You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@solr.apache.org by GitBox <gi...@apache.org> on 2022/02/24 22:35:38 UTC
[GitHub] [solr] risdenk opened a new pull request #705: SOLR-14920: Spotless formatting for core
risdenk opened a new pull request #705:
URL: https://github.com/apache/solr/pull/705
https://issues.apache.org/jira/browse/SOLR-14920
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] risdenk commented on a change in pull request #705: SOLR-14920: Spotless formatting for core - non-test only
Posted by GitBox <gi...@apache.org>.
risdenk commented on a change in pull request #705:
URL: https://github.com/apache/solr/pull/705#discussion_r816372703
##########
File path: solr/core/src/java/org/apache/solr/highlight/DefaultSolrHighlighter.java
##########
@@ -735,11 +818,13 @@ protected Object doHighlightingByHighlighter(SolrDocument doc, int docId, Schema
if (value instanceof IndexableField) {
strValue = fieldType.toExternal((IndexableField) value);
} else {
- strValue = value.toString(); // TODO FieldType needs an API for this, e.g. toExternalFromDv()
+ strValue =
+ value.toString(); // TODO FieldType needs an API for this, e.g. toExternalFromDv()
}
result.add(strValue);
- maxCharsToAnalyze -= strValue.length();//we exit early if we'll never get to analyze the value
+ maxCharsToAnalyze -=
+ strValue.length(); // we exit early if we'll never get to analyze the value
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/highlight/DefaultSolrHighlighter.java
##########
@@ -455,9 +477,11 @@ protected BoundaryScanner getBoundaryScanner(String fieldName, SolrParams params
} else {
returnFields = new SolrReturnFields(new String[0], req);
}
- FvhContainer fvhContainer = new FvhContainer(null, null); // Lazy container for fvh and fieldQuery
+ FvhContainer fvhContainer =
+ new FvhContainer(null, null); // Lazy container for fvh and fieldQuery
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/index/SlowCompositeReaderWrapper.java
##########
@@ -117,13 +112,17 @@ public CacheHelper getCoreCacheHelper() {
public Terms terms(String field) throws IOException {
ensureOpen();
try {
- return cachedTerms.computeIfAbsent(field, f -> {
- try {
- return MultiTerms.getTerms(in, f);
- } catch (IOException e) { // yuck! ...sigh... checked exceptions with built-in lambdas are a pain
- throw new RuntimeException("unwrapMe", e);
- }
- });
+ return cachedTerms.computeIfAbsent(
+ field,
+ f -> {
+ try {
+ return MultiTerms.getTerms(in, f);
+ } catch (
+ IOException
+ e) { // yuck! ...sigh... checked exceptions with built-in lambdas are a pain
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/highlight/DefaultSolrHighlighter.java
##########
@@ -684,21 +760,24 @@ protected Object doHighlightingByHighlighter(SolrDocument doc, int docId, Schema
// Highlight!
try {
TextFragment[] bestTextFragments =
- highlighter.getBestTextFragments(tstream, thisText, mergeContiguousFragments, numFragments);
+ highlighter.getBestTextFragments(
+ tstream, thisText, mergeContiguousFragments, numFragments);
for (TextFragment bestTextFragment : bestTextFragments) {
- if (bestTextFragment == null)//can happen via mergeContiguousFragments
- continue;
- // normally we want a score (must be highlighted), but if preserveMulti then we return a snippet regardless.
+ if (bestTextFragment == null) // can happen via mergeContiguousFragments
+ continue;
+ // normally we want a score (must be highlighted), but if preserveMulti then we return a
+ // snippet regardless.
if (bestTextFragment.getScore() > 0 || preserveMulti) {
frags.add(bestTextFragment);
if (bestTextFragment.getScore() > 0)
- --mvToMatch; // note: limits fragments (for multi-valued fields), not quite the number of values
+ --mvToMatch; // note: limits fragments (for multi-valued fields), not quite the
+ // number of values
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/parser/SolrQueryParserBase.java
##########
@@ -1062,14 +1082,17 @@ protected Query getFieldQuery(String field, String queryText, boolean quoted) th
private String lastFieldName;
private SchemaField lastField;
- // if raw==true, then it's possible for this method to return a RawQuery that will need to be transformed
+ // if raw==true, then it's possible for this method to return a RawQuery that will need to be
+ // transformed
// further before using.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/parser/SolrQueryParserBase.java
##########
@@ -1062,14 +1082,17 @@ protected Query getFieldQuery(String field, String queryText, boolean quoted) th
private String lastFieldName;
private SchemaField lastField;
- // if raw==true, then it's possible for this method to return a RawQuery that will need to be transformed
+ // if raw==true, then it's possible for this method to return a RawQuery that will need to be
+ // transformed
// further before using.
- protected Query getFieldQuery(String field, String queryText, boolean quoted, boolean raw) throws SyntaxError {
+ protected Query getFieldQuery(String field, String queryText, boolean quoted, boolean raw)
+ throws SyntaxError {
checkNullField(field);
SchemaField sf;
if (field.equals(lastFieldName)) {
- // only look up the SchemaField on a field change... this helps with memory allocation of dynamic fields
+ // only look up the SchemaField on a field change... this helps with memory allocation of
+ // dynamic fields
// and large queries like foo_i:(1 2 3 4 5 6 7 8 9 10) when we are passed "foo_i" each time.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/request/SimpleFacets.java
##########
@@ -767,35 +849,36 @@ public LeafCollector getLeafCollector(LeafReaderContext context) throws IOExcept
}
}
-
- static final Executor directExecutor = new Executor() {
- @Override
- public void execute(Runnable r) {
- r.run();
- }
- };
+ static final Executor directExecutor =
+ new Executor() {
+ @Override
+ public void execute(Runnable r) {
+ r.run();
+ }
+ };
private final Executor facetExecutor;
-
+
/**
- * Returns a list of value constraints and the associated facet counts
- * for each facet field specified in the params.
+ * Returns a list of value constraints and the associated facet counts for each facet field
+ * specified in the params.
*
* @see FacetParams#FACET_FIELD
* @see #getFieldMissingCount
* @see #getFacetTermEnumCounts
*/
- public NamedList<Object> getFacetFieldCounts()
- throws IOException, SyntaxError {
+ public NamedList<Object> getFacetFieldCounts() throws IOException, SyntaxError {
NamedList<Object> res = new SimpleOrderedMap<>();
String[] facetFs = global.getParams(FacetParams.FACET_FIELD);
if (null == facetFs) {
return res;
}
- // Passing a negative number for FACET_THREADS implies an unlimited number of threads is acceptable.
- // Also, a subtlety of directExecutor is that no matter how many times you "submit" a job, it's really
+ // Passing a negative number for FACET_THREADS implies an unlimited number of threads is
+ // acceptable.
+ // Also, a subtlety of directExecutor is that no matter how many times you "submit" a job, it's
+ // really
// just a method call in that it's run by the calling thread.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/request/SolrQueryRequestBase.java
##########
@@ -104,32 +98,34 @@ public void setParams(SolrParams params) {
this.params = params;
}
-
// Get the start time of this request in milliseconds
@Override
public long getStartTime() {
return startTime;
}
@Override
- public RTimerTree getRequestTimer () {
+ public RTimerTree getRequestTimer() {
return requestTimer;
}
// The index searcher associated with this request
protected RefCounted<SolrIndexSearcher> searcherHolder;
+
@Override
public SolrIndexSearcher getSearcher() {
- if(core == null) return null;//a request for a core admin will not have a core
+ if (core == null) return null; // a request for a core admin will not have a core
// should this reach out and get a searcher from the core singleton, or
// should the core populate one in a factory method to create requests?
// or there could be a setSearcher() method that Solr calls
- if (searcherHolder==null) {
+ if (searcherHolder == null) {
searcherHolder = core.getSearcher();
- // We start tracking here instead of at construction, because if getSearcher is never called, it's
- // not fatal to forget close(), and lots of test code is sloppy about it. However, when we get another
+ // We start tracking here instead of at construction, because if getSearcher is never called,
+ // it's
+ // not fatal to forget close(), and lots of test code is sloppy about it. However, when we get
+ // another
// searcher reference, having this tracked may be a good hint about where the leak comes from.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/highlight/DefaultSolrHighlighter.java
##########
@@ -237,27 +243,35 @@ protected Highlighter getHighlighter(Query query, String fieldName, SolrQueryReq
}
/**
- * Return a {@link org.apache.lucene.search.highlight.QueryScorer} suitable for this Query and field.
+ * Return a {@link org.apache.lucene.search.highlight.QueryScorer} suitable for this Query and
+ * field.
*
- * @param query The current query
- * @param tokenStream document text tokenStream that implements reset() efficiently (e.g. CachingTokenFilter).
- * If it's used, call reset() first.
- * @param fieldName The name of the field
- * @param request The SolrQueryRequest
+ * @param query The current query
+ * @param tokenStream document text tokenStream that implements reset() efficiently (e.g.
+ * CachingTokenFilter). If it's used, call reset() first.
+ * @param fieldName The name of the field
+ * @param request The SolrQueryRequest
*/
- protected QueryScorer getSpanQueryScorer(Query query, String fieldName, TokenStream tokenStream, SolrQueryRequest request) {
- QueryScorer scorer = new QueryScorer(query,
- request.getParams().getFieldBool(fieldName, HighlightParams.FIELD_MATCH, false) ? fieldName : null) {
- @Override
- protected WeightedSpanTermExtractor newTermExtractor(String defaultField) {
- return new CustomSpanTermExtractor(defaultField);
- }
- };
- scorer.setExpandMultiTermQuery(request.getParams().getBool(HighlightParams.HIGHLIGHT_MULTI_TERM, true));
+ protected QueryScorer getSpanQueryScorer(
+ Query query, String fieldName, TokenStream tokenStream, SolrQueryRequest request) {
+ QueryScorer scorer =
+ new QueryScorer(
+ query,
+ request.getParams().getFieldBool(fieldName, HighlightParams.FIELD_MATCH, false)
+ ? fieldName
+ : null) {
+ @Override
+ protected WeightedSpanTermExtractor newTermExtractor(String defaultField) {
+ return new CustomSpanTermExtractor(defaultField);
+ }
+ };
+ scorer.setExpandMultiTermQuery(
+ request.getParams().getBool(HighlightParams.HIGHLIGHT_MULTI_TERM, true));
- boolean defaultPayloads = true;//overwritten below
+ boolean defaultPayloads = true; // overwritten below
try {
- // It'd be nice to know if payloads are on the tokenStream but the presence of the attribute isn't a good
+ // It'd be nice to know if payloads are on the tokenStream but the presence of the attribute
+ // isn't a good
// indicator.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/packagemanager/PackageManager.java
##########
@@ -194,51 +228,79 @@ public void uninstall(String packageName, String version) {
public Map<String, SolrPackageInstance> getPackagesDeployed(String collection) {
Map<String, String> packages = null;
try {
- NavigableObject result = (NavigableObject) Utils.executeGET(solrClient.getHttpClient(),
- solrBaseUrl + PackageUtils.getCollectionParamsPath(collection) + "/PKG_VERSIONS?omitHeader=true&wt=javabin", Utils.JAVABINCONSUMER);
- packages = (Map<String, String>) result._get("/response/params/PKG_VERSIONS", Collections.emptyMap());
+ NavigableObject result =
+ (NavigableObject)
+ Utils.executeGET(
+ solrClient.getHttpClient(),
+ solrBaseUrl
+ + PackageUtils.getCollectionParamsPath(collection)
+ + "/PKG_VERSIONS?omitHeader=true&wt=javabin",
+ Utils.JAVABINCONSUMER);
+ packages =
+ (Map<String, String>)
+ result._get("/response/params/PKG_VERSIONS", Collections.emptyMap());
} catch (PathNotFoundException ex) {
- // Don't worry if PKG_VERSION wasn't found. It just means this collection was never touched by the package manager.
+ // Don't worry if PKG_VERSION wasn't found. It just means this collection was never touched by
+ // the package manager.
}
if (packages == null) return Collections.emptyMap();
Map<String, SolrPackageInstance> ret = new HashMap<>();
- for (String packageName: packages.keySet()) {
- if (Strings.isNullOrEmpty(packageName) == false && // There can be an empty key, storing the version here
- packages.get(packageName) != null) { // null means the package was undeployed from this package before
+ for (String packageName : packages.keySet()) {
+ if (Strings.isNullOrEmpty(packageName) == false
+ && // There can be an empty key, storing the version here
+ packages.get(packageName)
+ != null) { // null means the package was undeployed from this package before
ret.put(packageName, getPackageInstance(packageName, packages.get(packageName)));
}
}
return ret;
}
/**
- * Get a map of packages (key: package name, value: package instance) that have their plugins deployed as cluster level plugins.
- * The returned packages also contain the "pluginMeta" from "clusterprops.json" as custom data.
+ * Get a map of packages (key: package name, value: package instance) that have their plugins
+ * deployed as cluster level plugins. The returned packages also contain the "pluginMeta" from
+ * "clusterprops.json" as custom data.
*/
@SuppressWarnings({"unchecked"})
public Map<String, SolrPackageInstance> getPackagesDeployedAsClusterLevelPlugins() {
Map<String, String> packageVersions = new HashMap<>();
- MultiValuedMap<String, PluginMeta> packagePlugins = new HashSetValuedHashMap<>(); // map of package name to multiple values of pluginMeta (Map<String, String>)
+ MultiValuedMap<String, PluginMeta> packagePlugins =
+ new HashSetValuedHashMap<>(); // map of package name to multiple values of pluginMeta
+ // (Map<String, String>)
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/legacy/BBoxStrategy.java
##########
@@ -124,7 +121,8 @@
final String field_maxY;
final String field_xdl; // crosses dateline
- private final FieldType optionsFieldType;//from constructor; aggregate field type used to express all options
+ private final FieldType
+ optionsFieldType; // from constructor; aggregate field type used to express all options
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/highlight/DefaultSolrHighlighter.java
##########
@@ -905,34 +1007,41 @@ public boolean incrementToken() throws IOException {
int startOffset;
}
- /** For use with term vectors of multi-valued fields. We want an offset based window into its TokenStream. */
+ /**
+ * For use with term vectors of multi-valued fields. We want an offset based window into its
+ * TokenStream.
+ */
static final class OffsetWindowTokenFilter extends TokenFilter {
private final OffsetAttribute offsetAtt = addAttribute(OffsetAttribute.class);
- private final PositionIncrementAttribute posIncAtt = addAttribute(PositionIncrementAttribute.class);
+ private final PositionIncrementAttribute posIncAtt =
+ addAttribute(PositionIncrementAttribute.class);
private int windowStartOffset;
- private int windowEndOffset = -1;//exclusive
+ private int windowEndOffset = -1; // exclusive
private boolean windowTokenIncremented = false;
private boolean inputWasReset = false;
- private State capturedState;//only used for first token of each subsequent window
+ private State capturedState; // only used for first token of each subsequent window
- OffsetWindowTokenFilter(TokenStream input) {//input should not have been reset already
+ OffsetWindowTokenFilter(TokenStream input) { // input should not have been reset already
super(input);
}
- //Called at the start of each value/window
+ // Called at the start of each value/window
OffsetWindowTokenFilter advanceToNextWindowOfLength(int length) {
- windowStartOffset = windowEndOffset + 1;//unclear why there's a single offset gap between values, but tests show it
+ windowStartOffset =
+ windowEndOffset
+ + 1; // unclear why there's a single offset gap between values, but tests show it
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/packagemanager/PackageManager.java
##########
@@ -369,51 +511,77 @@ private boolean deployPackage(SolrPackageInstance packageInstance, boolean pegTo
throw new SolrException(ErrorCode.SERVER_ERROR, ex);
}
} else {
- throw new SolrException(ErrorCode.BAD_REQUEST, "Non-POST method not supported for setup commands");
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST, "Non-POST method not supported for setup commands");
}
} else {
- PackageUtils.printRed("There is no setup command to execute for plugin: " + plugin.name);
+ PackageUtils.printRed(
+ "There is no setup command to execute for plugin: " + plugin.name);
}
}
}
// Set the package version in the collection's parameters
try {
- SolrCLI.postJsonToSolr(solrClient, PackageUtils.getCollectionParamsPath(collection),
- "{update:{PKG_VERSIONS:{'" + packageInstance.name + "' : '" + (pegToLatest? PackageLoader.LATEST: packageInstance.version) + "'}}}");
+ SolrCLI.postJsonToSolr(
+ solrClient,
+ PackageUtils.getCollectionParamsPath(collection),
+ "{update:{PKG_VERSIONS:{'"
+ + packageInstance.name
+ + "' : '"
+ + (pegToLatest ? PackageLoader.LATEST : packageInstance.version)
+ + "'}}}");
} catch (Exception ex) {
throw new SolrException(ErrorCode.SERVER_ERROR, ex);
}
}
if (previouslyDeployed.isEmpty() == false) {
- PackageUtils.printRed("Already Deployed on " + previouslyDeployed + ", package: " + packageInstance.name + ", version: " + packageInstance.version);
+ PackageUtils.printRed(
+ "Already Deployed on "
+ + previouslyDeployed
+ + ", package: "
+ + packageInstance.name
+ + ", version: "
+ + packageInstance.version);
}
- List<String> deployedCollections = collections.stream().filter(c -> !previouslyDeployed.contains(c)).collect(Collectors.toList());
+ List<String> deployedCollections =
+ collections.stream()
+ .filter(c -> !previouslyDeployed.contains(c))
+ .collect(Collectors.toList());
return new Pair<List<String>, List<String>>(deployedCollections, previouslyDeployed);
}
@SuppressWarnings("unchecked")
- private boolean deployClusterPackage(SolrPackageInstance packageInstance, boolean isUpdate, boolean noprompt, String[] overrides) {
+ private boolean deployClusterPackage(
+ SolrPackageInstance packageInstance, boolean isUpdate, boolean noprompt, String[] overrides) {
boolean clusterPluginFailed = false;
int numberOfClusterPluginsDeployed = 0;
if (isUpdate) {
- for (Plugin plugin: packageInstance.plugins) {
+ for (Plugin plugin : packageInstance.plugins) {
if ("cluster".equalsIgnoreCase(plugin.type) == false) continue;
- SolrPackageInstance deployedPackage = getPackagesDeployedAsClusterLevelPlugins().get(packageInstance.name);
+ SolrPackageInstance deployedPackage =
+ getPackagesDeployedAsClusterLevelPlugins().get(packageInstance.name);
if (deployedPackage == null) {
- PackageUtils.printRed("Cluster level plugin " + plugin.name + " from package " + packageInstance.name + " not deployed. To deploy, remove the --update parameter.");
+ PackageUtils.printRed(
+ "Cluster level plugin "
+ + plugin.name
+ + " from package "
+ + packageInstance.name
+ + " not deployed. To deploy, remove the --update parameter.");
clusterPluginFailed = true;
continue;
}
- for (PluginMeta pluginMeta: (List<PluginMeta>)deployedPackage.getCustomData()) {
+ for (PluginMeta pluginMeta : (List<PluginMeta>) deployedPackage.getCustomData()) {
PackageUtils.printGreen("Updating this plugin: " + pluginMeta);
try {
- pluginMeta.version = packageInstance.version; // just update the version, let the other metadata same
+ pluginMeta.version =
+ packageInstance.version; // just update the version, let the other metadata same
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/metrics/MetricsMap.java
##########
@@ -132,21 +138,25 @@ public Object getAttribute(String attribute) throws AttributeNotFoundException,
val = stats.get(attribute);
if (val != null) {
- // It's String or one of the simple types, just return it as JMX suggests direct support for such types
+ // It's String or one of the simple types, just return it as JMX suggests direct support for
+ // such types
for (String simpleTypeName : SimpleType.ALLOWED_CLASSNAMES_LIST) {
if (val.getClass().getName().equals(simpleTypeName)) {
return val;
}
}
- // It's an arbitrary object which could be something complex and odd, return its toString, assuming that is
+ // It's an arbitrary object which could be something complex and odd, return its toString,
+ // assuming that is
// a workable representation of the object
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/legacy/BBoxStrategy.java
##########
@@ -45,63 +45,60 @@
import org.locationtech.spatial4j.shape.Rectangle;
import org.locationtech.spatial4j.shape.Shape;
-
/**
- * A SpatialStrategy for indexing and searching Rectangles by storing its
- * coordinates in numeric fields. It supports all {@link SpatialOperation}s and
- * has a custom overlap relevancy. It is based on GeoPortal's <a
+ * A SpatialStrategy for indexing and searching Rectangles by storing its coordinates in numeric
+ * fields. It supports all {@link SpatialOperation}s and has a custom overlap relevancy. It is based
+ * on GeoPortal's <a
* href="https://geoportal.sourceforge.io/javadoc/1.0/com/esri/gpt/catalog/lucene/SpatialClauseAdapter.html">SpatialClauseAdapter</a>.
- * <p>
- * <b>Characteristics:</b>
- * <br>
+ *
+ * <p><b>Characteristics:</b> <br>
+ *
* <ul>
- * <li>Only indexes Rectangles; just one per field value. Other shapes can be provided
- * and the bounding box will be used.</li>
- * <li>Can query only by a Rectangle. Providing other shapes is an error.</li>
- * <li>Supports most {@link SpatialOperation}s but not Overlaps.</li>
- * <li>Uses the DocValues API for any sorting / relevancy.</li>
+ * <li>Only indexes Rectangles; just one per field value. Other shapes can be provided and the
+ * bounding box will be used.
+ * <li>Can query only by a Rectangle. Providing other shapes is an error.
+ * <li>Supports most {@link SpatialOperation}s but not Overlaps.
+ * <li>Uses the DocValues API for any sorting / relevancy.
* </ul>
- * <p>
- * <b>Implementation:</b>
- * <p>
- * This uses 4 double fields for minX, maxX, minY, maxY
- * and a boolean to mark a dateline cross. Depending on the particular {@link
- * SpatialOperation}s, there are a variety of range queries on {@link DoublePoint}s to be
- * done.
- * The {@link #makeOverlapRatioValueSource(org.locationtech.spatial4j.shape.Rectangle, double)}
- * works by calculating the query bbox overlap percentage against the indexed
- * shape overlap percentage. The indexed shape's coordinates are retrieved from
- * {@link org.apache.lucene.index.LeafReader#getNumericDocValues}.
+ *
+ * <p><b>Implementation:</b>
+ *
+ * <p>This uses 4 double fields for minX, maxX, minY, maxY and a boolean to mark a dateline cross.
+ * Depending on the particular {@link SpatialOperation}s, there are a variety of range queries on
+ * {@link DoublePoint}s to be done. The {@link
+ * #makeOverlapRatioValueSource(org.locationtech.spatial4j.shape.Rectangle, double)} works by
+ * calculating the query bbox overlap percentage against the indexed shape overlap percentage. The
+ * indexed shape's coordinates are retrieved from {@link
+ * org.apache.lucene.index.LeafReader#getNumericDocValues}.
*
* @lucene.experimental
*/
@Deprecated
public class BBoxStrategy extends SpatialStrategy {
- // note: we use a FieldType to articulate the options we want on the field. We don't use it as-is with a Field, we
+ // note: we use a FieldType to articulate the options we want on the field. We don't use it as-is
+ // with a Field, we
// create more than one Field.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/parser/SolrQueryParserBase.java
##########
@@ -66,58 +69,56 @@
import org.apache.solr.search.QParser;
import org.apache.solr.search.QueryUtils;
import org.apache.solr.search.SyntaxError;
-import org.apache.lucene.queryparser.charstream.CharStream;
-import org.apache.lucene.queryparser.charstream.FastCharStream;
-
-import static org.apache.solr.parser.SolrQueryParserBase.SynonymQueryStyle.AS_SAME_TERM;
-/** This class is overridden by QueryParser in QueryParser.jj
- * and acts to separate the majority of the Java code from the .jj grammar file.
+/**
+ * This class is overridden by QueryParser in QueryParser.jj and acts to separate the majority of
+ * the Java code from the .jj grammar file.
*/
public abstract class SolrQueryParserBase extends QueryBuilder {
- protected static final String REVERSE_WILDCARD_LOWER_BOUND = new String(new char[]{ReverseStringFilter.START_OF_HEADING_MARKER + 1});
+ protected static final String REVERSE_WILDCARD_LOWER_BOUND =
+ new String(new char[] {ReverseStringFilter.START_OF_HEADING_MARKER + 1});
- public static final int TERMS_QUERY_THRESHOLD = 16; // @lucene.internal Set to a low value temporarily for better test coverage
+ public static final int TERMS_QUERY_THRESHOLD =
+ 16; // @lucene.internal Set to a low value temporarily for better test coverage
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/highlight/DefaultSolrHighlighter.java
##########
@@ -649,17 +721,20 @@ protected Object doHighlightingByHighlighter(SolrDocument doc, int docId, Schema
Highlighter highlighter;
if (params.getFieldBool(fieldName, HighlightParams.USE_PHRASE_HIGHLIGHTER, true)) {
- // We're going to call getPhraseHighlighter and it might consume the tokenStream. If it does, the tokenStream
+ // We're going to call getPhraseHighlighter and it might consume the tokenStream. If it
+ // does, the tokenStream
// needs to implement reset() efficiently.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/packagemanager/PackageManager.java
##########
@@ -634,121 +898,205 @@ public SolrPackageInstance getPackageInstance(String packageName, String version
}
}
}
- if (version == null || version.equalsIgnoreCase(PackageUtils.LATEST) || version.equalsIgnoreCase(PackageLoader.LATEST)) {
+ if (version == null
+ || version.equalsIgnoreCase(PackageUtils.LATEST)
+ || version.equalsIgnoreCase(PackageLoader.LATEST)) {
return latest;
} else return null;
}
/**
* Deploys a version of a package to a list of collections.
- * @param version If null, the most recent version is deployed.
- * EXPERT FEATURE: If version is PackageUtils.LATEST, this collection will be auto updated whenever a newer version of this package is installed.
- * @param isUpdate Is this a fresh deployment or is it an update (i.e. there is already a version of this package deployed on this collection)
+ *
+ * @param version If null, the most recent version is deployed. EXPERT FEATURE: If version is
+ * PackageUtils.LATEST, this collection will be auto updated whenever a newer version of this
+ * package is installed.
+ * @param isUpdate Is this a fresh deployment or is it an update (i.e. there is already a version
+ * of this package deployed on this collection)
* @param noprompt If true, don't prompt before executing setup commands.
*/
- public void deploy(String packageName, String version, String[] collections, boolean shouldInstallClusterPlugins, String[] parameters,
- boolean isUpdate, boolean noprompt) throws SolrException {
+ public void deploy(
+ String packageName,
+ String version,
+ String[] collections,
+ boolean shouldInstallClusterPlugins,
+ String[] parameters,
+ boolean isUpdate,
+ boolean noprompt)
+ throws SolrException {
ensureCollectionsExist(Arrays.asList(collections));
- boolean pegToLatest = PackageUtils.LATEST.equals(version); // User wants to peg this package's version to the latest installed (for auto-update, i.e. no explicit deploy step)
+ boolean pegToLatest =
+ PackageUtils.LATEST.equals(
+ version); // User wants to peg this package's version to the latest installed (for
+ // auto-update, i.e. no explicit deploy step)
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/packagemanager/PackageManager.java
##########
@@ -194,51 +228,79 @@ public void uninstall(String packageName, String version) {
public Map<String, SolrPackageInstance> getPackagesDeployed(String collection) {
Map<String, String> packages = null;
try {
- NavigableObject result = (NavigableObject) Utils.executeGET(solrClient.getHttpClient(),
- solrBaseUrl + PackageUtils.getCollectionParamsPath(collection) + "/PKG_VERSIONS?omitHeader=true&wt=javabin", Utils.JAVABINCONSUMER);
- packages = (Map<String, String>) result._get("/response/params/PKG_VERSIONS", Collections.emptyMap());
+ NavigableObject result =
+ (NavigableObject)
+ Utils.executeGET(
+ solrClient.getHttpClient(),
+ solrBaseUrl
+ + PackageUtils.getCollectionParamsPath(collection)
+ + "/PKG_VERSIONS?omitHeader=true&wt=javabin",
+ Utils.JAVABINCONSUMER);
+ packages =
+ (Map<String, String>)
+ result._get("/response/params/PKG_VERSIONS", Collections.emptyMap());
} catch (PathNotFoundException ex) {
- // Don't worry if PKG_VERSION wasn't found. It just means this collection was never touched by the package manager.
+ // Don't worry if PKG_VERSION wasn't found. It just means this collection was never touched by
+ // the package manager.
}
if (packages == null) return Collections.emptyMap();
Map<String, SolrPackageInstance> ret = new HashMap<>();
- for (String packageName: packages.keySet()) {
- if (Strings.isNullOrEmpty(packageName) == false && // There can be an empty key, storing the version here
- packages.get(packageName) != null) { // null means the package was undeployed from this package before
+ for (String packageName : packages.keySet()) {
+ if (Strings.isNullOrEmpty(packageName) == false
+ && // There can be an empty key, storing the version here
+ packages.get(packageName)
+ != null) { // null means the package was undeployed from this package before
ret.put(packageName, getPackageInstance(packageName, packages.get(packageName)));
}
}
return ret;
}
/**
- * Get a map of packages (key: package name, value: package instance) that have their plugins deployed as cluster level plugins.
- * The returned packages also contain the "pluginMeta" from "clusterprops.json" as custom data.
+ * Get a map of packages (key: package name, value: package instance) that have their plugins
+ * deployed as cluster level plugins. The returned packages also contain the "pluginMeta" from
+ * "clusterprops.json" as custom data.
*/
@SuppressWarnings({"unchecked"})
public Map<String, SolrPackageInstance> getPackagesDeployedAsClusterLevelPlugins() {
Map<String, String> packageVersions = new HashMap<>();
- MultiValuedMap<String, PluginMeta> packagePlugins = new HashSetValuedHashMap<>(); // map of package name to multiple values of pluginMeta (Map<String, String>)
+ MultiValuedMap<String, PluginMeta> packagePlugins =
+ new HashSetValuedHashMap<>(); // map of package name to multiple values of pluginMeta
+ // (Map<String, String>)
Map<String, Object> result;
try {
- result = (Map<String, Object>) Utils.executeGET(solrClient.getHttpClient(),
- solrBaseUrl + PackageUtils.CLUSTERPROPS_PATH, Utils.JSONCONSUMER);
+ result =
+ (Map<String, Object>)
+ Utils.executeGET(
+ solrClient.getHttpClient(),
+ solrBaseUrl + PackageUtils.CLUSTERPROPS_PATH,
+ Utils.JSONCONSUMER);
} catch (SolrException ex) {
if (ex.code() == ErrorCode.NOT_FOUND.code) {
- result = Collections.emptyMap(); // Cluster props doesn't exist, that means there are no cluster level plugins installed.
+ result =
+ Collections
+ .emptyMap(); // Cluster props doesn't exist, that means there are no cluster level
+ // plugins installed.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/highlight/DefaultSolrHighlighter.java
##########
@@ -735,11 +818,13 @@ protected Object doHighlightingByHighlighter(SolrDocument doc, int docId, Schema
if (value instanceof IndexableField) {
strValue = fieldType.toExternal((IndexableField) value);
} else {
- strValue = value.toString(); // TODO FieldType needs an API for this, e.g. toExternalFromDv()
+ strValue =
+ value.toString(); // TODO FieldType needs an API for this, e.g. toExternalFromDv()
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/packagemanager/PackageManager.java
##########
@@ -298,68 +379,129 @@ private boolean deployPackage(SolrPackageInstance packageInstance, boolean pegTo
/**
* @return list of collections on which packages deployed on
*/
- private Pair<List<String>, List<String>> deployCollectionPackage(SolrPackageInstance packageInstance, boolean pegToLatest, boolean isUpdate,
- boolean noprompt, List<String> collections, String[] overrides) {
- List<String> previouslyDeployed = new ArrayList<>(); // collections where package is already deployed in
- for (String collection: collections) {
- SolrPackageInstance deployedPackage = getPackagesDeployed(collection).get(packageInstance.name);
+ private Pair<List<String>, List<String>> deployCollectionPackage(
+ SolrPackageInstance packageInstance,
+ boolean pegToLatest,
+ boolean isUpdate,
+ boolean noprompt,
+ List<String> collections,
+ String[] overrides) {
+ List<String> previouslyDeployed =
+ new ArrayList<>(); // collections where package is already deployed in
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/parser/SolrQueryParserBase.java
##########
@@ -733,8 +734,10 @@ protected Query getBooleanQuery(List<BooleanClause> clauses) throws SyntaxError
if (sfield != rawq.sfield) {
sfield = rawq.sfield;
fieldValues = fmap.get(sfield);
- // If this field isn't indexed, or if it is indexed and we want to use TermsQuery, then collect this value.
- // We are currently relying on things like PointField not being marked as indexed in order to bypass
+ // If this field isn't indexed, or if it is indexed and we want to use TermsQuery, then
+ // collect this value.
+ // We are currently relying on things like PointField not being marked as indexed in
+ // order to bypass
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/request/json/RequestUtil.java
##########
@@ -161,15 +176,15 @@ public static void processParams(SolrRequestHandler handler, SolrQueryRequest re
}
// Set these params as soon as possible so if there is an error processing later, things like
// "wt=json" will take effect from the defaults.
- SolrParams newParams = new MultiMapSolrParams(newMap); // newMap may still change below, but that should be OK
+ SolrParams newParams =
+ new MultiMapSolrParams(newMap); // newMap may still change below, but that should be OK
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/parser/SolrQueryParserBase.java
##########
@@ -262,24 +261,26 @@ void restoreFlags(int flagsToRestore) {
flags = flagsToRestore;
}
- /** Parses a query string, returning a {@link org.apache.lucene.search.Query}.
- * @param query the query string to be parsed.
- */
+ /**
+ * Parses a query string, returning a {@link org.apache.lucene.search.Query}.
+ *
+ * @param query the query string to be parsed.
+ */
public Query parse(String query) throws SyntaxError {
ReInit(new FastCharStream(new StringReader(query)));
try {
// TopLevelQuery is a Query followed by the end-of-input (EOF)
- Query res = TopLevelQuery(null); // pass null so we can tell later if an explicit field was provided or not
- return res!=null ? res : newBooleanQuery().build();
- }
- catch (ParseException | TokenMgrError tme) {
- throw new SyntaxError("Cannot parse '" +query+ "': " + tme.getMessage(), tme);
+ Query res =
+ TopLevelQuery(
+ null); // pass null so we can tell later if an explicit field was provided or not
+ return res != null ? res : newBooleanQuery().build();
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/packagemanager/PackageManager.java
##########
@@ -634,121 +898,205 @@ public SolrPackageInstance getPackageInstance(String packageName, String version
}
}
}
- if (version == null || version.equalsIgnoreCase(PackageUtils.LATEST) || version.equalsIgnoreCase(PackageLoader.LATEST)) {
+ if (version == null
+ || version.equalsIgnoreCase(PackageUtils.LATEST)
+ || version.equalsIgnoreCase(PackageLoader.LATEST)) {
return latest;
} else return null;
}
/**
* Deploys a version of a package to a list of collections.
- * @param version If null, the most recent version is deployed.
- * EXPERT FEATURE: If version is PackageUtils.LATEST, this collection will be auto updated whenever a newer version of this package is installed.
- * @param isUpdate Is this a fresh deployment or is it an update (i.e. there is already a version of this package deployed on this collection)
+ *
+ * @param version If null, the most recent version is deployed. EXPERT FEATURE: If version is
+ * PackageUtils.LATEST, this collection will be auto updated whenever a newer version of this
+ * package is installed.
+ * @param isUpdate Is this a fresh deployment or is it an update (i.e. there is already a version
+ * of this package deployed on this collection)
* @param noprompt If true, don't prompt before executing setup commands.
*/
- public void deploy(String packageName, String version, String[] collections, boolean shouldInstallClusterPlugins, String[] parameters,
- boolean isUpdate, boolean noprompt) throws SolrException {
+ public void deploy(
+ String packageName,
+ String version,
+ String[] collections,
+ boolean shouldInstallClusterPlugins,
+ String[] parameters,
+ boolean isUpdate,
+ boolean noprompt)
+ throws SolrException {
ensureCollectionsExist(Arrays.asList(collections));
- boolean pegToLatest = PackageUtils.LATEST.equals(version); // User wants to peg this package's version to the latest installed (for auto-update, i.e. no explicit deploy step)
+ boolean pegToLatest =
+ PackageUtils.LATEST.equals(
+ version); // User wants to peg this package's version to the latest installed (for
+ // auto-update, i.e. no explicit deploy step)
SolrPackageInstance packageInstance = getPackageInstance(packageName, version);
if (packageInstance == null) {
- PackageUtils.printRed("Package instance doesn't exist: " + packageName + ":" + version + ". Use install command to install this version first.");
+ PackageUtils.printRed(
+ "Package instance doesn't exist: "
+ + packageName
+ + ":"
+ + version
+ + ". Use install command to install this version first.");
System.exit(1);
}
Manifest manifest = packageInstance.manifest;
try {
if (!SolrVersion.LATEST.satisfies(manifest.versionConstraint)) {
- PackageUtils.printRed("Version incompatible! Solr version: "
- + SolrVersion.LATEST + ", package version constraint: " + manifest.versionConstraint);
+ PackageUtils.printRed(
+ "Version incompatible! Solr version: "
+ + SolrVersion.LATEST
+ + ", package version constraint: "
+ + manifest.versionConstraint);
System.exit(1);
}
} catch (SolrVersion.InvalidSemVerExpressionException ex) {
- PackageUtils.printRed("Error in version constraint given in package manifest: " + manifest.versionConstraint + ". It does not a valid SemVer expression.");
+ PackageUtils.printRed(
+ "Error in version constraint given in package manifest: "
+ + manifest.versionConstraint
+ + ". It does not a valid SemVer expression.");
System.exit(1);
}
- boolean res = deployPackage(packageInstance, pegToLatest, isUpdate, noprompt,
- Arrays.asList(collections), shouldInstallClusterPlugins, parameters);
- PackageUtils.print(res? PackageUtils.GREEN: PackageUtils.RED, res? "Deployment successful": "Deployment failed");
+ boolean res =
+ deployPackage(
+ packageInstance,
+ pegToLatest,
+ isUpdate,
+ noprompt,
+ Arrays.asList(collections),
+ shouldInstallClusterPlugins,
+ parameters);
+ PackageUtils.print(
+ res ? PackageUtils.GREEN : PackageUtils.RED,
+ res ? "Deployment successful" : "Deployment failed");
}
- /**
- * Undeploys a package from given collections.
- */
- public void undeploy(String packageName, String[] collections, boolean shouldUndeployClusterPlugins) throws SolrException {
+ /** Undeploys a package from given collections. */
+ public void undeploy(
+ String packageName, String[] collections, boolean shouldUndeployClusterPlugins)
+ throws SolrException {
ensureCollectionsExist(Arrays.asList(collections));
-
+
// Undeploy cluster level plugins
if (shouldUndeployClusterPlugins) {
- SolrPackageInstance deployedPackage = getPackagesDeployedAsClusterLevelPlugins().get(packageName);
+ SolrPackageInstance deployedPackage =
+ getPackagesDeployedAsClusterLevelPlugins().get(packageName);
if (deployedPackage == null) {
- PackageUtils.printRed("Cluster level plugins from package "+packageName+" not deployed.");
+ PackageUtils.printRed(
+ "Cluster level plugins from package " + packageName + " not deployed.");
} else {
- for (Plugin plugin: deployedPackage.plugins) {
- if (!shouldUndeployClusterPlugins || "cluster".equalsIgnoreCase(plugin.type) == false) continue;
-
- Map<String, String> systemParams = Map.of("package-name", deployedPackage.name, "package-version", deployedPackage.version, "plugin-name", plugin.name);
+ for (Plugin plugin : deployedPackage.plugins) {
+ if (!shouldUndeployClusterPlugins || "cluster".equalsIgnoreCase(plugin.type) == false)
+ continue;
+
+ Map<String, String> systemParams =
+ Map.of(
+ "package-name",
+ deployedPackage.name,
+ "package-version",
+ deployedPackage.version,
+ "plugin-name",
+ plugin.name);
Command cmd = plugin.uninstallCommand;
if (cmd != null && !Strings.isNullOrEmpty(cmd.method)) {
if ("POST".equalsIgnoreCase(cmd.method)) {
try {
- String payload = PackageUtils.resolve(getMapper().writeValueAsString(cmd.payload), deployedPackage.parameterDefaults, Collections.emptyMap(), systemParams);
- String path = PackageUtils.resolve(cmd.path, deployedPackage.parameterDefaults, Collections.emptyMap(), systemParams);
+ String payload =
+ PackageUtils.resolve(
+ getMapper().writeValueAsString(cmd.payload),
+ deployedPackage.parameterDefaults,
+ Collections.emptyMap(),
+ systemParams);
+ String path =
+ PackageUtils.resolve(
+ cmd.path,
+ deployedPackage.parameterDefaults,
+ Collections.emptyMap(),
+ systemParams);
PackageUtils.printGreen("Executing " + payload + " for path:" + path);
SolrCLI.postJsonToSolr(solrClient, path, payload);
} catch (Exception ex) {
throw new SolrException(ErrorCode.SERVER_ERROR, ex);
}
} else {
- throw new SolrException(ErrorCode.BAD_REQUEST, "Non-POST method not supported for uninstall commands");
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST, "Non-POST method not supported for uninstall commands");
}
} else {
- PackageUtils.printRed("There is no uninstall command to execute for plugin: " + plugin.name);
+ PackageUtils.printRed(
+ "There is no uninstall command to execute for plugin: " + plugin.name);
}
-
}
}
}
// Undeploy collection level plugins
- for (String collection: collections) {
+ for (String collection : collections) {
SolrPackageInstance deployedPackage = getPackagesDeployed(collection).get(packageName);
if (deployedPackage == null) {
- PackageUtils.printRed("Package "+packageName+" not deployed on collection "+collection);
+ PackageUtils.printRed(
+ "Package " + packageName + " not deployed on collection " + collection);
continue;
}
Map<String, String> collectionParameterOverrides = getPackageParams(packageName, collection);
// Run the uninstall command for all plugins
- for (Plugin plugin: deployedPackage.plugins) {
+ for (Plugin plugin : deployedPackage.plugins) {
if ("collection".equalsIgnoreCase(plugin.type) == false) continue;
- Map<String, String> systemParams = Map.of("collection", collection, "package-name", deployedPackage.name, "package-version", deployedPackage.version, "plugin-name", plugin.name);
+ Map<String, String> systemParams =
+ Map.of(
+ "collection",
+ collection,
+ "package-name",
+ deployedPackage.name,
+ "package-version",
+ deployedPackage.version,
+ "plugin-name",
+ plugin.name);
Command cmd = plugin.uninstallCommand;
if (cmd != null && !Strings.isNullOrEmpty(cmd.method)) {
if ("POST".equalsIgnoreCase(cmd.method)) {
try {
- String payload = PackageUtils.resolve(getMapper().writeValueAsString(cmd.payload), deployedPackage.parameterDefaults, collectionParameterOverrides, systemParams);
- String path = PackageUtils.resolve(cmd.path, deployedPackage.parameterDefaults, collectionParameterOverrides, systemParams);
+ String payload =
+ PackageUtils.resolve(
+ getMapper().writeValueAsString(cmd.payload),
+ deployedPackage.parameterDefaults,
+ collectionParameterOverrides,
+ systemParams);
+ String path =
+ PackageUtils.resolve(
+ cmd.path,
+ deployedPackage.parameterDefaults,
+ collectionParameterOverrides,
+ systemParams);
PackageUtils.printGreen("Executing " + payload + " for path:" + path);
SolrCLI.postJsonToSolr(solrClient, path, payload);
} catch (Exception ex) {
throw new SolrException(ErrorCode.SERVER_ERROR, ex);
}
} else {
- throw new SolrException(ErrorCode.BAD_REQUEST, "Non-POST method not supported for uninstall commands");
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST, "Non-POST method not supported for uninstall commands");
}
} else {
- PackageUtils.printRed("There is no uninstall command to execute for plugin: " + plugin.name);
+ PackageUtils.printRed(
+ "There is no uninstall command to execute for plugin: " + plugin.name);
}
}
// Set the package version in the collection's parameters
try {
- SolrCLI.postJsonToSolr(solrClient, PackageUtils.getCollectionParamsPath(collection),
- "{set: {PKG_VERSIONS: {"+packageName+": null}}}"); // Is it better to "unset"? If so, build support in params API for "unset"
- SolrCLI.postJsonToSolr(solrClient, PackageUtils.PACKAGE_PATH, "{\"refresh\": \"" + packageName + "\"}");
+ SolrCLI.postJsonToSolr(
+ solrClient,
+ PackageUtils.getCollectionParamsPath(collection),
+ "{set: {PKG_VERSIONS: {"
+ + packageName
+ + ": null}}}"); // Is it better to "unset"? If so, build support in params API for
+ // "unset"
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/request/DocValuesFacets.java
##########
@@ -130,7 +165,8 @@ private DocValuesFacets() {}
List<LeafReaderContext> leaves = searcher.getTopReaderContext().leaves();
for (int subIndex = 0; subIndex < leaves.size(); subIndex++) {
LeafReaderContext leaf = leaves.get(subIndex);
- final DocIdSetIterator disi = docs.iterator(leaf); // solr docsets already exclude any deleted docs
+ final DocIdSetIterator disi =
+ docs.iterator(leaf); // solr docsets already exclude any deleted docs
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/request/NumericFacets.java
##########
@@ -496,18 +557,20 @@ protected boolean lessThan(Entry a, Entry b) {
while (pq.size() > offset) {
counts.addFirst(pq.pop());
}
-
+
for (Entry entry : counts) {
result.add(bitsToStringValue(ft, entry.bits), entry.count); // TODO: convert to correct value
}
-
- // Once facet.mincount=0 is supported we'll need to add logic similar to the SingleValue case, but obtaining values
+
+ // Once facet.mincount=0 is supported we'll need to add logic similar to the SingleValue case,
+ // but obtaining values
// with count 0 from DocValues
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/parser/SolrQueryParserBase.java
##########
@@ -1159,9 +1202,11 @@ protected Query getFieldQuery(String field, List<String> queryTerms, boolean raw
for (String queryTerm : queryTerms) {
try {
subqs.add(ft.getFieldQuery(parser, sf, queryTerm));
- } catch (Exception e) { // assumption: raw = false only when called from ExtendedDismaxQueryParser.getQuery()
- // ExtendedDismaxQueryParser is a lenient query parser
- // This happens when a field tries to parse a query term that has a type incompatible with the field
+ } catch (Exception e) { // assumption: raw = false only when called from
+ // ExtendedDismaxQueryParser.getQuery()
+ // ExtendedDismaxQueryParser is a lenient query parser
+ // This happens when a field tries to parse a query term that has a type
+ // incompatible with the field
// e.g.
// a numerical field trying to parse a textual query term
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/request/IntervalFacets.java
##########
@@ -274,7 +291,8 @@ private void getCountString() throws IOException {
List<LeafReaderContext> leaves = searcher.getTopReaderContext().leaves();
for (int subIndex = 0; subIndex < leaves.size(); subIndex++) {
LeafReaderContext leaf = leaves.get(subIndex);
- final DocIdSetIterator disi = docs.iterator(leaf); // solr docsets already exclude any deleted docs
+ final DocIdSetIterator disi =
+ docs.iterator(leaf); // solr docsets already exclude any deleted docs
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/request/NumericFacets.java
##########
@@ -439,12 +495,14 @@ protected boolean lessThan(Entry a, Entry b) {
hashTable.add(l, 1);
for (int i = 1, count = longs.docValueCount(); i < count; i++) {
long lnew = longs.nextValue();
- if (lnew > l) { // Skip the value if it's equal to the last one, we don't want to double-count it
+ if (lnew
+ > l) { // Skip the value if it's equal to the last one, we don't want to double-count
+ // it
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/request/PerSegmentSingleValuedFaceting.java
##########
@@ -201,16 +231,15 @@ protected boolean lessThan(SegFacet a, SegFacet b) {
}
// if mincount>0 then seg.pos++ can skip ahead to the next non-zero entry.
- do{
+ do {
++seg.pos;
- }
- while(
- (seg.pos < seg.endTermIndex) //stop incrementing before we run off the end
- && (seg.tenum.next() != null || true) //move term enum forward with position -- dont care about value
- && (mincount > 0) //only skip ahead if mincount > 0
- && (seg.counts[seg.pos - seg.startTermIndex] == 0) //check zero count
+ } while ((seg.pos < seg.endTermIndex) // stop incrementing before we run off the end
+ && (seg.tenum.next() != null
+ || true) // move term enum forward with position -- dont care about value
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/parser/SolrQueryParserBase.java
##########
@@ -1112,12 +1144,14 @@ protected Query getFieldQuery(String field, String queryText, boolean quoted, bo
}
// Assumption: quoted is always false
- protected Query getFieldQuery(String field, List<String> queryTerms, boolean raw) throws SyntaxError {
+ protected Query getFieldQuery(String field, List<String> queryTerms, boolean raw)
+ throws SyntaxError {
checkNullField(field);
SchemaField sf;
if (field.equals(lastFieldName)) {
- // only look up the SchemaField on a field change... this helps with memory allocation of dynamic fields
+ // only look up the SchemaField on a field change... this helps with memory allocation of
+ // dynamic fields
// and large queries like foo_i:(1 2 3 4 5 6 7 8 9 10) when we are passed "foo_i" each time.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/request/json/JsonQueryConverter.java
##########
@@ -38,31 +37,41 @@ String toLocalParams(Object jsonQueryObject, Map<String, String[]> additionalPar
}
private String putParam(String val, Map<String, String[]> additionalParams) {
- String name = "_tt"+(numParams++);
- additionalParams.put(name, new String[]{val});
+ String name = "_tt" + (numParams++);
+ additionalParams.put(name, new String[] {val});
return name;
}
- // when isQParser==true, "val" is a query object of the form {query_type:{param1:val1, param2:val2}}
- // when isQParser==false, "val" is a parameter on an existing qparser (which could be a simple parameter like 42, or a sub-query)
+ // when isQParser==true, "val" is a query object of the form {query_type:{param1:val1,
+ // param2:val2}}
+ // when isQParser==false, "val" is a parameter on an existing qparser (which could be a simple
+ // parameter like 42, or a sub-query)
@SuppressWarnings({"unchecked"})
- private void buildLocalParams(StringBuilder builder, Object val, boolean isQParser, Map<String, String[]> additionalParams) {
+ private void buildLocalParams(
+ StringBuilder builder,
+ Object val,
+ boolean isQParser,
+ Map<String, String[]> additionalParams) {
if (!isQParser && !(val instanceof Map)) {
// val is value of a query parser, and it is not a map
- throw new SolrException(SolrException.ErrorCode.BAD_REQUEST,
- "Error when parsing json query, expect a json object here, but found : "+val);
- // NOTE: a top-level query *can* be a String, so we should really allow it here. This currently only works because
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Error when parsing json query, expect a json object here, but found : " + val);
+ // NOTE: a top-level query *can* be a String, so we should really allow it here. This
+ // currently only works because
// we special-case String in toLocalParams() and don't call this method.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/request/json/RequestUtil.java
##########
@@ -35,51 +37,64 @@
import org.noggit.JSONParser;
import org.noggit.ObjectBuilder;
-import static org.apache.solr.common.params.CommonParams.JSON;
-import static org.apache.solr.common.params.CommonParams.SORT;
-
public class RequestUtil {
/**
- * Set default-ish params on a SolrQueryRequest as well as do standard macro processing and JSON request parsing.
+ * Set default-ish params on a SolrQueryRequest as well as do standard macro processing and JSON
+ * request parsing.
*
- * @param handler The search handler this is for (may be null if you don't want this method touching the content streams)
+ * @param handler The search handler this is for (may be null if you don't want this method
+ * touching the content streams)
* @param req The request whose params we are interested in
* @param defaults values to be used if no values are specified in the request params
- * @param appends values to be appended to those from the request (or defaults) when dealing with multi-val params, or treated as another layer of defaults for singl-val params.
- * @param invariants values which will be used instead of any request, or default values, regardless of context.
+ * @param appends values to be appended to those from the request (or defaults) when dealing with
+ * multi-val params, or treated as another layer of defaults for singl-val params.
+ * @param invariants values which will be used instead of any request, or default values,
+ * regardless of context.
*/
- public static void processParams(SolrRequestHandler handler, SolrQueryRequest req, SolrParams defaults,
- SolrParams appends, SolrParams invariants) {
+ public static void processParams(
+ SolrRequestHandler handler,
+ SolrQueryRequest req,
+ SolrParams defaults,
+ SolrParams appends,
+ SolrParams invariants) {
boolean searchHandler = handler instanceof SearchHandler;
SolrParams params = req.getParams();
// Handle JSON stream for search requests
if (searchHandler && req.getContentStreams() != null) {
- Map<String,String[]> map = MultiMapSolrParams.asMultiMap(params, false);
+ Map<String, String[]> map = MultiMapSolrParams.asMultiMap(params, false);
if (!(params instanceof MultiMapSolrParams || params instanceof ModifiableSolrParams)) {
// need to set params on request since we weren't able to access the original map
params = new MultiMapSolrParams(map);
req.setParams(params);
}
- String[] jsonFromParams = map.remove(JSON); // params from the query string should come after (and hence override) JSON content streams
+ String[] jsonFromParams =
+ map.remove(
+ JSON); // params from the query string should come after (and hence override) JSON
+ // content streams
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/request/json/RequestUtil.java
##########
@@ -93,20 +108,21 @@ public static void processParams(SolrRequestHandler handler, SolrQueryRequest re
String[] jsonS = params.getParams(JSON);
- boolean hasAdditions = defaults != null || invariants != null || appends != null || jsonS != null;
+ boolean hasAdditions =
+ defaults != null || invariants != null || appends != null || jsonS != null;
// short circuit processing
if (!hasAdditions && !params.getBool("expandMacros", true)) {
- return; // nothing to do...
+ return; // nothing to do...
}
boolean isShard = params.getBool("isShard", false);
Map<String, String[]> newMap = MultiMapSolrParams.asMultiMap(params, hasAdditions);
-
// see if the json has a "params" section
- // TODO: we should currently *not* do this if this is a leaf of a distributed search since it could overwrite parameters set by the top-level
+ // TODO: we should currently *not* do this if this is a leaf of a distributed search since it
+ // could overwrite parameters set by the top-level
// The parameters we extract will be propagated anyway.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/request/json/RequestUtil.java
##########
@@ -161,15 +176,15 @@ public static void processParams(SolrRequestHandler handler, SolrQueryRequest re
}
// Set these params as soon as possible so if there is an error processing later, things like
// "wt=json" will take effect from the defaults.
- SolrParams newParams = new MultiMapSolrParams(newMap); // newMap may still change below, but that should be OK
+ SolrParams newParams =
+ new MultiMapSolrParams(newMap); // newMap may still change below, but that should be OK
req.setParams(newParams);
-
- // Skip the rest of the processing (including json processing for now) if this isn't a search handler.
+ // Skip the rest of the processing (including json processing for now) if this isn't a search
+ // handler.
// For example json.command started to be used in SOLR-6294, and that caused errors here.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/request/json/RequestUtil.java
##########
@@ -306,13 +338,14 @@ private static void getParamsFromJSON(Map<String, String[]> params, String json)
return;
}
- parser.nextEvent(); // advance to the value
+ parser.nextEvent(); // advance to the value
Object o = ObjectBuilder.getVal(parser);
if (!(o instanceof Map)) return;
@SuppressWarnings("unchecked")
- Map<String,Object> map = (Map<String,Object>)o;
- // To make consistent with json.param handling, we should make query params come after json params (i.e. query params should
+ Map<String, Object> map = (Map<String, Object>) o;
+ // To make consistent with json.param handling, we should make query params come after json
+ // params (i.e. query params should
// appear to overwrite json params.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/response/DocsStreamer.java
##########
@@ -103,63 +100,65 @@ public SolrDocument next() {
transformer.transform(sdoc, id);
}
} catch (IOException e) {
- throw new SolrException(SolrException.ErrorCode.SERVER_ERROR, "Error applying transformer", e);
+ throw new SolrException(
+ SolrException.ErrorCode.SERVER_ERROR, "Error applying transformer", e);
}
}
return sdoc;
-
}
/**
- * This method is less efficient then the 3 arg version because it may convert some fields that
+ * This method is less efficient then the 3 arg version because it may convert some fields that
* are not needed
*
* @deprecated use the 3 arg version for better performance
* @see #convertLuceneDocToSolrDoc(Document,IndexSchema,ReturnFields)
*/
@Deprecated
public static SolrDocument convertLuceneDocToSolrDoc(Document doc, final IndexSchema schema) {
- return convertLuceneDocToSolrDoc(doc,schema, new SolrReturnFields());
+ return convertLuceneDocToSolrDoc(doc, schema, new SolrReturnFields());
}
-
+
/**
* Converts the specified <code>Document</code> into a <code>SolrDocument</code>.
- * <p>
- * The use of {@link ReturnFields} can be important even when it was already used to retrieve the
- * {@link Document} from {@link SolrDocumentFetcher} because the Document may have been cached with
- * more fields then are desired.
- * </p>
- *
+ *
+ * <p>The use of {@link ReturnFields} can be important even when it was already used to retrieve
+ * the {@link Document} from {@link SolrDocumentFetcher} because the Document may have been cached
+ * with more fields then are desired.
+ *
* @param doc <code>Document</code> to be converted, must not be null
- * @param schema <code>IndexSchema</code> containing the field/fieldType details for the index
- * the <code>Document</code> came from, must not be null.
- * @param fields <code>ReturnFields</code> instance that can be use to limit the set of fields
- * that will be converted, must not be null
+ * @param schema <code>IndexSchema</code> containing the field/fieldType details for the index the
+ * <code>Document</code> came from, must not be null.
+ * @param fields <code>ReturnFields</code> instance that can be use to limit the set of fields
+ * that will be converted, must not be null
*/
- public static SolrDocument convertLuceneDocToSolrDoc(Document doc,
- final IndexSchema schema,
- final ReturnFields fields) {
- // TODO move to SolrDocumentFetcher ? Refactor to also call docFetcher.decorateDocValueFields(...) ?
+ public static SolrDocument convertLuceneDocToSolrDoc(
+ Document doc, final IndexSchema schema, final ReturnFields fields) {
+ // TODO move to SolrDocumentFetcher ? Refactor to also call
+ // docFetcher.decorateDocValueFields(...) ?
assert null != doc;
assert null != schema;
assert null != fields;
-
+
// can't just use fields.wantsField(String)
// because that doesn't include extra fields needed by transformers
final Set<String> fieldNamesNeeded = fields.getLuceneFieldNames();
BinaryResponseWriter.MaskCharSeqSolrDocument masked = null;
- final SolrDocument out = ResultContext.READASBYTES.get() == null ?
- new SolrDocument() :
- (masked = new BinaryResponseWriter.MaskCharSeqSolrDocument());
+ final SolrDocument out =
+ ResultContext.READASBYTES.get() == null
+ ? new SolrDocument()
+ : (masked = new BinaryResponseWriter.MaskCharSeqSolrDocument());
// NOTE: it would be tempting to try and optimize this to loop over fieldNamesNeeded
- // when it's smaller then the IndexableField[] in the Document -- but that's actually *less* effecient
- // since Document.getFields(String) does a full (internal) iteration over the full IndexableField[]
+ // when it's smaller then the IndexableField[] in the Document -- but that's actually *less*
+ // effecient
+ // since Document.getFields(String) does a full (internal) iteration over the full
+ // IndexableField[]
// see SOLR-11891
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/request/macro/MacroExpander.java
##########
@@ -116,25 +117,25 @@ private String _expand(String val) {
int idx = val.indexOf(macroStart.charAt(0));
if (idx < 0) return val;
- int start = 0; // start of the unprocessed part of the string
+ int start = 0; // start of the unprocessed part of the string
StringBuilder sb = null;
- for (;;) {
+ for (; ; ) {
assert idx >= start;
idx = val.indexOf(macroStart, idx);
// check if escaped
if (idx > 0) {
// check if escaped...
- // TODO: what if you *want* to actually have a backslash... perhaps that's when we allow changing
+ // TODO: what if you *want* to actually have a backslash... perhaps that's when we allow
+ // changing
// of the escape character?
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/rest/schema/FieldTypeXmlAdapter.java
##########
@@ -99,88 +97,102 @@ protected static Element transformSimilarity(Document doc, Map<String,?> json, S
return similarity;
}
- /** Convert types produced by noggit's ObjectBuilder (Boolean, Double, Long, String) to plugin param XML tags. */
+ /**
+ * Convert types produced by noggit's ObjectBuilder (Boolean, Double, Long, String) to plugin
+ * param XML tags.
+ */
protected static String classToXmlTag(Class<?> clazz) {
switch (clazz.getSimpleName()) {
- case "Boolean": return "bool";
- case "Double": return "double";
- case "Long": return "long";
- case "String": return "str";
+ case "Boolean":
+ return "bool";
+ case "Double":
+ return "double";
+ case "Long":
+ return "long";
+ case "String":
+ return "str";
}
- throw new SolrException(ErrorCode.BAD_REQUEST, "Unsupported object type '" + clazz.getSimpleName() + "'");
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST, "Unsupported object type '" + clazz.getSimpleName() + "'");
}
-
+
@SuppressWarnings("unchecked")
- protected static Element transformAnalyzer(Document doc, Map<String,?> json, String jsonFieldName, String analyzerType) {
+ protected static Element transformAnalyzer(
+ Document doc, Map<String, ?> json, String jsonFieldName, String analyzerType) {
Object jsonField = json.get(jsonFieldName);
- if (jsonField == null)
- return null; // it's ok for this field to not exist in the JSON map
-
+ if (jsonField == null) return null; // it's ok for this field to not exist in the JSON map
+
if (!(jsonField instanceof Map))
- throw new SolrException(ErrorCode.BAD_REQUEST, "Invalid fieldType definition! Expected JSON object for "+
- jsonFieldName+" not a "+jsonField.getClass().getName());
-
- return createAnalyzerElement(doc, analyzerType, (Map<String,?>)jsonField);
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST,
+ "Invalid fieldType definition! Expected JSON object for "
+ + jsonFieldName
+ + " not a "
+ + jsonField.getClass().getName());
+
+ return createAnalyzerElement(doc, analyzerType, (Map<String, ?>) jsonField);
}
-
+
@SuppressWarnings("unchecked")
- protected static Element createAnalyzerElement(Document doc, String type, Map<String,?> analyzer) {
+ protected static Element createAnalyzerElement(
+ Document doc, String type, Map<String, ?> analyzer) {
Element analyzerElem = appendAttrs(doc.createElement("analyzer"), analyzer);
- if (type != null)
- analyzerElem.setAttribute("type", type);
+ if (type != null) analyzerElem.setAttribute("type", type);
- List<Map<String,?>> charFilters = (List<Map<String,?>>)analyzer.get("charFilters");
- Map<String,?> tokenizer = (Map<String,?>)analyzer.get("tokenizer");
- List<Map<String,?>> filters = (List<Map<String,?>>)analyzer.get("filters");
+ List<Map<String, ?>> charFilters = (List<Map<String, ?>>) analyzer.get("charFilters");
+ Map<String, ?> tokenizer = (Map<String, ?>) analyzer.get("tokenizer");
+ List<Map<String, ?>> filters = (List<Map<String, ?>>) analyzer.get("filters");
if (analyzer.get("class") == null) {
- if (charFilters != null)
- appendFilterElements(doc, analyzerElem, "charFilter", charFilters);
+ if (charFilters != null) appendFilterElements(doc, analyzerElem, "charFilter", charFilters);
if (tokenizer == null)
throw new SolrException(ErrorCode.BAD_REQUEST, "Analyzer must define a tokenizer!");
if (tokenizer.get("class") == null && tokenizer.get("name") == null)
- throw new SolrException(ErrorCode.BAD_REQUEST, "Every tokenizer must define a class or name property!");
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST, "Every tokenizer must define a class or name property!");
analyzerElem.appendChild(appendAttrs(doc.createElement("tokenizer"), tokenizer));
- if (filters != null)
- appendFilterElements(doc, analyzerElem, "filter", filters);
+ if (filters != null) appendFilterElements(doc, analyzerElem, "filter", filters);
- } else { // When analyzer class is specified: char filters, tokenizers, and filters are disallowed
+ } else { // When analyzer class is specified: char filters, tokenizers, and filters are
+ // disallowed
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/response/transform/TransformerFactory.java
##########
@@ -19,94 +19,105 @@
import java.util.HashMap;
import java.util.Map;
import java.util.Set;
-
import org.apache.solr.common.params.SolrParams;
import org.apache.solr.common.util.NamedList;
import org.apache.solr.request.SolrQueryRequest;
import org.apache.solr.util.plugin.NamedListInitializedPlugin;
-/**
- * New instance for each request
- *
- *
- */
-public abstract class TransformerFactory implements NamedListInitializedPlugin
-{
+/** New instance for each request */
+public abstract class TransformerFactory implements NamedListInitializedPlugin {
protected String defaultUserArgs = null;
@Override
public void init(NamedList<?> args) {
- defaultUserArgs = (String)args.get( "args" );
+ defaultUserArgs = (String) args.get("args");
}
public abstract DocTransformer create(String field, SolrParams params, SolrQueryRequest req);
/**
- * The {@link FieldRenamer} interface should be implemented by any {@link TransformerFactory} capable of generating
- * transformers that might rename fields, and should implement {@link #create(String, SolrParams, SolrQueryRequest, Map, Set)}
- * in place of {@link #create(String, SolrParams, SolrQueryRequest)} (with the latter method
- * overridden to throw {@link UnsupportedOperationException}).
+ * The {@link FieldRenamer} interface should be implemented by any {@link TransformerFactory}
+ * capable of generating transformers that might rename fields, and should implement {@link
+ * #create(String, SolrParams, SolrQueryRequest, Map, Set)} in place of {@link #create(String,
+ * SolrParams, SolrQueryRequest)} (with the latter method overridden to throw {@link
+ * UnsupportedOperationException}).
*
- * {@link DocTransformer}s returned via {@link #create(String, SolrParams, SolrQueryRequest, Map, Set)}
- * will be added in a second pass, allowing simplified logic in {@link TransformerFactory#create(String, SolrParams, SolrQueryRequest)}
- * for non-renaming factories.
+ * <p>{@link DocTransformer}s returned via {@link #create(String, SolrParams, SolrQueryRequest,
+ * Map, Set)} will be added in a second pass, allowing simplified logic in {@link
+ * TransformerFactory#create(String, SolrParams, SolrQueryRequest)} for non-renaming factories.
*
- * {@link #create(String, SolrParams, SolrQueryRequest, Map, Set)} must implement extra logic to be aware of
- * preceding field renames, and to make subsequent {@link FieldRenamer} transformers aware of its own field renames.
+ * <p>{@link #create(String, SolrParams, SolrQueryRequest, Map, Set)} must implement extra logic
+ * to be aware of preceding field renames, and to make subsequent {@link FieldRenamer}
+ * transformers aware of its own field renames.
*
- * It is harmless for a {@link DocTransformer} that does _not_ in practice rename fields to be returned from a
- * factory that implements this interface (e.g., for conditional renames?); but doing so opens the possibility of
- * {@link #create(String, SolrParams, SolrQueryRequest, Map, Set)} being called _after_ fields have been renamed,
- * so such implementations must still check whether the field with which they are concerned has been renamed ...
- * and if it _has_, must copy the field back to its original name. This situation also demonstrates the
- * motivation for separating the creation of {@link DocTransformer}s into two phases: an initial phase involving
- * no field renames, and a subsequent phase that implement extra logic to properly handle field renames.
+ * <p>It is harmless for a {@link DocTransformer} that does _not_ in practice rename fields to be
+ * returned from a factory that implements this interface (e.g., for conditional renames?); but
+ * doing so opens the possibility of {@link #create(String, SolrParams, SolrQueryRequest, Map,
+ * Set)} being called _after_ fields have been renamed, so such implementations must still check
+ * whether the field with which they are concerned has been renamed ... and if it _has_, must copy
+ * the field back to its original name. This situation also demonstrates the motivation for
+ * separating the creation of {@link DocTransformer}s into two phases: an initial phase involving
+ * no field renames, and a subsequent phase that implement extra logic to properly handle field
+ * renames.
*/
public interface FieldRenamer {
- // TODO: Behavior is undefined in the event of a "destination field" collision (e.g., a user maps two fields to
- // the same "destination field", or maps a field to a top-level requested field). In the future, the easiest way
- // to detect such a case would be by "failing fast" upon renaming to a field that already has an associated value,
+ // TODO: Behavior is undefined in the event of a "destination field" collision (e.g., a user
+ // maps two fields to
+ // the same "destination field", or maps a field to a top-level requested field). In the
+ // future, the easiest way
+ // to detect such a case would be by "failing fast" upon renaming to a field that already has
+ // an associated value,
// or support for this feature could be expressly added via a hypothetical
// `combined_field:[consolidate fl=field_1,field_2]` transformer.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/response/transform/ChildDocTransformer.java
##########
@@ -154,15 +162,22 @@ public void transform(SolrDocument rootDoc, int rootDocId) {
}
// we'll need this soon...
- final SortedDocValues segPathDocValues = DocValues.getSorted(leafReaderContext.reader(), NEST_PATH_FIELD_NAME);
- // passing a different SortedDocValues obj since the child documents which come after are of smaller docIDs,
+ final SortedDocValues segPathDocValues =
+ DocValues.getSorted(leafReaderContext.reader(), NEST_PATH_FIELD_NAME);
+ // passing a different SortedDocValues obj since the child documents which come after are of
+ // smaller docIDs,
// and the iterator can not be reversed.
- // The root doc is the input document to be transformed, and is not necessarily the root doc of the block of docs.
- final String rootDocPath = getPathByDocId(segRootId, DocValues.getSorted(leafReaderContext.reader(), NEST_PATH_FIELD_NAME));
-
- // the key in the Map is the document's ancestors key (one above the parent), while the key in the intermediate
+ // The root doc is the input document to be transformed, and is not necessarily the root doc
+ // of the block of docs.
+ final String rootDocPath =
+ getPathByDocId(
+ segRootId, DocValues.getSorted(leafReaderContext.reader(), NEST_PATH_FIELD_NAME));
+
+ // the key in the Map is the document's ancestors key (one above the parent), while the key in
+ // the intermediate
// MultiMap is the direct child document's key(of the parent document)
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/schema/AbstractSpatialPrefixTreeFieldType.java
##########
@@ -152,38 +178,41 @@ public Analyzer getIndexAnalyzer() {
protected TokenStreamComponents createComponents(String fieldName) {
PrefixTreeStrategy s = newSpatialStrategy(fieldName == null ? getTypeName() : fieldName);
PrefixTreeStrategy.ShapeTokenStream ts = s.tokenStream();
- return new TokenStreamComponents(r -> {
- try {
- ts.setShape(parseShape(IOUtils.toString(r)));
- } catch (IOException e) {
- throw new RuntimeException(e);
- }
- }, ts);
+ return new TokenStreamComponents(
+ r -> {
+ try {
+ ts.setShape(parseShape(IOUtils.toString(r)));
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ },
+ ts);
}
};
}
-
+
@Override
- public Analyzer getQueryAnalyzer()
- {
+ public Analyzer getQueryAnalyzer() {
return getIndexAnalyzer();
}
@Override
protected T newSpatialStrategy(String fieldName) {
T strat = newPrefixTreeStrategy(fieldName);
- if (distErrPct != null)
- strat.setDistErrPct(distErrPct);
+ if (distErrPct != null) strat.setDistErrPct(distErrPct);
if (defaultFieldValuesArrayLen != null)
strat.setDefaultFieldValuesArrayLen(defaultFieldValuesArrayLen);
if (log.isInfoEnabled()) {
- log.info("{} strat: {} maxLevels: {}", this, strat, grid.getMaxLevels());//TODO output maxDetailKm
+ log.info(
+ "{} strat: {} maxLevels: {}",
+ this,
+ strat,
+ grid.getMaxLevels()); // TODO output maxDetailKm
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/schema/AbstractSpatialFieldType.java
##########
@@ -310,30 +326,39 @@ public Query createSpatialQuery(QParser parser, SpatialOptions options) {
double distDeg = DistanceUtils.dist2Degrees(options.distance, options.radius);
Shape shape = ctx.makeCircle(pt, distDeg);
- if (options.bbox)
- shape = shape.getBoundingBox();
+ if (options.bbox) shape = shape.getBoundingBox();
SpatialArgs spatialArgs = new SpatialArgs(SpatialOperation.Intersects, shape);
return getQueryFromSpatialArgs(parser, options.field, spatialArgs);
}
@Override
- protected Query getSpecializedRangeQuery(QParser parser, SchemaField field, String part1, String part2, boolean minInclusive, boolean maxInclusive) {
+ protected Query getSpecializedRangeQuery(
+ QParser parser,
+ SchemaField field,
+ String part1,
+ String part2,
+ boolean minInclusive,
+ boolean maxInclusive) {
if (!minInclusive || !maxInclusive)
- throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "Both sides of spatial range query must be inclusive: " + field.getName());
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Both sides of spatial range query must be inclusive: " + field.getName());
Point p1 = SpatialUtils.parsePointSolrException(part1, ctx);
Point p2 = SpatialUtils.parsePointSolrException(part2, ctx);
Rectangle bbox = ctx.makeRectangle(p1, p2);
SpatialArgs spatialArgs = new SpatialArgs(SpatialOperation.Intersects, bbox);
- return getQueryFromSpatialArgs(parser, field, spatialArgs);//won't score by default
+ return getQueryFromSpatialArgs(parser, field, spatialArgs); // won't score by default
}
@Override
public ValueSource getValueSource(SchemaField field, QParser parser) {
- //This is different from Solr 3 LatLonType's approach which uses the MultiValueSource concept to directly expose
+ // This is different from Solr 3 LatLonType's approach which uses the MultiValueSource concept
+ // to directly expose
// the x & y pair of FieldCache value sources.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/schema/BBoxField.java
##########
@@ -103,22 +104,23 @@ public void inform(IndexSchema schema) {
}
}
- private void registerSubFields(IndexSchema schema, String name, FieldType numberType, FieldType booleanType) {
+ private void registerSubFields(
+ IndexSchema schema, String name, FieldType numberType, FieldType booleanType) {
register(schema, name + BBoxStrategy.SUFFIX_MINX, numberType);
register(schema, name + BBoxStrategy.SUFFIX_MAXX, numberType);
register(schema, name + BBoxStrategy.SUFFIX_MINY, numberType);
register(schema, name + BBoxStrategy.SUFFIX_MAXY, numberType);
register(schema, name + BBoxStrategy.SUFFIX_XDL, booleanType);
}
- // note: Registering the field is probably optional; it makes it show up in the schema browser and may have other
+ // note: Registering the field is probably optional; it makes it show up in the schema browser and
+ // may have other
// benefits.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/schema/CurrencyField.java
##########
@@ -67,17 +67,20 @@ protected void init(IndexSchema schema, Map<String, String> args) {
args.remove(PARAM_PRECISION_STEP);
}
- // NOTE: because we're not using the PluginLoader to register these field types, they aren't "real"
+ // NOTE: because we're not using the PluginLoader to register these field types, they aren't
+ // "real"
// field types and never get Schema default properties (based on schema.xml's version attribute)
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/schema/DateRangeField.java
##########
@@ -130,28 +138,36 @@ public Date parseMath(Date now, String rawval) {
@Override
protected SpatialArgs parseSpatialArgs(QParser parser, String externalVal) {
- //We avoid SpatialArgsParser entirely because it isn't very Solr-friendly
+ // We avoid SpatialArgsParser entirely because it isn't very Solr-friendly
final Shape shape = parseShape(externalVal);
final SolrParams localParams = parser.getLocalParams();
SpatialOperation op = SpatialOperation.Intersects;
if (localParams != null) {
String opStr = localParams.get(OP_PARAM);
- if (opStr != null)
- op = SpatialOperation.get(opStr);
+ if (opStr != null) op = SpatialOperation.get(opStr);
}
return new SpatialArgs(op, shape);
}
@Override
- protected Query getSpecializedRangeQuery(QParser parser, SchemaField field, String startStr, String endStr, boolean minInclusive, boolean maxInclusive) {
- if (parser == null) {//null when invoked by SimpleFacets. But getQueryFromSpatialArgs expects to get localParams.
+ protected Query getSpecializedRangeQuery(
+ QParser parser,
+ SchemaField field,
+ String startStr,
+ String endStr,
+ boolean minInclusive,
+ boolean maxInclusive) {
+ if (parser
+ == null) { // null when invoked by SimpleFacets. But getQueryFromSpatialArgs expects to get
+ // localParams.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/response/transform/ChildDocTransformer.java
##########
@@ -154,15 +162,22 @@ public void transform(SolrDocument rootDoc, int rootDocId) {
}
// we'll need this soon...
- final SortedDocValues segPathDocValues = DocValues.getSorted(leafReaderContext.reader(), NEST_PATH_FIELD_NAME);
- // passing a different SortedDocValues obj since the child documents which come after are of smaller docIDs,
+ final SortedDocValues segPathDocValues =
+ DocValues.getSorted(leafReaderContext.reader(), NEST_PATH_FIELD_NAME);
+ // passing a different SortedDocValues obj since the child documents which come after are of
+ // smaller docIDs,
// and the iterator can not be reversed.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/schema/LatLonPointSpatialField.java
##########
@@ -88,22 +91,27 @@ public String toExternal(IndexableField f) {
/**
* Decodes the docValues number into latitude and longitude components, formatting as "lat,lon".
- * The encoding is governed by {@code LatLonDocValuesField}. The decimal output representation is reflective
- * of the available precision.
+ * The encoding is governed by {@code LatLonDocValuesField}. The decimal output representation is
+ * reflective of the available precision.
+ *
* @param value Non-null; stored location field data
* @return Non-null; "lat, lon"
*/
public static String decodeDocValueToString(long value) {
final double latDouble = GeoEncodingUtils.decodeLatitude((int) (value >> 32));
final double lonDouble = GeoEncodingUtils.decodeLongitude((int) (value & 0xFFFFFFFFL));
- // This # decimal places gets us close to our available precision to 1.40cm; we have a test for it.
- // CEILING round-trips (decode then re-encode then decode to get identical results). Others did not. It also
+ // This # decimal places gets us close to our available precision to 1.40cm; we have a test for
+ // it.
+ // CEILING round-trips (decode then re-encode then decode to get identical results). Others did
+ // not. It also
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/schema/ManagedIndexSchema.java
##########
@@ -429,8 +476,10 @@ public ManagedIndexSchema addFields(Collection<SchemaField> newFields,
newSchema.refreshAnalyzers();
- if(persist) {
- success = newSchema.persistManagedSchema(false); // don't just create - update it if it already exists
+ if (persist) {
+ success =
+ newSchema.persistManagedSchema(
+ false); // don't just create - update it if it already exists
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/schema/ManagedIndexSchema.java
##########
@@ -1036,62 +1148,75 @@ public ManagedIndexSchema deleteFieldTypes(Collection<String> names) {
}
return newSchema;
}
-
- private Map<String,List<CopyField>> cloneCopyFieldsMap(Map<String,List<CopyField>> original) {
- Map<String,List<CopyField>> clone = new HashMap<>(original.size());
- Iterator<Map.Entry<String,List<CopyField>>> iterator = original.entrySet().iterator();
+
+ private Map<String, List<CopyField>> cloneCopyFieldsMap(Map<String, List<CopyField>> original) {
+ Map<String, List<CopyField>> clone = new HashMap<>(original.size());
+ Iterator<Map.Entry<String, List<CopyField>>> iterator = original.entrySet().iterator();
while (iterator.hasNext()) {
- Map.Entry<String,List<CopyField>> entry = iterator.next();
+ Map.Entry<String, List<CopyField>> entry = iterator.next();
clone.put(entry.getKey(), new ArrayList<>(entry.getValue()));
}
return clone;
}
@Override
@SuppressWarnings({"unchecked"})
- public ManagedIndexSchema replaceFieldType(String typeName, String replacementClassName, Map<String,Object> replacementArgs) {
+ public ManagedIndexSchema replaceFieldType(
+ String typeName, String replacementClassName, Map<String, Object> replacementArgs) {
ManagedIndexSchema newSchema;
if (isMutable) {
- if ( ! fieldTypes.containsKey(typeName)) {
- String msg = "The field type '" + typeName + "' is not present in this schema, and so cannot be replaced.";
+ if (!fieldTypes.containsKey(typeName)) {
+ String msg =
+ "The field type '"
+ + typeName
+ + "' is not present in this schema, and so cannot be replaced.";
throw new SolrException(ErrorCode.BAD_REQUEST, msg);
}
newSchema = shallowCopy(true);
// clone data structures before modifying them
- newSchema.fieldTypes = (Map<String,FieldType>)((HashMap<String,FieldType>)fieldTypes).clone();
+ newSchema.fieldTypes =
+ (Map<String, FieldType>) ((HashMap<String, FieldType>) fieldTypes).clone();
newSchema.copyFieldsMap = cloneCopyFieldsMap(copyFieldsMap);
- newSchema.copyFieldTargetCounts
- = (Map<SchemaField,Integer>)((HashMap<SchemaField,Integer>)copyFieldTargetCounts).clone();
+ newSchema.copyFieldTargetCounts =
+ (Map<SchemaField, Integer>)
+ ((HashMap<SchemaField, Integer>) copyFieldTargetCounts).clone();
newSchema.dynamicCopyFields = new DynamicCopy[dynamicCopyFields.length];
- System.arraycopy(dynamicCopyFields, 0, newSchema.dynamicCopyFields, 0, dynamicCopyFields.length);
+ System.arraycopy(
+ dynamicCopyFields, 0, newSchema.dynamicCopyFields, 0, dynamicCopyFields.length);
newSchema.dynamicFields = new DynamicField[dynamicFields.length];
System.arraycopy(dynamicFields, 0, newSchema.dynamicFields, 0, dynamicFields.length);
-
+
newSchema.fieldTypes.remove(typeName);
- FieldType replacementFieldType = newSchema.newFieldType(typeName, replacementClassName, replacementArgs);
+ FieldType replacementFieldType =
+ newSchema.newFieldType(typeName, replacementClassName, replacementArgs);
newSchema.fieldTypes.put(typeName, replacementFieldType);
// Rebuild fields of the type being replaced
List<CopyField> copyFieldsToRebuild = new ArrayList<>();
List<SchemaField> replacementFields = new ArrayList<>();
- Iterator<Map.Entry<String,SchemaField>> fieldsIter = newSchema.fields.entrySet().iterator();
+ Iterator<Map.Entry<String, SchemaField>> fieldsIter = newSchema.fields.entrySet().iterator();
while (fieldsIter.hasNext()) {
- Map.Entry<String,SchemaField> entry = fieldsIter.next();
+ Map.Entry<String, SchemaField> entry = fieldsIter.next();
SchemaField oldField = entry.getValue();
if (oldField.getType().getTypeName().equals(typeName)) {
String fieldName = oldField.getName();
-
+
// Drop the old field
fieldsIter.remove();
newSchema.fieldsWithDefaultValue.remove(oldField);
newSchema.requiredFields.remove(oldField);
-
+
// Add the replacement field
- SchemaField replacementField = SchemaField.create(fieldName, replacementFieldType, oldField.getArgs());
- replacementFields.add(replacementField); // Save the new field to be added after iteration is finished
+ SchemaField replacementField =
+ SchemaField.create(fieldName, replacementFieldType, oldField.getArgs());
+ replacementFields.add(
+ replacementField); // Save the new field to be added after iteration is finished
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/schema/LatLonPointSpatialField.java
##########
@@ -193,40 +203,43 @@ protected Query makeQueryFromDocValues(Shape shape) {
if (shape instanceof Circle) {
Circle circle = (Circle) shape;
double radiusMeters = circle.getRadius() * DistanceUtils.DEG_TO_KM * 1000;
- return LatLonDocValuesField.newSlowDistanceQuery(getFieldName(),
- circle.getCenter().getY(), circle.getCenter().getX(),
- radiusMeters);
+ return LatLonDocValuesField.newSlowDistanceQuery(
+ getFieldName(), circle.getCenter().getY(), circle.getCenter().getX(), radiusMeters);
} else if (shape instanceof Rectangle) {
Rectangle rect = (Rectangle) shape;
- return LatLonDocValuesField.newSlowBoxQuery(getFieldName(),
- rect.getMinY(), rect.getMaxY(), rect.getMinX(), rect.getMaxX());
+ return LatLonDocValuesField.newSlowBoxQuery(
+ getFieldName(), rect.getMinY(), rect.getMaxY(), rect.getMinX(), rect.getMaxX());
} else if (shape instanceof Point) {
Point point = (Point) shape;
- return LatLonDocValuesField.newSlowDistanceQuery(getFieldName(),
- point.getY(), point.getX(), 0);
+ return LatLonDocValuesField.newSlowDistanceQuery(
+ getFieldName(), point.getY(), point.getX(), 0);
} else {
- throw new UnsupportedOperationException("Shape " + shape.getClass() + " is not supported by " + getClass());
+ throw new UnsupportedOperationException(
+ "Shape " + shape.getClass() + " is not supported by " + getClass());
}
-// } else if (shape instanceof LucenePolygonShape) {
-// // TODO support multi-polygon
-// Polygon poly = ((LucenePolygonShape)shape).lucenePolygon;
-// return LatLonPoint.newPolygonQuery(getFieldName(), poly);
+ // } else if (shape instanceof LucenePolygonShape) {
+ // // TODO support multi-polygon
+ // Polygon poly = ((LucenePolygonShape)shape).lucenePolygon;
+ // return LatLonPoint.newPolygonQuery(getFieldName(), poly);
}
@Override
public DoubleValuesSource makeDistanceValueSource(Point queryPoint, double multiplier) {
if (!docValues) {
- throw new SolrException(SolrException.ErrorCode.BAD_REQUEST,
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
getFieldName() + " must have docValues enabled to support this feature");
}
- // Internally, the distance from LatLonPointSortField/Comparator is in meters. So we must also go from meters to
+ // Internally, the distance from LatLonPointSortField/Comparator is in meters. So we must also
+ // go from meters to
// degrees, which is what Lucene spatial-extras is oriented around.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/schema/ManagedIndexSchemaFactory.java
##########
@@ -414,7 +465,8 @@ private void zkUgradeToManagedSchema() {
// some other node already started the upgrade, or an error occurred - bail out
return;
}
- schema.persistManagedSchemaToZooKeeper(true); // Only create, don't update it if it already exists
+ schema.persistManagedSchemaToZooKeeper(
+ true); // Only create, don't update it if it already exists
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/schema/ManagedIndexSchema.java
##########
@@ -603,7 +670,9 @@ public ManagedIndexSchema addDynamicFields(Collection<SchemaField> newDynamicFie
newSchema.postReadInform();
newSchema.refreshAnalyzers();
if (persist) {
- success = newSchema.persistManagedSchema(false); // don't just create - update it if it already exists
+ success =
+ newSchema.persistManagedSchema(
+ false); // don't just create - update it if it already exists
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/schema/ManagedIndexSchema.java
##########
@@ -759,22 +849,25 @@ public ManagedIndexSchema deleteDynamicFields(Collection<String> fieldNamePatter
}
@Override
- public ManagedIndexSchema addCopyFields(Map<String, Collection<String>> copyFields, boolean persist) {
+ public ManagedIndexSchema addCopyFields(
+ Map<String, Collection<String>> copyFields, boolean persist) {
ManagedIndexSchema newSchema;
if (isMutable) {
boolean success = false;
newSchema = shallowCopy(true);
for (Map.Entry<String, Collection<String>> entry : copyFields.entrySet()) {
- //Key is the name of the field, values are the destinations
+ // Key is the name of the field, values are the destinations
for (String destination : entry.getValue()) {
newSchema.registerCopyField(entry.getKey(), destination);
}
}
newSchema.postReadInform();
newSchema.refreshAnalyzers();
- if(persist) {
- success = newSchema.persistManagedSchema(false); // don't just create - update it if it already exists
+ if (persist) {
+ success =
+ newSchema.persistManagedSchema(
+ false); // don't just create - update it if it already exists
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/schema/NumericFieldType.java
##########
@@ -144,19 +161,22 @@ protected Query getRangeQueryForFloatDoubleDocValues(SchemaField sf, String min,
// If min is negative (or -0d) and max is positive (or +0d), then issue a FunctionRangeQuery
if (minNegative && !maxNegative) {
ValueSource vs = getValueSource(sf, null);
- query = new FunctionRangeQuery(new ValueSourceRangeFilter(vs, minVal.toString(), maxVal.toString(), true, true));
- } else if (minNegative && maxNegative) {// If both max and min are negative (or -0d), then issue range query with max and min reversed
- query = numericDocValuesRangeQuery
- (fieldName, maxBits, minBits, true, true, false);
+ query =
+ new FunctionRangeQuery(
+ new ValueSourceRangeFilter(vs, minVal.toString(), maxVal.toString(), true, true));
+ } else if (minNegative
+ && maxNegative) { // If both max and min are negative (or -0d), then issue range query with
+ // max and min reversed
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/schema/PointType.java
##########
@@ -209,54 +222,62 @@ public Query createSpatialQuery(QParser parser, SpatialOptions options) {
private static final double SIN_PI_DIV_4 = Math.sin(Math.PI / 4);
/**
- * Return the coordinates of a vector that is the corner of a box (upper right or lower left), assuming a Rectangular
- * coordinate system. Note, this does not apply for points on a sphere or ellipse (although it could be used as an
- * approximation).
+ * Return the coordinates of a vector that is the corner of a box (upper right or lower left),
+ * assuming a Rectangular coordinate system. Note, this does not apply for points on a sphere or
+ * ellipse (although it could be used as an approximation).
*
- * @param center The center point
- * @param result Holds the result, potentially resizing if needed.
- * @param distance The d from the center to the corner
- * @param upperRight If true, return the coords for the upper right corner, else return the lower left.
+ * @param center The center point
+ * @param result Holds the result, potentially resizing if needed.
+ * @param distance The d from the center to the corner
+ * @param upperRight If true, return the coords for the upper right corner, else return the lower
+ * left.
* @return The point, either the upperLeft or the lower right
*/
- public static double[] vectorBoxCorner(double[] center, double[] result, double distance, boolean upperRight) {
+ public static double[] vectorBoxCorner(
+ double[] center, double[] result, double distance, boolean upperRight) {
if (result == null || result.length != center.length) {
result = new double[center.length];
}
if (upperRight == false) {
distance = -distance;
}
- //We don't care about the power here,
+ // We don't care about the power here,
// b/c we are always in a rectangular coordinate system, so any norm can be used by
- //using the definition of sine
- distance = SIN_PI_DIV_4 * distance; // sin(Pi/4) == (2^0.5)/2 == opp/hyp == opp/distance, solve for opp, similarly for cosine
+ // using the definition of sine
+ distance =
+ SIN_PI_DIV_4
+ * distance; // sin(Pi/4) == (2^0.5)/2 == opp/hyp == opp/distance, solve for opp,
+ // similarly for cosine
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/schema/PointType.java
##########
@@ -273,26 +294,31 @@ public Query createSpatialQuery(QParser parser, SpatialOptions options) {
}
}
if (i != dimension) {
- throw new SolrException(SolrException.ErrorCode.BAD_REQUEST,
- "incompatible dimension (" + dimension +
- ") and values (" + externalVal + "). Only " + i + " values specified");
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "incompatible dimension ("
+ + dimension
+ + ") and values ("
+ + externalVal
+ + "). Only "
+ + i
+ + " values specified");
}
return out;
}
@Override
public double getSphereRadius() {
- // This won't likely be used. You should probably be using LatLonPointSpatialField instead if you felt the need for this.
+ // This won't likely be used. You should probably be using LatLonPointSpatialField instead if
+ // you felt the need for this.
// This is here just for backward compatibility reasons.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/schema/RandomSortField.java
##########
@@ -102,69 +103,69 @@ public ValueSource getValueSource(SchemaField field, QParser qparser) {
}
@Override
- public void write(TextResponseWriter writer, String name, IndexableField f) throws IOException { }
-
-
- private static FieldComparatorSource randomComparatorSource = new FieldComparatorSource() {
- @Override
- public FieldComparator<Integer> newComparator(final String fieldname, final int numHits, int sortPos, boolean reversed) {
- return new SimpleFieldComparator<Integer>() {
- int seed;
- private final int[] values = new int[numHits];
- int bottomVal;
- int topVal;
-
- @Override
- public int compare(int slot1, int slot2) {
- return values[slot1] - values[slot2]; // values will be positive... no overflow possible.
- }
-
- @Override
- public void setBottom(int slot) {
- bottomVal = values[slot];
- }
-
- @Override
- public void setTopValue(Integer value) {
- topVal = value.intValue();
- }
-
- @Override
- public int compareBottom(int doc) {
- return bottomVal - hash(doc+seed);
- }
-
- @Override
- public void copy(int slot, int doc) {
- values[slot] = hash(doc+seed);
- }
-
- @Override
- protected void doSetNextReader(LeafReaderContext context) {
- seed = getSeed(fieldname, context);
- }
-
- @Override
- public Integer value(int slot) {
- return values[slot];
- }
+ public void write(TextResponseWriter writer, String name, IndexableField f) throws IOException {}
+ private static FieldComparatorSource randomComparatorSource =
+ new FieldComparatorSource() {
@Override
- public int compareTop(int doc) {
- // values will be positive... no overflow possible.
- return topVal - hash(doc+seed);
+ public FieldComparator<Integer> newComparator(
+ final String fieldname, final int numHits, int sortPos, boolean reversed) {
+ return new SimpleFieldComparator<Integer>() {
+ int seed;
+ private final int[] values = new int[numHits];
+ int bottomVal;
+ int topVal;
+
+ @Override
+ public int compare(int slot1, int slot2) {
+ return values[slot1]
+ - values[slot2]; // values will be positive... no overflow possible.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/schema/PointType.java
##########
@@ -209,54 +222,62 @@ public Query createSpatialQuery(QParser parser, SpatialOptions options) {
private static final double SIN_PI_DIV_4 = Math.sin(Math.PI / 4);
/**
- * Return the coordinates of a vector that is the corner of a box (upper right or lower left), assuming a Rectangular
- * coordinate system. Note, this does not apply for points on a sphere or ellipse (although it could be used as an
- * approximation).
+ * Return the coordinates of a vector that is the corner of a box (upper right or lower left),
+ * assuming a Rectangular coordinate system. Note, this does not apply for points on a sphere or
+ * ellipse (although it could be used as an approximation).
*
- * @param center The center point
- * @param result Holds the result, potentially resizing if needed.
- * @param distance The d from the center to the corner
- * @param upperRight If true, return the coords for the upper right corner, else return the lower left.
+ * @param center The center point
+ * @param result Holds the result, potentially resizing if needed.
+ * @param distance The d from the center to the corner
+ * @param upperRight If true, return the coords for the upper right corner, else return the lower
+ * left.
* @return The point, either the upperLeft or the lower right
*/
- public static double[] vectorBoxCorner(double[] center, double[] result, double distance, boolean upperRight) {
+ public static double[] vectorBoxCorner(
+ double[] center, double[] result, double distance, boolean upperRight) {
if (result == null || result.length != center.length) {
result = new double[center.length];
}
if (upperRight == false) {
distance = -distance;
}
- //We don't care about the power here,
+ // We don't care about the power here,
// b/c we are always in a rectangular coordinate system, so any norm can be used by
- //using the definition of sine
- distance = SIN_PI_DIV_4 * distance; // sin(Pi/4) == (2^0.5)/2 == opp/hyp == opp/distance, solve for opp, similarly for cosine
+ // using the definition of sine
+ distance =
+ SIN_PI_DIV_4
+ * distance; // sin(Pi/4) == (2^0.5)/2 == opp/hyp == opp/distance, solve for opp,
+ // similarly for cosine
for (int i = 0; i < center.length; i++) {
result[i] = center[i] + distance;
}
return result;
}
/**
- * Given a string containing <i>dimension</i> values encoded in it, separated by commas,
- * return a String array of length <i>dimension</i> containing the values.
+ * Given a string containing <i>dimension</i> values encoded in it, separated by commas, return a
+ * String array of length <i>dimension</i> containing the values.
*
* @param externalVal The value to parse
- * @param dimension The expected number of values for the point
+ * @param dimension The expected number of values for the point
* @return An array of the values that make up the point (aka vector)
* @throws SolrException if the dimension specified does not match the number found
*/
- public static String[] parseCommaSeparatedList(String externalVal, int dimension) throws SolrException {
- //TODO: Should we support sparse vectors?
+ public static String[] parseCommaSeparatedList(String externalVal, int dimension)
+ throws SolrException {
+ // TODO: Should we support sparse vectors?
String[] out = new String[dimension];
int idx = externalVal.indexOf(',');
int end = idx;
int start = 0;
int i = 0;
- if (idx == -1 && dimension == 1 && externalVal.length() > 0) {//we have a single point, dimension better be 1
+ if (idx == -1
+ && dimension == 1
+ && externalVal.length() > 0) { // we have a single point, dimension better be 1
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/schema/RptWithGeometrySpatialField.java
##########
@@ -157,17 +169,19 @@ public ShapeValues getValues(LeafReaderContext readerContext) throws IOException
@Override
public Shape value() throws IOException {
- //lookup in cache
+ // lookup in cache
IndexReader.CacheHelper cacheHelper = readerContext.reader().getCoreCacheHelper();
if (cacheHelper == null) {
- throw new IllegalStateException("Leaf " + readerContext.reader() + " is not suited for caching");
+ throw new IllegalStateException(
+ "Leaf " + readerContext.reader() + " is not suited for caching");
}
PerSegCacheKey key = new PerSegCacheKey(cacheHelper.getKey(), docId);
Shape shape = cache.computeIfAbsent(key, k -> targetFuncValues.value());
if (shape != null) {
- //optimize shape on a cache hit if possible. This must be thread-safe and it is.
+ // optimize shape on a cache hit if possible. This must be thread-safe and it is.
if (shape instanceof JtsGeometry) {
- ((JtsGeometry) shape).index(); // TODO would be nice if some day we didn't have to cast
+ ((JtsGeometry) shape)
+ .index(); // TODO would be nice if some day we didn't have to cast
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/schema/SchemaManager.java
##########
@@ -121,36 +122,46 @@ public SchemaManager(SolrQueryRequest req){
try {
managedIndexSchema.persist(sw);
} catch (IOException e) {
- throw new SolrException(SolrException.ErrorCode.SERVER_ERROR, "unable to serialize schema");
- //unlikely
+ throw new SolrException(
+ SolrException.ErrorCode.SERVER_ERROR, "unable to serialize schema");
+ // unlikely
}
try {
- latestVersion = ZkController.persistConfigResourceToZooKeeper
- (zkLoader, managedIndexSchema.getSchemaZkVersion(), managedIndexSchema.getResourceName(),
- sw.toString().getBytes(StandardCharsets.UTF_8), true);
- req.getCore().getCoreContainer().reload(req.getCore().getName(), req.getCore().uniqueId);
+ latestVersion =
+ ZkController.persistConfigResourceToZooKeeper(
+ zkLoader,
+ managedIndexSchema.getSchemaZkVersion(),
+ managedIndexSchema.getResourceName(),
+ sw.toString().getBytes(StandardCharsets.UTF_8),
+ true);
+ req.getCore()
+ .getCoreContainer()
+ .reload(req.getCore().getName(), req.getCore().uniqueId);
break;
} catch (ZkController.ResourceModifiedInZkException e) {
log.info("Schema was modified by another node. Retrying..");
}
} else {
try {
- //only for non cloud stuff
+ // only for non cloud stuff
managedIndexSchema.persistManagedSchema(false);
core.setLatestSchema(managedIndexSchema);
core.getCoreContainer().reload(core.getName(), core.uniqueId);
} catch (SolrException e) {
log.warn(errorMsg);
- errors = singletonList(singletonMap(CommandOperation.ERR_MSGS, errorMsg + e.getMessage()));
+ errors =
+ singletonList(singletonMap(CommandOperation.ERR_MSGS, errorMsg + e.getMessage()));
}
break;
}
}
}
if (req.getCore().getResourceLoader() instanceof ZkSolrResourceLoader) {
- // Don't block further schema updates while waiting for a pending update to propagate to other replicas.
- // This reduces the likelihood of a (time-limited) distributed deadlock during concurrent schema updates.
+ // Don't block further schema updates while waiting for a pending update to propagate to other
+ // replicas.
+ // This reduces the likelihood of a (time-limited) distributed deadlock during concurrent
+ // schema updates.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/schema/RptWithGeometrySpatialField.java
##########
@@ -49,13 +52,17 @@
@Override
protected void init(IndexSchema schema, Map<String, String> args) {
- Map<String, String> origArgs = new HashMap<>(args); // clone so we can feed it to an aggregated field type
+ Map<String, String> origArgs =
+ new HashMap<>(args); // clone so we can feed it to an aggregated field type
super.init(schema, origArgs);
- //TODO Move this check to a call from AbstractSpatialFieldType.createFields() so the type can declare
- // if it supports multi-valued or not. It's insufficient here; we can't see if you set multiValued on the field.
+ // TODO Move this check to a call from AbstractSpatialFieldType.createFields() so the type can
+ // declare
+ // if it supports multi-valued or not. It's insufficient here; we can't see if you set
+ // multiValued on the field.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/schema/StrFieldSource.java
##########
@@ -68,13 +68,14 @@ public String toString(int doc) throws IOException {
@Override
public boolean equals(Object o) {
- return o instanceof StrFieldSource
- && super.equals(o);
+ return o instanceof StrFieldSource && super.equals(o);
}
private static int hcode = StrFieldSource.class.hashCode();
+
@Override
public int hashCode() {
return hcode + super.hashCode();
- };
+ }
+ ;
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/schema/SchemaManager.java
##########
@@ -193,28 +211,36 @@ private void waitForOtherReplicasToUpdate(TimeOut timeOut, int latestVersion) {
}
},
ADD_COPY_FIELD("add-copy-field") {
- @Override public boolean perform(CommandOperation op, SchemaManager mgr) {
- String src = op.getStr(SOURCE);
+ @Override
+ public boolean perform(CommandOperation op, SchemaManager mgr) {
+ String src = op.getStr(SOURCE);
List<String> dests = op.getStrs(DESTINATION);
- int maxChars = CopyField.UNLIMITED; // If maxChars is not specified, there is no limit on copied chars
+ int maxChars =
+ CopyField.UNLIMITED; // If maxChars is not specified, there is no limit on copied chars
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/CollapsingQParserPlugin.java
##########
@@ -1915,96 +2112,125 @@ public SortedDocValues getSorted(FieldInfo ignored) throws IOException {
int leafCount = searcher.getTopReaderContext().leaves().size();
SolrRequestInfo req = SolrRequestInfo.getRequestInfo();
- boolean collectElevatedDocsWhenCollapsing = req != null && req.getReq().getParams().getBool(COLLECT_ELEVATED_DOCS_WHEN_COLLAPSING, true);
+ boolean collectElevatedDocsWhenCollapsing =
+ req != null
+ && req.getReq().getParams().getBool(COLLECT_ELEVATED_DOCS_WHEN_COLLAPSING, true);
if (GroupHeadSelectorType.SCORE.equals(groupHeadSelector.type)) {
if (collapseFieldType instanceof StrField) {
if (blockCollapse) {
return new BlockOrdScoreCollector(collapseField, nullPolicy, boostDocs);
}
- return new OrdScoreCollector(maxDoc, leafCount, docValuesProducer, nullPolicy, boostDocs, searcher, collectElevatedDocsWhenCollapsing);
+ return new OrdScoreCollector(
+ maxDoc,
+ leafCount,
+ docValuesProducer,
+ nullPolicy,
+ boostDocs,
+ searcher,
+ collectElevatedDocsWhenCollapsing);
} else if (isNumericCollapsible(collapseFieldType)) {
if (blockCollapse) {
return new BlockIntScoreCollector(collapseField, nullPolicy, boostDocs);
}
- return new IntScoreCollector(maxDoc, leafCount, nullPolicy, size, collapseField, boostDocs, searcher, collectElevatedDocsWhenCollapsing);
+ return new IntScoreCollector(
+ maxDoc,
+ leafCount,
+ nullPolicy,
+ size,
+ collapseField,
+ boostDocs,
+ searcher,
+ collectElevatedDocsWhenCollapsing);
} else {
- throw new SolrException(SolrException.ErrorCode.BAD_REQUEST,
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
"Collapsing field should be of either String, Int or Float type");
}
} else { // min, max, sort, etc.. something other then just "score"
if (collapseFieldType instanceof StrField) {
if (blockCollapse) {
- // NOTE: for now we don't worry about wether this is a sortSpec of min/max groupHeadSelector,
- // we use a "sort spec' based block collector unless/until there is some (performance?) reason to specialize
- return new BlockOrdSortSpecCollector(collapseField, nullPolicy, boostDocs,
- BlockOrdSortSpecCollector.getSort(groupHeadSelector,
- sortSpec, funcQuery, searcher),
- needsScores || needsScores4Collapsing);
+ // NOTE: for now we don't worry about wether this is a sortSpec of min/max
+ // groupHeadSelector,
+ // we use a "sort spec' based block collector unless/until there is some (performance?)
+ // reason to specialize
+ return new BlockOrdSortSpecCollector(
+ collapseField,
+ nullPolicy,
+ boostDocs,
+ BlockOrdSortSpecCollector.getSort(groupHeadSelector, sortSpec, funcQuery, searcher),
+ needsScores || needsScores4Collapsing);
}
- return new OrdFieldValueCollector(maxDoc,
- leafCount,
- docValuesProducer,
- nullPolicy,
- groupHeadSelector,
- sortSpec,
- needsScores4Collapsing,
- needsScores,
- minMaxFieldType,
- boostDocs,
- funcQuery,
- searcher,
- collectElevatedDocsWhenCollapsing);
+ return new OrdFieldValueCollector(
+ maxDoc,
+ leafCount,
+ docValuesProducer,
+ nullPolicy,
+ groupHeadSelector,
+ sortSpec,
+ needsScores4Collapsing,
+ needsScores,
+ minMaxFieldType,
+ boostDocs,
+ funcQuery,
+ searcher,
+ collectElevatedDocsWhenCollapsing);
} else if (isNumericCollapsible(collapseFieldType)) {
if (blockCollapse) {
- // NOTE: for now we don't worry about wether this is a sortSpec of min/max groupHeadSelector,
- // we use a "sort spec' based block collector unless/until there is some (performance?) reason to specialize
- return new BlockIntSortSpecCollector(collapseField, nullPolicy, boostDocs,
- BlockOrdSortSpecCollector.getSort(groupHeadSelector,
- sortSpec, funcQuery, searcher),
- needsScores || needsScores4Collapsing);
+ // NOTE: for now we don't worry about wether this is a sortSpec of min/max
+ // groupHeadSelector,
+ // we use a "sort spec' based block collector unless/until there is some (performance?)
+ // reason to specialize
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/BitDocSet.java
##########
@@ -26,36 +26,36 @@
import org.apache.lucene.util.RamUsageEstimator;
/**
- * A {@link FixedBitSet} based implementation of a {@link DocSet}. Good for medium/large sets.
+ * A {@link FixedBitSet} based implementation of a {@link DocSet}. Good for medium/large sets.
*
* @since solr 0.9
*/
public class BitDocSet extends DocSet {
- private static final long BASE_RAM_BYTES_USED = RamUsageEstimator.shallowSizeOfInstance(BitDocSet.class)
- + RamUsageEstimator.shallowSizeOfInstance(FixedBitSet.class)
- + RamUsageEstimator.NUM_BYTES_ARRAY_HEADER; // for the array object inside the FixedBitSet. long[] array won't change alignment, so no need to calculate it.
+ private static final long BASE_RAM_BYTES_USED =
+ RamUsageEstimator.shallowSizeOfInstance(BitDocSet.class)
+ + RamUsageEstimator.shallowSizeOfInstance(FixedBitSet.class)
+ + RamUsageEstimator
+ .NUM_BYTES_ARRAY_HEADER; // for the array object inside the FixedBitSet. long[] array
+ // won't change alignment, so no need to calculate it.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/CollapsingQParserPlugin.java
##########
@@ -3346,46 +3625,47 @@ public boolean testAndSetGroupValues(int collapseKey, int contextDoc) throws IOE
}
return testAndSetGroupValues(getOrInitGroupHeadValues(collapseKey), contextDoc);
}
-
+
/**
- * Compares the SortField values of the specified contextDoc with the existing group head
- * values for the null group, and overwrites them (and returns true) if this document
- * should become the new group head in accordance with the SortFields.
- * (otherwise returns false)
+ * Compares the SortField values of the specified contextDoc with the existing group head values
+ * for the null group, and overwrites them (and returns true) if this document should become the
+ * new group head in accordance with the SortFields. (otherwise returns false)
*/
public boolean testAndSetNullGroupValues(int contextDoc) throws IOException {
return testAndSetGroupValues(nullGroupValues, contextDoc);
}
/**
- * Compares the SortField values of the specified contextDoc with the existing values
- * array, and overwrites them (and returns true) if this document is the new group head in
- * accordance with the SortFields.
- * (otherwise returns false)
+ * Compares the SortField values of the specified contextDoc with the existing values array, and
+ * overwrites them (and returns true) if this document is the new group head in accordance with
+ * the SortFields. (otherwise returns false)
*/
@SuppressWarnings({"unchecked", "rawtypes"})
private boolean testAndSetGroupValues(Object[] values, int contextDoc) throws IOException {
Object[] stash = new Object[numClauses];
int lastCompare = 0;
int testClause = 0;
- for (/* testClause */; testClause < numClauses; testClause++) {
+ for (
+ /* testClause */ ; testClause < numClauses; testClause++) {
leafFieldComparators[testClause].copy(0, contextDoc);
FieldComparator fcomp = fieldComparators[testClause];
stash[testClause] = cloneIfBytesRef(fcomp.value(0));
- lastCompare = reverseMul[testClause] * fcomp.compareValues(stash[testClause], values[testClause]);
-
+ lastCompare =
+ reverseMul[testClause] * fcomp.compareValues(stash[testClause], values[testClause]);
+
if (0 != lastCompare) {
// no need to keep checking additional clauses
break;
}
}
if (0 <= lastCompare) {
- // we're either not competitive, or we're completely tied with another doc that's already group head
+ // we're either not competitive, or we're completely tied with another doc that's already
+ // group head
// that's already been selected
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/CollapsingQParserPlugin.java
##########
@@ -3130,33 +3399,36 @@ public boolean collectInNullGroupIfBoosted(int globalDoc) {
return false;
}
- /**
- * Kludgy API neccessary to deal with diff collectors/strategies using diff
- * data structs for tracking collapse keys...
+ /**
+ * Kludgy API neccessary to deal with diff collectors/strategies using diff data structs for
+ * tracking collapse keys...
*/
- public void purgeGroupsThatHaveBoostedDocs(final FixedBitSet collapsedSet,
- final IntProcedure removeGroupKey,
- final Runnable resetNullGroupHead) {
+ public void purgeGroupsThatHaveBoostedDocs(
+ final FixedBitSet collapsedSet,
+ final IntProcedure removeGroupKey,
+ final Runnable resetNullGroupHead) {
// Add the (collected) boosted docs to the collapsedSet
- boostedDocs.forEach(new IntProcedure() {
- public void apply(int globalDoc) {
- collapsedSet.set(globalDoc);
- }
- });
+ boostedDocs.forEach(
+ new IntProcedure() {
+ public void apply(int globalDoc) {
+ collapsedSet.set(globalDoc);
+ }
+ });
// Remove any group heads that are in the same groups as (collected) boosted documents.
boostedKeys.forEach(removeGroupKey);
if (boostedNullGroup) {
// If we're using IGNORE then no (matching) null docs were collected (by caller)
// If we're using EXPAND then all (matching) null docs were already collected (by us)
- // ...and that's *good* because each is treated like it's own group, our boosts don't matter
- // We only have to worry about removing null docs when using COLLAPSE, in which case any boosted null doc
+ // ...and that's *good* because each is treated like it's own group, our boosts don't
+ // matter
+ // We only have to worry about removing null docs when using COLLAPSE, in which case any
+ // boosted null doc
// means we clear the group head of the null group..
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/CollapsingQParserPlugin.java
##########
@@ -1915,96 +2112,125 @@ public SortedDocValues getSorted(FieldInfo ignored) throws IOException {
int leafCount = searcher.getTopReaderContext().leaves().size();
SolrRequestInfo req = SolrRequestInfo.getRequestInfo();
- boolean collectElevatedDocsWhenCollapsing = req != null && req.getReq().getParams().getBool(COLLECT_ELEVATED_DOCS_WHEN_COLLAPSING, true);
+ boolean collectElevatedDocsWhenCollapsing =
+ req != null
+ && req.getReq().getParams().getBool(COLLECT_ELEVATED_DOCS_WHEN_COLLAPSING, true);
if (GroupHeadSelectorType.SCORE.equals(groupHeadSelector.type)) {
if (collapseFieldType instanceof StrField) {
if (blockCollapse) {
return new BlockOrdScoreCollector(collapseField, nullPolicy, boostDocs);
}
- return new OrdScoreCollector(maxDoc, leafCount, docValuesProducer, nullPolicy, boostDocs, searcher, collectElevatedDocsWhenCollapsing);
+ return new OrdScoreCollector(
+ maxDoc,
+ leafCount,
+ docValuesProducer,
+ nullPolicy,
+ boostDocs,
+ searcher,
+ collectElevatedDocsWhenCollapsing);
} else if (isNumericCollapsible(collapseFieldType)) {
if (blockCollapse) {
return new BlockIntScoreCollector(collapseField, nullPolicy, boostDocs);
}
- return new IntScoreCollector(maxDoc, leafCount, nullPolicy, size, collapseField, boostDocs, searcher, collectElevatedDocsWhenCollapsing);
+ return new IntScoreCollector(
+ maxDoc,
+ leafCount,
+ nullPolicy,
+ size,
+ collapseField,
+ boostDocs,
+ searcher,
+ collectElevatedDocsWhenCollapsing);
} else {
- throw new SolrException(SolrException.ErrorCode.BAD_REQUEST,
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
"Collapsing field should be of either String, Int or Float type");
}
} else { // min, max, sort, etc.. something other then just "score"
if (collapseFieldType instanceof StrField) {
if (blockCollapse) {
- // NOTE: for now we don't worry about wether this is a sortSpec of min/max groupHeadSelector,
- // we use a "sort spec' based block collector unless/until there is some (performance?) reason to specialize
- return new BlockOrdSortSpecCollector(collapseField, nullPolicy, boostDocs,
- BlockOrdSortSpecCollector.getSort(groupHeadSelector,
- sortSpec, funcQuery, searcher),
- needsScores || needsScores4Collapsing);
+ // NOTE: for now we don't worry about wether this is a sortSpec of min/max
+ // groupHeadSelector,
+ // we use a "sort spec' based block collector unless/until there is some (performance?)
+ // reason to specialize
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/DocSetBuilder.java
##########
@@ -176,21 +171,21 @@ private static int dedup(int[] arr, int length, FixedBitSet acceptDocs) {
return pos;
}
-
-
public DocSet build(FixedBitSet filter) {
if (bitSet != null) {
if (filter != null) {
bitSet.and(filter);
}
return new BitDocSet(bitSet);
- // TODO - if this set will be cached, should we make it smaller if it's below DocSetUtil.smallSetSize?
+ // TODO - if this set will be cached, should we make it smaller if it's below
+ // DocSetUtil.smallSetSize?
} else {
LSBRadixSorter sorter = new LSBRadixSorter();
sorter.sort(PackedInts.bitsRequired(maxDoc - 1), buffer, pos);
final int l = dedup(buffer, pos, filter);
assert l <= pos;
- return new SortedIntDocSet(buffer, l); // TODO: have option to not shrink in the future if it will be a temporary set
+ return new SortedIntDocSet(
+ buffer, l); // TODO: have option to not shrink in the future if it will be a temporary set
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/DocSetBuilder.java
##########
@@ -207,9 +202,8 @@ public DocSet buildUniqueInOrder(FixedBitSet filter) {
if (filter != null) {
l = dedup(buffer, pos, filter);
}
- return new SortedIntDocSet(buffer, l); // TODO: have option to not shrink in the future if it will be a temporary set
+ return new SortedIntDocSet(
+ buffer, l); // TODO: have option to not shrink in the future if it will be a temporary set
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/ExtendedDismaxQParser.java
##########
@@ -315,66 +308,82 @@ protected void addPhraseFieldQueries(BooleanQuery.Builder query, List<Clause> cl
}
// create a map of {wordGram, [phraseField]}
- Multimap<Integer, FieldParams> phraseFieldsByWordGram = Multimaps.index(allPhraseFields, FieldParams::getWordGrams);
-
- // for each {wordGram, [phraseField]} entry, create and add shingled field queries to the main user query
- for (Map.Entry<Integer, Collection<FieldParams>> phraseFieldsByWordGramEntry : phraseFieldsByWordGram.asMap().entrySet()) {
-
- // group the fields within this wordGram collection by their associated slop (it's possible that the same
- // field appears multiple times for the same wordGram count but with different slop values. In this case, we
+ Multimap<Integer, FieldParams> phraseFieldsByWordGram =
+ Multimaps.index(allPhraseFields, FieldParams::getWordGrams);
+
+ // for each {wordGram, [phraseField]} entry, create and add shingled field queries to the main
+ // user query
+ for (Map.Entry<Integer, Collection<FieldParams>> phraseFieldsByWordGramEntry :
+ phraseFieldsByWordGram.asMap().entrySet()) {
+
+ // group the fields within this wordGram collection by their associated slop (it's possible
+ // that the same
+ // field appears multiple times for the same wordGram count but with different slop values.
+ // In this case, we
// should take the *sum* of those phrase queries, rather than the max across them).
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/FunctionQParser.java
##########
@@ -36,17 +35,23 @@
public class FunctionQParser extends QParser {
- public static final int FLAG_CONSUME_DELIMITER = 0x01; // consume delimiter after parsing arg
+ public static final int FLAG_CONSUME_DELIMITER = 0x01; // consume delimiter after parsing arg
public static final int FLAG_IS_AGG = 0x02;
- public static final int FLAG_USE_FIELDNAME_SOURCE = 0x04; // When a field name is encountered, use the placeholder FieldNameValueSource instead of resolving to a real ValueSource
+ public static final int FLAG_USE_FIELDNAME_SOURCE =
+ 0x04; // When a field name is encountered, use the placeholder FieldNameValueSource instead of
+ // resolving to a real ValueSource
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/DocSetUtil.java
##########
@@ -158,12 +166,15 @@ public static DocSet createDocSet(SolrIndexSearcher searcher, Term term) throws
int maxCount = 0;
int firstReader = -1;
List<LeafReaderContext> leaves = reader.leaves();
- PostingsEnum[] postList = new PostingsEnum[leaves.size()]; // use array for slightly higher scanning cost, but fewer memory allocations
+ PostingsEnum[] postList =
+ new PostingsEnum
+ [leaves.size()]; // use array for slightly higher scanning cost, but fewer memory
+ // allocations
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/ExtendedDismaxQParser.java
##########
@@ -413,20 +425,23 @@ protected Query parseOriginalQuery(ExtendedSolrQueryParser up,
String mmSpec = config.minShouldMatch;
if (foundOperators(clauses, config.lowercaseOperators)) {
- mmSpec = config.solrParams.get(DisMaxParams.MM, "0%"); // Use provided mm spec if present, otherwise turn off mm processing
+ mmSpec =
+ config.solrParams.get(
+ DisMaxParams.MM,
+ "0%"); // Use provided mm spec if present, otherwise turn off mm processing
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/GraphTermsQParserPlugin.java
##########
@@ -656,7 +701,8 @@ public void visit(int docID, byte[] packedValue) {
if (crosses) {
return PointValues.Relation.CELL_CROSSES_QUERY;
} else {
- // NOTE: we only hit this if we are on a cell whose min and max values are exactly equal to our point,
+ // NOTE: we only hit this if we are on a cell whose min and max values are exactly equal to
+ // our point,
// which can easily happen if many docs share this one value
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/ExtendedDismaxQParser.java
##########
@@ -1320,16 +1337,18 @@ protected void addMultiTermClause(List<BooleanClause> clauses, Query q) {
if (q instanceof BooleanQuery) {
boolean allOptionalDisMaxQueries = true;
- for (BooleanClause c : ((BooleanQuery)q).clauses()) {
- if (c.getOccur() != BooleanClause.Occur.SHOULD || ! (c.getQuery() instanceof DisjunctionMaxQuery)) {
+ for (BooleanClause c : ((BooleanQuery) q).clauses()) {
+ if (c.getOccur() != BooleanClause.Occur.SHOULD
+ || !(c.getQuery() instanceof DisjunctionMaxQuery)) {
allOptionalDisMaxQueries = false;
break;
}
}
if (allOptionalDisMaxQueries) {
- // getAliasedMultiTermQuery() constructed a BooleanQuery containing only SHOULD DisjunctionMaxQuery-s.
+ // getAliasedMultiTermQuery() constructed a BooleanQuery containing only SHOULD
+ // DisjunctionMaxQuery-s.
// Unwrap the query and add a clause for each contained DisMax query.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/GraphTermsQParserPlugin.java
##########
@@ -333,14 +337,13 @@ private void collectTermStates(IndexReader reader,
int maxDocFreq = Integer.MAX_VALUE;
final long ramBytesUsed; // cache
- /**
- * Iterator of encoded point values.
- */
+ /** Iterator of encoded point values. */
// TODO: if we want to stream, maybe we should use jdk stream class?
- public static abstract class Stream implements BytesRefIterator {
+ public abstract static class Stream implements BytesRefIterator {
@Override
public abstract BytesRef next();
- };
+ }
+ ;
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/Grouping.java
##########
@@ -464,67 +481,60 @@ public static int getMax(int offset, int len, int max) {
}
/**
- * Returns whether a cache warning should be send to the client.
- * The value <code>true</code> is returned when the cache is emptied because the caching limits where met, otherwise
- * <code>false</code> is returned.
+ * Returns whether a cache warning should be send to the client. The value <code>true</code> is
+ * returned when the cache is emptied because the caching limits where met, otherwise <code>false
+ * </code> is returned.
*
* @return whether a cache warning should be send to the client
*/
public boolean isSignalCacheWarning() {
return signalCacheWarning;
}
- //====================================== Inner classes =============================================================
+ // ====================================== Inner classes
+ // =============================================================
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/Grouping.java
##########
@@ -104,28 +102,30 @@
private boolean getDocList; // doclist needed for debugging or highlighting
private Query query;
private NamedList<Object> grouped = new SimpleOrderedMap<>();
- private Set<Integer> idSet = new LinkedHashSet<>(); // used for tracking unique docs when we need a doclist
- private int maxMatches; // max number of matches from any grouping command
- private float maxScore = Float.NaN; // max score seen in any doclist
+ private Set<Integer> idSet =
+ new LinkedHashSet<>(); // used for tracking unique docs when we need a doclist
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/HashQParserPlugin.java
##########
@@ -40,42 +39,46 @@
import org.apache.solr.common.util.Hash;
import org.apache.solr.request.SolrQueryRequest;
-/**
- * syntax fq={!hash workers=11 worker=4 keys=field1,field2}
- */
+/** syntax fq={!hash workers=11 worker=4 keys=field1,field2} */
public class HashQParserPlugin extends QParserPlugin {
public static final String NAME = "hash";
- public QParser createParser(String query, SolrParams localParams, SolrParams params, SolrQueryRequest request) {
+ public QParser createParser(
+ String query, SolrParams localParams, SolrParams params, SolrQueryRequest request) {
return new HashQParser(query, localParams, params, request);
}
private static class HashQParser extends QParser {
- public HashQParser(String query, SolrParams localParams, SolrParams params, SolrQueryRequest request) {
+ public HashQParser(
+ String query, SolrParams localParams, SolrParams params, SolrQueryRequest request) {
super(query, localParams, params, request);
}
public Query parse() {
int workers = localParams.getInt("workers", 0);
if (workers < 2) {
- throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "workers needs to be more than 1");
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST, "workers needs to be more than 1");
}
int worker = localParams.getInt("worker", 0);
String keyParam = params.get("partitionKeys");
String[] keys = keyParam.replace(" ", "").split(",");
- Arrays.stream(keys).forEach(field -> req.getSchema().getField(field)); // validate all fields exist
+ Arrays.stream(keys)
+ .forEach(field -> req.getSchema().getField(field)); // validate all fields exist
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/Grouping.java
##########
@@ -104,28 +102,30 @@
private boolean getDocList; // doclist needed for debugging or highlighting
private Query query;
private NamedList<Object> grouped = new SimpleOrderedMap<>();
- private Set<Integer> idSet = new LinkedHashSet<>(); // used for tracking unique docs when we need a doclist
- private int maxMatches; // max number of matches from any grouping command
- private float maxScore = Float.NaN; // max score seen in any doclist
+ private Set<Integer> idSet =
+ new LinkedHashSet<>(); // used for tracking unique docs when we need a doclist
+ private int maxMatches; // max number of matches from any grouping command
+ private float maxScore = Float.NaN; // max score seen in any doclist
private boolean signalCacheWarning = false;
private TimeLimitingCollector timeLimitingCollector;
-
- public DocList mainResult; // output if one of the grouping commands should be used as the main result.
+ public DocList
+ mainResult; // output if one of the grouping commands should be used as the main result.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/ExtendedDismaxQParser.java
##########
@@ -1688,20 +1705,24 @@ public String toString() {
protected boolean lowercaseOperators;
- protected String[] boostFuncs;
+ protected String[] boostFuncs;
protected boolean splitOnWhitespace;
protected IndexSchema schema;
- public ExtendedDismaxConfiguration(SolrParams localParams,
- SolrParams params, SolrQueryRequest req) {
+ public ExtendedDismaxConfiguration(
+ SolrParams localParams, SolrParams params, SolrQueryRequest req) {
solrParams = SolrParams.wrapDefaults(localParams, params);
schema = req.getSchema();
- minShouldMatch = DisMaxQParser.parseMinShouldMatch(schema, solrParams); // req.getSearcher() here causes searcher refcount imbalance
+ minShouldMatch =
+ DisMaxQParser.parseMinShouldMatch(
+ schema, solrParams); // req.getSearcher() here causes searcher refcount imbalance
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/JoinQuery.java
##########
@@ -112,23 +113,28 @@ public JoinQueryWeight(SolrIndexSearcher searcher, ScoreMode scoreMode, float bo
this.fromSearcher = searcher;
} else {
if (info == null) {
- throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "Cross-core join must have SolrRequestInfo");
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST, "Cross-core join must have SolrRequestInfo");
}
CoreContainer container = searcher.getCore().getCoreContainer();
final SolrCore fromCore = container.getCore(fromIndex);
if (fromCore == null) {
- throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "Cross-core join: no such core " + fromIndex);
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST, "Cross-core join: no such core " + fromIndex);
}
if (info.getReq().getCore() == fromCore) {
- // if this is the same core, use the searcher passed in... otherwise we could be warming and
+ // if this is the same core, use the searcher passed in... otherwise we could be warming
+ // and
// get an older searcher from the core.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/JoinQuery.java
##########
@@ -58,7 +57,8 @@
class JoinQuery extends Query {
String fromField;
String toField;
- String fromIndex; // TODO: name is missleading here compared to JoinQParserPlugin usage - here it must be a core name
+ String fromIndex; // TODO: name is missleading here compared to JoinQParserPlugin usage - here it
+ // must be a core name
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/GraphTermsQParserPlugin.java
##########
@@ -465,14 +480,26 @@ public PointSetQuery(String field, int numDims, int bytesPerDim, Stream packedPo
this.bytesPerDim = bytesPerDim;
this.numDims = numDims;
- // In the 1D case this works well (the more points, the more common prefixes they share, typically), but in
- // the > 1 D case, where we are only looking at the first dimension's prefix bytes, it can at worst not hurt:
+ // In the 1D case this works well (the more points, the more common prefixes they share,
+ // typically), but in
+ // the > 1 D case, where we are only looking at the first dimension's prefix bytes, it can at
+ // worst not hurt:
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/JoinQuery.java
##########
@@ -112,23 +113,28 @@ public JoinQueryWeight(SolrIndexSearcher searcher, ScoreMode scoreMode, float bo
this.fromSearcher = searcher;
} else {
if (info == null) {
- throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "Cross-core join must have SolrRequestInfo");
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST, "Cross-core join must have SolrRequestInfo");
}
CoreContainer container = searcher.getCore().getCoreContainer();
final SolrCore fromCore = container.getCore(fromIndex);
if (fromCore == null) {
- throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "Cross-core join: no such core " + fromIndex);
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST, "Cross-core join: no such core " + fromIndex);
}
if (info.getReq().getCore() == fromCore) {
- // if this is the same core, use the searcher passed in... otherwise we could be warming and
+ // if this is the same core, use the searcher passed in... otherwise we could be warming
+ // and
// get an older searcher from the core.
fromSearcher = searcher;
} else {
- // This could block if there is a static warming query with a join in it, and if useColdSearcher is true.
- // Deadlock could result if two cores both had useColdSearcher and had joins that used eachother.
+ // This could block if there is a static warming query with a join in it, and if
+ // useColdSearcher is true.
+ // Deadlock could result if two cores both had useColdSearcher and had joins that used
+ // eachother.
// This would be very predictable though (should happen every time if misconfigured)
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/PayloadCheckQParserPlugin.java
##########
@@ -104,19 +107,19 @@ public Query parse() throws SyntaxError {
}
if (encoder == null) {
- throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "invalid encoder: " + e + " for field: " + field);
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "invalid encoder: " + e + " for field: " + field);
}
List<BytesRef> payloads = new ArrayList<>();
- String[] rawPayloads = p.split(" "); // since payloads (most likely) came in whitespace delimited, just split
+ String[] rawPayloads =
+ p.split(" "); // since payloads (most likely) came in whitespace delimited, just split
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/PayloadCheckQParserPlugin.java
##########
@@ -92,7 +94,8 @@ public Query parse() throws SyntaxError {
PayloadEncoder encoder = null;
String e = PayloadUtils.getPayloadEncoder(ft);
PayloadType payloadType = null;
- if ("float".equals(e)) { // TODO: centralize this string->PayloadEncoder logic (see DelimitedPayloadTokenFilterFactory)
+ if ("float".equals(e)) { // TODO: centralize this string->PayloadEncoder logic (see
+ // DelimitedPayloadTokenFilterFactory)
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/PointMerger.java
##########
@@ -167,8 +172,8 @@ public int getCurrentCount() {
// sets the next value and returns getCurrentCount()
public int setNextValue() throws IOException {
return 0;
- };
-
+ }
+ ;
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/HashQParserPlugin.java
##########
@@ -93,43 +96,60 @@ public LongValues getValues(LeafReaderContext ctx, DoubleValues scores) throws I
final NumericDocValues numericDocValues = ctx.reader().getNumericDocValues(field);
if (numericDocValues != null) {
// Numeric
- resultValues[i] = new LongValues() {
- // Even if not a Long field; could be int, double, float and this still works because DocValues numerics
- // are based on a Long.
- final NumericDocValues values = numericDocValues;
- boolean atDoc = false;
- @Override
- public boolean advanceExact(int doc) throws IOException { atDoc = values.advanceExact(doc); return true; }
- @Override
- public long longValue() throws IOException { return atDoc ? Long.hashCode(values.longValue()) : 0; }
- };
+ resultValues[i] =
+ new LongValues() {
+ // Even if not a Long field; could be int, double, float and this still works
+ // because DocValues numerics
+ // are based on a Long.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/PayloadCheckQParserPlugin.java
##########
@@ -48,16 +47,19 @@
private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
@Override
- public QParser createParser(String qstr, SolrParams localParams, SolrParams params, SolrQueryRequest req) {
+ public QParser createParser(
+ String qstr, SolrParams localParams, SolrParams params, SolrQueryRequest req) {
return new QParser(qstr, localParams, params, req) {
@Override
public Query parse() throws SyntaxError {
String field = localParams.get(QueryParsing.F);
String value = localParams.get(QueryParsing.V);
String p = localParams.get("payloads");
- // payloads and op parameter are probably mutually exclusive. we could consider making a different query
- // not a span payload check query, but something that just operates on payloads without the span?
+ // payloads and op parameter are probably mutually exclusive. we could consider making a
+ // different query
+ // not a span payload check query, but something that just operates on payloads without the
+ // span?
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/SimpleQParserPlugin.java
##########
@@ -78,52 +71,55 @@
/* Setup the map of possible operators. */
static {
- OPERATORS.put(SimpleParams.AND_OPERATOR, SimpleQueryParser.AND_OPERATOR);
- OPERATORS.put(SimpleParams.NOT_OPERATOR, SimpleQueryParser.NOT_OPERATOR);
- OPERATORS.put(SimpleParams.OR_OPERATOR, SimpleQueryParser.OR_OPERATOR);
- OPERATORS.put(SimpleParams.PREFIX_OPERATOR, SimpleQueryParser.PREFIX_OPERATOR);
- OPERATORS.put(SimpleParams.PHRASE_OPERATOR, SimpleQueryParser.PHRASE_OPERATOR);
+ OPERATORS.put(SimpleParams.AND_OPERATOR, SimpleQueryParser.AND_OPERATOR);
+ OPERATORS.put(SimpleParams.NOT_OPERATOR, SimpleQueryParser.NOT_OPERATOR);
+ OPERATORS.put(SimpleParams.OR_OPERATOR, SimpleQueryParser.OR_OPERATOR);
+ OPERATORS.put(SimpleParams.PREFIX_OPERATOR, SimpleQueryParser.PREFIX_OPERATOR);
+ OPERATORS.put(SimpleParams.PHRASE_OPERATOR, SimpleQueryParser.PHRASE_OPERATOR);
OPERATORS.put(SimpleParams.PRECEDENCE_OPERATORS, SimpleQueryParser.PRECEDENCE_OPERATORS);
- OPERATORS.put(SimpleParams.ESCAPE_OPERATOR, SimpleQueryParser.ESCAPE_OPERATOR);
- OPERATORS.put(SimpleParams.WHITESPACE_OPERATOR, SimpleQueryParser.WHITESPACE_OPERATOR);
- OPERATORS.put(SimpleParams.FUZZY_OPERATOR, SimpleQueryParser.FUZZY_OPERATOR);
- OPERATORS.put(SimpleParams.NEAR_OPERATOR, SimpleQueryParser.NEAR_OPERATOR);
+ OPERATORS.put(SimpleParams.ESCAPE_OPERATOR, SimpleQueryParser.ESCAPE_OPERATOR);
+ OPERATORS.put(SimpleParams.WHITESPACE_OPERATOR, SimpleQueryParser.WHITESPACE_OPERATOR);
+ OPERATORS.put(SimpleParams.FUZZY_OPERATOR, SimpleQueryParser.FUZZY_OPERATOR);
+ OPERATORS.put(SimpleParams.NEAR_OPERATOR, SimpleQueryParser.NEAR_OPERATOR);
}
/** Returns a QParser that will create a query by using Lucene's SimpleQueryParser. */
@Override
- public QParser createParser(String qstr, SolrParams localParams, SolrParams params, SolrQueryRequest req) {
+ public QParser createParser(
+ String qstr, SolrParams localParams, SolrParams params, SolrQueryRequest req) {
return new SimpleQParser(qstr, localParams, params, req);
}
private static class SimpleQParser extends QParser {
private SimpleQueryParser parser;
- public SimpleQParser (String qstr, SolrParams localParams, SolrParams params, SolrQueryRequest req) {
+ public SimpleQParser(
+ String qstr, SolrParams localParams, SolrParams params, SolrQueryRequest req) {
super(qstr, localParams, params, req);
// Some of the parameters may come in through localParams, so combine them with params.
SolrParams defaultParams = SolrParams.wrapDefaults(localParams, params);
// This will be used to specify what fields and boosts will be used by SimpleQueryParser.
- Map<String, Float> queryFields = SolrPluginUtils.parseFieldBoosts(defaultParams.get(SimpleParams.QF));
+ Map<String, Float> queryFields =
+ SolrPluginUtils.parseFieldBoosts(defaultParams.get(SimpleParams.QF));
if (queryFields.isEmpty()) {
// It qf is not specified setup up the queryFields map to use the defaultField.
String defaultField = defaultParams.get(CommonParams.DF);
if (defaultField == null) {
// A query cannot be run without having a field or set of fields to run against.
- throw new IllegalStateException("Neither " + SimpleParams.QF + " nor " + CommonParams.DF
- + " are present.");
+ throw new IllegalStateException(
+ "Neither " + SimpleParams.QF + " nor " + CommonParams.DF + " are present.");
}
queryFields.put(defaultField, 1.0F);
- }
- else {
+ } else {
for (Map.Entry<String, Float> queryField : queryFields.entrySet()) {
if (queryField.getValue() == null) {
- // Some fields may be specified without a boost, so default the boost to 1.0 since a null value
+ // Some fields may be specified without a boost, so default the boost to 1.0 since a
+ // null value
// will not be accepted by SimpleQueryParser.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/QParser.java
##########
@@ -233,7 +247,9 @@ public QParser subQuery(String q, String defaultType) throws SyntaxError {
defaultType = localParams.get(QueryParsing.DEFTYPE);
}
QParser nestedParser = getParser(q, defaultType, getReq());
- nestedParser.flags = this.flags; // TODO: this would be better passed in to the constructor... change to a ParserContext object?
+ nestedParser.flags =
+ this.flags; // TODO: this would be better passed in to the constructor... change to a
+ // ParserContext object?
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/QueryParsing.java
##########
@@ -214,14 +219,16 @@ static void writeFieldVal(BytesRef val, FieldType ft, Appendable out, int flags)
}
}
-
- private static int FLAG_BOOSTED=0x01;
- private static int FLAG_IS_CLAUSE=0x02;
+ private static int FLAG_BOOSTED = 0x01;
+ private static int FLAG_IS_CLAUSE = 0x02;
/**
* @see #toString(Query,IndexSchema)
*/
- public static void toString(Query query, IndexSchema schema, Appendable out, int flags) throws IOException {
- int subflag = flags & ~(FLAG_BOOSTED|FLAG_IS_CLAUSE); // clear the boosted / is clause flags for recursion
+ public static void toString(Query query, IndexSchema schema, Appendable out, int flags)
+ throws IOException {
+ int subflag =
+ flags
+ & ~(FLAG_BOOSTED | FLAG_IS_CLAUSE); // clear the boosted / is clause flags for recursion
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/PointMerger.java
##########
@@ -43,8 +41,10 @@
* @lucene.experimental
*/
public class PointMerger {
- public static int TOTAL_BUFFER_SIZE = 1000000; // target number of elements to cache across all segments
- public static int MIN_SEG_BUFFER_SIZE = 100; // minimum buffer size on any segment (to limit unnecessary exception throws)
+ public static int TOTAL_BUFFER_SIZE =
+ 1000000; // target number of elements to cache across all segments
+ public static int MIN_SEG_BUFFER_SIZE =
+ 100; // minimum buffer size on any segment (to limit unnecessary exception throws)
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/SolrDocumentFetcher.java
##########
@@ -422,7 +452,8 @@ public IndexableFieldType fieldType() {
@Override
public TokenStream tokenStream(Analyzer analyzer, TokenStream reuse) {
- return analyzer.tokenStream(name(), stringValue()); // or we could throw unsupported exception?
+ return analyzer.tokenStream(
+ name(), stringValue()); // or we could throw unsupported exception?
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/SolrDocumentFetcher.java
##########
@@ -599,33 +637,38 @@ else if (schemaField.multiValued() == false) {
}
private Object decodeNumberFromDV(SchemaField schemaField, long value, boolean sortableNumeric) {
- // note: This special-case is unfortunate; if we have to add any more than perhaps the fieldType should
+ // note: This special-case is unfortunate; if we have to add any more than perhaps the fieldType
+ // should
// have this method so that specific field types can customize it.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/SolrDocumentFetcher.java
##########
@@ -190,11 +205,14 @@ public boolean isLazyFieldLoadingEnabled() {
final String fieldName = fieldInfo.name;
try {
SchemaField field = searcher.getSchema().getField(fieldName);
- if (field.stored() && ((field.getType() instanceof org.apache.solr.schema.TextField)
- || (field.getType() instanceof org.apache.solr.schema.StrField))) {
+ if (field.stored()
+ && ((field.getType() instanceof org.apache.solr.schema.TextField)
+ || (field.getType() instanceof org.apache.solr.schema.StrField))) {
storedHighlightFieldNames.add(fieldName);
}
- } catch (RuntimeException e) { // getField() throws a SolrException, but it arrives as a RuntimeException
+ } catch (
+ RuntimeException
+ e) { // getField() throws a SolrException, but it arrives as a RuntimeException
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/SolrDocumentFetcher.java
##########
@@ -309,14 +329,15 @@ public void stringField(FieldInfo fieldInfo, String value) throws IOException {
} else {
super.stringField(fieldInfo, value);
}
-
}
@Override
public Status needsField(FieldInfo fieldInfo) throws IOException {
Status status = super.needsField(fieldInfo);
assert status != Status.STOP : "Status.STOP not supported or expected";
- if (addLargeFieldsLazily && largeFields.contains(fieldInfo.name)) { // load "large" fields using this lazy mechanism
+ if (addLargeFieldsLazily
+ && largeFields.contains(
+ fieldInfo.name)) { // load "large" fields using this lazy mechanism
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/SolrDocumentFetcher.java
##########
@@ -124,7 +133,9 @@
final Set<String> dvsCanSubstituteStored = new HashSet<>();
final Set<String> allStoreds = new HashSet<>();
- for (FieldInfo fieldInfo : searcher.getFieldInfos()) { // can find materialized dynamic fields, unlike using the Solr IndexSchema.
+ for (FieldInfo fieldInfo :
+ searcher.getFieldInfos()) { // can find materialized dynamic fields, unlike using the Solr
+ // IndexSchema.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/SolrIndexSearcher.java
##########
@@ -782,7 +857,8 @@ public void cacheDocSet(Query query, DocSet optionalAnswer, boolean mustCache) t
}
private BitDocSet makeBitDocSet(DocSet answer) {
- // TODO: this should be implemented in DocSet, most likely with a getBits method that takes a maxDoc argument
+ // TODO: this should be implemented in DocSet, most likely with a getBits method that takes a
+ // maxDoc argument
// or make DocSet instances remember maxDoc
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/SortedIntDocSet.java
##########
@@ -76,11 +78,13 @@ public static int intersectionSize(int[] smallerSortedList, int[] biggerSortedLi
// The next doc we are looking for will be much closer to the last position we tried
// than it will be to the midpoint between last and high... so probe ahead using
// a function of the ratio of the sizes of the sets.
- int step = (b.length/a.length)+1;
+ int step = (b.length / a.length) + 1;
// Since the majority of probes should be misses, we'll already be above the last probe
- // and shouldn't need to move larger than the step size on average to step over our target (and thus lower
- // the high upper bound a lot.)... but if we don't go over our target, it's a big miss... so double it.
+ // and shouldn't need to move larger than the step size on average to step over our target (and
+ // thus lower
+ // the high upper bound a lot.)... but if we don't go over our target, it's a big miss... so
+ // double it.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/SortedIntDocSet.java
##########
@@ -677,10 +672,12 @@ public DocSet union(DocSet other) {
int lastLimit = 0;
int lastLimitDoc = docs[0]; // docs.length != 0
for (LeafReaderContext lrc : leaves) {
- // sanity check that initial `lastLimit*` values are valid (and consequently that our initial
+ // sanity check that initial `lastLimit*` values are valid (and consequently that our
+ // initial
// setting of `startIdx` for context.ord==0 won't inadvertently include invalid docs).
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/SolrDocumentFetcher.java
##########
@@ -737,13 +778,18 @@ private ReturnFields getReturnFields() {
}
} else if (fnames != null) {
storedFields.addAll(fnames);
- storedFields.removeIf((String name) -> {
- SchemaField schemaField = searcher.getSchema().getFieldOrNull(name);
- if (schemaField == null) return false; // Get it from the stored fields if, for some reasonm, we can't get the schema.
- if (schemaField.stored() && schemaField.multiValued()) return false; // must return multivalued fields from stored data if possible.
- if (schemaField.stored() == false) return true; // if it's not stored, no choice but to return from DV.
- return false;
- });
+ storedFields.removeIf(
+ (String name) -> {
+ SchemaField schemaField = searcher.getSchema().getFieldOrNull(name);
+ if (schemaField == null)
+ return false; // Get it from the stored fields if, for some reasonm, we can't get
+ // the schema.
+ if (schemaField.stored() && schemaField.multiValued())
+ return false; // must return multivalued fields from stored data if possible.
+ if (schemaField.stored() == false)
+ return true; // if it's not stored, no choice but to return from DV.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/SolrIndexSearcher.java
##########
@@ -839,10 +920,14 @@ private DocSet getAndCacheDocSet(Query query) throws IOException {
assert filterCache != null : "must check for caching before calling this method";
if (SolrQueryTimeoutImpl.getInstance().isTimeoutEnabled()) {
- // If there is a possibility of timeout for this query, then don't reserve a computation slot. Further, we can't
- // naively wait for an in progress computation to finish, because if we time out before it does then we won't
- // even have partial results to provide. We could possibly wait for the query to finish in parallel with our own
- // results and if they complete first use that instead, but we'll leave that to implement later.
+ // If there is a possibility of timeout for this query, then don't reserve a computation slot.
+ // Further, we can't
+ // naively wait for an in progress computation to finish, because if we time out before it
+ // does then we won't
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/SolrReturnFields.java
##########
@@ -315,61 +324,69 @@ private void add(String fl, Deque<DeferredRenameEntry> deferred, DocTransformers
if (funcStr.startsWith("[")) {
ModifiableSolrParams augmenterParams = new ModifiableSolrParams();
- int end = QueryParsing.parseLocalParams(funcStr, 0, augmenterParams, req.getParams(), "[", ']');
+ int end =
+ QueryParsing.parseLocalParams(funcStr, 0, augmenterParams, req.getParams(), "[", ']');
sp.pos += end;
// [foo] is short for [type=foo] in localParams syntax
String augmenterName = augmenterParams.get("type");
augmenterParams.remove("type");
String disp = key;
- if( disp == null ) {
- disp = '['+augmenterName+']';
+ if (disp == null) {
+ disp = '[' + augmenterName + ']';
}
- TransformerFactory factory = req.getCore().getTransformerFactory( augmenterName );
+ TransformerFactory factory = req.getCore().getTransformerFactory(augmenterName);
if (factory instanceof TransformerFactory.FieldRenamer) {
- // NOTE: `deferred` is a Deque because some TransformerFactories (e.g., `GeoTransformerFactory`) can
- // subtly modify the representation of the associated value (i.e., it's not just a straight rename). This
- // subverts the "update source field" phase of `FieldRenamer.create(...)`. We _know_ however that "simple"
- // field renames don't do any value modification whatsoever, so those are added to the beginning of
- // the `deferred` Deque so that they will be processed first, and all other `FieldRenamers` are
- // added (here) to the front or back of the Deque, depending on the return value of `mayModifyValue()`.
- final DeferredRenameEntry deferredEntry = new DeferredRenameEntry(disp, augmenterParams, req, (TransformerFactory.FieldRenamer) factory);
+ // NOTE: `deferred` is a Deque because some TransformerFactories (e.g.,
+ // `GeoTransformerFactory`) can
+ // subtly modify the representation of the associated value (i.e., it's not just a
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/SortedIntDocSet.java
##########
@@ -700,7 +697,8 @@ public DocIdSetIterator iterator(LeafReaderContext context) {
if (docs.length == 0 || context.reader().maxDoc() < 1) {
// empty docset or entirely empty segment (verified that the latter actually happens)
- // NOTE: wrt the "empty docset" case, this is not just an optimization; this shortcircuits also
+ // NOTE: wrt the "empty docset" case, this is not just an optimization; this shortcircuits
+ // also
// to prevent the static DocSet.EmptyLazyHolder.INSTANCE from having cachedOrdIdxMap initiated
// across different IndexReaders.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/SolrIndexSearcher.java
##########
@@ -937,23 +1029,27 @@ public DocSet getDocSet(List<Query> queries) throws IOException {
}
/**
- * INTERNAL: The response object from {@link #getProcessedFilter(DocSet, List)}.
- * Holds a filter and postFilter pair that together match a set of documents.
- * Either of them may be null, in which case the semantics are to match everything.
+ * INTERNAL: The response object from {@link #getProcessedFilter(DocSet, List)}. Holds a filter
+ * and postFilter pair that together match a set of documents. Either of them may be null, in
+ * which case the semantics are to match everything.
+ *
* @see #getProcessedFilter(DocSet, List)
*/
public static class ProcessedFilter {
- public DocSet answer; // maybe null. Sometimes we have a docSet answer that represents the complete answer / result.
+ public DocSet
+ answer; // maybe null. Sometimes we have a docSet answer that represents the complete answer
+ // / result.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/facet/FacetBucket.java
##########
@@ -20,24 +20,30 @@
import java.util.Collection;
import java.util.HashMap;
import java.util.Map;
-
import org.apache.solr.common.util.SimpleOrderedMap;
public class FacetBucket {
@SuppressWarnings("rawtypes")
final FacetModule.FacetBucketMerger parent;
+
@SuppressWarnings({"rawtypes"})
final Comparable bucketValue;
- final int bucketNumber; // this is just for internal correlation (the first bucket created is bucket 0, the next bucket 1, across all field buckets)
+
+ final int
+ bucketNumber; // this is just for internal correlation (the first bucket created is bucket 0,
+ // the next bucket 1, across all field buckets)
long count;
Map<String, FacetMerger> subs;
- public FacetBucket(@SuppressWarnings("rawtypes") FacetModule.FacetBucketMerger parent
- , @SuppressWarnings("rawtypes") Comparable bucketValue, FacetMerger.Context mcontext) {
+ public FacetBucket(
+ @SuppressWarnings("rawtypes") FacetModule.FacetBucketMerger parent,
+ @SuppressWarnings("rawtypes") Comparable bucketValue,
+ FacetMerger.Context mcontext) {
this.parent = parent;
this.bucketValue = bucketValue;
- this.bucketNumber = mcontext.getNewBucketNumber(); // TODO: we don't need bucket numbers for all buckets...
+ this.bucketNumber =
+ mcontext.getNewBucketNumber(); // TODO: we don't need bucket numbers for all buckets...
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/facet/FacetContext.java
##########
@@ -26,16 +25,18 @@
public class FacetContext {
// Context info for actually executing a local facet command
- public static final int IS_SHARD=0x01;
- public static final int IS_REFINEMENT=0x02;
- public static final int SKIP_FACET=0x04; // refinement: skip calculating this immediate facet, but proceed to specific sub-facets based on facetInfo
+ public static final int IS_SHARD = 0x01;
+ public static final int IS_REFINEMENT = 0x02;
+ public static final int SKIP_FACET =
+ 0x04; // refinement: skip calculating this immediate facet, but proceed to specific sub-facets
+ // based on facetInfo
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/facet/FacetBucket.java
##########
@@ -20,24 +20,30 @@
import java.util.Collection;
import java.util.HashMap;
import java.util.Map;
-
import org.apache.solr.common.util.SimpleOrderedMap;
public class FacetBucket {
@SuppressWarnings("rawtypes")
final FacetModule.FacetBucketMerger parent;
+
@SuppressWarnings({"rawtypes"})
final Comparable bucketValue;
- final int bucketNumber; // this is just for internal correlation (the first bucket created is bucket 0, the next bucket 1, across all field buckets)
+
+ final int
+ bucketNumber; // this is just for internal correlation (the first bucket created is bucket 0,
+ // the next bucket 1, across all field buckets)
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/facet/BlockJoin.java
##########
@@ -26,33 +25,44 @@
import org.apache.solr.search.DocSetCollector;
import org.apache.solr.search.QueryContext;
-/** @lucene.experimental */
+/**
+ * @lucene.experimental
+ */
public class BlockJoin {
- /** acceptDocs will normally be used to avoid deleted documents from being generated as part of the answer DocSet (just use *:*)
- * although it can be used to further constrain the generated documents.
+ /**
+ * acceptDocs will normally be used to avoid deleted documents from being generated as part of the
+ * answer DocSet (just use *:*) although it can be used to further constrain the generated
+ * documents.
*/
- public static DocSet toChildren(DocSet parentInput, BitDocSet parentList, DocSet acceptDocs, QueryContext qcontext) throws IOException {
+ public static DocSet toChildren(
+ DocSet parentInput, BitDocSet parentList, DocSet acceptDocs, QueryContext qcontext)
+ throws IOException {
FixedBitSet parentBits = parentList.getBits();
DocSetCollector collector = new DocSetCollector(qcontext.searcher().maxDoc());
DocIterator iter = parentInput.iterator();
while (iter.hasNext()) {
int parentDoc = iter.nextDoc();
- if (!parentList.exists(parentDoc) || parentDoc == 0) { // test for parentDoc==0 here to avoid passing -1 to prevSetBit later on
+ if (!parentList.exists(parentDoc)
+ || parentDoc
+ == 0) { // test for parentDoc==0 here to avoid passing -1 to prevSetBit later on
// not a parent, or parent has no children
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/facet/FacetField.java
##########
@@ -28,13 +27,15 @@
public static final int DEFAULT_FACET_LIMIT = 10;
String field;
boolean missing;
- boolean allBuckets; // show cumulative stats across all buckets (this can be different than non-bucketed stats across all docs because of multi-valued docs)
+ boolean allBuckets; // show cumulative stats across all buckets (this can be different than
+ // non-bucketed stats across all docs because of multi-valued docs)
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/ValueSourceParser.java
##########
@@ -124,975 +123,1194 @@ private static void alias(String source, String dest) {
}
static {
- addParser("testfunc", new ValueSourceParser() {
- @Override
- public ValueSource parse(FunctionQParser fp) throws SyntaxError {
- final ValueSource source = fp.parseValueSource();
- return new TestValueSource(source);
- }
- });
- addParser("ord", new ValueSourceParser() {
- @Override
- public ValueSource parse(FunctionQParser fp) throws SyntaxError {
- String field = fp.parseId();
- return new OrdFieldSource(field);
- }
- });
- addParser("literal", new ValueSourceParser() {
- @Override
- public ValueSource parse(FunctionQParser fp) throws SyntaxError {
- return new LiteralValueSource(fp.parseArg());
- }
- });
- addParser("threadid", new ValueSourceParser() {
- @Override
- public ValueSource parse(FunctionQParser fp) throws SyntaxError {
- return new LongConstValueSource(Thread.currentThread().getId());
- }
- });
- addParser("sleep", new ValueSourceParser() {
- @Override
- public ValueSource parse(FunctionQParser fp) throws SyntaxError {
- int ms = fp.parseInt();
- ValueSource source = fp.parseValueSource();
- try {
- Thread.sleep(ms);
- } catch (InterruptedException e) {
- throw new RuntimeException(e);
- }
- return source;
- }
- });
- addParser("rord", new ValueSourceParser() {
- @Override
- public ValueSource parse(FunctionQParser fp) throws SyntaxError {
- String field = fp.parseId();
- return new ReverseOrdFieldSource(field);
- }
- });
- addParser("top", new ValueSourceParser() {
- @Override
- public ValueSource parse(FunctionQParser fp) throws SyntaxError {
- // top(vs) is now a no-op
- ValueSource source = fp.parseValueSource();
- return source;
- }
- });
- addParser("linear", new ValueSourceParser() {
- @Override
- public ValueSource parse(FunctionQParser fp) throws SyntaxError {
- ValueSource source = fp.parseValueSource();
- float slope = fp.parseFloat();
- float intercept = fp.parseFloat();
- return new LinearFloatFunction(source, slope, intercept);
- }
- });
- addParser("recip", new ValueSourceParser() {
- @Override
- public ValueSource parse(FunctionQParser fp) throws SyntaxError {
- ValueSource source = fp.parseValueSource();
- float m = fp.parseFloat();
- float a = fp.parseFloat();
- float b = fp.parseFloat();
- return new ReciprocalFloatFunction(source, m, a, b);
- }
- });
- addParser("scale", new ValueSourceParser() {
- @Override
- public ValueSource parse(FunctionQParser fp) throws SyntaxError {
- ValueSource source = fp.parseValueSource();
- float min = fp.parseFloat();
- float max = fp.parseFloat();
- return new ScaleFloatFunction(source, min, max);
- }
- });
- addParser("div", new ValueSourceParser() {
- @Override
- public ValueSource parse(FunctionQParser fp) throws SyntaxError {
- ValueSource a = fp.parseValueSource();
- ValueSource b = fp.parseValueSource();
- return new DivFloatFunction(a, b);
- }
- });
- addParser("mod", new ValueSourceParser() {
- @Override
- public ValueSource parse(FunctionQParser fp) throws SyntaxError {
- ValueSource a = fp.parseValueSource();
- ValueSource b = fp.parseValueSource();
- return new DualFloatFunction(a, b) {
+ addParser(
+ "testfunc",
+ new ValueSourceParser() {
@Override
- protected String name() {
- return "mod";
+ public ValueSource parse(FunctionQParser fp) throws SyntaxError {
+ final ValueSource source = fp.parseValueSource();
+ return new TestValueSource(source);
}
+ });
+ addParser(
+ "ord",
+ new ValueSourceParser() {
@Override
- protected float func(int doc, FunctionValues aVals, FunctionValues bVals) throws IOException {
- return aVals.floatVal(doc) % bVals.floatVal(doc);
+ public ValueSource parse(FunctionQParser fp) throws SyntaxError {
+ String field = fp.parseId();
+ return new OrdFieldSource(field);
}
- };
- }
- });
- addParser("map", new ValueSourceParser() {
- @Override
- public ValueSource parse(FunctionQParser fp) throws SyntaxError {
- ValueSource source = fp.parseValueSource();
- float min = fp.parseFloat();
- float max = fp.parseFloat();
- ValueSource target = fp.parseValueSource();
- ValueSource def = fp.hasMoreArguments() ? fp.parseValueSource() : null;
- return new RangeMapFloatFunction(source, min, max, target, def);
- }
- });
-
- addParser("abs", new ValueSourceParser() {
- @Override
- public ValueSource parse(FunctionQParser fp) throws SyntaxError {
- ValueSource source = fp.parseValueSource();
- return new SimpleFloatFunction(source) {
+ });
+ addParser(
+ "literal",
+ new ValueSourceParser() {
@Override
- protected String name() {
- return "abs";
+ public ValueSource parse(FunctionQParser fp) throws SyntaxError {
+ return new LiteralValueSource(fp.parseArg());
}
-
+ });
+ addParser(
+ "threadid",
+ new ValueSourceParser() {
@Override
- protected float func(int doc, FunctionValues vals) throws IOException {
- return Math.abs(vals.floatVal(doc));
+ public ValueSource parse(FunctionQParser fp) throws SyntaxError {
+ return new LongConstValueSource(Thread.currentThread().getId());
}
- };
- }
- });
- addParser("cscore", new ValueSourceParser() {
- @Override
- public ValueSource parse(FunctionQParser fp) throws SyntaxError {
- return new CollapseScoreFunction();
- }
- });
- addParser("sum", new ValueSourceParser() {
- @Override
- public ValueSource parse(FunctionQParser fp) throws SyntaxError {
- List<ValueSource> sources = fp.parseValueSourceList();
- return new SumFloatFunction(sources.toArray(new ValueSource[sources.size()]));
- }
- });
- alias("sum","add");
+ });
+ addParser(
+ "sleep",
+ new ValueSourceParser() {
+ @Override
+ public ValueSource parse(FunctionQParser fp) throws SyntaxError {
+ int ms = fp.parseInt();
+ ValueSource source = fp.parseValueSource();
+ try {
+ Thread.sleep(ms);
+ } catch (InterruptedException e) {
+ throw new RuntimeException(e);
+ }
+ return source;
+ }
+ });
+ addParser(
+ "rord",
+ new ValueSourceParser() {
+ @Override
+ public ValueSource parse(FunctionQParser fp) throws SyntaxError {
+ String field = fp.parseId();
+ return new ReverseOrdFieldSource(field);
+ }
+ });
+ addParser(
+ "top",
+ new ValueSourceParser() {
+ @Override
+ public ValueSource parse(FunctionQParser fp) throws SyntaxError {
+ // top(vs) is now a no-op
+ ValueSource source = fp.parseValueSource();
+ return source;
+ }
+ });
+ addParser(
+ "linear",
+ new ValueSourceParser() {
+ @Override
+ public ValueSource parse(FunctionQParser fp) throws SyntaxError {
+ ValueSource source = fp.parseValueSource();
+ float slope = fp.parseFloat();
+ float intercept = fp.parseFloat();
+ return new LinearFloatFunction(source, slope, intercept);
+ }
+ });
+ addParser(
+ "recip",
+ new ValueSourceParser() {
+ @Override
+ public ValueSource parse(FunctionQParser fp) throws SyntaxError {
+ ValueSource source = fp.parseValueSource();
+ float m = fp.parseFloat();
+ float a = fp.parseFloat();
+ float b = fp.parseFloat();
+ return new ReciprocalFloatFunction(source, m, a, b);
+ }
+ });
+ addParser(
+ "scale",
+ new ValueSourceParser() {
+ @Override
+ public ValueSource parse(FunctionQParser fp) throws SyntaxError {
+ ValueSource source = fp.parseValueSource();
+ float min = fp.parseFloat();
+ float max = fp.parseFloat();
+ return new ScaleFloatFunction(source, min, max);
+ }
+ });
+ addParser(
+ "div",
+ new ValueSourceParser() {
+ @Override
+ public ValueSource parse(FunctionQParser fp) throws SyntaxError {
+ ValueSource a = fp.parseValueSource();
+ ValueSource b = fp.parseValueSource();
+ return new DivFloatFunction(a, b);
+ }
+ });
+ addParser(
+ "mod",
+ new ValueSourceParser() {
+ @Override
+ public ValueSource parse(FunctionQParser fp) throws SyntaxError {
+ ValueSource a = fp.parseValueSource();
+ ValueSource b = fp.parseValueSource();
+ return new DualFloatFunction(a, b) {
+ @Override
+ protected String name() {
+ return "mod";
+ }
+
+ @Override
+ protected float func(int doc, FunctionValues aVals, FunctionValues bVals)
+ throws IOException {
+ return aVals.floatVal(doc) % bVals.floatVal(doc);
+ }
+ };
+ }
+ });
+ addParser(
+ "map",
+ new ValueSourceParser() {
+ @Override
+ public ValueSource parse(FunctionQParser fp) throws SyntaxError {
+ ValueSource source = fp.parseValueSource();
+ float min = fp.parseFloat();
+ float max = fp.parseFloat();
+ ValueSource target = fp.parseValueSource();
+ ValueSource def = fp.hasMoreArguments() ? fp.parseValueSource() : null;
+ return new RangeMapFloatFunction(source, min, max, target, def);
+ }
+ });
- addParser("product", new ValueSourceParser() {
- @Override
- public ValueSource parse(FunctionQParser fp) throws SyntaxError {
- List<ValueSource> sources = fp.parseValueSourceList();
- return new ProductFloatFunction(sources.toArray(new ValueSource[sources.size()]));
- }
- });
- alias("product","mul");
+ addParser(
+ "abs",
+ new ValueSourceParser() {
+ @Override
+ public ValueSource parse(FunctionQParser fp) throws SyntaxError {
+ ValueSource source = fp.parseValueSource();
+ return new SimpleFloatFunction(source) {
+ @Override
+ protected String name() {
+ return "abs";
+ }
+
+ @Override
+ protected float func(int doc, FunctionValues vals) throws IOException {
+ return Math.abs(vals.floatVal(doc));
+ }
+ };
+ }
+ });
+ addParser(
+ "cscore",
+ new ValueSourceParser() {
+ @Override
+ public ValueSource parse(FunctionQParser fp) throws SyntaxError {
+ return new CollapseScoreFunction();
+ }
+ });
+ addParser(
+ "sum",
+ new ValueSourceParser() {
+ @Override
+ public ValueSource parse(FunctionQParser fp) throws SyntaxError {
+ List<ValueSource> sources = fp.parseValueSourceList();
+ return new SumFloatFunction(sources.toArray(new ValueSource[sources.size()]));
+ }
+ });
+ alias("sum", "add");
- addParser("sub", new ValueSourceParser() {
- @Override
- public ValueSource parse(FunctionQParser fp) throws SyntaxError {
- ValueSource a = fp.parseValueSource();
- ValueSource b = fp.parseValueSource();
- return new DualFloatFunction(a, b) {
+ addParser(
+ "product",
+ new ValueSourceParser() {
@Override
- protected String name() {
- return "sub";
+ public ValueSource parse(FunctionQParser fp) throws SyntaxError {
+ List<ValueSource> sources = fp.parseValueSourceList();
+ return new ProductFloatFunction(sources.toArray(new ValueSource[sources.size()]));
}
+ });
+ alias("product", "mul");
+ addParser(
+ "sub",
+ new ValueSourceParser() {
@Override
- protected float func(int doc, FunctionValues aVals, FunctionValues bVals) throws IOException {
- return aVals.floatVal(doc) - bVals.floatVal(doc);
+ public ValueSource parse(FunctionQParser fp) throws SyntaxError {
+ ValueSource a = fp.parseValueSource();
+ ValueSource b = fp.parseValueSource();
+ return new DualFloatFunction(a, b) {
+ @Override
+ protected String name() {
+ return "sub";
+ }
+
+ @Override
+ protected float func(int doc, FunctionValues aVals, FunctionValues bVals)
+ throws IOException {
+ return aVals.floatVal(doc) - bVals.floatVal(doc);
+ }
+ };
}
- };
- }
- });
- addParser("vector", new ValueSourceParser(){
- @Override
- public ValueSource parse(FunctionQParser fp) throws SyntaxError {
- return new VectorValueSource(fp.parseValueSourceList());
- }
- });
- addParser("query", new ValueSourceParser() {
- // boost(query($q),rating)
- @Override
- public ValueSource parse(FunctionQParser fp) throws SyntaxError {
- Query q = fp.parseNestedQuery();
- float defVal = 0.0f;
- if (fp.hasMoreArguments()) {
- defVal = fp.parseFloat();
- }
- return new QueryValueSource(q, defVal);
- }
- });
- addParser("boost", new ValueSourceParser() {
- @Override
- public ValueSource parse(FunctionQParser fp) throws SyntaxError {
- Query q = fp.parseNestedQuery();
- ValueSource vs = fp.parseValueSource();
- return new QueryValueSource(FunctionScoreQuery.boostByValue(q, vs.asDoubleValuesSource()), 0.0f);
- }
- });
- addParser("joindf", new ValueSourceParser() {
- @Override
- public ValueSource parse(FunctionQParser fp) throws SyntaxError {
- String f0 = fp.parseArg();
- String qf = fp.parseArg();
- return new JoinDocFreqValueSource( f0, qf );
- }
- });
+ });
+ addParser(
+ "vector",
+ new ValueSourceParser() {
+ @Override
+ public ValueSource parse(FunctionQParser fp) throws SyntaxError {
+ return new VectorValueSource(fp.parseValueSourceList());
+ }
+ });
+ addParser(
+ "query",
+ new ValueSourceParser() {
+ // boost(query($q),rating)
+ @Override
+ public ValueSource parse(FunctionQParser fp) throws SyntaxError {
+ Query q = fp.parseNestedQuery();
+ float defVal = 0.0f;
+ if (fp.hasMoreArguments()) {
+ defVal = fp.parseFloat();
+ }
+ return new QueryValueSource(q, defVal);
+ }
+ });
+ addParser(
+ "boost",
+ new ValueSourceParser() {
+ @Override
+ public ValueSource parse(FunctionQParser fp) throws SyntaxError {
+ Query q = fp.parseNestedQuery();
+ ValueSource vs = fp.parseValueSource();
+ return new QueryValueSource(
+ FunctionScoreQuery.boostByValue(q, vs.asDoubleValuesSource()), 0.0f);
+ }
+ });
+ addParser(
+ "joindf",
+ new ValueSourceParser() {
+ @Override
+ public ValueSource parse(FunctionQParser fp) throws SyntaxError {
+ String f0 = fp.parseArg();
+ String qf = fp.parseArg();
+ return new JoinDocFreqValueSource(f0, qf);
+ }
+ });
addParser("geodist", new GeoDistValueSourceParser());
- addParser("hsin", new ValueSourceParser() {
- @Override
- public ValueSource parse(FunctionQParser fp) throws SyntaxError {
-
- double radius = fp.parseDouble();
- //SOLR-2114, make the convert flag required, since the parser doesn't support much in the way of lookahead or the ability to convert a String into a ValueSource
- boolean convert = Boolean.parseBoolean(fp.parseArg());
-
- MultiValueSource pv1;
- MultiValueSource pv2;
-
- ValueSource one = fp.parseValueSource();
- ValueSource two = fp.parseValueSource();
- if (fp.hasMoreArguments()) {
- pv1 = new VectorValueSource(Arrays.asList(one, two));//x1, y1
- pv2 = new VectorValueSource(Arrays.asList(fp.parseValueSource(), fp.parseValueSource()));//x2, y2
- } else {
- //check to see if we have multiValue source
- if (one instanceof MultiValueSource && two instanceof MultiValueSource){
- pv1 = (MultiValueSource) one;
- pv2 = (MultiValueSource) two;
- } else {
- throw new SolrException(SolrException.ErrorCode.BAD_REQUEST,
+ addParser(
+ "hsin",
+ new ValueSourceParser() {
+ @Override
+ public ValueSource parse(FunctionQParser fp) throws SyntaxError {
+
+ double radius = fp.parseDouble();
+ // SOLR-2114, make the convert flag required, since the parser doesn't support much in
+ // the way of lookahead or the ability to convert a String into a ValueSource
+ boolean convert = Boolean.parseBoolean(fp.parseArg());
+
+ MultiValueSource pv1;
+ MultiValueSource pv2;
+
+ ValueSource one = fp.parseValueSource();
+ ValueSource two = fp.parseValueSource();
+ if (fp.hasMoreArguments()) {
+ pv1 = new VectorValueSource(Arrays.asList(one, two)); // x1, y1
+ pv2 =
+ new VectorValueSource(
+ Arrays.asList(fp.parseValueSource(), fp.parseValueSource())); // x2, y2
+ } else {
+ // check to see if we have multiValue source
+ if (one instanceof MultiValueSource && two instanceof MultiValueSource) {
+ pv1 = (MultiValueSource) one;
+ pv2 = (MultiValueSource) two;
+ } else {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
"Input must either be 2 MultiValueSources, or there must be 4 ValueSources");
- }
- }
+ }
+ }
- return new HaversineFunction(pv1, pv2, radius, convert);
- }
- });
+ return new HaversineFunction(pv1, pv2, radius, convert);
+ }
+ });
- addParser("ghhsin", new ValueSourceParser() {
- @Override
- public ValueSource parse(FunctionQParser fp) throws SyntaxError {
- double radius = fp.parseDouble();
+ addParser(
+ "ghhsin",
+ new ValueSourceParser() {
+ @Override
+ public ValueSource parse(FunctionQParser fp) throws SyntaxError {
+ double radius = fp.parseDouble();
- ValueSource gh1 = fp.parseValueSource();
- ValueSource gh2 = fp.parseValueSource();
+ ValueSource gh1 = fp.parseValueSource();
+ ValueSource gh2 = fp.parseValueSource();
- return new GeohashHaversineFunction(gh1, gh2, radius);
- }
- });
+ return new GeohashHaversineFunction(gh1, gh2, radius);
+ }
+ });
- addParser("geohash", new ValueSourceParser() {
- @Override
- public ValueSource parse(FunctionQParser fp) throws SyntaxError {
+ addParser(
+ "geohash",
+ new ValueSourceParser() {
+ @Override
+ public ValueSource parse(FunctionQParser fp) throws SyntaxError {
- ValueSource lat = fp.parseValueSource();
- ValueSource lon = fp.parseValueSource();
+ ValueSource lat = fp.parseValueSource();
+ ValueSource lon = fp.parseValueSource();
- return new GeohashFunction(lat, lon);
- }
- });
- addParser("strdist", new ValueSourceParser() {
- @Override
- public ValueSource parse(FunctionQParser fp) throws SyntaxError {
-
- ValueSource str1 = fp.parseValueSource();
- ValueSource str2 = fp.parseValueSource();
- String distClass = fp.parseArg();
-
- StringDistance dist = null;
- if (distClass.equalsIgnoreCase("jw")) {
- dist = new JaroWinklerDistance();
- } else if (distClass.equalsIgnoreCase("edit")) {
- dist = new LevenshteinDistance();
- } else if (distClass.equalsIgnoreCase("ngram")) {
- int ngram = 2;
- if (fp.hasMoreArguments()) {
- ngram = fp.parseInt();
- }
- dist = new NGramDistance(ngram);
- } else {
- dist = fp.req.getCore().getResourceLoader().newInstance(distClass, StringDistance.class);
- }
- return new StringDistanceFunction(str1, str2, dist);
- }
- });
- addParser("field", new ValueSourceParser() {
- @Override
- public ValueSource parse(FunctionQParser fp) throws SyntaxError {
-
- String fieldName = fp.parseArg();
- SchemaField f = fp.getReq().getSchema().getField(fieldName);
- if (fp.hasMoreArguments()) {
- // multivalued selector option
- String s = fp.parseArg();
- FieldType.MultiValueSelector selector = FieldType.MultiValueSelector.lookup(s);
- if (null == selector) {
- throw new SolrException(SolrException.ErrorCode.BAD_REQUEST,
- "Multi-Valued field selector '"+s+"' not supported");
- }
- return f.getType().getSingleValueSource(selector, f, fp);
- }
- // simple field ValueSource
- return f.getType().getValueSource(f, fp);
- }
- });
- addParser("currency", new ValueSourceParser() {
- @Override
- public ValueSource parse(FunctionQParser fp) throws SyntaxError {
-
- String fieldName = fp.parseArg();
- SchemaField f = fp.getReq().getSchema().getField(fieldName);
- if (! (f.getType() instanceof CurrencyFieldType)) {
- throw new SolrException(SolrException.ErrorCode.BAD_REQUEST,
- "Currency function input must be the name of a CurrencyFieldType: " + fieldName);
- }
- CurrencyFieldType ft = (CurrencyFieldType) f.getType();
- String code = fp.hasMoreArguments() ? fp.parseArg() : null;
- return ft.getConvertedValueSource(code, ft.getValueSource(f, fp));
- }
- });
+ return new GeohashFunction(lat, lon);
+ }
+ });
+ addParser(
+ "strdist",
+ new ValueSourceParser() {
+ @Override
+ public ValueSource parse(FunctionQParser fp) throws SyntaxError {
+
+ ValueSource str1 = fp.parseValueSource();
+ ValueSource str2 = fp.parseValueSource();
+ String distClass = fp.parseArg();
+
+ StringDistance dist = null;
+ if (distClass.equalsIgnoreCase("jw")) {
+ dist = new JaroWinklerDistance();
+ } else if (distClass.equalsIgnoreCase("edit")) {
+ dist = new LevenshteinDistance();
+ } else if (distClass.equalsIgnoreCase("ngram")) {
+ int ngram = 2;
+ if (fp.hasMoreArguments()) {
+ ngram = fp.parseInt();
+ }
+ dist = new NGramDistance(ngram);
+ } else {
+ dist =
+ fp.req.getCore().getResourceLoader().newInstance(distClass, StringDistance.class);
+ }
+ return new StringDistanceFunction(str1, str2, dist);
+ }
+ });
+ addParser(
+ "field",
+ new ValueSourceParser() {
+ @Override
+ public ValueSource parse(FunctionQParser fp) throws SyntaxError {
+
+ String fieldName = fp.parseArg();
+ SchemaField f = fp.getReq().getSchema().getField(fieldName);
+ if (fp.hasMoreArguments()) {
+ // multivalued selector option
+ String s = fp.parseArg();
+ FieldType.MultiValueSelector selector = FieldType.MultiValueSelector.lookup(s);
+ if (null == selector) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Multi-Valued field selector '" + s + "' not supported");
+ }
+ return f.getType().getSingleValueSource(selector, f, fp);
+ }
+ // simple field ValueSource
+ return f.getType().getValueSource(f, fp);
+ }
+ });
+ addParser(
+ "currency",
+ new ValueSourceParser() {
+ @Override
+ public ValueSource parse(FunctionQParser fp) throws SyntaxError {
+
+ String fieldName = fp.parseArg();
+ SchemaField f = fp.getReq().getSchema().getField(fieldName);
+ if (!(f.getType() instanceof CurrencyFieldType)) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Currency function input must be the name of a CurrencyFieldType: " + fieldName);
+ }
+ CurrencyFieldType ft = (CurrencyFieldType) f.getType();
+ String code = fp.hasMoreArguments() ? fp.parseArg() : null;
+ return ft.getConvertedValueSource(code, ft.getValueSource(f, fp));
+ }
+ });
- addParser(new DoubleParser("rad") {
- @Override
- public double func(int doc, FunctionValues vals) throws IOException {
- return vals.doubleVal(doc) * DistanceUtils.DEGREES_TO_RADIANS;
- }
- });
- addParser(new DoubleParser("deg") {
- @Override
- public double func(int doc, FunctionValues vals) throws IOException {
- return vals.doubleVal(doc) * DistanceUtils.RADIANS_TO_DEGREES;
- }
- });
- addParser(new DoubleParser("sqrt") {
- @Override
- public double func(int doc, FunctionValues vals) throws IOException {
- return Math.sqrt(vals.doubleVal(doc));
- }
- });
- addParser(new DoubleParser("cbrt") {
- @Override
- public double func(int doc, FunctionValues vals) throws IOException {
- return Math.cbrt(vals.doubleVal(doc));
- }
- });
- addParser(new DoubleParser("log") {
- @Override
- public double func(int doc, FunctionValues vals) throws IOException {
- return Math.log10(vals.doubleVal(doc));
- }
- });
- addParser(new DoubleParser("ln") {
- @Override
- public double func(int doc, FunctionValues vals) throws IOException {
- return Math.log(vals.doubleVal(doc));
- }
- });
- addParser(new DoubleParser("exp") {
- @Override
- public double func(int doc, FunctionValues vals) throws IOException {
- return Math.exp(vals.doubleVal(doc));
- }
- });
- addParser(new DoubleParser("sin") {
- @Override
- public double func(int doc, FunctionValues vals) throws IOException {
- return Math.sin(vals.doubleVal(doc));
- }
- });
- addParser(new DoubleParser("cos") {
- @Override
- public double func(int doc, FunctionValues vals) throws IOException {
- return Math.cos(vals.doubleVal(doc));
- }
- });
- addParser(new DoubleParser("tan") {
- @Override
- public double func(int doc, FunctionValues vals) throws IOException {
- return Math.tan(vals.doubleVal(doc));
- }
- });
- addParser(new DoubleParser("asin") {
- @Override
- public double func(int doc, FunctionValues vals) throws IOException {
- return Math.asin(vals.doubleVal(doc));
- }
- });
- addParser(new DoubleParser("acos") {
- @Override
- public double func(int doc, FunctionValues vals) throws IOException {
- return Math.acos(vals.doubleVal(doc));
- }
- });
- addParser(new DoubleParser("atan") {
- @Override
- public double func(int doc, FunctionValues vals) throws IOException {
- return Math.atan(vals.doubleVal(doc));
- }
- });
- addParser(new DoubleParser("sinh") {
- @Override
- public double func(int doc, FunctionValues vals) throws IOException {
- return Math.sinh(vals.doubleVal(doc));
- }
- });
- addParser(new DoubleParser("cosh") {
- @Override
- public double func(int doc, FunctionValues vals) throws IOException {
- return Math.cosh(vals.doubleVal(doc));
- }
- });
- addParser(new DoubleParser("tanh") {
- @Override
- public double func(int doc, FunctionValues vals) throws IOException {
- return Math.tanh(vals.doubleVal(doc));
- }
- });
- addParser(new DoubleParser("ceil") {
- @Override
- public double func(int doc, FunctionValues vals) throws IOException {
- return Math.ceil(vals.doubleVal(doc));
- }
- });
- addParser(new DoubleParser("floor") {
- @Override
- public double func(int doc, FunctionValues vals) throws IOException {
- return Math.floor(vals.doubleVal(doc));
- }
- });
- addParser(new DoubleParser("rint") {
- @Override
- public double func(int doc, FunctionValues vals) throws IOException {
- return Math.rint(vals.doubleVal(doc));
- }
- });
- addParser(new Double2Parser("pow") {
- @Override
- public double func(int doc, FunctionValues a, FunctionValues b) throws IOException {
- return Math.pow(a.doubleVal(doc), b.doubleVal(doc));
- }
- });
- addParser(new Double2Parser("hypot") {
- @Override
- public double func(int doc, FunctionValues a, FunctionValues b) throws IOException {
- return Math.hypot(a.doubleVal(doc), b.doubleVal(doc));
- }
- });
- addParser(new Double2Parser("atan2") {
- @Override
- public double func(int doc, FunctionValues a, FunctionValues b) throws IOException {
- return Math.atan2(a.doubleVal(doc), b.doubleVal(doc));
- }
- });
- addParser("max", new ValueSourceParser() {
- @Override
- public ValueSource parse(FunctionQParser fp) throws SyntaxError {
- List<ValueSource> sources = fp.parseValueSourceList();
- return new MaxFloatFunction(sources.toArray(new ValueSource[sources.size()]));
- }
- });
- addParser("min", new ValueSourceParser() {
- @Override
- public ValueSource parse(FunctionQParser fp) throws SyntaxError {
- List<ValueSource> sources = fp.parseValueSourceList();
- return new MinFloatFunction(sources.toArray(new ValueSource[sources.size()]));
- }
- });
+ addParser(
+ new DoubleParser("rad") {
+ @Override
+ public double func(int doc, FunctionValues vals) throws IOException {
+ return vals.doubleVal(doc) * DistanceUtils.DEGREES_TO_RADIANS;
+ }
+ });
+ addParser(
+ new DoubleParser("deg") {
+ @Override
+ public double func(int doc, FunctionValues vals) throws IOException {
+ return vals.doubleVal(doc) * DistanceUtils.RADIANS_TO_DEGREES;
+ }
+ });
+ addParser(
+ new DoubleParser("sqrt") {
+ @Override
+ public double func(int doc, FunctionValues vals) throws IOException {
+ return Math.sqrt(vals.doubleVal(doc));
+ }
+ });
+ addParser(
+ new DoubleParser("cbrt") {
+ @Override
+ public double func(int doc, FunctionValues vals) throws IOException {
+ return Math.cbrt(vals.doubleVal(doc));
+ }
+ });
+ addParser(
+ new DoubleParser("log") {
+ @Override
+ public double func(int doc, FunctionValues vals) throws IOException {
+ return Math.log10(vals.doubleVal(doc));
+ }
+ });
+ addParser(
+ new DoubleParser("ln") {
+ @Override
+ public double func(int doc, FunctionValues vals) throws IOException {
+ return Math.log(vals.doubleVal(doc));
+ }
+ });
+ addParser(
+ new DoubleParser("exp") {
+ @Override
+ public double func(int doc, FunctionValues vals) throws IOException {
+ return Math.exp(vals.doubleVal(doc));
+ }
+ });
+ addParser(
+ new DoubleParser("sin") {
+ @Override
+ public double func(int doc, FunctionValues vals) throws IOException {
+ return Math.sin(vals.doubleVal(doc));
+ }
+ });
+ addParser(
+ new DoubleParser("cos") {
+ @Override
+ public double func(int doc, FunctionValues vals) throws IOException {
+ return Math.cos(vals.doubleVal(doc));
+ }
+ });
+ addParser(
+ new DoubleParser("tan") {
+ @Override
+ public double func(int doc, FunctionValues vals) throws IOException {
+ return Math.tan(vals.doubleVal(doc));
+ }
+ });
+ addParser(
+ new DoubleParser("asin") {
+ @Override
+ public double func(int doc, FunctionValues vals) throws IOException {
+ return Math.asin(vals.doubleVal(doc));
+ }
+ });
+ addParser(
+ new DoubleParser("acos") {
+ @Override
+ public double func(int doc, FunctionValues vals) throws IOException {
+ return Math.acos(vals.doubleVal(doc));
+ }
+ });
+ addParser(
+ new DoubleParser("atan") {
+ @Override
+ public double func(int doc, FunctionValues vals) throws IOException {
+ return Math.atan(vals.doubleVal(doc));
+ }
+ });
+ addParser(
+ new DoubleParser("sinh") {
+ @Override
+ public double func(int doc, FunctionValues vals) throws IOException {
+ return Math.sinh(vals.doubleVal(doc));
+ }
+ });
+ addParser(
+ new DoubleParser("cosh") {
+ @Override
+ public double func(int doc, FunctionValues vals) throws IOException {
+ return Math.cosh(vals.doubleVal(doc));
+ }
+ });
+ addParser(
+ new DoubleParser("tanh") {
+ @Override
+ public double func(int doc, FunctionValues vals) throws IOException {
+ return Math.tanh(vals.doubleVal(doc));
+ }
+ });
+ addParser(
+ new DoubleParser("ceil") {
+ @Override
+ public double func(int doc, FunctionValues vals) throws IOException {
+ return Math.ceil(vals.doubleVal(doc));
+ }
+ });
+ addParser(
+ new DoubleParser("floor") {
+ @Override
+ public double func(int doc, FunctionValues vals) throws IOException {
+ return Math.floor(vals.doubleVal(doc));
+ }
+ });
+ addParser(
+ new DoubleParser("rint") {
+ @Override
+ public double func(int doc, FunctionValues vals) throws IOException {
+ return Math.rint(vals.doubleVal(doc));
+ }
+ });
+ addParser(
+ new Double2Parser("pow") {
+ @Override
+ public double func(int doc, FunctionValues a, FunctionValues b) throws IOException {
+ return Math.pow(a.doubleVal(doc), b.doubleVal(doc));
+ }
+ });
+ addParser(
+ new Double2Parser("hypot") {
+ @Override
+ public double func(int doc, FunctionValues a, FunctionValues b) throws IOException {
+ return Math.hypot(a.doubleVal(doc), b.doubleVal(doc));
+ }
+ });
+ addParser(
+ new Double2Parser("atan2") {
+ @Override
+ public double func(int doc, FunctionValues a, FunctionValues b) throws IOException {
+ return Math.atan2(a.doubleVal(doc), b.doubleVal(doc));
+ }
+ });
+ addParser(
+ "max",
+ new ValueSourceParser() {
+ @Override
+ public ValueSource parse(FunctionQParser fp) throws SyntaxError {
+ List<ValueSource> sources = fp.parseValueSourceList();
+ return new MaxFloatFunction(sources.toArray(new ValueSource[sources.size()]));
+ }
+ });
+ addParser(
+ "min",
+ new ValueSourceParser() {
+ @Override
+ public ValueSource parse(FunctionQParser fp) throws SyntaxError {
+ List<ValueSource> sources = fp.parseValueSourceList();
+ return new MinFloatFunction(sources.toArray(new ValueSource[sources.size()]));
+ }
+ });
- addParser("sqedist", new ValueSourceParser() {
- @Override
- public ValueSource parse(FunctionQParser fp) throws SyntaxError {
- List<ValueSource> sources = fp.parseValueSourceList();
- MVResult mvr = getMultiValueSources(sources);
+ addParser(
+ "sqedist",
+ new ValueSourceParser() {
+ @Override
+ public ValueSource parse(FunctionQParser fp) throws SyntaxError {
+ List<ValueSource> sources = fp.parseValueSourceList();
+ MVResult mvr = getMultiValueSources(sources);
- return new SquaredEuclideanFunction(mvr.mv1, mvr.mv2);
- }
- });
+ return new SquaredEuclideanFunction(mvr.mv1, mvr.mv2);
+ }
+ });
- addParser("dist", new ValueSourceParser() {
- @Override
- public ValueSource parse(FunctionQParser fp) throws SyntaxError {
- float power = fp.parseFloat();
- List<ValueSource> sources = fp.parseValueSourceList();
- MVResult mvr = getMultiValueSources(sources);
- return new VectorDistanceFunction(power, mvr.mv1, mvr.mv2);
- }
- });
+ addParser(
+ "dist",
+ new ValueSourceParser() {
+ @Override
+ public ValueSource parse(FunctionQParser fp) throws SyntaxError {
+ float power = fp.parseFloat();
+ List<ValueSource> sources = fp.parseValueSourceList();
+ MVResult mvr = getMultiValueSources(sources);
+ return new VectorDistanceFunction(power, mvr.mv1, mvr.mv2);
+ }
+ });
addParser("ms", new DateValueSourceParser());
-
- addParser("pi", new ValueSourceParser() {
- @Override
- public ValueSource parse(FunctionQParser fp) {
- return new DoubleConstValueSource(Math.PI);
- }
- });
- addParser("e", new ValueSourceParser() {
- @Override
- public ValueSource parse(FunctionQParser fp) {
- return new DoubleConstValueSource(Math.E);
- }
- });
-
-
- addParser("docfreq", new ValueSourceParser() {
- @Override
- public ValueSource parse(FunctionQParser fp) throws SyntaxError {
- TInfo tinfo = parseTerm(fp);
- return new DocFreqValueSource(tinfo.field, tinfo.val, tinfo.indexedField, tinfo.indexedBytes.get());
- }
- });
-
- addParser("totaltermfreq", new ValueSourceParser() {
- @Override
- public ValueSource parse(FunctionQParser fp) throws SyntaxError {
- TInfo tinfo = parseTerm(fp);
- return new TotalTermFreqValueSource(tinfo.field, tinfo.val, tinfo.indexedField, tinfo.indexedBytes.get());
- }
- });
- alias("totaltermfreq","ttf");
-
- addParser("sumtotaltermfreq", new ValueSourceParser() {
- @Override
- public ValueSource parse(FunctionQParser fp) throws SyntaxError {
- String field = fp.parseArg();
- return new SumTotalTermFreqValueSource(field);
- }
- });
- alias("sumtotaltermfreq","sttf");
-
- addParser("idf", new ValueSourceParser() {
- @Override
- public ValueSource parse(FunctionQParser fp) throws SyntaxError {
- TInfo tinfo = parseTerm(fp);
- return new IDFValueSource(tinfo.field, tinfo.val, tinfo.indexedField, tinfo.indexedBytes.get());
- }
- });
-
- addParser("termfreq", new ValueSourceParser() {
- @Override
- public ValueSource parse(FunctionQParser fp) throws SyntaxError {
- TInfo tinfo = parseTerm(fp);
- return new TermFreqValueSource(tinfo.field, tinfo.val, tinfo.indexedField, tinfo.indexedBytes.get());
- }
- });
+ addParser(
+ "pi",
+ new ValueSourceParser() {
+ @Override
+ public ValueSource parse(FunctionQParser fp) {
+ return new DoubleConstValueSource(Math.PI);
+ }
+ });
+ addParser(
+ "e",
+ new ValueSourceParser() {
+ @Override
+ public ValueSource parse(FunctionQParser fp) {
+ return new DoubleConstValueSource(Math.E);
+ }
+ });
- addParser("tf", new ValueSourceParser() {
- @Override
- public ValueSource parse(FunctionQParser fp) throws SyntaxError {
- TInfo tinfo = parseTerm(fp);
- return new TFValueSource(tinfo.field, tinfo.val, tinfo.indexedField, tinfo.indexedBytes.get());
- }
- });
+ addParser(
+ "docfreq",
+ new ValueSourceParser() {
+ @Override
+ public ValueSource parse(FunctionQParser fp) throws SyntaxError {
+ TInfo tinfo = parseTerm(fp);
+ return new DocFreqValueSource(
+ tinfo.field, tinfo.val, tinfo.indexedField, tinfo.indexedBytes.get());
+ }
+ });
- addParser("norm", new ValueSourceParser() {
- @Override
- public ValueSource parse(FunctionQParser fp) throws SyntaxError {
- String field = fp.parseArg();
- return new NormValueSource(field);
- }
- });
+ addParser(
+ "totaltermfreq",
+ new ValueSourceParser() {
+ @Override
+ public ValueSource parse(FunctionQParser fp) throws SyntaxError {
+ TInfo tinfo = parseTerm(fp);
+ return new TotalTermFreqValueSource(
+ tinfo.field, tinfo.val, tinfo.indexedField, tinfo.indexedBytes.get());
+ }
+ });
+ alias("totaltermfreq", "ttf");
- addParser("maxdoc", new ValueSourceParser() {
- @Override
- public ValueSource parse(FunctionQParser fp) {
- return new MaxDocValueSource();
- }
- });
+ addParser(
+ "sumtotaltermfreq",
+ new ValueSourceParser() {
+ @Override
+ public ValueSource parse(FunctionQParser fp) throws SyntaxError {
+ String field = fp.parseArg();
+ return new SumTotalTermFreqValueSource(field);
+ }
+ });
+ alias("sumtotaltermfreq", "sttf");
- addParser("numdocs", new ValueSourceParser() {
- @Override
- public ValueSource parse(FunctionQParser fp) {
- return new NumDocsValueSource();
- }
- });
+ addParser(
+ "idf",
+ new ValueSourceParser() {
+ @Override
+ public ValueSource parse(FunctionQParser fp) throws SyntaxError {
+ TInfo tinfo = parseTerm(fp);
+ return new IDFValueSource(
+ tinfo.field, tinfo.val, tinfo.indexedField, tinfo.indexedBytes.get());
+ }
+ });
- addParser("payload", new ValueSourceParser() {
- @Override
- public ValueSource parse(FunctionQParser fp) throws SyntaxError {
- // payload(field,value[,default, ['min|max|average|first']])
- // defaults to "average" and 0.0 default value
+ addParser(
+ "termfreq",
+ new ValueSourceParser() {
+ @Override
+ public ValueSource parse(FunctionQParser fp) throws SyntaxError {
+ TInfo tinfo = parseTerm(fp);
+ return new TermFreqValueSource(
+ tinfo.field, tinfo.val, tinfo.indexedField, tinfo.indexedBytes.get());
+ }
+ });
- TInfo tinfo = parseTerm(fp); // would have made this parser a new separate class and registered it, but this handy method is private :/
+ addParser(
+ "tf",
+ new ValueSourceParser() {
+ @Override
+ public ValueSource parse(FunctionQParser fp) throws SyntaxError {
+ TInfo tinfo = parseTerm(fp);
+ return new TFValueSource(
+ tinfo.field, tinfo.val, tinfo.indexedField, tinfo.indexedBytes.get());
+ }
+ });
- ValueSource defaultValueSource;
- if (fp.hasMoreArguments()) {
- defaultValueSource = fp.parseValueSource();
- } else {
- defaultValueSource = new ConstValueSource(0.0f);
- }
+ addParser(
+ "norm",
+ new ValueSourceParser() {
+ @Override
+ public ValueSource parse(FunctionQParser fp) throws SyntaxError {
+ String field = fp.parseArg();
+ return new NormValueSource(field);
+ }
+ });
- PayloadFunction payloadFunction = null;
- String func = "average";
- if (fp.hasMoreArguments()) {
- func = fp.parseArg();
- }
- payloadFunction = PayloadUtils.getPayloadFunction(func);
+ addParser(
+ "maxdoc",
+ new ValueSourceParser() {
+ @Override
+ public ValueSource parse(FunctionQParser fp) {
+ return new MaxDocValueSource();
+ }
+ });
- // Support func="first" by payloadFunction=null
- if(payloadFunction == null && !"first".equals(func)) {
- // not "first" (or average, min, or max)
- throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "Invalid payload function: " + func);
- }
+ addParser(
+ "numdocs",
+ new ValueSourceParser() {
+ @Override
+ public ValueSource parse(FunctionQParser fp) {
+ return new NumDocsValueSource();
+ }
+ });
- IndexSchema schema = fp.getReq().getCore().getLatestSchema();
- PayloadDecoder decoder = schema.getPayloadDecoder(tinfo.field);
+ addParser(
+ "payload",
+ new ValueSourceParser() {
+ @Override
+ public ValueSource parse(FunctionQParser fp) throws SyntaxError {
+ // payload(field,value[,default, ['min|max|average|first']])
+ // defaults to "average" and 0.0 default value
+
+ TInfo tinfo =
+ parseTerm(
+ fp); // would have made this parser a new separate class and registered it, but
+ // this handy method is private :/
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/facet/FacetFieldMerger.java
##########
@@ -22,21 +22,19 @@
import java.util.HashMap;
import java.util.List;
import java.util.Map;
-
import org.apache.solr.common.util.SimpleOrderedMap;
-
// TODO: refactor more out to base class
public class FacetFieldMerger extends FacetRequestSortedMerger<FacetField> {
FacetBucket missingBucket;
FacetBucket allBuckets;
FacetMerger numBuckets;
- int[] numReturnedPerShard; // TODO: this is currently unused?
+ int[] numReturnedPerShard; // TODO: this is currently unused?
// LinkedHashMap<Object,FacetBucket> buckets = new LinkedHashMap<>();
// List<FacetBucket> sortedBuckets;
- int numReturnedBuckets; // the number of buckets in the bucket lists returned from all of the shards
-
+ int numReturnedBuckets; // the number of buckets in the bucket lists returned from all of the
+ // shards
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/facet/FacetFieldProcessor.java
##########
@@ -316,9 +339,12 @@ private static long applyDefaultOverrequest(long offset, long limit) {
if (fcontext.isShard()) {
if (freq.overrequest > 0) {
// NOTE: although _default_ distrib overrequest is disabled for the "index sort" case (see
- // below), we _do_ want to respect an _explicit_ `overrequest` value, if present. Overrequest
- // is always relevant (regardless of prelim sort) for the `resort` case; but even in the case of
- // "index sort, no resort", overrequest can be relevant in some edge cases of the "shard" case,
+ // below), we _do_ want to respect an _explicit_ `overrequest` value, if present.
+ // Overrequest
+ // is always relevant (regardless of prelim sort) for the `resort` case; but even in the
+ // case of
+ // "index sort, no resort", overrequest can be relevant in some edge cases of the "shard"
+ // case,
// where it can affect the behavior of `isBucketComplete()` (see SOLR-14595).
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/facet/FacetFieldProcessor.java
##########
@@ -44,62 +45,72 @@
import org.apache.solr.search.facet.SlotAcc.SweepableSlotAcc;
import org.apache.solr.search.facet.SlotAcc.SweepingCountSlotAcc;
-import static org.apache.solr.search.facet.FacetContext.SKIP_FACET;
-
/**
* Facet processing based on field values. (not range nor by query)
+ *
* @see FacetField
*/
abstract class FacetFieldProcessor extends FacetProcessor<FacetField> {
SchemaField sf;
SlotAcc indexOrderAcc;
int effectiveMincount;
final boolean singlePassSlotAccCollection;
- final FacetRequest.FacetSort sort; // never null (may be the user's requested sort, or the prelim_sort)
+ final FacetRequest.FacetSort
+ sort; // never null (may be the user's requested sort, or the prelim_sort)
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/facet/FacetFieldProcessor.java
##########
@@ -573,52 +623,56 @@ private void fillBucketFromSlot(SimpleOrderedMap<Object> target, Slot slot,
processSubs(target, filter, subDomain, false, null);
}
- /**
+ /**
* Helper method that resorts the slots (if needed).
- *
- * @return a SlotAcc that should be used {@link SlotAcc#setValues} on the final buckets via
- * {@link Slot#resortSlotNum} or null if no special SlotAcc was needed (ie: no resorting, or resorting
- * on something already known/collected)
+ *
+ * @return a SlotAcc that should be used {@link SlotAcc#setValues} on the final buckets via {@link
+ * Slot#resortSlotNum} or null if no special SlotAcc was needed (ie: no resorting, or
+ * resorting on something already known/collected)
*/
private SlotAcc resortSlots(Slot[] slots) throws IOException {
if (null == this.resort) {
return null; // Nothing to do.
}
- assert ! fcontext.isShard();
+ assert !fcontext.isShard();
// NOTE: getMultiplier() is confusing and weird and ment for use in PriorityQueue.lessThan,
// so it's backwards from what you'd expect in a Comparator...
final int resortMul = -1 * resort.sortDirection.getMultiplier();
-
+
SlotAcc resortAcc = getTrivialSortingSlotAcc(this.resort);
if (null != resortAcc) {
- // resorting on count or index is rare (and not particularly useful) but if someone chooses to do
+ // resorting on count or index is rare (and not particularly useful) but if someone chooses to
+ // do
// either of these we don't need to re-collect ... instead just re-sort the slots based on
// the previously collected values using the originally collected slot numbers...
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/facet/FacetFieldProcessor.java
##########
@@ -186,14 +198,15 @@ void createCollectAcc(int numDocs, int numSlots) throws IOException {
sortAcc = getTrivialSortingSlotAcc(this.sort);
if (this.singlePassSlotAccCollection) {
- // If we are going to return all buckets, and if there are no subfacets (that would need a domain),
+ // If we are going to return all buckets, and if there are no subfacets (that would need a
+ // domain),
// then don't defer any aggregation calculations to a second phase.
// This way we can avoid calculating domains for each bucket, which can be expensive.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/facet/FacetFieldProcessor.java
##########
@@ -44,62 +45,72 @@
import org.apache.solr.search.facet.SlotAcc.SweepableSlotAcc;
import org.apache.solr.search.facet.SlotAcc.SweepingCountSlotAcc;
-import static org.apache.solr.search.facet.FacetContext.SKIP_FACET;
-
/**
* Facet processing based on field values. (not range nor by query)
+ *
* @see FacetField
*/
abstract class FacetFieldProcessor extends FacetProcessor<FacetField> {
SchemaField sf;
SlotAcc indexOrderAcc;
int effectiveMincount;
final boolean singlePassSlotAccCollection;
- final FacetRequest.FacetSort sort; // never null (may be the user's requested sort, or the prelim_sort)
+ final FacetRequest.FacetSort
+ sort; // never null (may be the user's requested sort, or the prelim_sort)
final FacetRequest.FacetSort resort; // typically null (unless the user specified a prelim_sort)
-
- final Map<String,AggValueSource> deferredAggs = new HashMap<String,AggValueSource>();
+
+ final Map<String, AggValueSource> deferredAggs = new HashMap<String, AggValueSource>();
// TODO: push any of this down to base class?
//
// For sort="x desc", collectAcc would point to "x", and sortAcc would also point to "x".
- // collectAcc would be used to accumulate all buckets, and sortAcc would be used to sort those buckets.
+ // collectAcc would be used to accumulate all buckets, and sortAcc would be used to sort those
+ // buckets.
//
- SlotAcc collectAcc; // Accumulator to collect across entire domain (in addition to the countAcc). May be null.
- SlotAcc sortAcc; // Accumulator to use for sorting *only* (i.e. not used for collection). May be an alias of countAcc, collectAcc, or indexOrderAcc
+ SlotAcc
+ collectAcc; // Accumulator to collect across entire domain (in addition to the countAcc). May
+ // be null.
+ SlotAcc
+ sortAcc; // Accumulator to use for sorting *only* (i.e. not used for collection). May be an
+ // alias of countAcc, collectAcc, or indexOrderAcc
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/facet/FacetFieldProcessor.java
##########
@@ -329,15 +355,19 @@ private static long applyDefaultOverrequest(long offset, long limit) {
case -1:
// default
if (!"index".equals(this.sort.sortVariable)) {
- // NOTE: even for distrib requests, `overrequest` is not directly relevant for "index" sort, hence
- // there is no default/implicit overrequest for "index sort" (even if `resort` is also specified --
- // overrequest that is exclusively for `resort` must be explicit, even in a distrib context)
+ // NOTE: even for distrib requests, `overrequest` is not directly relevant for
+ // "index" sort, hence
+ // there is no default/implicit overrequest for "index sort" (even if `resort` is
+ // also specified --
+ // overrequest that is exclusively for `resort` must be explicit, even in a distrib
+ // context)
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/facet/FacetFieldProcessor.java
##########
@@ -348,38 +378,48 @@ private static long applyDefaultOverrequest(long offset, long limit) {
final int sortMul = sort.sortDirection.getMultiplier();
- int maxTopVals = (int) (effectiveLimit >= 0 ? Math.min(freq.offset + effectiveLimit, Integer.MAX_VALUE - 1) : Integer.MAX_VALUE - 1);
+ int maxTopVals =
+ (int)
+ (effectiveLimit >= 0
+ ? Math.min(freq.offset + effectiveLimit, Integer.MAX_VALUE - 1)
+ : Integer.MAX_VALUE - 1);
maxTopVals = Math.min(maxTopVals, slotCardinality);
final SlotAcc sortAcc = this.sortAcc, indexOrderAcc = this.indexOrderAcc;
- final BiPredicate<Slot,Slot> orderPredicate;
+ final BiPredicate<Slot, Slot> orderPredicate;
if (indexOrderAcc != null && indexOrderAcc != sortAcc) {
- orderPredicate = (a, b) -> {
- int cmp = sortAcc.compare(a.slot, b.slot) * sortMul;
- return cmp == 0 ? (indexOrderAcc.compare(a.slot, b.slot) > 0) : cmp < 0;
- };
+ orderPredicate =
+ (a, b) -> {
+ int cmp = sortAcc.compare(a.slot, b.slot) * sortMul;
+ return cmp == 0 ? (indexOrderAcc.compare(a.slot, b.slot) > 0) : cmp < 0;
+ };
} else {
- orderPredicate = (a, b) -> {
- int cmp = sortAcc.compare(a.slot, b.slot) * sortMul;
- return cmp == 0 ? b.slot < a.slot : cmp < 0;
- };
- }
- final PriorityQueue<Slot> queue = new PriorityQueue<>(maxTopVals) {
- @Override
- protected boolean lessThan(Slot a, Slot b) { return orderPredicate.test(a, b); }
- };
+ orderPredicate =
+ (a, b) -> {
+ int cmp = sortAcc.compare(a.slot, b.slot) * sortMul;
+ return cmp == 0 ? b.slot < a.slot : cmp < 0;
+ };
+ }
+ final PriorityQueue<Slot> queue =
+ new PriorityQueue<>(maxTopVals) {
+ @Override
+ protected boolean lessThan(Slot a, Slot b) {
+ return orderPredicate.test(a, b);
+ }
+ };
// note: We avoid object allocation by having a Slot and re-using the 'bottom'.
Slot bottom = null;
Slot scratchSlot = new Slot();
- boolean shardHasMoreBuckets = false; // This shard has more buckets than were returned
+ boolean shardHasMoreBuckets = false; // This shard has more buckets than were returned
for (int slotNum = 0; slotNum < numSlots; slotNum++) {
// screen out buckets not matching mincount
if (effectiveMincount > 0) {
long count = countAcc.getCount(slotNum);
- if (count < effectiveMincount) {
+ if (count < effectiveMincount) {
if (count > 0)
- numBuckets++; // Still increment numBuckets as long as we have some count. This is for consistency between distrib and non-distrib mode.
+ numBuckets++; // Still increment numBuckets as long as we have some count. This is for
+ // consistency between distrib and non-distrib mode.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/facet/FacetFieldProcessor.java
##########
@@ -388,7 +428,8 @@ private static long applyDefaultOverrequest(long offset, long limit) {
if (bottom != null) {
shardHasMoreBuckets = true;
- scratchSlot.slot = slotNum; // scratchSlot is only used to hold this slotNum for the following line
+ scratchSlot.slot =
+ slotNum; // scratchSlot is only used to hold this slotNum for the following line
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/facet/FacetFieldProcessor.java
##########
@@ -44,62 +45,72 @@
import org.apache.solr.search.facet.SlotAcc.SweepableSlotAcc;
import org.apache.solr.search.facet.SlotAcc.SweepingCountSlotAcc;
-import static org.apache.solr.search.facet.FacetContext.SKIP_FACET;
-
/**
* Facet processing based on field values. (not range nor by query)
+ *
* @see FacetField
*/
abstract class FacetFieldProcessor extends FacetProcessor<FacetField> {
SchemaField sf;
SlotAcc indexOrderAcc;
int effectiveMincount;
final boolean singlePassSlotAccCollection;
- final FacetRequest.FacetSort sort; // never null (may be the user's requested sort, or the prelim_sort)
+ final FacetRequest.FacetSort
+ sort; // never null (may be the user's requested sort, or the prelim_sort)
final FacetRequest.FacetSort resort; // typically null (unless the user specified a prelim_sort)
-
- final Map<String,AggValueSource> deferredAggs = new HashMap<String,AggValueSource>();
+
+ final Map<String, AggValueSource> deferredAggs = new HashMap<String, AggValueSource>();
// TODO: push any of this down to base class?
//
// For sort="x desc", collectAcc would point to "x", and sortAcc would also point to "x".
- // collectAcc would be used to accumulate all buckets, and sortAcc would be used to sort those buckets.
+ // collectAcc would be used to accumulate all buckets, and sortAcc would be used to sort those
+ // buckets.
//
- SlotAcc collectAcc; // Accumulator to collect across entire domain (in addition to the countAcc). May be null.
- SlotAcc sortAcc; // Accumulator to use for sorting *only* (i.e. not used for collection). May be an alias of countAcc, collectAcc, or indexOrderAcc
+ SlotAcc
+ collectAcc; // Accumulator to collect across entire domain (in addition to the countAcc). May
+ // be null.
+ SlotAcc
+ sortAcc; // Accumulator to use for sorting *only* (i.e. not used for collection). May be an
+ // alias of countAcc, collectAcc, or indexOrderAcc
SlotAcc[] otherAccs; // Accumulators that do not need to be calculated across all buckets.
- SpecialSlotAcc allBucketsAcc; // this can internally refer to otherAccs and/or collectAcc. setNextReader should be called on otherAccs directly if they exist.
+ SpecialSlotAcc
+ allBucketsAcc; // this can internally refer to otherAccs and/or collectAcc. setNextReader
+ // should be called on otherAccs directly if they exist.
FacetFieldProcessor(FacetContext fcontext, FacetField freq, SchemaField sf) {
super(fcontext, freq);
this.sf = sf;
- this.effectiveMincount = (int)(fcontext.isShard() ? Math.min(1 , freq.mincount) : freq.mincount);
+ this.effectiveMincount =
+ (int) (fcontext.isShard() ? Math.min(1, freq.mincount) : freq.mincount);
this.singlePassSlotAccCollection = (freq.limit == -1 && freq.subFacets.size() == 0);
- if ( null == freq.prelim_sort ) {
+ if (null == freq.prelim_sort) {
// If the user has not specified any preliminary sort, then things are very simple.
// Just use the "sort" as is w/o needing any re-sorting
this.sort = freq.sort;
this.resort = null;
} else {
assert null != freq.prelim_sort;
-
- if ( fcontext.isShard() ) {
- // for a shard request, we can ignore the users requested "sort" and focus solely on the prelim_sort
+
+ if (fcontext.isShard()) {
+ // for a shard request, we can ignore the users requested "sort" and focus solely on the
+ // prelim_sort
// the merger will worry about the final sorting -- we don't need to resort anything...
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/facet/FacetField.java
##########
@@ -89,48 +98,57 @@ public static FacetMethod fromString(String method) {
// ensure we can support the requested options for numeric faceting:
if (ntype != null) {
if (prefix != null) {
- throw new SolrException(SolrException.ErrorCode.BAD_REQUEST,
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
"Doesn't make sense to set facet prefix on a numeric field");
}
if (mincount == 0) {
- throw new SolrException(SolrException.ErrorCode.BAD_REQUEST,
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
"Numeric fields do not support facet mincount=0; try indexing as terms");
// TODO if indexed=true then we could add support
}
}
- // TODO auto-pick ENUM/STREAM SOLR-9351 when index asc and DocSet cardinality is *not* much smaller than term cardinality
- if (method == FacetMethod.ENUM) {// at the moment these two are the same
+ // TODO auto-pick ENUM/STREAM SOLR-9351 when index asc and DocSet cardinality is *not* much
+ // smaller than term cardinality
+ if (method == FacetMethod.ENUM) { // at the moment these two are the same
method = FacetMethod.STREAM;
}
- if (method == FacetMethod.STREAM && sf.indexed() && !ft.isPointField() &&
+ if (method == FacetMethod.STREAM
+ && sf.indexed()
+ && !ft.isPointField()
+ &&
// streaming doesn't support allBuckets, numBuckets or missing
// so, don't use stream processor if anyone of them is enabled
- !(allBuckets || numBuckets || missing) &&
+ !(allBuckets || numBuckets || missing)
+ &&
// whether we can use stream processing depends on whether this is a shard request, whether
// re-sorting has been requested, and if the effective sort during collection is "index asc"
- ( fcontext.isShard()
- // for a shard request, the effective per-shard sort must be index asc
- ? FacetSort.INDEX_ASC.equals(null == prelim_sort ? sort : prelim_sort)
- // for a non-shard request, we can only use streaming if there is no pre-sorting
- : (null == prelim_sort && FacetSort.INDEX_ASC.equals( sort ) ) ) ) {
-
+ (fcontext.isShard()
+ // for a shard request, the effective per-shard sort must be index asc
+ ? FacetSort.INDEX_ASC.equals(null == prelim_sort ? sort : prelim_sort)
+ // for a non-shard request, we can only use streaming if there is no pre-sorting
+ : (null == prelim_sort && FacetSort.INDEX_ASC.equals(sort)))) {
+
return new FacetFieldProcessorByEnumTermsStream(fcontext, this, sf);
}
- // TODO if method=UIF and not single-valued numerics then simply choose that now? TODO add FieldType.getDocValuesType()
+ // TODO if method=UIF and not single-valued numerics then simply choose that now? TODO add
+ // FieldType.getDocValuesType()
if (!multiToken) {
if (mincount > 0 && prefix == null && (ntype != null || method == FacetMethod.DVHASH)) {
- // TODO can we auto-pick for strings when term cardinality is much greater than DocSet cardinality?
+ // TODO can we auto-pick for strings when term cardinality is much greater than DocSet
+ // cardinality?
// or if we don't know cardinality but DocSet size is very small
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/facet/FacetFieldProcessor.java
##########
@@ -44,62 +45,72 @@
import org.apache.solr.search.facet.SlotAcc.SweepableSlotAcc;
import org.apache.solr.search.facet.SlotAcc.SweepingCountSlotAcc;
-import static org.apache.solr.search.facet.FacetContext.SKIP_FACET;
-
/**
* Facet processing based on field values. (not range nor by query)
+ *
* @see FacetField
*/
abstract class FacetFieldProcessor extends FacetProcessor<FacetField> {
SchemaField sf;
SlotAcc indexOrderAcc;
int effectiveMincount;
final boolean singlePassSlotAccCollection;
- final FacetRequest.FacetSort sort; // never null (may be the user's requested sort, or the prelim_sort)
+ final FacetRequest.FacetSort
+ sort; // never null (may be the user's requested sort, or the prelim_sort)
final FacetRequest.FacetSort resort; // typically null (unless the user specified a prelim_sort)
-
- final Map<String,AggValueSource> deferredAggs = new HashMap<String,AggValueSource>();
+
+ final Map<String, AggValueSource> deferredAggs = new HashMap<String, AggValueSource>();
// TODO: push any of this down to base class?
//
// For sort="x desc", collectAcc would point to "x", and sortAcc would also point to "x".
- // collectAcc would be used to accumulate all buckets, and sortAcc would be used to sort those buckets.
+ // collectAcc would be used to accumulate all buckets, and sortAcc would be used to sort those
+ // buckets.
//
- SlotAcc collectAcc; // Accumulator to collect across entire domain (in addition to the countAcc). May be null.
- SlotAcc sortAcc; // Accumulator to use for sorting *only* (i.e. not used for collection). May be an alias of countAcc, collectAcc, or indexOrderAcc
+ SlotAcc
+ collectAcc; // Accumulator to collect across entire domain (in addition to the countAcc). May
+ // be null.
+ SlotAcc
+ sortAcc; // Accumulator to use for sorting *only* (i.e. not used for collection). May be an
+ // alias of countAcc, collectAcc, or indexOrderAcc
SlotAcc[] otherAccs; // Accumulators that do not need to be calculated across all buckets.
- SpecialSlotAcc allBucketsAcc; // this can internally refer to otherAccs and/or collectAcc. setNextReader should be called on otherAccs directly if they exist.
+ SpecialSlotAcc
+ allBucketsAcc; // this can internally refer to otherAccs and/or collectAcc. setNextReader
+ // should be called on otherAccs directly if they exist.
FacetFieldProcessor(FacetContext fcontext, FacetField freq, SchemaField sf) {
super(fcontext, freq);
this.sf = sf;
- this.effectiveMincount = (int)(fcontext.isShard() ? Math.min(1 , freq.mincount) : freq.mincount);
+ this.effectiveMincount =
+ (int) (fcontext.isShard() ? Math.min(1, freq.mincount) : freq.mincount);
this.singlePassSlotAccCollection = (freq.limit == -1 && freq.subFacets.size() == 0);
- if ( null == freq.prelim_sort ) {
+ if (null == freq.prelim_sort) {
// If the user has not specified any preliminary sort, then things are very simple.
// Just use the "sort" as is w/o needing any re-sorting
this.sort = freq.sort;
this.resort = null;
} else {
assert null != freq.prelim_sort;
-
- if ( fcontext.isShard() ) {
- // for a shard request, we can ignore the users requested "sort" and focus solely on the prelim_sort
+
+ if (fcontext.isShard()) {
+ // for a shard request, we can ignore the users requested "sort" and focus solely on the
+ // prelim_sort
// the merger will worry about the final sorting -- we don't need to resort anything...
this.sort = freq.prelim_sort;
this.resort = null;
-
+
} else { // non shard...
- if ( singlePassSlotAccCollection ) { // special case situation...
+ if (singlePassSlotAccCollection) { // special case situation...
// when we can do a single pass SlotAcc collection on non-shard request, there is
// no point re-sorting. Ignore the freq.prelim_sort and use the freq.sort option as is...
this.sort = freq.sort;
this.resort = null;
} else {
- // for a non-shard request, we will use the prelim_sort as our initial sort option if it exists
+ // for a non-shard request, we will use the prelim_sort as our initial sort option if it
+ // exists
// then later we will re-sort on the final desired sort...
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/facet/FacetFieldProcessor.java
##########
@@ -44,62 +45,72 @@
import org.apache.solr.search.facet.SlotAcc.SweepableSlotAcc;
import org.apache.solr.search.facet.SlotAcc.SweepingCountSlotAcc;
-import static org.apache.solr.search.facet.FacetContext.SKIP_FACET;
-
/**
* Facet processing based on field values. (not range nor by query)
+ *
* @see FacetField
*/
abstract class FacetFieldProcessor extends FacetProcessor<FacetField> {
SchemaField sf;
SlotAcc indexOrderAcc;
int effectiveMincount;
final boolean singlePassSlotAccCollection;
- final FacetRequest.FacetSort sort; // never null (may be the user's requested sort, or the prelim_sort)
+ final FacetRequest.FacetSort
+ sort; // never null (may be the user's requested sort, or the prelim_sort)
final FacetRequest.FacetSort resort; // typically null (unless the user specified a prelim_sort)
-
- final Map<String,AggValueSource> deferredAggs = new HashMap<String,AggValueSource>();
+
+ final Map<String, AggValueSource> deferredAggs = new HashMap<String, AggValueSource>();
// TODO: push any of this down to base class?
//
// For sort="x desc", collectAcc would point to "x", and sortAcc would also point to "x".
- // collectAcc would be used to accumulate all buckets, and sortAcc would be used to sort those buckets.
+ // collectAcc would be used to accumulate all buckets, and sortAcc would be used to sort those
+ // buckets.
//
- SlotAcc collectAcc; // Accumulator to collect across entire domain (in addition to the countAcc). May be null.
- SlotAcc sortAcc; // Accumulator to use for sorting *only* (i.e. not used for collection). May be an alias of countAcc, collectAcc, or indexOrderAcc
+ SlotAcc
+ collectAcc; // Accumulator to collect across entire domain (in addition to the countAcc). May
+ // be null.
+ SlotAcc
+ sortAcc; // Accumulator to use for sorting *only* (i.e. not used for collection). May be an
+ // alias of countAcc, collectAcc, or indexOrderAcc
SlotAcc[] otherAccs; // Accumulators that do not need to be calculated across all buckets.
- SpecialSlotAcc allBucketsAcc; // this can internally refer to otherAccs and/or collectAcc. setNextReader should be called on otherAccs directly if they exist.
+ SpecialSlotAcc
+ allBucketsAcc; // this can internally refer to otherAccs and/or collectAcc. setNextReader
+ // should be called on otherAccs directly if they exist.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/facet/FacetFieldProcessor.java
##########
@@ -937,67 +1026,85 @@ public void resize(Resizer resizer) {
}
}
-
/*
- "qfacet":{"cat2":{"_l":["A"]}},
- "all":{"_s":[[
- "all",
- {"cat3":{"_l":["A"]}}]]},
- "cat1":{"_l":["A"]}}}
- */
+ "qfacet":{"cat2":{"_l":["A"]}},
+ "all":{"_s":[[
+ "all",
+ {"cat3":{"_l":["A"]}}]]},
+ "cat1":{"_l":["A"]}}}
+ */
@SuppressWarnings({"unchecked"})
static <T> List<T> asList(Object list) {
- return list != null ? (List<T>)list : Collections.emptyList();
+ return list != null ? (List<T>) list : Collections.emptyList();
}
@SuppressWarnings({"rawtypes", "unchecked"})
protected SimpleOrderedMap<Object> refineFacets() throws IOException {
boolean skipThisFacet = (fcontext.flags & SKIP_FACET) != 0;
-
- List leaves = asList(fcontext.facetInfo.get("_l")); // We have not seen this bucket: do full faceting for this bucket, including all sub-facets
- List<List> skip = asList(fcontext.facetInfo.get("_s")); // We have seen this bucket, so skip stats on it, and skip sub-facets except for the specified sub-facets that should calculate specified buckets.
- List<List> partial = asList(fcontext.facetInfo.get("_p")); // We have not seen this bucket, do full faceting for this bucket, and most sub-facets... but some sub-facets are partial and should only visit specified buckets.
-
- // For leaf refinements, we do full faceting for each leaf bucket. Any sub-facets of these buckets will be fully evaluated. Because of this, we should never
+ List leaves =
+ asList(
+ fcontext.facetInfo.get(
+ "_l")); // We have not seen this bucket: do full faceting for this bucket, including
+ // all sub-facets
+ List<List> skip =
+ asList(
+ fcontext.facetInfo.get(
+ "_s")); // We have seen this bucket, so skip stats on it, and skip sub-facets except
+ // for the specified sub-facets that should calculate specified buckets.
+ List<List> partial =
+ asList(
+ fcontext.facetInfo.get(
+ "_p")); // We have not seen this bucket, do full faceting for this bucket, and most
+ // sub-facets... but some sub-facets are partial and should only visit
+ // specified buckets.
+
+ // For leaf refinements, we do full faceting for each leaf bucket. Any sub-facets of these
+ // buckets will be fully evaluated. Because of this, we should never
// encounter leaf refinements that have sub-facets that return partial results.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/facet/FacetFieldProcessorByArrayDV.java
##########
@@ -93,37 +92,48 @@ protected void findStartAndEndOrds() throws IOException {
protected void collectDocs() throws IOException {
int domainSize = fcontext.base.size();
- if (nTerms <= 0 || domainSize < effectiveMincount) { // TODO: what about allBuckets? missing bucket?
+ if (nTerms <= 0
+ || domainSize < effectiveMincount) { // TODO: what about allBuckets? missing bucket?
return;
}
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/facet/FacetFieldProcessor.java
##########
@@ -1014,14 +1121,20 @@ public void resize(Resizer resizer) {
return res;
}
- private SimpleOrderedMap<Object> refineBucket(Object bucketVal, boolean skip, Map<String,Object> facetInfo) throws IOException {
+ private SimpleOrderedMap<Object> refineBucket(
+ Object bucketVal, boolean skip, Map<String, Object> facetInfo) throws IOException {
SimpleOrderedMap<Object> bucket = new SimpleOrderedMap<>();
FieldType ft = sf.getType();
- bucketVal = ft.toNativeType(bucketVal); // refinement info passed in as JSON will cause int->long and float->double
+ bucketVal =
+ ft.toNativeType(
+ bucketVal); // refinement info passed in as JSON will cause int->long and float->double
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/facet/FacetFieldProcessorByEnumTermsStream.java
##########
@@ -89,59 +89,67 @@ public void close() throws IOException {
public void process() throws IOException {
super.process();
- // We need to keep the fcontext open after processing is done (since we will be streaming in the response writer).
+ // We need to keep the fcontext open after processing is done (since we will be streaming in the
+ // response writer).
// But if the connection is broken, we want to clean up.
// fcontext.base.incref(); // OFF-HEAP
fcontext.qcontext.addCloseHook(this);
setup();
response = new SimpleOrderedMap<>();
- response.add("buckets", new Iterator<>() {
- boolean retrieveNext = true;
- Object val;
-
- @Override
- public boolean hasNext() {
- if (retrieveNext) {
- val = nextBucket();
- }
- retrieveNext = false;
- return val != null;
- }
+ response.add(
+ "buckets",
+ new Iterator<>() {
+ boolean retrieveNext = true;
+ Object val;
+
+ @Override
+ public boolean hasNext() {
+ if (retrieveNext) {
+ val = nextBucket();
+ }
+ retrieveNext = false;
+ return val != null;
+ }
- @Override
- public Object next() {
- if (retrieveNext) {
- val = nextBucket();
- }
- retrieveNext = true;
- if (val == null) {
- // Last value, so clean up. In the case that we are doing streaming facets within streaming facets,
- // the number of close hooks could grow very large, so we want to remove ourselves.
- boolean removed = fcontext.qcontext.removeCloseHook(FacetFieldProcessorByEnumTermsStream.this);
- assert removed;
- try {
- close();
- } catch (IOException e) {
- throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "Error during facet streaming close", e);
+ @Override
+ public Object next() {
+ if (retrieveNext) {
+ val = nextBucket();
+ }
+ retrieveNext = true;
+ if (val == null) {
+ // Last value, so clean up. In the case that we are doing streaming facets within
+ // streaming facets,
+ // the number of close hooks could grow very large, so we want to remove ourselves.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/facet/FacetFieldProcessorByHashDV.java
##########
@@ -40,24 +39,27 @@
import org.apache.solr.search.facet.SlotAcc.SlotContext;
/**
- * Facets numbers into a hash table. The number is either a raw numeric DocValues value, or
- * a term global ordinal integer.
- * Limitations:
+ * Facets numbers into a hash table. The number is either a raw numeric DocValues value, or a term
+ * global ordinal integer. Limitations:
+ *
* <ul>
- * <li>doesn't handle prefix, but could easily be added</li>
- * <li>doesn't handle mincount==0 -- you're better off with an array alg</li>
+ * <li>doesn't handle prefix, but could easily be added
+ * <li>doesn't handle mincount==0 -- you're better off with an array alg
* </ul>
*/
class FacetFieldProcessorByHashDV extends FacetFieldProcessor {
- static int MAXIMUM_STARTING_TABLE_SIZE=1024; // must be a power of two, non-final to support setting by tests
+ static int MAXIMUM_STARTING_TABLE_SIZE =
+ 1024; // must be a power of two, non-final to support setting by tests
/** a hash table with long keys (what we're counting) and integer values (counts) */
private static class LongCounts {
static final float LOAD_FACTOR = 0.7f;
long[] vals;
- long[] counts; // maintain the counts here since we need them to tell if there was actually a value anyway
+ long[]
+ counts; // maintain the counts here since we need them to tell if there was actually a value
+ // anyway
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/facet/FacetHeatmap.java
##########
@@ -300,8 +343,10 @@ public FacetMerger createFacetMerger(Object prototype) {
NamedList<Object> mergedResult; // except counts, which we add in when done
int[] counts;
- // note: there appears to be no mechanism to modify the shard requests in this API. If we could, we'd
- // change the format to png. Instead, we have the facet processor recognize it's a shard request and ignore
+ // note: there appears to be no mechanism to modify the shard requests in this API. If we
+ // could, we'd
+ // change the format to png. Instead, we have the facet processor recognize it's a shard
+ // request and ignore
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/facet/FacetFieldProcessorByArrayDV.java
##########
@@ -93,37 +92,48 @@ protected void findStartAndEndOrds() throws IOException {
protected void collectDocs() throws IOException {
int domainSize = fcontext.base.size();
- if (nTerms <= 0 || domainSize < effectiveMincount) { // TODO: what about allBuckets? missing bucket?
+ if (nTerms <= 0
+ || domainSize < effectiveMincount) { // TODO: what about allBuckets? missing bucket?
return;
}
final SweepCountAccStruct base = SweepingCountSlotAcc.baseStructOf(this);
final List<SweepCountAccStruct> others = SweepingCountSlotAcc.otherStructsOf(this);
assert null != base;
-
+
// TODO: refactor some of this logic into a base class
- boolean countOnly = collectAcc==null && allBucketsAcc==null;
+ boolean countOnly = collectAcc == null && allBucketsAcc == null;
boolean fullRange = startTermIndex == 0 && endTermIndex == si.getValueCount();
// Are we expecting many hits per bucket?
// FUTURE: pro-rate for nTerms?
- // FUTURE: better take into account number of values in multi-valued fields. This info is available for indexed fields.
+ // FUTURE: better take into account number of values in multi-valued fields. This info is
+ // available for indexed fields.
// FUTURE: take into account that bigger ord maps are more expensive than smaller ones
- // One test: 5M doc index, faceting on a single-valued field with almost 1M unique values, crossover point where global counting was slower
- // than per-segment counting was a domain of 658k docs. At that point, top 10 buckets had 6-7 matches each.
+ // One test: 5M doc index, faceting on a single-valued field with almost 1M unique values,
+ // crossover point where global counting was slower
+ // than per-segment counting was a domain of 658k docs. At that point, top 10 buckets had 6-7
+ // matches each.
// this was for heap docvalues produced by UninvertingReader
- // Since these values were randomly distributed, lets round our domain multiplier up to account for less random real world data.
+ // Since these values were randomly distributed, lets round our domain multiplier up to account
+ // for less random real world data.
long domainMultiplier = multiValuedField ? 4L : 2L;
- boolean manyHitsPerBucket = domainSize * domainMultiplier > (si.getValueCount() + 3); // +3 to increase test coverage with small tests
-
- // If we're only calculating counts, we're not prefixing, and we expect to collect many documents per unique value,
- // then collect per-segment before mapping to global ords at the end. This will save redundant seg->global ord mappings.
- // FUTURE: there are probably some other non "countOnly" cases where we can use this as well (i.e. those where
+ boolean manyHitsPerBucket =
+ domainSize * domainMultiplier
+ > (si.getValueCount() + 3); // +3 to increase test coverage with small tests
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/facet/FacetFieldProcessorByHashDV.java
##########
@@ -111,13 +114,14 @@ int add(long val) {
protected void rehash() {
long[] oldVals = vals;
- long[] oldCounts = counts; // after retrieving the count, this array is reused as a mapping to new array
+ long[] oldCounts =
+ counts; // after retrieving the count, this array is reused as a mapping to new array
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/facet/FacetHeatmap.java
##########
@@ -279,16 +319,19 @@ private static Object formatCountsVal(String format, int columns, int rows, int[
}
switch (format) {
- case FORMAT_INTS2D: //A List of List of Integers. Good for small heatmaps and ease of consumption
+ case FORMAT_INTS2D: // A List of List of Integers. Good for small heatmaps and ease of
+ // consumption
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/facet/FacetFieldProcessorByHashDV.java
##########
@@ -81,7 +83,8 @@ private int hash(long val) {
// Many values will only have significant bits just to the right of that,
// and the leftmost bits will all be zero.
- // For now, lets just settle to get first 8 significant mantissa bits of double or float in the lowest bits of our hash
+ // For now, lets just settle to get first 8 significant mantissa bits of double or float in
+ // the lowest bits of our hash
// The upper bits of our hash will be irrelevant.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/facet/FacetHeatmap.java
##########
@@ -352,14 +403,15 @@ public Object getMergedResult() {
@VisibleForTesting
static List<List<Integer>> asInts2D(final int columns, final int rows, final int[] counts) {
- //Returns a view versus returning a copy. This saves memory.
- //The data is oriented naturally for human/developer viewing: one row at a time top-down
+ // Returns a view versus returning a copy. This saves memory.
+ // The data is oriented naturally for human/developer viewing: one row at a time top-down
return new AbstractList<List<Integer>>() {
@Override
- public List<Integer> get(final int rowIdx) {//top-down remember; the heatmap.counts is bottom up
- //check if all zeroes and return null if so
+ public List<Integer> get(
+ final int rowIdx) { // top-down remember; the heatmap.counts is bottom up
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/facet/FacetFieldProcessorByArray.java
##########
@@ -118,12 +117,15 @@ public void process() throws IOException {
if (refineResult != null) {
if (freq.allBuckets) {
// count is irrelevant, but hardcoded in collect(...), so intercept/mask normal counts.
- // Set here to prevent createAccs(...) from creating a 1-slot countAcc that will fail with AIOOBE
- // NOTE: because collectAcc will be null, it is fine/irrelevant to set a countAcc that doesn't support sweeping
+ // Set here to prevent createAccs(...) from creating a 1-slot countAcc that will fail with
+ // AIOOBE
+ // NOTE: because collectAcc will be null, it is fine/irrelevant to set a countAcc that
+ // doesn't support sweeping
countAcc = SlotAcc.DEV_NULL_SLOT_ACC;
createAccs(nDocs, 1);
assert collectAcc == null;
- otherAccs = accs; // accs is created above and set on allBucketsAcc; but during collection, setNextReader is called on otherAccs.
+ otherAccs = accs; // accs is created above and set on allBucketsAcc; but during collection,
+ // setNextReader is called on otherAccs.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/facet/FacetFieldProcessorByEnumTermsStream.java
##########
@@ -89,59 +89,67 @@ public void close() throws IOException {
public void process() throws IOException {
super.process();
- // We need to keep the fcontext open after processing is done (since we will be streaming in the response writer).
+ // We need to keep the fcontext open after processing is done (since we will be streaming in the
+ // response writer).
// But if the connection is broken, we want to clean up.
// fcontext.base.incref(); // OFF-HEAP
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/facet/FacetFieldProcessorByEnumTermsStream.java
##########
@@ -150,10 +158,14 @@ private void setup() throws IOException {
// other negative values are not supported
throw new IllegalArgumentException("Illegal `overrequest` specified: " + freq.overrequest);
} else if (freq.overrequest > 0 && (fcontext.isShard() || null != resort)) {
- // NOTE: "index sort" _never_ applies a default overrequest. In both the shard case and the resort case, the
- // default overrequest is `0`. However, if `overrequest` is explicitly specified, we respect it except for
- // non-distrib, no-resort request. Overrequest is relevant for the `resort` case; but it can also be relevant
- // in some edge cases of the "shard" case, where it can affect the behavior of `isBucketComplete()`
+ // NOTE: "index sort" _never_ applies a default overrequest. In both the shard case and the
+ // resort case, the
+ // default overrequest is `0`. However, if `overrequest` is explicitly specified, we respect
+ // it except for
+ // non-distrib, no-resort request. Overrequest is relevant for the `resort` case; but it can
+ // also be relevant
+ // in some edge cases of the "shard" case, where it can affect the behavior of
+ // `isBucketComplete()`
// (see SOLR-14595).
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/facet/FacetProcessor.java
##########
@@ -455,46 +490,52 @@ void processSubs(SimpleOrderedMap<Object> response, Query filter, DocSet domain,
continue;
}
- Map<String,Object>facetInfoSub = null;
+ Map<String, Object> facetInfoSub = null;
if (facetInfo != null) {
- facetInfoSub = (Map<String,Object>)facetInfo.get(sub.getKey());
+ facetInfoSub = (Map<String, Object>) facetInfo.get(sub.getKey());
}
- // If we're skipping this node, then we only need to process sub-facets that have facet info specified.
+ // If we're skipping this node, then we only need to process sub-facets that have facet info
+ // specified.
if (skip && facetInfoSub == null) continue;
// make a new context for each sub-facet since they can change the domain
FacetContext subContext = fcontext.sub(filter, domain);
subContext.facetInfo = facetInfoSub;
- if (!skip) subContext.flags &= ~FacetContext.SKIP_FACET; // turn off the skip flag if we're not skipping this bucket
+ if (!skip)
+ subContext.flags &=
+ ~FacetContext.SKIP_FACET; // turn off the skip flag if we're not skipping this bucket
- if (fcontext.getDebugInfo() != null) { // if fcontext.debugInfo != null, it means rb.debug() == true
+ if (fcontext.getDebugInfo()
+ != null) { // if fcontext.debugInfo != null, it means rb.debug() == true
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/facet/FacetModule.java
##########
@@ -263,12 +275,13 @@ public void modifyRequest(ResponseBuilder rb, SearchComponent who, ShardRequest
if ((sreq.purpose & ShardRequest.PURPOSE_GET_TOP_IDS) != 0) {
sreq.purpose |= FacetModule.PURPOSE_GET_JSON_FACETS;
- sreq.params.set(FACET_INFO, "{}"); // The presence of FACET_INFO (_facet_) turns on json faceting
+ sreq.params.set(
+ FACET_INFO, "{}"); // The presence of FACET_INFO (_facet_) turns on json faceting
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/facet/FacetHeatmap.java
##########
@@ -109,46 +120,58 @@ public FacetHeatmap parse(Object argsObj) {
Map<String, Object> argsMap = (Map<String, Object>) argsObj;
String fieldName = getField(argsMap);
- //get the strategy from the field type
+ // get the strategy from the field type
final SchemaField schemaField = getSchema().getField(fieldName);
final FieldType type = schemaField.getType();
final PrefixTreeStrategy strategy;
final DistanceUnits distanceUnits;
- // note: the two instanceof conditions is not ideal, versus one. If we start needing to add more then refactor.
+ // note: the two instanceof conditions is not ideal, versus one. If we start needing to add
+ // more then refactor.
if ((type instanceof AbstractSpatialPrefixTreeFieldType)) {
- AbstractSpatialPrefixTreeFieldType<?> rptType = (AbstractSpatialPrefixTreeFieldType<?>) type;
+ AbstractSpatialPrefixTreeFieldType<?> rptType =
+ (AbstractSpatialPrefixTreeFieldType<?>) type;
strategy = rptType.getStrategy(fieldName);
distanceUnits = rptType.getDistanceUnits();
} else if (type instanceof RptWithGeometrySpatialField) {
- RptWithGeometrySpatialField rptSdvType = (RptWithGeometrySpatialField) type;
+ RptWithGeometrySpatialField rptSdvType = (RptWithGeometrySpatialField) type;
strategy = rptSdvType.getStrategy(fieldName).getIndexStrategy();
distanceUnits = rptSdvType.getDistanceUnits();
} else {
- //FYI we support the term query one too but few people use that one
- throw err("heatmap field needs to be of type " + SpatialRecursivePrefixTreeFieldType.class + " or " + RptWithGeometrySpatialField.class);
+ // FYI we support the term query one too but few people use that one
+ throw err(
+ "heatmap field needs to be of type "
+ + SpatialRecursivePrefixTreeFieldType.class
+ + " or "
+ + RptWithGeometrySpatialField.class);
}
final SpatialContext ctx = strategy.getSpatialContext();
- //get the bbox (query Rectangle)
+ // get the bbox (query Rectangle)
String geomStr = getString(argsMap, GEOM_PARAM, null);
- final Shape boundsShape = geomStr == null ? ctx.getWorldBounds() : SpatialUtils.parseGeomSolrException(geomStr, ctx);
+ final Shape boundsShape =
+ geomStr == null
+ ? ctx.getWorldBounds()
+ : SpatialUtils.parseGeomSolrException(geomStr, ctx);
- //get the grid level (possibly indirectly via distErr or distErrPct)
+ // get the grid level (possibly indirectly via distErr or distErrPct)
final int gridLevel;
final Long gridLevelObj = getLongOrNull(argsMap, LEVEL_PARAM, false);
final int maxGridLevel = strategy.getGrid().getMaxLevels();
if (gridLevelObj != null) {
gridLevel = gridLevelObj.intValue();
if (gridLevel <= 0 || gridLevel > maxGridLevel) {
- throw err(LEVEL_PARAM +" should be > 0 and <= " + maxGridLevel);
+ throw err(LEVEL_PARAM + " should be > 0 and <= " + maxGridLevel);
}
} else {
- //SpatialArgs has utility methods to resolve a 'distErr' from optionally set distErr & distErrPct. Arguably that
+ // SpatialArgs has utility methods to resolve a 'distErr' from optionally set distErr &
+ // distErrPct. Arguably that
// should be refactored to feel less weird than using it like this.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/facet/FacetHeatmap.java
##########
@@ -413,86 +470,93 @@ public int size() {
@VisibleForTesting
static class PngHelper {
- static final ImageReaderSpi imageReaderSpi;//thread-safe
+ static final ImageReaderSpi imageReaderSpi; // thread-safe
+
static {
final Iterator<ImageReader> imageReaders = ImageIO.getImageReadersByFormatName("png");
if (!imageReaders.hasNext()) {
- throw new SolrException(SolrException.ErrorCode.SERVER_ERROR, "Can't find png image reader, neaded for heatmaps!");
+ throw new SolrException(
+ SolrException.ErrorCode.SERVER_ERROR,
+ "Can't find png image reader, neaded for heatmaps!");
}
ImageReader imageReader = imageReaders.next();
imageReaderSpi = imageReader.getOriginatingProvider();
}
static BufferedImage readImage(final byte[] bytes) {
- // Wrap ImageInputStream around the bytes. We could use MemoryCacheImageInputStream but it will
+ // Wrap ImageInputStream around the bytes. We could use MemoryCacheImageInputStream but it
+ // will
// cache the data which is quite unnecessary given we have it all in-memory already.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/facet/FacetFieldProcessorByArrayDV.java
##########
@@ -93,37 +92,48 @@ protected void findStartAndEndOrds() throws IOException {
protected void collectDocs() throws IOException {
int domainSize = fcontext.base.size();
- if (nTerms <= 0 || domainSize < effectiveMincount) { // TODO: what about allBuckets? missing bucket?
+ if (nTerms <= 0
+ || domainSize < effectiveMincount) { // TODO: what about allBuckets? missing bucket?
return;
}
final SweepCountAccStruct base = SweepingCountSlotAcc.baseStructOf(this);
final List<SweepCountAccStruct> others = SweepingCountSlotAcc.otherStructsOf(this);
assert null != base;
-
+
// TODO: refactor some of this logic into a base class
- boolean countOnly = collectAcc==null && allBucketsAcc==null;
+ boolean countOnly = collectAcc == null && allBucketsAcc == null;
boolean fullRange = startTermIndex == 0 && endTermIndex == si.getValueCount();
// Are we expecting many hits per bucket?
// FUTURE: pro-rate for nTerms?
- // FUTURE: better take into account number of values in multi-valued fields. This info is available for indexed fields.
+ // FUTURE: better take into account number of values in multi-valued fields. This info is
+ // available for indexed fields.
// FUTURE: take into account that bigger ord maps are more expensive than smaller ones
- // One test: 5M doc index, faceting on a single-valued field with almost 1M unique values, crossover point where global counting was slower
- // than per-segment counting was a domain of 658k docs. At that point, top 10 buckets had 6-7 matches each.
+ // One test: 5M doc index, faceting on a single-valued field with almost 1M unique values,
+ // crossover point where global counting was slower
+ // than per-segment counting was a domain of 658k docs. At that point, top 10 buckets had 6-7
+ // matches each.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/facet/FacetProcessor.java
##########
@@ -455,46 +490,52 @@ void processSubs(SimpleOrderedMap<Object> response, Query filter, DocSet domain,
continue;
}
- Map<String,Object>facetInfoSub = null;
+ Map<String, Object> facetInfoSub = null;
if (facetInfo != null) {
- facetInfoSub = (Map<String,Object>)facetInfo.get(sub.getKey());
+ facetInfoSub = (Map<String, Object>) facetInfo.get(sub.getKey());
}
- // If we're skipping this node, then we only need to process sub-facets that have facet info specified.
+ // If we're skipping this node, then we only need to process sub-facets that have facet info
+ // specified.
if (skip && facetInfoSub == null) continue;
// make a new context for each sub-facet since they can change the domain
FacetContext subContext = fcontext.sub(filter, domain);
subContext.facetInfo = facetInfoSub;
- if (!skip) subContext.flags &= ~FacetContext.SKIP_FACET; // turn off the skip flag if we're not skipping this bucket
+ if (!skip)
+ subContext.flags &=
+ ~FacetContext.SKIP_FACET; // turn off the skip flag if we're not skipping this bucket
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/facet/FacetRangeProcessor.java
##########
@@ -94,11 +98,13 @@
ranges = null;
}
- // Under the normal mincount=0, each shard will need to return 0 counts since we don't calculate buckets at the top level.
+ // Under the normal mincount=0, each shard will need to return 0 counts since we don't calculate
+ // buckets at the top level.
// If mincount>0 then we could *potentially* set our sub mincount to 1...
// ...but that would require sorting the buckets (by their val) at the top level
//
- // Rather then do that, which could be complicated by non trivial field types, we'll force the sub-shard effectiveMincount
+ // Rather then do that, which could be complicated by non trivial field types, we'll force the
+ // sub-shard effectiveMincount
// to be 0, ensuring that we can trivially merge all the buckets from every shard
// (we have to filter the merged buckets by the original mincount either way)
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/facet/FacetRangeProcessor.java
##########
@@ -885,46 +964,50 @@ protected Object parseGap(final String rawval) throws java.text.ParseException {
@Override
@SuppressWarnings({"rawtypes"})
- protected Comparable parseAndAddGap(Comparable value, String gap) throws java.text.ParseException{
+ protected Comparable parseAndAddGap(Comparable value, String gap)
+ throws java.text.ParseException {
if (value == null) {
throw new NullPointerException("Cannot perform range faceting on null CurrencyValue");
}
CurrencyValue val = (CurrencyValue) value;
- CurrencyValue gapCurrencyValue =
- CurrencyValue.parse(gap, defaultCurrencyCode);
+ CurrencyValue gapCurrencyValue = CurrencyValue.parse(gap, defaultCurrencyCode);
long gapAmount =
- CurrencyValue.convertAmount(this.exchangeRateProvider,
+ CurrencyValue.convertAmount(
+ this.exchangeRateProvider,
gapCurrencyValue.getCurrencyCode(),
gapCurrencyValue.getAmount(),
val.getCurrencyCode());
- return new CurrencyValue(val.getAmount() + gapAmount,
- val.getCurrencyCode());
-
+ return new CurrencyValue(val.getAmount() + gapAmount, val.getCurrencyCode());
}
-
}
@SuppressWarnings({"unchecked", "rawtypes"})
protected SimpleOrderedMap<Object> refineFacets() throws IOException {
// this refineFacets method is patterned after FacetFieldProcessor.refineFacets such that
- // the same "_s" skip bucket syntax is used and FacetRangeMerger can subclass FacetRequestSortedMerger
+ // the same "_s" skip bucket syntax is used and FacetRangeMerger can subclass
+ // FacetRequestSortedMerger
// for dealing with them & the refinement requests.
//
// But range faceting does *NOT* use the "leaves" and "partial" syntax
//
- // If/When range facet becomes more like field facet in it's ability to sort and limit the "range buckets"
+ // If/When range facet becomes more like field facet in it's ability to sort and limit the
+ // "range buckets"
// FacetRangeProcessor and FacetFieldProcessor should probably be refactored to share more code.
boolean skipThisFacet = (fcontext.flags & SKIP_FACET) != 0;
- List<List> skip = FacetFieldProcessor.asList(fcontext.facetInfo.get("_s")); // We have seen this bucket, so skip stats on it, and skip sub-facets except for the specified sub-facets that should calculate specified buckets.
+ List<List> skip =
+ FacetFieldProcessor.asList(
+ fcontext.facetInfo.get(
+ "_s")); // We have seen this bucket, so skip stats on it, and skip sub-facets except
+ // for the specified sub-facets that should calculate specified buckets.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/facet/FacetModule.java
##########
@@ -153,17 +158,17 @@ public void process(ResponseBuilder rb) throws IOException {
rb.rsp.add("facets", results);
}
-
private void clearFaceting(List<ShardRequest> outgoing) {
// turn off faceting for requests not marked as being for faceting refinements
for (ShardRequest sreq : outgoing) {
if ((sreq.purpose & PURPOSE_REFINE_JSON_FACETS) != 0) continue;
- sreq.params.remove("json.facet"); // this just saves space... the presence of FACET_INFO is enough to control the faceting
+ sreq.params.remove(
+ "json.facet"); // this just saves space... the presence of FACET_INFO is enough to control
+ // the faceting
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/facet/FacetFieldProcessorByHashDV.java
##########
@@ -339,88 +348,109 @@ private void collectDocs() throws IOException {
// TODO support SortedSetDocValues
SortedDocValues globalDocValues = FieldUtil.getSortedDocValues(fcontext.qcontext, sf, null);
- ((TermOrdCalc)calc).lookupOrdFunction = ord -> {
- try {
- return globalDocValues.lookupOrd(ord);
- } catch (IOException e) {
- throw new RuntimeException(e);
- }
- };
-
- DocSetUtil.collectSortedDocSet(fcontext.base, fcontext.searcher.getIndexReader(), new SimpleCollector() {
- SortedDocValues docValues = globalDocValues; // this segment/leaf. NN
- LongValues toGlobal = LongValues.IDENTITY; // this segment to global ordinal. NN
-
- @Override public ScoreMode scoreMode() { return ScoreMode.COMPLETE_NO_SCORES; }
+ ((TermOrdCalc) calc).lookupOrdFunction =
+ ord -> {
+ try {
+ return globalDocValues.lookupOrd(ord);
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ };
+
+ DocSetUtil.collectSortedDocSet(
+ fcontext.base,
+ fcontext.searcher.getIndexReader(),
+ new SimpleCollector() {
+ SortedDocValues docValues = globalDocValues; // this segment/leaf. NN
+ LongValues toGlobal = LongValues.IDENTITY; // this segment to global ordinal. NN
+
+ @Override
+ public ScoreMode scoreMode() {
+ return ScoreMode.COMPLETE_NO_SCORES;
+ }
- @Override
- protected void doSetNextReader(LeafReaderContext ctx) throws IOException {
- setNextReaderFirstPhase(ctx);
- if (globalDocValues instanceof MultiDocValues.MultiSortedDocValues) {
- MultiDocValues.MultiSortedDocValues multiDocValues = (MultiDocValues.MultiSortedDocValues) globalDocValues;
- docValues = multiDocValues.values[ctx.ord];
- toGlobal = multiDocValues.mapping.getGlobalOrds(ctx.ord);
+ @Override
+ protected void doSetNextReader(LeafReaderContext ctx) throws IOException {
+ setNextReaderFirstPhase(ctx);
+ if (globalDocValues instanceof MultiDocValues.MultiSortedDocValues) {
+ MultiDocValues.MultiSortedDocValues multiDocValues =
+ (MultiDocValues.MultiSortedDocValues) globalDocValues;
+ docValues = multiDocValues.values[ctx.ord];
+ toGlobal = multiDocValues.mapping.getGlobalOrds(ctx.ord);
+ }
}
- }
- @Override
- public void collect(int segDoc) throws IOException {
- if (docValues.advanceExact(segDoc)) {
- long val = toGlobal.get(docValues.ordValue());
- collectValFirstPhase(segDoc, val);
+ @Override
+ public void collect(int segDoc) throws IOException {
+ if (docValues.advanceExact(segDoc)) {
+ long val = toGlobal.get(docValues.ordValue());
+ collectValFirstPhase(segDoc, val);
+ }
}
- }
- });
+ });
} else { // Numeric:
if (sf.multiValued()) {
- DocSetUtil.collectSortedDocSet(fcontext.base, fcontext.searcher.getIndexReader(), new SimpleCollector() {
- SortedNumericDocValues values = null; //NN
-
- @Override public ScoreMode scoreMode() { return ScoreMode.COMPLETE_NO_SCORES; }
+ DocSetUtil.collectSortedDocSet(
+ fcontext.base,
+ fcontext.searcher.getIndexReader(),
+ new SimpleCollector() {
+ SortedNumericDocValues values = null; // NN
+
+ @Override
+ public ScoreMode scoreMode() {
+ return ScoreMode.COMPLETE_NO_SCORES;
+ }
- @Override
- protected void doSetNextReader(LeafReaderContext ctx) throws IOException {
- setNextReaderFirstPhase(ctx);
- values = DocValues.getSortedNumeric(ctx.reader(), sf.getName());
- }
+ @Override
+ protected void doSetNextReader(LeafReaderContext ctx) throws IOException {
+ setNextReaderFirstPhase(ctx);
+ values = DocValues.getSortedNumeric(ctx.reader(), sf.getName());
+ }
- @Override
- public void collect(int segDoc) throws IOException {
- if (values.advanceExact(segDoc)) {
- long l = values.nextValue(); // This document must have at least one value
- collectValFirstPhase(segDoc, l);
- for (int i = 1, count = values.docValueCount(); i < count; i++) {
- long lnew = values.nextValue();
- if (lnew != l) { // Skip the value if it's equal to the last one, we don't want to double-count it
- collectValFirstPhase(segDoc, lnew);
+ @Override
+ public void collect(int segDoc) throws IOException {
+ if (values.advanceExact(segDoc)) {
+ long l = values.nextValue(); // This document must have at least one value
+ collectValFirstPhase(segDoc, l);
+ for (int i = 1, count = values.docValueCount(); i < count; i++) {
+ long lnew = values.nextValue();
+ if (lnew
+ != l) { // Skip the value if it's equal to the last one, we don't want to
+ // double-count it
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/facet/FacetRangeProcessor.java
##########
@@ -885,46 +964,50 @@ protected Object parseGap(final String rawval) throws java.text.ParseException {
@Override
@SuppressWarnings({"rawtypes"})
- protected Comparable parseAndAddGap(Comparable value, String gap) throws java.text.ParseException{
+ protected Comparable parseAndAddGap(Comparable value, String gap)
+ throws java.text.ParseException {
if (value == null) {
throw new NullPointerException("Cannot perform range faceting on null CurrencyValue");
}
CurrencyValue val = (CurrencyValue) value;
- CurrencyValue gapCurrencyValue =
- CurrencyValue.parse(gap, defaultCurrencyCode);
+ CurrencyValue gapCurrencyValue = CurrencyValue.parse(gap, defaultCurrencyCode);
long gapAmount =
- CurrencyValue.convertAmount(this.exchangeRateProvider,
+ CurrencyValue.convertAmount(
+ this.exchangeRateProvider,
gapCurrencyValue.getCurrencyCode(),
gapCurrencyValue.getAmount(),
val.getCurrencyCode());
- return new CurrencyValue(val.getAmount() + gapAmount,
- val.getCurrencyCode());
-
+ return new CurrencyValue(val.getAmount() + gapAmount, val.getCurrencyCode());
}
-
}
@SuppressWarnings({"unchecked", "rawtypes"})
protected SimpleOrderedMap<Object> refineFacets() throws IOException {
// this refineFacets method is patterned after FacetFieldProcessor.refineFacets such that
- // the same "_s" skip bucket syntax is used and FacetRangeMerger can subclass FacetRequestSortedMerger
+ // the same "_s" skip bucket syntax is used and FacetRangeMerger can subclass
+ // FacetRequestSortedMerger
// for dealing with them & the refinement requests.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/facet/FacetProcessor.java
##########
@@ -45,14 +44,15 @@
import org.apache.solr.search.facet.SlotAcc.SlotContext;
/** Base abstraction for a class that computes facets. This is fairly internal to the module. */
-public abstract class FacetProcessor<T extends FacetRequest> {
+public abstract class FacetProcessor<T extends FacetRequest> {
SimpleOrderedMap<Object> response;
FacetContext fcontext;
// TODO : I'm not sure this needs to be generic but come back to this later
T freq;
- DocSet filter; // additional filters specified by "filter" // TODO: do these need to be on the context to support recomputing during multi-select?
- LinkedHashMap<String,SlotAcc> accMap;
+ DocSet filter; // additional filters specified by "filter" // TODO: do these need to be on the
+ // context to support recomputing during multi-select?
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/facet/FacetRequestSortedMerger.java
##########
@@ -57,7 +58,8 @@ public void merge(Object facetResult, Context mcontext) {
private static class SortVal implements Comparable<SortVal> {
FacetBucket bucket;
- FacetModule.FacetSortableMerger merger; // make this class inner and access merger , direction in parent?
+ FacetModule.FacetSortableMerger
+ merger; // make this class inner and access merger , direction in parent?
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/facet/FacetRequest.java
##########
@@ -190,32 +206,42 @@ private JoinField(String from, String to, String method) {
* validates if a '<code>join</code>' is specified, and if so creates a <code>JoinField</code>
* and sets it on the <code>Domain</code>.
*
- * (params must not be null)
+ * <p>(params must not be null)
*/
- public static void createJoinField(FacetRequest.Domain domain, Map<String,Object> domainMap) {
+ public static void createJoinField(
+ FacetRequest.Domain domain, Map<String, Object> domainMap) {
assert null != domain;
assert null != domainMap;
final Object queryJoin = domainMap.get("join");
if (null != queryJoin) {
- // TODO: maybe allow simple string (instead of map) to mean "self join on this field name" ?
- if (! (queryJoin instanceof Map)) {
- throw new SolrException(SolrException.ErrorCode.BAD_REQUEST,
+ // TODO: maybe allow simple string (instead of map) to mean "self join on this field name"
+ // ?
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/facet/FacetRequest.java
##########
@@ -115,51 +122,59 @@ public int getMultiplier() {
// NONE is distinct from null since we may want to know if refinement was explicitly turned off.
public static FacetRequest.RefineMethod fromObj(Object method) {
if (method == null) return null;
- if (method instanceof Boolean) {
- return ((Boolean)method) ? SIMPLE : NONE;
+ if (method instanceof Boolean) {
+ return ((Boolean) method) ? SIMPLE : NONE;
}
if ("simple".equals(method)) {
return SIMPLE;
} else if ("none".equals(method)) {
return NONE;
} else {
- throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "Unknown RefineMethod method " + method);
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST, "Unknown RefineMethod method " + method);
}
}
}
-
- protected Map<String,AggValueSource> facetStats; // per-bucket statistics
- protected Map<String,FacetRequest> subFacets; // per-bucket sub-facets
+ protected Map<String, AggValueSource> facetStats; // per-bucket statistics
+ protected Map<String, FacetRequest> subFacets; // per-bucket sub-facets
protected boolean processEmpty;
protected Domain domain;
// domain changes
public static class Domain {
- /**
- * An explicit query domain, <em>ignoring all parent context</em>, expressed in JSON query format.
- * Mutually exclusive to {@link #excludeTags}
+ /**
+ * An explicit query domain, <em>ignoring all parent context</em>, expressed in JSON query
+ * format. Mutually exclusive to {@link #excludeTags}
*/
public List<Object> explicitQueries; // list of symbolic filters (JSON query format)
/**
- * Specifies query/filter tags that should be excluded to re-compute the domain from the parent context.
- * Mutually exclusive to {@link #explicitQueries}
+ * Specifies query/filter tags that should be excluded to re-compute the domain from the parent
+ * context. Mutually exclusive to {@link #explicitQueries}
*/
public List<String> excludeTags;
+
public JoinField joinField;
public GraphField graphField;
public boolean toParent;
public boolean toChildren;
- public String parents; // identifies the parent filter... the full set of parent documents for any block join operation
+ public String
+ parents; // identifies the parent filter... the full set of parent documents for any block
+ // join operation
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/facet/FacetRequestSortedMerger.java
##########
@@ -175,34 +162,51 @@ boolean isBucketComplete(FacetBucket bucket, Context mcontext) {
@Override
public Map<String, Object> getRefinement(Context mcontext) {
- // step 1) If this facet request has refining, then we need to fully request top buckets that were not seen by this shard.
- // step 2) If this facet does not have refining, but some sub-facets do, we need to check/recurse those sub-facets in *every* top bucket.
- // A combination of the two is possible and makes step 2 redundant for any buckets we fully requested in step 1.
+ // step 1) If this facet request has refining, then we need to fully request top buckets that
+ // were not seen by this shard.
+ // step 2) If this facet does not have refining, but some sub-facets do, we need to
+ // check/recurse those sub-facets in *every* top bucket.
+ // A combination of the two is possible and makes step 2 redundant for any buckets we fully
+ // requested in step 1.
- Map<String,Object> refinement = null;
+ Map<String, Object> refinement = null;
Collection<String> tags = mcontext.getSubsWithRefinement(freq);
if (tags.isEmpty() && !freq.doRefine()) {
// we don't have refining, and neither do our subs
return null;
}
- final FacetRequest.FacetSort initial_sort = null == freq.prelim_sort ? freq.sort : freq.prelim_sort;
-
+ final FacetRequest.FacetSort initial_sort =
+ null == freq.prelim_sort ? freq.sort : freq.prelim_sort;
+
// Tags for sub facets that have partial facets somewhere in their children.
- // If we are missing a bucket for this shard, we'll need to get the specific buckets that need refining.
+ // If we are missing a bucket for this shard, we'll need to get the specific buckets that need
+ // refining.
Collection<String> tagsWithPartial = mcontext.getSubsWithPartial(freq);
- boolean thisMissing = mcontext.bucketWasMissing(); // Was this whole facet missing (i.e. inside a bucket that was missing)?
- boolean shardHasMore = shardHasMoreBuckets != null && shardHasMoreBuckets.get(mcontext.shardNum); // shard indicated it has more buckets
- shardHasMore |= thisMissing; // if we didn't hear from the shard at all, assume it as more buckets
-
- // If we know we've seen all the buckets from a shard, then we don't have to add to leafBuckets or partialBuckets, only skipBuckets
- boolean isCommandPartial = freq.returnsPartial() || freq.processEmpty; // TODO: should returnsPartial() check processEmpty internally?
- boolean returnedAllBuckets = !shardHasMore && !freq.processEmpty; // did the shard return all of the possible buckets at this level? (pretend it didn't if processEmpty is set)
+ boolean thisMissing =
+ mcontext.bucketWasMissing(); // Was this whole facet missing (i.e. inside a bucket that was
+ // missing)?
+ boolean shardHasMore =
+ shardHasMoreBuckets != null
+ && shardHasMoreBuckets.get(mcontext.shardNum); // shard indicated it has more buckets
+ shardHasMore |=
+ thisMissing; // if we didn't hear from the shard at all, assume it as more buckets
+
+ // If we know we've seen all the buckets from a shard, then we don't have to add to leafBuckets
+ // or partialBuckets, only skipBuckets
+ boolean isCommandPartial =
+ freq.returnsPartial()
+ || freq.processEmpty; // TODO: should returnsPartial() check processEmpty internally?
+ boolean returnedAllBuckets =
+ !shardHasMore
+ && !freq.processEmpty; // did the shard return all of the possible buckets at this
+ // level? (pretend it didn't if processEmpty is set)
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/facet/FacetRangeProcessor.java
##########
@@ -534,9 +583,16 @@ private void rangeStats(Range range, int slot, boolean hasSubFacets) throws IOEx
DocSet intersection = fcontext.searcher.getDocSet(rangeQ, fcontext.base);
if (hasSubFacets) {
filters[slot] = rangeQ;
- intersections[slot] = intersection; // save for later // TODO: only save if number of slots is small enough?
- }
- long num = collect(intersection, slot, slotNum -> { return new SlotAcc.SlotContext(rangeQ); });
+ intersections[slot] =
+ intersection; // save for later // TODO: only save if number of slots is small enough?
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/facet/FacetRequestSortedMerger.java
##########
@@ -175,34 +162,51 @@ boolean isBucketComplete(FacetBucket bucket, Context mcontext) {
@Override
public Map<String, Object> getRefinement(Context mcontext) {
- // step 1) If this facet request has refining, then we need to fully request top buckets that were not seen by this shard.
- // step 2) If this facet does not have refining, but some sub-facets do, we need to check/recurse those sub-facets in *every* top bucket.
- // A combination of the two is possible and makes step 2 redundant for any buckets we fully requested in step 1.
+ // step 1) If this facet request has refining, then we need to fully request top buckets that
+ // were not seen by this shard.
+ // step 2) If this facet does not have refining, but some sub-facets do, we need to
+ // check/recurse those sub-facets in *every* top bucket.
+ // A combination of the two is possible and makes step 2 redundant for any buckets we fully
+ // requested in step 1.
- Map<String,Object> refinement = null;
+ Map<String, Object> refinement = null;
Collection<String> tags = mcontext.getSubsWithRefinement(freq);
if (tags.isEmpty() && !freq.doRefine()) {
// we don't have refining, and neither do our subs
return null;
}
- final FacetRequest.FacetSort initial_sort = null == freq.prelim_sort ? freq.sort : freq.prelim_sort;
-
+ final FacetRequest.FacetSort initial_sort =
+ null == freq.prelim_sort ? freq.sort : freq.prelim_sort;
+
// Tags for sub facets that have partial facets somewhere in their children.
- // If we are missing a bucket for this shard, we'll need to get the specific buckets that need refining.
+ // If we are missing a bucket for this shard, we'll need to get the specific buckets that need
+ // refining.
Collection<String> tagsWithPartial = mcontext.getSubsWithPartial(freq);
- boolean thisMissing = mcontext.bucketWasMissing(); // Was this whole facet missing (i.e. inside a bucket that was missing)?
- boolean shardHasMore = shardHasMoreBuckets != null && shardHasMoreBuckets.get(mcontext.shardNum); // shard indicated it has more buckets
- shardHasMore |= thisMissing; // if we didn't hear from the shard at all, assume it as more buckets
-
- // If we know we've seen all the buckets from a shard, then we don't have to add to leafBuckets or partialBuckets, only skipBuckets
- boolean isCommandPartial = freq.returnsPartial() || freq.processEmpty; // TODO: should returnsPartial() check processEmpty internally?
- boolean returnedAllBuckets = !shardHasMore && !freq.processEmpty; // did the shard return all of the possible buckets at this level? (pretend it didn't if processEmpty is set)
+ boolean thisMissing =
+ mcontext.bucketWasMissing(); // Was this whole facet missing (i.e. inside a bucket that was
+ // missing)?
+ boolean shardHasMore =
+ shardHasMoreBuckets != null
+ && shardHasMoreBuckets.get(mcontext.shardNum); // shard indicated it has more buckets
+ shardHasMore |=
+ thisMissing; // if we didn't hear from the shard at all, assume it as more buckets
+
+ // If we know we've seen all the buckets from a shard, then we don't have to add to leafBuckets
+ // or partialBuckets, only skipBuckets
+ boolean isCommandPartial =
+ freq.returnsPartial()
+ || freq.processEmpty; // TODO: should returnsPartial() check processEmpty internally?
+ boolean returnedAllBuckets =
+ !shardHasMore
+ && !freq.processEmpty; // did the shard return all of the possible buckets at this
+ // level? (pretend it didn't if processEmpty is set)
if (returnedAllBuckets && tags.isEmpty() && tagsWithPartial.isEmpty()) {
- // this shard returned all of its possible buckets, and there were no sub-facets with partial results
+ // this shard returned all of its possible buckets, and there were no sub-facets with partial
+ // results
// or sub-facets that require refining
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/facet/RelatednessAgg.java
##########
@@ -385,23 +430,26 @@ public void collect(int perSegDocId, int slot, IntFunction<SlotContext> slotCont
}
@Override
- public int collect(DocSet docs, int slot, IntFunction<SlotContext> slotContext) throws IOException {
+ public int collect(DocSet docs, int slot, IntFunction<SlotContext> slotContext)
+ throws IOException {
// NOTE: we don't actaully care about the doc set being collected for the bucket
// so we only worry about ensuring that every "slot" / bucket is processed exactly once
-
+
// if we're doing bulk collection, we better not be getting asked to re-use slots
assert null == slotvalues[slot];
processSlot(slot, slotContext);
- // we don't do any filtering, we collect the whole docset, so return that as out collected count
- // (as a stat, we're actually required to return this by assertions in FacetFieldProcessor.processStats)
+ // we don't do any filtering, we collect the whole docset, so return that as out collected
+ // count
+ // (as a stat, we're actually required to return this by assertions in
+ // FacetFieldProcessor.processStats)
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/facet/FacetRequestSortedMerger.java
##########
@@ -313,11 +327,11 @@ boolean isBucketComplete(FacetBucket bucket, Context mcontext) {
return refinement;
}
- // utility method for subclasses to override to finish calculating faceting (special buckets in field facets)... this feels hacky and we
+ // utility method for subclasses to override to finish calculating faceting (special buckets in
+ // field facets)... this feels hacky and we
// should find a better way.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/facet/RelatednessAgg.java
##########
@@ -297,63 +325,80 @@ public void close() throws IOException {
private final DocSet bgSet;
private final long fgSize;
private final long bgSize;
- public SKGSlotAcc(final RelatednessAgg agg, final FacetContext fcontext, final int numSlots,
- final DocSet fgSet, final DocSet bgSet) throws IOException {
+
+ public SKGSlotAcc(
+ final RelatednessAgg agg,
+ final FacetContext fcontext,
+ final int numSlots,
+ final DocSet fgSet,
+ final DocSet bgSet)
+ throws IOException {
super(fcontext);
this.agg = agg;
this.fgSet = fgSet;
this.bgSet = bgSet;
// cache the set sizes for frequent re-use on every slot
this.fgSize = fgSet.size();
this.bgSize = bgSet.size();
- this.slotvalues = new BucketData[numSlots]; //TODO: avoid initializing array until we know we're not doing sweep collection?
+ this.slotvalues =
+ new BucketData
+ [numSlots]; // TODO: avoid initializing array until we know we're not doing sweep
+ // collection?
reset();
}
/**
- * If called, may register SweepingAccs for fg and bg set based on whether
- * user indicated sweeping should be used (default)
+ * If called, may register SweepingAccs for fg and bg set based on whether user indicated
+ * sweeping should be used (default)
*
- * @returns null if any SweepingAccs were registered since no other collection is needed for relatedness
+ * @returns null if any SweepingAccs were registered since no other collection is needed for
+ * relatedness
*/
@Override
public SKGSlotAcc registerSweepingAccs(SweepingCountSlotAcc baseSweepingAcc) {
if (!this.agg.useSweep) {
return this;
} else {
- final ReadOnlyCountSlotAcc fgCount = baseSweepingAcc.add(key + "!fg", fgSet, slotvalues.length);
- final ReadOnlyCountSlotAcc bgCount = baseSweepingAcc.add(key + "!bg", bgSet, slotvalues.length);
- SweepSKGSlotAcc readOnlyReplacement = new SweepSKGSlotAcc(agg.min_pop, fcontext, slotvalues.length, fgSize, bgSize, fgCount, bgCount);
+ final ReadOnlyCountSlotAcc fgCount =
+ baseSweepingAcc.add(key + "!fg", fgSet, slotvalues.length);
+ final ReadOnlyCountSlotAcc bgCount =
+ baseSweepingAcc.add(key + "!bg", bgSet, slotvalues.length);
+ SweepSKGSlotAcc readOnlyReplacement =
+ new SweepSKGSlotAcc(
+ agg.min_pop, fcontext, slotvalues.length, fgSize, bgSize, fgCount, bgCount);
readOnlyReplacement.key = key;
baseSweepingAcc.registerMapping(this, readOnlyReplacement);
return null;
}
}
private void processSlot(int slot, IntFunction<SlotContext> slotContext) throws IOException {
-
+
assert null != slotContext;
-
+
final BucketData slotVal = new BucketData(agg);
slotvalues[slot] = slotVal;
-
+
final SlotContext ctx = slotContext.apply(slot);
if (ctx.isAllBuckets()) {
// relatedness is meaningless for allBuckets (see SOLR-14467)
// our current (implied & empty) BucketData is all we need
//
// NOTE: it might be temping to use 'slotvalues[slot] = null' in this case
// since getValue() will also ultimately generate an implied bucket in that case,
- // but by using a non-null bucket we let collect(int,...) know it doesn't need to keep calling
+ // but by using a non-null bucket we let collect(int,...) know it doesn't need to keep
+ // calling
// processSlot over and over.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/facet/FacetRequestSortedMerger.java
##########
@@ -251,23 +256,30 @@ boolean isBucketComplete(FacetBucket bucket, Context mcontext) {
} else {
// don't re-sort (the prerefinement values) if our subclass already did it
if (sortedBuckets == null) {
- sortBuckets(initial_sort); // todo: make sure this filters buckets as well
+ sortBuckets(initial_sort); // todo: make sure this filters buckets as well
}
bucketList = sortedBuckets;
}
- ArrayList<Object> leafBuckets = null; // "_l" missing buckets specified by bucket value only (no need to specify anything further)
- ArrayList<Object> partialBuckets = null; // "_p" missing buckets that have a partial sub-facet that need to specify those bucket values... each entry is [bucketval, subs]
- ArrayList<Object> skipBuckets = null; // "_s" present buckets that we need to recurse into because children facets have refinement requirements. each entry is [bucketval, subs]
+ ArrayList<Object> leafBuckets =
+ null; // "_l" missing buckets specified by bucket value only (no need to specify anything
+ // further)
+ ArrayList<Object> partialBuckets =
+ null; // "_p" missing buckets that have a partial sub-facet that need to specify those
+ // bucket values... each entry is [bucketval, subs]
+ ArrayList<Object> skipBuckets =
+ null; // "_s" present buckets that we need to recurse into because children facets have
+ // refinement requirements. each entry is [bucketval, subs]
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/facet/MinMaxAgg.java
##########
@@ -38,41 +37,47 @@
import org.apache.solr.search.function.FieldNameValueSource;
public class MinMaxAgg extends SimpleAggValueSource {
- final int minmax; // a multiplier to reverse the normal order of compare if this is max instead of min (i.e. max will be -1)
+ final int
+ minmax; // a multiplier to reverse the normal order of compare if this is max instead of min
+ // (i.e. max will be -1)
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/facet/RelatednessAgg.java
##########
@@ -586,77 +646,77 @@ public int compareTo(BucketData that) {
}
return r;
}
-
+
/**
* @see SlotAcc#getValue
* @see Merger#getMergedResult
*/
-
public SimpleOrderedMap<Object> externalize(final boolean isShardRequest) {
SimpleOrderedMap<Object> result = new SimpleOrderedMap<>();
// if counts are non-zero, then this bucket must not be implied
- assert 0 == fg_count || ! implied : "Implied bucket has non-zero fg_count";
- assert 0 == bg_count || ! implied : "Implied bucket has non-zero bg_count";
-
+ assert 0 == fg_count || !implied : "Implied bucket has non-zero fg_count";
+ assert 0 == bg_count || !implied : "Implied bucket has non-zero bg_count";
+
if (isShardRequest) {
// shard responses must include size info, but don't need the derived stats
//
// NOTE: sizes will be the same for every slot...
// TODO: it would be nice to put them directly in the parent facet, instead of every bucket,
// in order to reduce the size of the response.
- result.add(FG_SIZE, fg_size);
+ result.add(FG_SIZE, fg_size);
result.add(BG_SIZE, bg_size);
-
+
if (implied) {
// for an implied bucket on this shard, we don't need to bother returning the (empty)
// counts, just the flag explaining that this bucket is (locally) implied...
result.add(IMPLIED_KEY, Boolean.TRUE);
} else {
- result.add(FG_COUNT, fg_count);
+ result.add(FG_COUNT, fg_count);
result.add(BG_COUNT, bg_count);
}
} else {
if (implied) {
- // When returning results to an external client, any bucket still 'implied' shouldn't return
+ // When returning results to an external client, any bucket still 'implied' shouldn't
+ // return
// any results at all.
// (practically speaking this should only happen for the 'allBuckets' bucket
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/facet/RelatednessAgg.java
##########
@@ -297,63 +325,80 @@ public void close() throws IOException {
private final DocSet bgSet;
private final long fgSize;
private final long bgSize;
- public SKGSlotAcc(final RelatednessAgg agg, final FacetContext fcontext, final int numSlots,
- final DocSet fgSet, final DocSet bgSet) throws IOException {
+
+ public SKGSlotAcc(
+ final RelatednessAgg agg,
+ final FacetContext fcontext,
+ final int numSlots,
+ final DocSet fgSet,
+ final DocSet bgSet)
+ throws IOException {
super(fcontext);
this.agg = agg;
this.fgSet = fgSet;
this.bgSet = bgSet;
// cache the set sizes for frequent re-use on every slot
this.fgSize = fgSet.size();
this.bgSize = bgSet.size();
- this.slotvalues = new BucketData[numSlots]; //TODO: avoid initializing array until we know we're not doing sweep collection?
+ this.slotvalues =
+ new BucketData
+ [numSlots]; // TODO: avoid initializing array until we know we're not doing sweep
+ // collection?
reset();
}
/**
- * If called, may register SweepingAccs for fg and bg set based on whether
- * user indicated sweeping should be used (default)
+ * If called, may register SweepingAccs for fg and bg set based on whether user indicated
+ * sweeping should be used (default)
*
- * @returns null if any SweepingAccs were registered since no other collection is needed for relatedness
+ * @returns null if any SweepingAccs were registered since no other collection is needed for
+ * relatedness
*/
@Override
public SKGSlotAcc registerSweepingAccs(SweepingCountSlotAcc baseSweepingAcc) {
if (!this.agg.useSweep) {
return this;
} else {
- final ReadOnlyCountSlotAcc fgCount = baseSweepingAcc.add(key + "!fg", fgSet, slotvalues.length);
- final ReadOnlyCountSlotAcc bgCount = baseSweepingAcc.add(key + "!bg", bgSet, slotvalues.length);
- SweepSKGSlotAcc readOnlyReplacement = new SweepSKGSlotAcc(agg.min_pop, fcontext, slotvalues.length, fgSize, bgSize, fgCount, bgCount);
+ final ReadOnlyCountSlotAcc fgCount =
+ baseSweepingAcc.add(key + "!fg", fgSet, slotvalues.length);
+ final ReadOnlyCountSlotAcc bgCount =
+ baseSweepingAcc.add(key + "!bg", bgSet, slotvalues.length);
+ SweepSKGSlotAcc readOnlyReplacement =
+ new SweepSKGSlotAcc(
+ agg.min_pop, fcontext, slotvalues.length, fgSize, bgSize, fgCount, bgCount);
readOnlyReplacement.key = key;
baseSweepingAcc.registerMapping(this, readOnlyReplacement);
return null;
}
}
private void processSlot(int slot, IntFunction<SlotContext> slotContext) throws IOException {
-
+
assert null != slotContext;
-
+
final BucketData slotVal = new BucketData(agg);
slotvalues[slot] = slotVal;
-
+
final SlotContext ctx = slotContext.apply(slot);
if (ctx.isAllBuckets()) {
// relatedness is meaningless for allBuckets (see SOLR-14467)
// our current (implied & empty) BucketData is all we need
//
// NOTE: it might be temping to use 'slotvalues[slot] = null' in this case
// since getValue() will also ultimately generate an implied bucket in that case,
- // but by using a non-null bucket we let collect(int,...) know it doesn't need to keep calling
+ // but by using a non-null bucket we let collect(int,...) know it doesn't need to keep
+ // calling
// processSlot over and over.
return;
}
-
+
Query slotQ = ctx.getSlotQuery();
if (null == slotQ) {
// extremeley special edge case...
- // the only way this should be possible is if our relatedness() function is used as a "top level"
- // stat w/o being nested under any facet, in which case it should be a FacetQuery w/no parent...
+ // the only way this should be possible is if our relatedness() function is used as a "top
+ // level"
+ // stat w/o being nested under any facet, in which case it should be a FacetQuery w/no
+ // parent...
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/facet/MinMaxAgg.java
##########
@@ -308,7 +322,9 @@ public int compare(int slotA, int slotB) {
boolean eb = b != 0 || exists.get(slotB);
if (ea != eb) {
- if (ea) return 1; // a exists and b doesn't TODO: we need context to be able to sort missing last! SOLR-10618
+ if (ea)
+ return 1; // a exists and b doesn't TODO: we need context to be able to sort missing last!
+ // SOLR-10618
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/facet/RelatednessAgg.java
##########
@@ -297,63 +325,80 @@ public void close() throws IOException {
private final DocSet bgSet;
private final long fgSize;
private final long bgSize;
- public SKGSlotAcc(final RelatednessAgg agg, final FacetContext fcontext, final int numSlots,
- final DocSet fgSet, final DocSet bgSet) throws IOException {
+
+ public SKGSlotAcc(
+ final RelatednessAgg agg,
+ final FacetContext fcontext,
+ final int numSlots,
+ final DocSet fgSet,
+ final DocSet bgSet)
+ throws IOException {
super(fcontext);
this.agg = agg;
this.fgSet = fgSet;
this.bgSet = bgSet;
// cache the set sizes for frequent re-use on every slot
this.fgSize = fgSet.size();
this.bgSize = bgSet.size();
- this.slotvalues = new BucketData[numSlots]; //TODO: avoid initializing array until we know we're not doing sweep collection?
+ this.slotvalues =
+ new BucketData
+ [numSlots]; // TODO: avoid initializing array until we know we're not doing sweep
+ // collection?
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/facet/StddevAgg.java
##########
@@ -83,46 +84,52 @@ public void merge(Object facetResult, Context mcontext1) {
public Object getMergedResult() {
return this.getDouble();
}
-
+
@Override
protected double getDouble() {
return AggUtil.stdDev(sumSq, sum, count);
- }
+ }
}
class StddevSortedNumericAcc extends DocValuesAcc.SDVSortedNumericAcc {
- public StddevSortedNumericAcc(FacetContext fcontext, SchemaField sf, int numSlots) throws IOException {
+ public StddevSortedNumericAcc(FacetContext fcontext, SchemaField sf, int numSlots)
+ throws IOException {
super(fcontext, sf, numSlots);
}
@Override
protected double computeVal(int slot) {
- return AggUtil.stdDev(result[slot], sum[slot], counts[slot]); // calc once and cache in result?
+ return AggUtil.stdDev(
+ result[slot], sum[slot], counts[slot]); // calc once and cache in result?
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/facet/SlotAcc.java
##########
@@ -567,7 +570,8 @@ public void resize(Resizer resizer) {
}
private double stdDev(int slot) {
- return AggUtil.stdDev(result[slot], sum[slot], counts[slot]); // calc once and cache in result?
+ return AggUtil.stdDev(
+ result[slot], sum[slot], counts[slot]); // calc once and cache in result?
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/facet/StddevAgg.java
##########
@@ -83,46 +84,52 @@ public void merge(Object facetResult, Context mcontext1) {
public Object getMergedResult() {
return this.getDouble();
}
-
+
@Override
protected double getDouble() {
return AggUtil.stdDev(sumSq, sum, count);
- }
+ }
}
class StddevSortedNumericAcc extends DocValuesAcc.SDVSortedNumericAcc {
- public StddevSortedNumericAcc(FacetContext fcontext, SchemaField sf, int numSlots) throws IOException {
+ public StddevSortedNumericAcc(FacetContext fcontext, SchemaField sf, int numSlots)
+ throws IOException {
super(fcontext, sf, numSlots);
}
@Override
protected double computeVal(int slot) {
- return AggUtil.stdDev(result[slot], sum[slot], counts[slot]); // calc once and cache in result?
+ return AggUtil.stdDev(
+ result[slot], sum[slot], counts[slot]); // calc once and cache in result?
}
}
class StddevSortedSetAcc extends DocValuesAcc.SDVSortedSetAcc {
- public StddevSortedSetAcc(FacetContext fcontext, SchemaField sf, int numSlots) throws IOException {
+ public StddevSortedSetAcc(FacetContext fcontext, SchemaField sf, int numSlots)
+ throws IOException {
super(fcontext, sf, numSlots);
}
@Override
protected double computeVal(int slot) {
- return AggUtil.stdDev(result[slot], sum[slot], counts[slot]); // calc once and cache in result?
+ return AggUtil.stdDev(
+ result[slot], sum[slot], counts[slot]); // calc once and cache in result?
}
}
class StddevUnInvertedFieldAcc extends UnInvertedFieldAcc.SDVUnInvertedFieldAcc {
- public StddevUnInvertedFieldAcc(FacetContext fcontext, SchemaField sf, int numSlots) throws IOException {
+ public StddevUnInvertedFieldAcc(FacetContext fcontext, SchemaField sf, int numSlots)
+ throws IOException {
super(fcontext, sf, numSlots);
}
@Override
protected double computeVal(int slot) {
- return AggUtil.stdDev(result[slot], sum[slot], counts[slot]); // calc once and cache in result?
+ return AggUtil.stdDev(
+ result[slot], sum[slot], counts[slot]); // calc once and cache in result?
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/facet/StddevAgg.java
##########
@@ -83,46 +84,52 @@ public void merge(Object facetResult, Context mcontext1) {
public Object getMergedResult() {
return this.getDouble();
}
-
+
@Override
protected double getDouble() {
return AggUtil.stdDev(sumSq, sum, count);
- }
+ }
}
class StddevSortedNumericAcc extends DocValuesAcc.SDVSortedNumericAcc {
- public StddevSortedNumericAcc(FacetContext fcontext, SchemaField sf, int numSlots) throws IOException {
+ public StddevSortedNumericAcc(FacetContext fcontext, SchemaField sf, int numSlots)
+ throws IOException {
super(fcontext, sf, numSlots);
}
@Override
protected double computeVal(int slot) {
- return AggUtil.stdDev(result[slot], sum[slot], counts[slot]); // calc once and cache in result?
+ return AggUtil.stdDev(
+ result[slot], sum[slot], counts[slot]); // calc once and cache in result?
}
}
class StddevSortedSetAcc extends DocValuesAcc.SDVSortedSetAcc {
- public StddevSortedSetAcc(FacetContext fcontext, SchemaField sf, int numSlots) throws IOException {
+ public StddevSortedSetAcc(FacetContext fcontext, SchemaField sf, int numSlots)
+ throws IOException {
super(fcontext, sf, numSlots);
}
@Override
protected double computeVal(int slot) {
- return AggUtil.stdDev(result[slot], sum[slot], counts[slot]); // calc once and cache in result?
+ return AggUtil.stdDev(
+ result[slot], sum[slot], counts[slot]); // calc once and cache in result?
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/facet/SlotAcc.java
##########
@@ -412,8 +409,11 @@ public SumSlotAcc(ValueSource values, FacetContext fcontext, int numSlots) {
super(values, fcontext, numSlots);
}
- public void collect(int doc, int slotNum, IntFunction<SlotContext> slotContext) throws IOException {
- double val = values.doubleVal(doc); // todo: worth trying to share this value across multiple stats that need it?
+ public void collect(int doc, int slotNum, IntFunction<SlotContext> slotContext)
+ throws IOException {
+ double val =
+ values.doubleVal(
+ doc); // todo: worth trying to share this value across multiple stats that need it?
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/facet/SlotAcc.java
##########
@@ -510,7 +511,8 @@ public void resize(Resizer resizer) {
}
private double variance(int slot) {
- return AggUtil.variance(result[slot], sum[slot], counts[slot]); // calc once and cache in result?
+ return AggUtil.variance(
+ result[slot], sum[slot], counts[slot]); // calc once and cache in result?
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/facet/UnInvertedField.java
##########
@@ -151,7 +152,8 @@ protected void visitTerm(TermsEnum te, int termNum) throws IOException {
deState = new SolrIndexSearcher.DocsEnumState();
deState.fieldName = field;
deState.liveDocs = searcher.getLiveDocsBits();
- deState.termsEnum = te; // TODO: check for MultiTermsEnum in SolrIndexSearcher could now fail?
+ deState.termsEnum =
+ te; // TODO: check for MultiTermsEnum in SolrIndexSearcher could now fail?
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/grouping/distributed/shardresultserializer/TopGroupsResultTransformer.java
##########
@@ -125,7 +133,8 @@ public TopGroupsResultTransformer(ResponseBuilder rb) {
String groupValue = commandResult.getName(i);
@SuppressWarnings("unchecked")
NamedList<Object> groupResult = (NamedList<Object>) commandResult.getVal(i);
- Number totalGroupHits = (Number) groupResult.get("totalHits"); // // previously Integer now Long
+ Number totalGroupHits =
+ (Number) groupResult.get("totalHits"); // // previously Integer now Long
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/facet/SweepCountAware.java
##########
@@ -154,25 +160,30 @@ public final void incrementCount(int segOrd, int inc, int maxIdx) {
}
/**
- * Maps accumulated per-segment term ords to global term ords and increments global slots on the specified countAccs
- * accordingly. The index of each CountSlotAcc in the specified countAccs array must correspond to the
- * the static index of its associated count accumulation doc domain and per-seg count array.
- *
- * @param countAccs - global-scope CountSlotAccs (one for each domain) to be incremented for the most recently accumulated
- * segment
- * @param toGlobal - mapping of per-segment term ords to global term ords for the most recently accumulated segment
+ * Maps accumulated per-segment term ords to global term ords and increments global slots on the
+ * specified countAccs accordingly. The index of each CountSlotAcc in the specified countAccs
+ * array must correspond to the the static index of its associated count accumulation doc domain
+ * and per-seg count array.
+ *
+ * @param countAccs - global-scope CountSlotAccs (one for each domain) to be incremented for the
+ * most recently accumulated segment
+ * @param toGlobal - mapping of per-segment term ords to global term ords for the most recently
+ * accumulated segment
* @param maxSegOrd - the max per-seg term ord for the most recently accumulated segment
*/
public void register(CountSlotAcc[] countAccs, LongValues toGlobal, int maxSegOrd) {
int segOrd = maxSegOrd;
- // NOTE: the `countAccs` array may be oversized (e.g., in the event that one or more of the domains supplies a <code>null</code>
- // DocIdSetIterator -- matches no docs -- for a given segment). But the relevant contents of `countAccs` are re-initialized
- // per-segment, so we can (and must!) safely set `maxIdx` based on `activeSegCounts.length` (as opposed to `countAccs.length`).
+ // NOTE: the `countAccs` array may be oversized (e.g., in the event that one or more of the
+ // domains supplies a <code>null</code>
+ // DocIdSetIterator -- matches no docs -- for a given segment). But the relevant contents of
+ // `countAccs` are re-initialized
+ // per-segment, so we can (and must!) safely set `maxIdx` based on `activeSegCounts.length`
+ // (as opposed to `countAccs.length`).
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/facet/UniqueMultivaluedSlotAcc.java
##########
@@ -29,12 +28,15 @@
private UnInvertedField uif;
private UnInvertedField.DocToTerm docToTerm;
- public UniqueMultivaluedSlotAcc(FacetContext fcontext, SchemaField field, int numSlots, HLLAgg.HLLFactory factory) throws IOException {
+ public UniqueMultivaluedSlotAcc(
+ FacetContext fcontext, SchemaField field, int numSlots, HLLAgg.HLLFactory factory)
+ throws IOException {
super(fcontext, field, numSlots, factory);
SolrIndexSearcher searcher = fcontext.qcontext.searcher();
uif = UnInvertedField.getUnInvertedField(field.getName(), searcher);
docToTerm = uif.new DocToTerm();
- fcontext.qcontext.addCloseHook(this); // TODO: find way to close accumulators instead of using close hook?
+ fcontext.qcontext.addCloseHook(
+ this); // TODO: find way to close accumulators instead of using close hook?
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/facet/VarianceAgg.java
##########
@@ -82,46 +83,52 @@ public void merge(Object facetResult, Context mcontext1) {
public Object getMergedResult() {
return this.getDouble();
}
-
+
@Override
protected double getDouble() {
return AggUtil.variance(sumSq, sum, count);
- }
+ }
}
class VarianceSortedNumericAcc extends DocValuesAcc.SDVSortedNumericAcc {
- public VarianceSortedNumericAcc(FacetContext fcontext, SchemaField sf, int numSlots) throws IOException {
+ public VarianceSortedNumericAcc(FacetContext fcontext, SchemaField sf, int numSlots)
+ throws IOException {
super(fcontext, sf, numSlots);
}
@Override
protected double computeVal(int slot) {
- return AggUtil.variance(result[slot], sum[slot], counts[slot]); // calc once and cache in result?
+ return AggUtil.variance(
+ result[slot], sum[slot], counts[slot]); // calc once and cache in result?
}
}
class VarianceSortedSetAcc extends DocValuesAcc.SDVSortedSetAcc {
- public VarianceSortedSetAcc(FacetContext fcontext, SchemaField sf, int numSlots) throws IOException {
+ public VarianceSortedSetAcc(FacetContext fcontext, SchemaField sf, int numSlots)
+ throws IOException {
super(fcontext, sf, numSlots);
}
@Override
protected double computeVal(int slot) {
- return AggUtil.variance(result[slot], sum[slot], counts[slot]); // calc once and cache in result?
+ return AggUtil.variance(
+ result[slot], sum[slot], counts[slot]); // calc once and cache in result?
}
}
class VarianceUnInvertedFieldAcc extends UnInvertedFieldAcc.SDVUnInvertedFieldAcc {
- public VarianceUnInvertedFieldAcc(FacetContext fcontext, SchemaField sf, int numSlots) throws IOException {
+ public VarianceUnInvertedFieldAcc(FacetContext fcontext, SchemaField sf, int numSlots)
+ throws IOException {
super(fcontext, sf, numSlots);
}
@Override
protected double computeVal(int slot) {
- return AggUtil.variance(result[slot], sum[slot], counts[slot]); // calc once and cache in result?
+ return AggUtil.variance(
+ result[slot], sum[slot], counts[slot]); // calc once and cache in result?
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/facet/VarianceAgg.java
##########
@@ -82,46 +83,52 @@ public void merge(Object facetResult, Context mcontext1) {
public Object getMergedResult() {
return this.getDouble();
}
-
+
@Override
protected double getDouble() {
return AggUtil.variance(sumSq, sum, count);
- }
+ }
}
class VarianceSortedNumericAcc extends DocValuesAcc.SDVSortedNumericAcc {
- public VarianceSortedNumericAcc(FacetContext fcontext, SchemaField sf, int numSlots) throws IOException {
+ public VarianceSortedNumericAcc(FacetContext fcontext, SchemaField sf, int numSlots)
+ throws IOException {
super(fcontext, sf, numSlots);
}
@Override
protected double computeVal(int slot) {
- return AggUtil.variance(result[slot], sum[slot], counts[slot]); // calc once and cache in result?
+ return AggUtil.variance(
+ result[slot], sum[slot], counts[slot]); // calc once and cache in result?
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/servlet/CoreContainerProvider.java
##########
@@ -90,14 +89,18 @@
private final CountDownLatch init = new CountDownLatch(1);
private String registryName;
private final boolean isV2Enabled = !"true".equals(System.getProperty("disable.v2.api", "false"));
- // AFAIK the only reason we need this is to support JettySolrRunner for tests. In tests we might have
+ // AFAIK the only reason we need this is to support JettySolrRunner for tests. In tests we might
+ // have
// multiple CoreContainers in the same JVM, but I *think* that doesn't happen in a real server.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/facet/UniqueMultivaluedSlotAcc.java
##########
@@ -43,21 +45,23 @@ protected BytesRef lookupOrd(int ord) throws IOException {
return docToTerm.lookupOrd(ord);
}
- private FixedBitSet bits; // bits for the current slot, only set for the callback
+ private FixedBitSet bits; // bits for the current slot, only set for the callback
@Override
public void call(int termNum) {
bits.set(termNum);
}
@Override
- public void collect(int doc, int slotNum, IntFunction<SlotContext> slotContext) throws IOException {
+ public void collect(int doc, int slotNum, IntFunction<SlotContext> slotContext)
+ throws IOException {
bits = arr[slotNum];
if (bits == null) {
bits = new FixedBitSet(nTerms);
arr[slotNum] = bits;
}
- docToTerm.getBigTerms(doc + currentDocBase, this); // this will call back to our Callback.call(int termNum)
+ docToTerm.getBigTerms(
+ doc + currentDocBase, this); // this will call back to our Callback.call(int termNum)
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/neural/KnnQParser.java
##########
@@ -29,73 +29,88 @@
public class KnnQParser extends QParser {
- static final String TOP_K = "topK";// retrieve the top K results based on the distance similarity function
- static final int DEFAULT_TOP_K = 10;
+ static final String TOP_K =
+ "topK"; // retrieve the top K results based on the distance similarity function
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/search/facet/VarianceAgg.java
##########
@@ -82,46 +83,52 @@ public void merge(Object facetResult, Context mcontext1) {
public Object getMergedResult() {
return this.getDouble();
}
-
+
@Override
protected double getDouble() {
return AggUtil.variance(sumSq, sum, count);
- }
+ }
}
class VarianceSortedNumericAcc extends DocValuesAcc.SDVSortedNumericAcc {
- public VarianceSortedNumericAcc(FacetContext fcontext, SchemaField sf, int numSlots) throws IOException {
+ public VarianceSortedNumericAcc(FacetContext fcontext, SchemaField sf, int numSlots)
+ throws IOException {
super(fcontext, sf, numSlots);
}
@Override
protected double computeVal(int slot) {
- return AggUtil.variance(result[slot], sum[slot], counts[slot]); // calc once and cache in result?
+ return AggUtil.variance(
+ result[slot], sum[slot], counts[slot]); // calc once and cache in result?
}
}
class VarianceSortedSetAcc extends DocValuesAcc.SDVSortedSetAcc {
- public VarianceSortedSetAcc(FacetContext fcontext, SchemaField sf, int numSlots) throws IOException {
+ public VarianceSortedSetAcc(FacetContext fcontext, SchemaField sf, int numSlots)
+ throws IOException {
super(fcontext, sf, numSlots);
}
@Override
protected double computeVal(int slot) {
- return AggUtil.variance(result[slot], sum[slot], counts[slot]); // calc once and cache in result?
+ return AggUtil.variance(
+ result[slot], sum[slot], counts[slot]); // calc once and cache in result?
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/security/MultiAuthPlugin.java
##########
@@ -162,16 +178,24 @@ public void initializeMetrics(SolrMetricsContext parentContext, String scope) {
private String getSchemeFromAuthHeader(final String authHeader) {
final int firstSpace = authHeader.indexOf(' ');
- return (firstSpace != -1) ? authHeader.substring(0, firstSpace).toLowerCase(Locale.ROOT) : UNKNOWN_SCHEME;
+ return (firstSpace != -1)
+ ? authHeader.substring(0, firstSpace).toLowerCase(Locale.ROOT)
+ : UNKNOWN_SCHEME;
}
@Override
- public boolean doAuthenticate(HttpServletRequest request, HttpServletResponse response, FilterChain filterChain) throws Exception {
+ public boolean doAuthenticate(
+ HttpServletRequest request, HttpServletResponse response, FilterChain filterChain)
+ throws Exception {
final String authHeader = request.getHeader(AUTHORIZATION_HEADER);
if (authHeader == null) {
- // no Authorization header but if it's an AJAX request, forward to the default scheme so it can handle it
+ // no Authorization header but if it's an AJAX request, forward to the default scheme so it
+ // can handle it
// otherwise, send to the first plugin that allows blockUnknown = false
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/servlet/HttpSolrCall.java
##########
@@ -147,8 +144,10 @@
public static final String INTERNAL_REQUEST_COUNT = "_forwardedCount";
public static final Random random;
+
static {
- // We try to make things reproducible in the context of our tests by initializing the random instance
+ // We try to make things reproducible in the context of our tests by initializing the random
+ // instance
// based on the current seed
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/security/RuleBasedAuthorizationPluginBase.java
##########
@@ -179,88 +189,115 @@ private boolean permissionAppliesToRequest(Permission permission, AuthorizationC
}
}
- private boolean predefinedPermissionAppliesToRequest(Permission predefinedPermission, AuthorizationContext context) {
+ private boolean predefinedPermissionAppliesToRequest(
+ Permission predefinedPermission, AuthorizationContext context) {
log.trace("Permission [{}] is a predefined perm", predefinedPermission);
if (predefinedPermission.wellknownName == PermissionNameProvider.Name.ALL) {
log.trace("'ALL' perm applies to all requests; perm applies.");
- return true; //'ALL' applies to everything!
- } else if (! (context.getHandler() instanceof PermissionNameProvider)) {
- // TODO: Is this code path needed anymore, now that all handlers implement the interface? context.getHandler returns Object and is not documented
+ return true; // 'ALL' applies to everything!
+ } else if (!(context.getHandler() instanceof PermissionNameProvider)) {
+ // TODO: Is this code path needed anymore, now that all handlers implement the interface?
+ // context.getHandler returns Object and is not documented
if (log.isTraceEnabled()) {
- log.trace("Request handler [{}] is not a PermissionNameProvider, perm doesnt apply", context.getHandler());
+ log.trace(
+ "Request handler [{}] is not a PermissionNameProvider, perm doesnt apply",
+ context.getHandler());
}
- return false; // We're not 'ALL', and the handler isn't associated with any other predefined permissions
+ return false; // We're not 'ALL', and the handler isn't associated with any other predefined
+ // permissions
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/servlet/CoreContainerProvider.java
##########
@@ -90,14 +89,18 @@
private final CountDownLatch init = new CountDownLatch(1);
private String registryName;
private final boolean isV2Enabled = !"true".equals(System.getProperty("disable.v2.api", "false"));
- // AFAIK the only reason we need this is to support JettySolrRunner for tests. In tests we might have
+ // AFAIK the only reason we need this is to support JettySolrRunner for tests. In tests we might
+ // have
// multiple CoreContainers in the same JVM, but I *think* that doesn't happen in a real server.
private static final Map<ContextInitializationKey, ServiceHolder> services =
Collections.synchronizedMap(new WeakHashMap<>());
- // todo: dependency injection instead, but for now this method and the associated map will have to suffice.
- // Note that this relies on ServletContext.equals() not implementing anything significantly different
- // than Object.equals for its .equals method (I've found no implementation that even implements it).
+ // todo: dependency injection instead, but for now this method and the associated map will have
+ // to suffice.
+ // Note that this relies on ServletContext.equals() not implementing anything significantly
+ // different
+ // than Object.equals for its .equals method (I've found no implementation that even implements
+ // it).
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/servlet/HttpSolrCall.java
##########
@@ -174,18 +173,25 @@
protected SolrConfig config;
protected Map<String, Integer> invalidStates;
- //The states of client that is invalid in this request
- protected String origCorename; // What's in the URL path; might reference a collection/alias or a Solr core name
- protected List<String> collectionsList; // The list of SolrCloud collections if in SolrCloud (usually 1)
+ // The states of client that is invalid in this request
+ protected String
+ origCorename; // What's in the URL path; might reference a collection/alias or a Solr core
+ // name
+ protected List<String>
+ collectionsList; // The list of SolrCloud collections if in SolrCloud (usually 1)
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/servlet/CoreContainerProvider.java
##########
@@ -233,39 +251,45 @@ public void init(ServletContext servletContext) {
if (log.isDebugEnabled()) {
log.debug("user.dir={}", System.getProperty("user.dir"));
}
- } catch( Throwable t ) {
+ } catch (Throwable t) {
// catch this so our filter still works
- log.error( "Could not start Solr. Check solr/home property and the logs");
- SolrCore.log( t );
+ log.error("Could not start Solr. Check solr/home property and the logs");
+ SolrCore.log(t);
if (t instanceof Error) {
throw (Error) t;
}
- } finally{
+ } finally {
log.trace("SolrDispatchFilter.init() done");
this.cores = coresInit; // crucially final assignment
- services.computeIfAbsent(new ContextInitializationKey(servletContext), ServiceHolder::new)
+ services
+ .computeIfAbsent(new ContextInitializationKey(servletContext), ServiceHolder::new)
.setService(this);
init.countDown();
}
}
-
private void logWelcomeBanner() {
- // _Really_ sorry about how clumsy this is as a result of the logging call checker, but this is the only one
+ // _Really_ sorry about how clumsy this is as a result of the logging call checker, but this is
+ // the only one
// that's so ugly so far.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/servlet/CoreContainerProvider.java
##########
@@ -407,7 +462,8 @@ private ContextInitializationKey(ServletContext ctx) {
throw new IllegalArgumentException("Context must not be null");
}
// if one of these is reachable both must be to avoid collection from weak hashmap, so
- // set an attribute holding this object to ensure we never get collected until the ServletContext
+ // set an attribute holding this object to ensure we never get collected until the
+ // ServletContext
// is eligible for collection too.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/servlet/HttpSolrCall.java
##########
@@ -261,7 +269,8 @@ protected void init() throws Exception {
if (core != null) {
path = path.substring(idx);
} else {
- if (cores.isCoreLoading(origCorename)) { // extra mem barriers, so don't look at this before trying to get core
+ if (cores.isCoreLoading(
+ origCorename)) { // extra mem barriers, so don't look at this before trying to get core
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/security/MultiAuthPlugin.java
##########
@@ -57,20 +58,27 @@
private final Map<String, AuthenticationPlugin> pluginMap = new LinkedHashMap<>();
private final ResourceLoader loader;
- private AuthenticationPlugin allowsUnknown = null; // the first of our plugins that allows anonymous requests
+ private AuthenticationPlugin allowsUnknown =
+ null; // the first of our plugins that allows anonymous requests
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/servlet/CoreContainerProvider.java
##########
@@ -147,21 +155,27 @@ private static void waitForCoreContainer(Supplier<CoreContainer> provider, Count
public void close() {
CoreContainer cc = cores;
-// if (cc != null) {
-// ZkController zkController = cc.getZkController();
-// if (zkController != null) {
-//
-// // Mark Miller suggested that we should be publishing that we are down before anything else which makes
-// // good sense, but the following causes test failures, so that improvement can be the subject of another
-// // PR/issue. Also, jetty might already be refusing requests by this point so that's a potential issue too.
-// // Digging slightly I see that there's a whole mess of code looking up collections and calculating state
-// // changes associated with this call, which smells a lot like we're duplicating node state in collection
-// // stuff, but it will take a lot of code reading to figure out if that's really what it is, why we
-// // did it and if there's room for improvement.
-//
-// zkController.publishNodeAsDown(zkController.getNodeName());
-// }
-// }
+ // if (cc != null) {
+ // ZkController zkController = cc.getZkController();
+ // if (zkController != null) {
+ //
+ // // Mark Miller suggested that we should be publishing that we are down before anything
+ // else which makes
+ // // good sense, but the following causes test failures, so that improvement can be the
+ // subject of another
+ // // PR/issue. Also, jetty might already be refusing requests by this point so that's a
+ // potential issue too.
+ // // Digging slightly I see that there's a whole mess of code looking up collections and
+ // calculating state
+ // // changes associated with this call, which smells a lot like we're duplicating node
+ // state in collection
+ // // stuff, but it will take a lot of code reading to figure out if that's really what
+ // it is, why we
+ // // did it and if there's room for improvement.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/servlet/HttpSolrCall.java
##########
@@ -279,16 +288,22 @@ protected void init() throws Exception {
if (cores.isZooKeeperAware()) {
// init collectionList (usually one name but not when there are aliases)
String def = core != null ? core.getCoreDescriptor().getCollectionName() : origCorename;
- collectionsList = resolveCollectionListOrAlias(queryParams.get(COLLECTION_PROP, def)); // &collection= takes precedence
+ collectionsList =
+ resolveCollectionListOrAlias(
+ queryParams.get(COLLECTION_PROP, def)); // &collection= takes precedence
if (core == null) {
// lookup core from collection, or route away if need to
- String collectionName = collectionsList.isEmpty() ? null : collectionsList.get(0); // route to 1st
- //TODO try the other collections if can't find a local replica of the first? (and do to V2HttpSolrCall)
+ String collectionName =
+ collectionsList.isEmpty() ? null : collectionsList.get(0); // route to 1st
+ // TODO try the other collections if can't find a local replica of the first? (and do to
+ // V2HttpSolrCall)
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/servlet/HttpSolrCall.java
##########
@@ -879,10 +951,14 @@ protected void addCollectionParamIfNeeded(List<String> collections) {
}
assert cores.isZooKeeperAware();
String collectionParam = queryParams.get(COLLECTION_PROP);
- // if there is no existing collection param and the core we go to is for the expected collection,
+ // if there is no existing collection param and the core we go to is for the expected
+ // collection,
// then we needn't add a collection param
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/servlet/ServletUtils.java
##########
@@ -96,14 +97,17 @@ public ServletInputStream getInputStream() throws IOException {
return new ServletInputStreamWrapper(super.getInputStream()) {
@Override
public void close() {
- // even though we skip closes, we let local tests know not to close so that a full understanding can take
+ // even though we skip closes, we let local tests know not to close so that a full
+ // understanding can take
// place
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/servlet/ServletUtils.java
##########
@@ -131,22 +135,30 @@ public ServletOutputStream getOutputStream() throws IOException {
return new ServletOutputStreamWrapper(super.getOutputStream()) {
@Override
public void close() {
- // even though we skip closes, we let local tests know not to close so that a full understanding can take
+ // even though we skip closes, we let local tests know not to close so that a full
+ // understanding can take
// place
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/servlet/HttpSolrCall.java
##########
@@ -420,25 +447,32 @@ protected void extractHandlerFromURLPath(SolrRequestParsers parser) throws Excep
throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "unknown handler: " + qt);
}
if (qt != null && qt.startsWith("/") && (handler instanceof ContentStreamHandlerBase)) {
- //For security reasons it's a bad idea to allow a leading '/', ex: /select?qt=/update see SOLR-3161
- //There was no restriction from Solr 1.4 thru 3.5 and it's not supported for update handlers.
- throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "Invalid Request Handler ('qt'). Do not use /select to access: " + qt);
+ // For security reasons it's a bad idea to allow a leading '/', ex: /select?qt=/update
+ // see SOLR-3161
+ // There was no restriction from Solr 1.4 thru 3.5 and it's not supported for update
+ // handlers.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/servlet/RequestRateLimiter.java
##########
@@ -19,24 +19,24 @@
import java.util.concurrent.Semaphore;
import java.util.concurrent.TimeUnit;
-
import org.apache.solr.common.annotation.SolrThreadSafe;
import org.apache.solr.core.RateLimiterConfig;
/**
* Handles rate limiting for a specific request type.
*
- * The control flow is as follows:
- * Handle request -- Check if slot is available -- If available, acquire slot and proceed --
- * else reject the same.
+ * <p>The control flow is as follows: Handle request -- Check if slot is available -- If available,
+ * acquire slot and proceed -- else reject the same.
*/
@SolrThreadSafe
public class RequestRateLimiter {
// Slots that are guaranteed for this request rate limiter.
private final Semaphore guaranteedSlotsPool;
- // Competitive slots pool that are available for this rate limiter as well as borrowing by other request rate limiters.
- // By competitive, the meaning is that there is no prioritization for the acquisition of these slots -- First Come First Serve,
+ // Competitive slots pool that are available for this rate limiter as well as borrowing by other
+ // request rate limiters.
+ // By competitive, the meaning is that there is no prioritization for the acquisition of these
+ // slots -- First Come First Serve,
// irrespective of whether the request is of this request rate limiter or other.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/servlet/SolrDispatchFilter.java
##########
@@ -38,43 +58,27 @@
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
-import javax.servlet.FilterChain;
-import javax.servlet.FilterConfig;
-import javax.servlet.ServletException;
-import javax.servlet.ServletRequest;
-import javax.servlet.ServletResponse;
-import javax.servlet.UnavailableException;
-import javax.servlet.http.HttpServletRequest;
-import javax.servlet.http.HttpServletResponse;
-import java.io.IOException;
-import java.lang.invoke.MethodHandles;
-import java.util.List;
-import java.util.concurrent.CountDownLatch;
-import java.util.concurrent.atomic.AtomicBoolean;
-import java.util.concurrent.atomic.AtomicReference;
-import java.util.regex.Pattern;
-
-import static org.apache.solr.security.AuditEvent.EventType;
-import static org.apache.solr.servlet.ServletUtils.closeShield;
-import static org.apache.solr.servlet.ServletUtils.configExcludes;
-import static org.apache.solr.servlet.ServletUtils.excludedPath;
-
/**
* This filter looks at the incoming URL maps them to handlers defined in solrconfig.xml
*
* @since solr 1.2
*/
-// todo: get rid of this class entirely! Request dispatch is the container's responsibility. Much of what we have here
-// should be several separate but composable servlet Filters, wrapping multiple servlets that are more focused in
-// scope. This should become possible now that we have a ServletContextListener for startup/shutdown of CoreContainer
-// that sets up a service from which things like CoreContainer can be requested. (or better yet injected)
+// todo: get rid of this class entirely! Request dispatch is the container's responsibility. Much of
+// what we have here
+// should be several separate but composable servlet Filters, wrapping multiple servlets that are
+// more focused in
+// scope. This should become possible now that we have a ServletContextListener for
+// startup/shutdown of CoreContainer
+// that sets up a service from which things like CoreContainer can be requested. (or better yet
+// injected)
public class SolrDispatchFilter extends BaseSolrFilter implements PathExcluder {
private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
public static final String ATTR_TRACING_SPAN = Span.class.getName();
public static final String ATTR_TRACING_TRACER = Tracer.class.getName();
public static final String ATTR_RATELIMIT_MANAGER = RateLimitManager.class.getName();
- // TODO: see if we can get rid of the holder here (Servlet spec actually guarantees ContextListeners run
+ // TODO: see if we can get rid of the holder here (Servlet spec actually guarantees
+ // ContextListeners run
// before filter init, but JettySolrRunner that we use for tests is complicated)
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/servlet/SolrDispatchFilter.java
##########
@@ -38,43 +58,27 @@
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
-import javax.servlet.FilterChain;
-import javax.servlet.FilterConfig;
-import javax.servlet.ServletException;
-import javax.servlet.ServletRequest;
-import javax.servlet.ServletResponse;
-import javax.servlet.UnavailableException;
-import javax.servlet.http.HttpServletRequest;
-import javax.servlet.http.HttpServletResponse;
-import java.io.IOException;
-import java.lang.invoke.MethodHandles;
-import java.util.List;
-import java.util.concurrent.CountDownLatch;
-import java.util.concurrent.atomic.AtomicBoolean;
-import java.util.concurrent.atomic.AtomicReference;
-import java.util.regex.Pattern;
-
-import static org.apache.solr.security.AuditEvent.EventType;
-import static org.apache.solr.servlet.ServletUtils.closeShield;
-import static org.apache.solr.servlet.ServletUtils.configExcludes;
-import static org.apache.solr.servlet.ServletUtils.excludedPath;
-
/**
* This filter looks at the incoming URL maps them to handlers defined in solrconfig.xml
*
* @since solr 1.2
*/
-// todo: get rid of this class entirely! Request dispatch is the container's responsibility. Much of what we have here
-// should be several separate but composable servlet Filters, wrapping multiple servlets that are more focused in
-// scope. This should become possible now that we have a ServletContextListener for startup/shutdown of CoreContainer
-// that sets up a service from which things like CoreContainer can be requested. (or better yet injected)
+// todo: get rid of this class entirely! Request dispatch is the container's responsibility. Much of
+// what we have here
+// should be several separate but composable servlet Filters, wrapping multiple servlets that are
+// more focused in
+// scope. This should become possible now that we have a ServletContextListener for
+// startup/shutdown of CoreContainer
+// that sets up a service from which things like CoreContainer can be requested. (or better yet
+// injected)
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/spelling/ConjunctionSolrSpellChecker.java
##########
@@ -102,122 +94,122 @@ public String init(NamedList<?> config, SolrCore core) {
initialized = true;
return dictionaryName;
}
-
+
@Override
public void build(SolrCore core, SolrIndexSearcher searcher) throws IOException {
for (SolrSpellChecker c : checkers) {
c.build(core, searcher);
}
}
-
+
@Override
- public SpellingResult getSuggestions(SpellingOptions options)
- throws IOException {
+ public SpellingResult getSuggestions(SpellingOptions options) throws IOException {
SpellingResult[] results = new SpellingResult[checkers.size()];
for (int i = 0; i < checkers.size(); i++) {
results[i] = checkers.get(i).getSuggestions(options);
}
return mergeCheckers(results, options.count);
}
-
+
@Override
- public SpellingResult mergeSuggestions(SpellCheckMergeData mergeData,
- int numSug, int count, boolean extendedResults) {
+ public SpellingResult mergeSuggestions(
+ SpellCheckMergeData mergeData, int numSug, int count, boolean extendedResults) {
SpellingResult[] results = new SpellingResult[checkers.size()];
for (int i = 0; i < checkers.size(); i++) {
- results[i] = checkers.get(i).mergeSuggestions(mergeData, numSug, count,
- extendedResults);
+ results[i] = checkers.get(i).mergeSuggestions(mergeData, numSug, count, extendedResults);
}
return mergeCheckers(results, numSug);
}
-
- //TODO: This just interleaves the results. In the future, we might want to let users give each checker its
+
+ // TODO: This just interleaves the results. In the future, we might want to let users give each
+ // checker its
// own weight and use that in combination to score & frequency to sort the results ?
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/spelling/AbstractLuceneSpellChecker.java
##########
@@ -210,62 +216,63 @@ protected IndexReader determineReader(IndexReader reader) {
@Override
public void reload(SolrCore core, SolrIndexSearcher searcher) throws IOException {
spellChecker.setSpellIndex(index);
-
}
/**
- * Initialize the {@link #index} variable based on the {@link #indexDir}. Does not actually create the spelling index.
+ * Initialize the {@link #index} variable based on the {@link #indexDir}. Does not actually create
+ * the spelling index.
*
* @throws IOException If there is a low-level I/O error.
*/
protected void initIndex() throws IOException {
if (indexDir != null) {
- // TODO: this is a workaround for SpellChecker repeatedly closing and opening a new IndexWriter while leaving readers open, which on
- // Windows causes problems because deleted files can't be opened. It would be better for SpellChecker to hold a single IW instance,
- // and close it on close, but Solr never seems to close its spell checkers. Wrapping as FilterDirectory prevents IndexWriter from
+ // TODO: this is a workaround for SpellChecker repeatedly closing and opening a new
+ // IndexWriter while leaving readers open, which on
+ // Windows causes problems because deleted files can't be opened. It would be better for
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/servlet/SolrRequestParsers.java
##########
@@ -721,75 +761,74 @@ public SolrParams parseParamsAndFillStreams(HttpServletRequest req, ArrayList<Co
return new MultiMapSolrParams(map);
}
-
@Override
- public SolrParams parseParamsAndFillStreams(HttpServletRequest req, ArrayList<ContentStream> streams ) throws Exception {
+ public SolrParams parseParamsAndFillStreams(
+ HttpServletRequest req, ArrayList<ContentStream> streams) throws Exception {
if (!isFormData(req)) {
- throw new SolrException(ErrorCode.BAD_REQUEST, "Not application/x-www-form-urlencoded content: " + req.getContentType());
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST,
+ "Not application/x-www-form-urlencoded content: " + req.getContentType());
}
return parseParamsAndFillStreams(req, streams, null);
}
- public static SolrException getParameterIncompatibilityException() {
- return new SolrException(ErrorCode.SERVER_ERROR,
- "Solr requires that request parameters sent using application/x-www-form-urlencoded " +
- "content-type can be read through the request input stream. Unfortunately, the " +
- "stream was empty / not available. This may be caused by another servlet filter calling " +
- "ServletRequest.getParameter*() before SolrDispatchFilter, please remove it."
- );
+ public static SolrException getParameterIncompatibilityException() {
+ return new SolrException(
+ ErrorCode.SERVER_ERROR,
+ "Solr requires that request parameters sent using application/x-www-form-urlencoded "
+ + "content-type can be read through the request input stream. Unfortunately, the "
+ + "stream was empty / not available. This may be caused by another servlet filter calling "
+ + "ServletRequest.getParameter*() before SolrDispatchFilter, please remove it.");
}
-
+
public boolean isFormData(HttpServletRequest req) {
String contentType = req.getContentType();
if (contentType != null) {
- int idx = contentType.indexOf( ';' );
- if( idx > 0 ) { // remove the charset definition "; charset=utf-8"
- contentType = contentType.substring( 0, idx );
+ int idx = contentType.indexOf(';');
+ if (idx > 0) { // remove the charset definition "; charset=utf-8"
+ contentType = contentType.substring(0, idx);
}
contentType = contentType.trim();
- if( "application/x-www-form-urlencoded".equalsIgnoreCase(contentType)) {
+ if ("application/x-www-form-urlencoded".equalsIgnoreCase(contentType)) {
return true;
}
}
return false;
}
}
-
- /**
- * The default Logic
- */
- static class StandardRequestParser implements SolrRequestParser
- {
+ /** The default Logic */
+ static class StandardRequestParser implements SolrRequestParser {
MultipartRequestParser multipart;
RawRequestParser raw;
FormDataRequestParser formdata;
- StandardRequestParser(MultipartRequestParser multi, RawRequestParser raw, FormDataRequestParser formdata)
- {
+ StandardRequestParser(
+ MultipartRequestParser multi, RawRequestParser raw, FormDataRequestParser formdata) {
this.multipart = multi;
this.raw = raw;
this.formdata = formdata;
}
@Override
- public SolrParams parseParamsAndFillStreams(final HttpServletRequest req, ArrayList<ContentStream> streams ) throws Exception {
+ public SolrParams parseParamsAndFillStreams(
+ final HttpServletRequest req, ArrayList<ContentStream> streams) throws Exception {
String contentType = req.getContentType();
String method = req.getMethod(); // No need to uppercase... HTTP verbs are case sensitive
String uri = req.getRequestURI();
boolean isV2 = getHttpSolrCall(req) instanceof V2HttpCall;
boolean isPost = "POST".equals(method);
- // SOLR-6787 changed the behavior of a POST without content type. Previously it would throw an exception,
+ // SOLR-6787 changed the behavior of a POST without content type. Previously it would throw
+ // an exception,
// but now it will use the raw request parser.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/servlet/SolrRequestParsers.java
##########
@@ -842,30 +881,40 @@ public SolrParams parseParamsAndFillStreams(final HttpServletRequest req, ArrayL
}
}
-
- private static final long WS_MASK=(1L<<' ')|(1L<<'\t')|(1L<<'\r')|(1L<<'\n')|(1L<<'#')|(1L<<'/')|(0x01); // set 1 bit so 0xA0 will be flagged as possible whitespace
+ private static final long WS_MASK =
+ (1L << ' ')
+ | (1L << '\t')
+ | (1L << '\r')
+ | (1L << '\n')
+ | (1L << '#')
+ | (1L << '/')
+ | (0x01); // set 1 bit so 0xA0 will be flagged as possible whitespace
/** Returns the parameter map if a different content type was auto-detected */
- private static SolrParams autodetect(HttpServletRequest req, ArrayList<ContentStream> streams, FastInputStream in) throws IOException {
+ private static SolrParams autodetect(
+ HttpServletRequest req, ArrayList<ContentStream> streams, FastInputStream in)
+ throws IOException {
String detectedContentType = null;
boolean shouldClose = true;
try {
- in.peek(); // should cause some bytes to be read
+ in.peek(); // should cause some bytes to be read
byte[] arr = in.getBuffer();
int pos = in.getPositionInBuffer();
int end = in.getEndInBuffer();
- for (int i = pos; i < end - 1; i++) { // we do "end-1" because we check "arr[i+1]" sometimes in the loop body
+ for (int i = pos;
+ i < end - 1;
+ i++) { // we do "end-1" because we check "arr[i+1]" sometimes in the loop body
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/servlet/SolrDispatchFilter.java
##########
@@ -284,44 +308,66 @@ private void authenticateRequest(HttpServletRequest request, HttpServletResponse
log.debug("Pass through PKI authentication endpoint");
return;
}
- // /solr/ (Admin UI) must be always open to allow displaying Admin UI with login page
+ // /solr/ (Admin UI) must be always open to allow displaying Admin UI with login page
if ("/solr/".equals(requestPath) || "/".equals(requestPath)) {
log.debug("Pass through Admin UI entry point");
return;
}
String header = request.getHeader(PKIAuthenticationPlugin.HEADER);
String headerV2 = request.getHeader(PKIAuthenticationPlugin.HEADER_V2);
- if ((header != null || headerV2 != null) && cores.getPkiAuthenticationSecurityBuilder() != null)
+ if ((header != null || headerV2 != null)
+ && cores.getPkiAuthenticationSecurityBuilder() != null)
authenticationPlugin = cores.getPkiAuthenticationSecurityBuilder();
try {
if (log.isDebugEnabled()) {
- log.debug("Request to authenticate: {}, domain: {}, port: {}", request, request.getLocalName(), request.getLocalPort());
+ log.debug(
+ "Request to authenticate: {}, domain: {}, port: {}",
+ request,
+ request.getLocalName(),
+ request.getLocalPort());
}
- // For legacy reasons, upon successful authentication this wants to call the chain's next filter, which
- // obfuscates the layout of the code since one usually expects to be able to find the call to doFilter()
- // in the implementation of javax.servlet.Filter. Supplying a trivial impl here to keep existing code happy
- // while making the flow clearer. Chain will be called after this method completes. Eventually auth all
- // moves to its own filter (hopefully). Most auth plugins simply return true after calling this anyway,
- // so they obviously don't care. Kerberos plugins seem to mostly use it to satisfy the api of a wrapped
- // instance of javax.servlet.Filter and neither of those seem to be doing anything fancy with the filter chain,
- // so this would seem to be a hack brought on by the fact that our auth code has been forced to be code
- // within dispatch filter, rather than being a filter itself. The HadoopAuthPlugin has a suspicious amount
- // of code after the call to doFilter() which seems to imply that anything in this chain can get executed before
- // authentication completes, and I can't figure out how that's a good idea in the first place.
- requestContinues = authenticationPlugin.authenticate(request, response, (req, rsp) -> {
- isAuthenticated.set(true);
- wrappedRequest.set((HttpServletRequest) req);
- });
+ // For legacy reasons, upon successful authentication this wants to call the chain's next
+ // filter, which
+ // obfuscates the layout of the code since one usually expects to be able to find the call
+ // to doFilter()
+ // in the implementation of javax.servlet.Filter. Supplying a trivial impl here to keep
+ // existing code happy
+ // while making the flow clearer. Chain will be called after this method completes.
+ // Eventually auth all
+ // moves to its own filter (hopefully). Most auth plugins simply return true after calling
+ // this anyway,
+ // so they obviously don't care. Kerberos plugins seem to mostly use it to satisfy the api
+ // of a wrapped
+ // instance of javax.servlet.Filter and neither of those seem to be doing anything fancy
+ // with the filter chain,
+ // so this would seem to be a hack brought on by the fact that our auth code has been
+ // forced to be code
+ // within dispatch filter, rather than being a filter itself. The HadoopAuthPlugin has a
+ // suspicious amount
+ // of code after the call to doFilter() which seems to imply that anything in this chain can
+ // get executed before
+ // authentication completes, and I can't figure out how that's a good idea in the first
+ // place.
+ requestContinues =
+ authenticationPlugin.authenticate(
+ request,
+ response,
+ (req, rsp) -> {
+ isAuthenticated.set(true);
+ wrappedRequest.set((HttpServletRequest) req);
+ });
} catch (Exception e) {
log.info("Error authenticating", e);
throw new SolrException(ErrorCode.SERVER_ERROR, "Error during request authentication, ", e);
}
}
// requestContinues is an optional short circuit, thus we still need to check isAuthenticated.
- // This is because the AuthenticationPlugin doesn't always have enough information to determine if
+ // This is because the AuthenticationPlugin doesn't always have enough information to determine
+ // if
// it should short circuit, e.g. the Kerberos Authentication Filter will send an error and not
// call later filters in chain, but doesn't throw an exception. We could force each Plugin
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/spelling/suggest/SolrSuggester.java
##########
@@ -94,48 +89,51 @@
private Analyzer contextFilterQueryAnalyzer;
/**
- * Uses the <code>config</code> and the <code>core</code> to initialize the underlying
- * Lucene suggester
- * */
+ * Uses the <code>config</code> and the <code>core</code> to initialize the underlying Lucene
+ * suggester
+ */
public String init(NamedList<?> config, SolrCore core) {
log.info("init: {}", config);
-
+
// read the config
- name = config.get(NAME) != null ? (String) config.get(NAME)
- : DEFAULT_DICT_NAME;
+ name = config.get(NAME) != null ? (String) config.get(NAME) : DEFAULT_DICT_NAME;
sourceLocation = (String) config.get(LOCATION);
lookupImpl = (String) config.get(LOOKUP_IMPL);
dictionaryImpl = (String) config.get(DICTIONARY_IMPL);
- String store = (String)config.get(STORE_DIR);
+ String store = (String) config.get(STORE_DIR);
if (lookupImpl == null) {
lookupImpl = LookupFactory.DEFAULT_FILE_BASED_DICT;
log.info("No {} parameter was provided falling back to {}", LOOKUP_IMPL, lookupImpl);
}
- contextFilterQueryAnalyzer = new TokenizerChain(new StandardTokenizerFactory(Collections.emptyMap()), null);
+ contextFilterQueryAnalyzer =
+ new TokenizerChain(new StandardTokenizerFactory(Collections.emptyMap()), null);
// initialize appropriate lookup instance
factory = core.getResourceLoader().newInstance(lookupImpl, LookupFactory.class);
lookup = factory.create(config, core);
-
+
if (lookup instanceof Closeable) {
- core.addCloseHook(new CloseHook() {
- @Override
- public void preClose(SolrCore core) {
- try {
- ((Closeable) lookup).close();
- } catch (IOException e) {
- log.warn("Could not close the suggester lookup.", e);
- }
- }
- });
+ core.addCloseHook(
+ new CloseHook() {
+ @Override
+ public void preClose(SolrCore core) {
+ try {
+ ((Closeable) lookup).close();
+ } catch (IOException e) {
+ log.warn("Could not close the suggester lookup.", e);
+ }
+ }
+ });
}
// if store directory is provided make it or load up the lookup with its content
if (store != null && !store.isEmpty()) {
storeDir = Path.of(store);
- storeDir = Path.of(core.getDataDir()).resolve(storeDir); // if store dir is absolute this won't change it
+ storeDir =
+ Path.of(core.getDataDir())
+ .resolve(storeDir); // if store dir is absolute this won't change it
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/servlet/SolrDispatchFilter.java
##########
@@ -284,44 +308,66 @@ private void authenticateRequest(HttpServletRequest request, HttpServletResponse
log.debug("Pass through PKI authentication endpoint");
return;
}
- // /solr/ (Admin UI) must be always open to allow displaying Admin UI with login page
+ // /solr/ (Admin UI) must be always open to allow displaying Admin UI with login page
if ("/solr/".equals(requestPath) || "/".equals(requestPath)) {
log.debug("Pass through Admin UI entry point");
return;
}
String header = request.getHeader(PKIAuthenticationPlugin.HEADER);
String headerV2 = request.getHeader(PKIAuthenticationPlugin.HEADER_V2);
- if ((header != null || headerV2 != null) && cores.getPkiAuthenticationSecurityBuilder() != null)
+ if ((header != null || headerV2 != null)
+ && cores.getPkiAuthenticationSecurityBuilder() != null)
authenticationPlugin = cores.getPkiAuthenticationSecurityBuilder();
try {
if (log.isDebugEnabled()) {
- log.debug("Request to authenticate: {}, domain: {}, port: {}", request, request.getLocalName(), request.getLocalPort());
+ log.debug(
+ "Request to authenticate: {}, domain: {}, port: {}",
+ request,
+ request.getLocalName(),
+ request.getLocalPort());
}
- // For legacy reasons, upon successful authentication this wants to call the chain's next filter, which
- // obfuscates the layout of the code since one usually expects to be able to find the call to doFilter()
- // in the implementation of javax.servlet.Filter. Supplying a trivial impl here to keep existing code happy
- // while making the flow clearer. Chain will be called after this method completes. Eventually auth all
- // moves to its own filter (hopefully). Most auth plugins simply return true after calling this anyway,
- // so they obviously don't care. Kerberos plugins seem to mostly use it to satisfy the api of a wrapped
- // instance of javax.servlet.Filter and neither of those seem to be doing anything fancy with the filter chain,
- // so this would seem to be a hack brought on by the fact that our auth code has been forced to be code
- // within dispatch filter, rather than being a filter itself. The HadoopAuthPlugin has a suspicious amount
- // of code after the call to doFilter() which seems to imply that anything in this chain can get executed before
- // authentication completes, and I can't figure out how that's a good idea in the first place.
- requestContinues = authenticationPlugin.authenticate(request, response, (req, rsp) -> {
- isAuthenticated.set(true);
- wrappedRequest.set((HttpServletRequest) req);
- });
+ // For legacy reasons, upon successful authentication this wants to call the chain's next
+ // filter, which
+ // obfuscates the layout of the code since one usually expects to be able to find the call
+ // to doFilter()
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/spelling/suggest/Suggester.java
##########
@@ -101,26 +98,29 @@ public String init(NamedList<?> config, SolrCore core) {
}
factory = core.getResourceLoader().newInstance(lookupImpl, LookupFactory.class);
-
+
lookup = factory.create(config, core);
- core.addCloseHook(new CloseHook() {
- @Override
- public void preClose(SolrCore core) {
- if (lookup instanceof Closeable) {
- try {
- ((Closeable) lookup).close();
- } catch (IOException e) {
- log.warn("Could not close the suggester lookup.", e);
+ core.addCloseHook(
+ new CloseHook() {
+ @Override
+ public void preClose(SolrCore core) {
+ if (lookup instanceof Closeable) {
+ try {
+ ((Closeable) lookup).close();
+ } catch (IOException e) {
+ log.warn("Could not close the suggester lookup.", e);
+ }
+ }
}
- }
- }
- });
+ });
// TODO: Duplicated with SolrSuggester
- String store = (String)config.get(STORE_DIR);
+ String store = (String) config.get(STORE_DIR);
if (store != null) {
storeDir = Path.of(store);
- storeDir = Path.of(core.getDataDir()).resolve(storeDir); // if store dir is absolute this won't change it
+ storeDir =
+ Path.of(core.getDataDir())
+ .resolve(storeDir); // if store dir is absolute this won't change it
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/uninverting/FieldCacheImpl.java
##########
@@ -1191,45 +1253,50 @@ public int length() {
// TODO: this if DocTermsIndex was already created, we
// should share it...
- public SortedSetDocValues getDocTermOrds(LeafReader reader, String field, BytesRef prefix) throws IOException {
+ public SortedSetDocValues getDocTermOrds(LeafReader reader, String field, BytesRef prefix)
+ throws IOException {
// not a general purpose filtering mechanism...
assert prefix == null || prefix == INT32_TERM_PREFIX || prefix == INT64_TERM_PREFIX;
-
+
SortedSetDocValues dv = reader.getSortedSetDocValues(field);
if (dv != null) {
return dv;
}
-
+
SortedDocValues sdv = reader.getSortedDocValues(field);
if (sdv != null) {
return DocValues.singleton(sdv);
}
-
+
final FieldInfo info = reader.getFieldInfos().fieldInfo(field);
if (info == null) {
return DocValues.emptySortedSet();
} else if (info.getDocValuesType() != DocValuesType.NONE) {
- throw new IllegalStateException("Type mismatch: " + field + " was indexed as " + info.getDocValuesType());
+ throw new IllegalStateException(
+ "Type mismatch: " + field + " was indexed as " + info.getDocValuesType());
} else if (info.getIndexOptions() == IndexOptions.NONE) {
return DocValues.emptySortedSet();
}
-
+
// ok we need to uninvert. check if we can optimize a bit.
-
+
Terms terms = reader.terms(field);
if (terms == null) {
return DocValues.emptySortedSet();
} else {
// if #postings = #docswithfield we know that the field is "single valued enough".
- // it's possible the same term might appear twice in the same document, but SORTED_SET discards frequency.
- // it's still ok with filtering (which we limit to numerics), it just means precisionStep = Inf
+ // it's possible the same term might appear twice in the same document, but SORTED_SET
+ // discards frequency.
+ // it's still ok with filtering (which we limit to numerics), it just means precisionStep =
+ // Inf
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/uninverting/DocTermOrds.java
##########
@@ -528,80 +534,85 @@ protected void uninvert(final LeafReader reader, Bits liveDocs, final BytesRef t
System.arraycopy(target, 0, newtarget, 0, pos);
target = newtarget;
}
-
+
tnums[pass] = target;
- if ((pass << 16) > maxDoc)
- break;
+ if ((pass << 16) > maxDoc) break;
}
-
}
indexedTermsArray = indexedTerms.toArray(new BytesRef[indexedTerms.size()]);
long endTime = System.nanoTime();
- total_time = (int) TimeUnit.MILLISECONDS.convert(endTime-startTime, TimeUnit.NANOSECONDS);
- phase1_time = (int) TimeUnit.MILLISECONDS.convert(midPoint-startTime, TimeUnit.NANOSECONDS);
+ total_time = (int) TimeUnit.MILLISECONDS.convert(endTime - startTime, TimeUnit.NANOSECONDS);
+ phase1_time = (int) TimeUnit.MILLISECONDS.convert(midPoint - startTime, TimeUnit.NANOSECONDS);
}
/** Number of bytes to represent an unsigned int as a vint. */
private static int vIntSize(int x) {
- // Tests outside of this code base shows that the previous conditional-based vIntSize is fairly slow until
+ // Tests outside of this code base shows that the previous conditional-based vIntSize is fairly
+ // slow until
// JITted and still about 1/3 slower after JIT than the numberOfLeadingZeros version below.
Review comment:
Fix this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] risdenk commented on pull request #705: SOLR-14920: Spotless formatting for core - non-test only
Posted by GitBox <gi...@apache.org>.
risdenk commented on pull request #705:
URL: https://github.com/apache/solr/pull/705#issuecomment-1056050253
Woot finally made all the review fixes. This is ready for review :D
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] risdenk commented on a change in pull request #705: SOLR-14920: Spotless formatting for core - non-test only
Posted by GitBox <gi...@apache.org>.
risdenk commented on a change in pull request #705:
URL: https://github.com/apache/solr/pull/705#discussion_r815471386
##########
File path: solr/core/src/java/org/apache/solr/cloud/ZkController.java
##########
@@ -317,144 +329,159 @@ public ZkController(final CoreContainer cc, String zkServerAddress, int zkClient
String zkCredentialsProviderClass = cloudConfig.getZkCredentialsProviderClass();
if (zkCredentialsProviderClass != null && zkCredentialsProviderClass.trim().length() > 0) {
- strat.setZkCredentialsToAddAutomatically(cc.getResourceLoader().newInstance(zkCredentialsProviderClass, ZkCredentialsProvider.class));
+ strat.setZkCredentialsToAddAutomatically(
+ cc.getResourceLoader()
+ .newInstance(zkCredentialsProviderClass, ZkCredentialsProvider.class));
} else {
strat.setZkCredentialsToAddAutomatically(new DefaultZkCredentialsProvider());
}
addOnReconnectListener(getConfigDirListener());
- zkClient = new SolrZkClient(zkServerAddress, clientTimeout, zkClientConnectTimeout, strat,
- // on reconnect, reload cloud info
- new OnReconnect() {
+ zkClient =
+ new SolrZkClient(
+ zkServerAddress,
+ clientTimeout,
+ zkClientConnectTimeout,
+ strat,
+ // on reconnect, reload cloud info
+ new OnReconnect() {
+
+ @Override
+ public void command() throws SessionExpiredException {
+ log.info(
+ "ZooKeeper session re-connected ... refreshing core states after session expiration.");
+ clearZkCollectionTerms();
+ try {
+ // recreate our watchers first so that they exist even on any problems below
+ zkStateReader.createClusterStateWatchersAndUpdate();
+
+ // this is troublesome - we dont want to kill anything the old
+ // leader accepted
+ // though I guess sync will likely get those updates back? But
+ // only if
+ // he is involved in the sync, and he certainly may not be
+ // ExecutorUtil.shutdownAndAwaitTermination(cc.getCmdDistribExecutor());
+ // we need to create all of our lost watches
+
+ // seems we dont need to do this again...
+ // Overseer.createClientNodes(zkClient, getNodeName());
+
+ // start the overseer first as following code may need it's processing
+ if (!zkRunOnly) {
+ ElectionContext context =
+ new OverseerElectionContext(zkClient, overseer, getNodeName());
+
+ ElectionContext prevContext = overseerElector.getContext();
+ if (prevContext != null) {
+ prevContext.cancelElection();
+ prevContext.close();
+ }
- @Override
- public void command() throws SessionExpiredException {
- log.info("ZooKeeper session re-connected ... refreshing core states after session expiration.");
- clearZkCollectionTerms();
- try {
- // recreate our watchers first so that they exist even on any problems below
- zkStateReader.createClusterStateWatchersAndUpdate();
-
- // this is troublesome - we dont want to kill anything the old
- // leader accepted
- // though I guess sync will likely get those updates back? But
- // only if
- // he is involved in the sync, and he certainly may not be
- // ExecutorUtil.shutdownAndAwaitTermination(cc.getCmdDistribExecutor());
- // we need to create all of our lost watches
-
- // seems we dont need to do this again...
- // Overseer.createClientNodes(zkClient, getNodeName());
-
- // start the overseer first as following code may need it's processing
- if (!zkRunOnly) {
- ElectionContext context = new OverseerElectionContext(zkClient,
- overseer, getNodeName());
-
- ElectionContext prevContext = overseerElector.getContext();
- if (prevContext != null) {
- prevContext.cancelElection();
- prevContext.close();
- }
+ overseerElector.setup(context);
- overseerElector.setup(context);
+ if (cc.nodeRoles.isOverseerAllowedOrPreferred()) {
+ overseerElector.joinElection(context, true);
+ }
+ }
- if (cc.nodeRoles.isOverseerAllowedOrPreferred()) {
- overseerElector.joinElection(context, true);
- }
- }
+ cc.cancelCoreRecoveries();
- cc.cancelCoreRecoveries();
-
- try {
- registerAllCoresAsDown(descriptorsSupplier, false);
- } catch (SessionExpiredException e) {
- // zk has to reconnect and this will all be tried again
- throw e;
- } catch (Exception e) {
- // this is really best effort - in case of races or failure cases where we now need to be the leader, if anything fails,
- // just continue
- log.warn("Exception while trying to register all cores as DOWN", e);
- }
-
- // we have to register as live first to pick up docs in the buffer
- createEphemeralLiveNode();
-
- List<CoreDescriptor> descriptors = descriptorsSupplier.get();
- // re register all descriptors
- ExecutorService executorService = (cc != null) ? cc.getCoreZkRegisterExecutorService() : null;
- if (descriptors != null) {
- for (CoreDescriptor descriptor : descriptors) {
- // TODO: we need to think carefully about what happens when it
- // was
- // a leader that was expired - as well as what to do about
- // leaders/overseers
- // with connection loss
try {
- // unload solrcores that have been 'failed over'
- throwErrorIfReplicaReplaced(descriptor);
+ registerAllCoresAsDown(descriptorsSupplier, false);
+ } catch (SessionExpiredException e) {
+ // zk has to reconnect and this will all be tried again
+ throw e;
+ } catch (Exception e) {
+ // this is really best effort - in case of races or failure cases where we now
+ // need to be the leader, if anything fails,
+ // just continue
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/ZkController.java
##########
@@ -2620,19 +2895,27 @@ public boolean checkIfCoreNodeNameAlreadyExists(CoreDescriptor dcore) {
public void publishNodeAsDown(String nodeName) {
log.info("Publish node={} as DOWN", nodeName);
if (distributedClusterStateUpdater.isDistributedStateUpdate()) {
- // Note that with the current implementation, when distributed cluster state updates are enabled, we mark the node
- // down synchronously from this thread, whereas the Overseer cluster state update frees this thread right away and
+ // Note that with the current implementation, when distributed cluster state updates are
+ // enabled, we mark the node
+ // down synchronously from this thread, whereas the Overseer cluster state update frees this
+ // thread right away and
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/api/collections/BackupCmd.java
##########
@@ -70,74 +69,97 @@ public BackupCmd(CollectionCommandContext ccc) {
}
@Override
- public void call(ClusterState state, ZkNodeProps message, NamedList<Object> results) throws Exception {
+ public void call(ClusterState state, ZkNodeProps message, NamedList<Object> results)
+ throws Exception {
String extCollectionName = message.getStr(COLLECTION_PROP);
boolean followAliases = message.getBool(FOLLOW_ALIASES, false);
String collectionName;
if (followAliases) {
- collectionName = ccc.getSolrCloudManager().getClusterStateProvider().resolveSimpleAlias(extCollectionName);
+ collectionName =
+ ccc.getSolrCloudManager().getClusterStateProvider().resolveSimpleAlias(extCollectionName);
} else {
collectionName = extCollectionName;
}
String backupName = message.getStr(NAME);
String repo = message.getStr(CoreAdminParams.BACKUP_REPOSITORY);
boolean incremental = message.getBool(CoreAdminParams.BACKUP_INCREMENTAL, true);
- String configName = ccc.getSolrCloudManager().getClusterStateProvider().getCollection(collectionName).getConfigName();
+ String configName =
+ ccc.getSolrCloudManager()
+ .getClusterStateProvider()
+ .getCollection(collectionName)
+ .getConfigName();
- BackupProperties backupProperties = BackupProperties.create(backupName, collectionName,
- extCollectionName, configName);
+ BackupProperties backupProperties =
+ BackupProperties.create(backupName, collectionName, extCollectionName, configName);
CoreContainer cc = ccc.getCoreContainer();
try (BackupRepository repository = cc.newBackupRepository(repo)) {
// Backup location
URI location = repository.createDirectoryURI(message.getStr(CoreAdminParams.BACKUP_LOCATION));
- final URI backupUri = createAndValidateBackupPath(repository, incremental, location, backupName, collectionName);
-
- BackupManager backupMgr = (incremental) ?
- BackupManager.forIncrementalBackup(repository, ccc.getZkStateReader(), backupUri) :
- BackupManager.forBackup(repository, ccc.getZkStateReader(), backupUri);
-
- String strategy = message.getStr(CollectionAdminParams.INDEX_BACKUP_STRATEGY, CollectionAdminParams.COPY_FILES_STRATEGY);
+ final URI backupUri =
+ createAndValidateBackupPath(
+ repository, incremental, location, backupName, collectionName);
+
+ BackupManager backupMgr =
+ (incremental)
+ ? BackupManager.forIncrementalBackup(repository, ccc.getZkStateReader(), backupUri)
+ : BackupManager.forBackup(repository, ccc.getZkStateReader(), backupUri);
+
+ String strategy =
+ message.getStr(
+ CollectionAdminParams.INDEX_BACKUP_STRATEGY,
+ CollectionAdminParams.COPY_FILES_STRATEGY);
switch (strategy) {
- case CollectionAdminParams.COPY_FILES_STRATEGY: {
- if (incremental) {
- try {
- incrementalCopyIndexFiles(backupUri, collectionName, message, results, backupProperties, backupMgr);
- } catch (SolrException e) {
- log.error("Error happened during incremental backup for collection: {}", collectionName, e);
- CollectionHandlingUtils.cleanBackup(repository, backupUri, backupMgr.getBackupId(), ccc);
- throw e;
+ case CollectionAdminParams.COPY_FILES_STRATEGY:
+ {
+ if (incremental) {
+ try {
+ incrementalCopyIndexFiles(
+ backupUri, collectionName, message, results, backupProperties, backupMgr);
+ } catch (SolrException e) {
+ log.error(
+ "Error happened during incremental backup for collection: {}",
+ collectionName,
+ e);
+ CollectionHandlingUtils.cleanBackup(
+ repository, backupUri, backupMgr.getBackupId(), ccc);
+ throw e;
+ }
+ } else {
+ copyIndexFiles(
+ backupUri, collectionName, message, results, backupProperties, backupMgr);
}
- } else {
- copyIndexFiles(backupUri, collectionName, message, results, backupProperties, backupMgr);
+ break;
+ }
+ case CollectionAdminParams.NO_INDEX_BACKUP_STRATEGY:
+ {
+ NamedList<Object> response = new SimpleOrderedMap<>();
+ response.add("collection", collectionName);
+ response.add("startTime", backupProperties.getStartTime());
+ results.add("response", response);
+ break;
}
- break;
- }
- case CollectionAdminParams.NO_INDEX_BACKUP_STRATEGY: {
- NamedList<Object> response = new SimpleOrderedMap<>();
- response.add("collection", collectionName);
- response.add("startTime", backupProperties.getStartTime());
- results.add("response", response);
- break;
- }
}
log.info("Starting to backup ZK data for backupName={}", backupName);
- //Download the configs
+ // Download the configs
backupMgr.downloadConfigDir(configName, cc.getConfigSetService());
- //Save the collection's state. Can be part of the monolithic clusterstate.json or a individual state.json
- //Since we don't want to distinguish we extract the state and back it up as a separate json
- DocCollection collectionState = ccc.getZkStateReader().getClusterState().getCollection(collectionName);
+ // Save the collection's state. Can be part of the monolithic clusterstate.json or a
+ // individual state.json
+ // Since we don't want to distinguish we extract the state and back it up as a separate json
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/SyncStrategy.java
##########
@@ -135,70 +142,100 @@ public SyncStrategy(CoreContainer cc) {
log.info("We have been closed, won't attempt to sync replicas back to leader");
return PeerSync.PeerSyncResult.failure();
}
-
+
if (success) {
log.info("Sync Success - now sync replicas to me");
-
- syncToMe(zkController, collection, shardId, leaderProps, core.getCoreDescriptor(), core.getUpdateHandler().getUpdateLog().getNumRecordsToKeep());
-
+
+ syncToMe(
+ zkController,
+ collection,
+ shardId,
+ leaderProps,
+ core.getCoreDescriptor(),
+ core.getUpdateHandler().getUpdateLog().getNumRecordsToKeep());
+
} else {
log.info("Leader's attempt to sync with shard failed, moving to the next candidate");
// lets see who seems ahead...
}
-
+
} catch (Exception e) {
SolrException.log(log, "Sync Failed", e);
}
-
+
return result == null ? PeerSync.PeerSyncResult.failure() : result;
}
-
- private PeerSync.PeerSyncResult syncWithReplicas(ZkController zkController, SolrCore core,
- ZkNodeProps props, String collection, String shardId, boolean peerSyncOnlyWithActive) throws Exception {
- List<ZkCoreNodeProps> nodes = zkController.getZkStateReader()
- .getReplicaProps(collection, shardId,core.getCoreDescriptor().getCloudDescriptor().getCoreNodeName());
-
+
+ private PeerSync.PeerSyncResult syncWithReplicas(
+ ZkController zkController,
+ SolrCore core,
+ ZkNodeProps props,
+ String collection,
+ String shardId,
+ boolean peerSyncOnlyWithActive)
+ throws Exception {
+ List<ZkCoreNodeProps> nodes =
+ zkController
+ .getZkStateReader()
+ .getReplicaProps(
+ collection,
+ shardId,
+ core.getCoreDescriptor().getCloudDescriptor().getCoreNodeName());
+
if (isClosed) {
log.info("We have been closed, won't sync with replicas");
return PeerSync.PeerSyncResult.failure();
}
-
+
if (nodes == null) {
// I have no replicas
return PeerSync.PeerSyncResult.success();
}
-
+
List<String> syncWith = new ArrayList<>(nodes.size());
for (ZkCoreNodeProps node : nodes) {
syncWith.add(node.getCoreUrl());
}
-
+
// if we can't reach a replica for sync, we still consider the overall sync a success
// TODO: as an assurance, we should still try and tell the sync nodes that we couldn't reach
// to recover once more?
- // Fingerprinting here is off because the we currently rely on having at least one of the nodes return "true", and if replicas are out-of-sync
- // we still need to pick one as leader. A followup sync from the replica to the new leader (with fingerprinting on) should then fail and
+ // Fingerprinting here is off because the we currently rely on having at least one of the nodes
+ // return "true", and if replicas are out-of-sync
+ // we still need to pick one as leader. A followup sync from the replica to the new leader
+ // (with fingerprinting on) should then fail and
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/api/collections/CreateAliasCmd.java
##########
@@ -71,31 +69,39 @@ public void call(ClusterState state, ZkNodeProps message, NamedList<Object> resu
//
// THIS IS A KLUDGE.
//
- // Solr's view of the cluster is eventually consistent. *Eventually* all nodes and CloudSolrClients will be aware of
- // alias changes, but not immediately. If a newly created alias is queried, things should work right away since Solr
- // will attempt to see if it needs to get the latest aliases when it can't otherwise resolve the name. However
+ // Solr's view of the cluster is eventually consistent. *Eventually* all nodes and
+ // CloudSolrClients will be aware of
+ // alias changes, but not immediately. If a newly created alias is queried, things should work
+ // right away since Solr
+ // will attempt to see if it needs to get the latest aliases when it can't otherwise resolve the
+ // name. However
// modifications to an alias will take some time.
//
- // We could levy this requirement on the client but they would probably always add an obligatory sleep, which is
- // just kicking the can down the road. Perhaps ideally at this juncture here we could somehow wait until all
+ // We could levy this requirement on the client but they would probably always add an obligatory
+ // sleep, which is
+ // just kicking the can down the road. Perhaps ideally at this juncture here we could somehow
+ // wait until all
// Solr nodes in the cluster have the latest aliases?
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/api/collections/CreateCollectionCmd.java
##########
@@ -128,75 +133,114 @@ public void call(ClusterState clusterState, ZkNodeProps message, NamedList<Objec
ZkStateReader zkStateReader = ccc.getZkStateReader();
message.getProperties().put(COLL_CONF, configName);
- Map<String,String> collectionParams = new HashMap<>();
- Map<String,Object> collectionProps = message.getProperties();
+ Map<String, String> collectionParams = new HashMap<>();
+ Map<String, Object> collectionProps = message.getProperties();
for (Map.Entry<String, Object> entry : collectionProps.entrySet()) {
String propName = entry.getKey();
if (propName.startsWith(ZkController.COLLECTION_PARAM_PREFIX)) {
- collectionParams.put(propName.substring(ZkController.COLLECTION_PARAM_PREFIX.length()), (String) entry.getValue());
+ collectionParams.put(
+ propName.substring(ZkController.COLLECTION_PARAM_PREFIX.length()),
+ (String) entry.getValue());
}
}
- createCollectionZkNode(ccc.getSolrCloudManager().getDistribStateManager(), collectionName, collectionParams, ccc.getCoreContainer().getConfigSetService());
+ createCollectionZkNode(
+ ccc.getSolrCloudManager().getDistribStateManager(),
+ collectionName,
+ collectionParams,
+ ccc.getCoreContainer().getConfigSetService());
- // Note that in code below there are two main execution paths: Overseer based cluster state updates and distributed
+ // Note that in code below there are two main execution paths: Overseer based cluster state
+ // updates and distributed
// cluster state updates (look for isDistributedStateUpdate() conditions).
//
- // PerReplicaStates (PRS) collections follow a hybrid approach. Even when the cluster is Overseer cluster state update based,
- // these collections are created locally then the cluster state updater is notified (look for usage of RefreshCollectionMessage).
- // This explains why PRS collections have less diverging execution paths between distributed or Overseer based cluster state updates.
+ // PerReplicaStates (PRS) collections follow a hybrid approach. Even when the cluster is
+ // Overseer cluster state update based,
+ // these collections are created locally then the cluster state updater is notified (look for
+ // usage of RefreshCollectionMessage).
+ // This explains why PRS collections have less diverging execution paths between distributed
+ // or Overseer based cluster state updates.
if (isPRS) {
- // In case of a PRS collection, create the collection structure directly instead of resubmitting
+ // In case of a PRS collection, create the collection structure directly instead of
+ // resubmitting
// to the overseer queue.
// TODO: Consider doing this for all collections, not just the PRS collections.
// TODO comment above achieved by switching the cluster to distributed state updates
- // This code directly updates Zookeeper by creating the collection state.json. It is compatible with both distributed
+ // This code directly updates Zookeeper by creating the collection state.json. It is
+ // compatible with both distributed
// cluster state updates and Overseer based cluster state updates.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/api/collections/CreateCollectionCmd.java
##########
@@ -347,9 +444,11 @@ public void call(ClusterState clusterState, ZkNodeProps message, NamedList<Objec
} else {
failure = true;
}
- // When cluster state updates are distributed, Overseer state updater is not used and doesn't have to be notified
+ // When cluster state updates are distributed, Overseer state updater is not used and
+ // doesn't have to be notified
// of a new collection created elsewhere (which is how all collections are created).
- // Note it is likely possibly to skip the the whole if (isPRS) bloc, but keeping distributed state updates as
+ // Note it is likely possibly to skip the the whole if (isPRS) bloc, but keeping distributed
+ // state updates as
// close in behavior to Overseer state updates for now.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/api/collections/CreateSnapshotCmd.java
##########
@@ -176,24 +193,38 @@ public void call(ClusterState state, ZkNodeProps message, NamedList<Object> resu
}
if (failedShards.isEmpty()) { // No failures.
- CollectionSnapshotMetaData meta = new CollectionSnapshotMetaData(commitName, SnapshotStatus.Successful, creationDate, replicas);
+ CollectionSnapshotMetaData meta =
+ new CollectionSnapshotMetaData(
+ commitName, SnapshotStatus.Successful, creationDate, replicas);
SolrSnapshotManager.updateCollectionLevelSnapshot(zkClient, collectionName, meta);
if (log.isInfoEnabled()) {
- log.info("Saved following snapshot information for collection={} with commitName={} in Zookeeper : {}", collectionName,
- commitName, meta.toNamedList());
+ log.info(
+ "Saved following snapshot information for collection={} with commitName={} in Zookeeper : {}",
+ collectionName,
+ commitName,
+ meta.toNamedList());
}
} else {
- log.warn("Failed to create a snapshot for collection {} with commitName = {}. Snapshot could not be captured for following shards {}",
- collectionName, commitName, failedShards);
- // Update the ZK meta-data to include only cores with the snapshot. This will enable users to figure out
+ log.warn(
+ "Failed to create a snapshot for collection {} with commitName = {}. Snapshot could not be captured for following shards {}",
+ collectionName,
+ commitName,
+ failedShards);
+ // Update the ZK meta-data to include only cores with the snapshot. This will enable users to
+ // figure out
// which cores have the named snapshot.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/ZkController.java
##########
@@ -1334,54 +1470,66 @@ private String getLeader(final CloudDescriptor cloudDesc, int timeoutms) {
// cluster state node that won't be updated for a moment
String leaderUrl;
try {
- leaderUrl = getLeaderProps(collection, cloudDesc.getShardId(), timeoutms)
- .getCoreUrl();
+ leaderUrl = getLeaderProps(collection, cloudDesc.getShardId(), timeoutms).getCoreUrl();
// now wait until our currently cloud state contains the latest leader
- String clusterStateLeaderUrl = zkStateReader.getLeaderUrl(collection,
- shardId, timeoutms * 2); // since we found it in zk, we are willing to
+ String clusterStateLeaderUrl =
+ zkStateReader.getLeaderUrl(
+ collection, shardId, timeoutms * 2); // since we found it in zk, we are willing to
// wait a while to find it in state
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/api/collections/DeleteCollectionCmd.java
##########
@@ -121,21 +126,31 @@ public void call(ClusterState state, ZkNodeProps message, NamedList<Object> resu
ZkNodeProps internalMsg = message.plus(NAME, collection);
- List<Replica> failedReplicas = CollectionHandlingUtils.collectionCmd(internalMsg, params, results, null, asyncId, okayExceptions, ccc, state);
+ List<Replica> failedReplicas =
+ CollectionHandlingUtils.collectionCmd(
+ internalMsg, params, results, null, asyncId, okayExceptions, ccc, state);
for (Replica failedReplica : failedReplicas) {
- boolean isSharedFS = failedReplica.getBool(ZkStateReader.SHARED_STORAGE_PROP, false) && failedReplica.get("dataDir") != null;
+ boolean isSharedFS =
+ failedReplica.getBool(ZkStateReader.SHARED_STORAGE_PROP, false)
+ && failedReplica.get("dataDir") != null;
if (isSharedFS) {
- // if the replica use a shared FS and it did not receive the unload message, then counter node should not be removed
- // because when a new collection with same name is created, new replicas may reuse the old dataDir
+ // if the replica use a shared FS and it did not receive the unload message, then counter
+ // node should not be removed
+ // because when a new collection with same name is created, new replicas may reuse the old
+ // dataDir
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/ZkController.java
##########
@@ -1568,15 +1727,18 @@ public void publish(final CoreDescriptor cd, final Replica.State state, boolean
}
}
} catch (SolrCoreInitializationException ex) {
- // The core had failed to initialize (in a previous request, not this one), hence nothing to do here.
+ // The core had failed to initialize (in a previous request, not this one), hence nothing to
+ // do here.
if (log.isInfoEnabled()) {
log.info("The core '{}' had failed to initialize before.", cd.getName());
}
}
// pull replicas are excluded because their terms are not considered
- if (state == Replica.State.RECOVERING && cd.getCloudDescriptor().getReplicaType() != Type.PULL) {
- // state is used by client, state of replica can change from RECOVERING to DOWN without needed to finish recovery
+ if (state == Replica.State.RECOVERING
+ && cd.getCloudDescriptor().getReplicaType() != Type.PULL) {
+ // state is used by client, state of replica can change from RECOVERING to DOWN without
+ // needed to finish recovery
// by calling this we will know that a replica actually finished recovery or not
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/api/collections/AliasCmd.java
##########
@@ -63,22 +70,27 @@ protected AliasCmd(CollectionCommandContext ccc) {
}
}
if (createReqParams.get(COLL_CONF) == null) {
- throw new SolrException(SolrException.ErrorCode.BAD_REQUEST,
- "We require an explicit " + COLL_CONF);
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST, "We require an explicit " + COLL_CONF);
}
createReqParams.set(NAME, createCollName);
- // a CollectionOperation reads params and produces a message (Map) that is supposed to be sent to the Overseer.
- // Although we could create the Map without it, there are a fair amount of rules we don't want to reproduce.
- final Map<String, Object> createMsgMap = CollectionsHandler.CollectionOperation.CREATE_OP.execute(
- new LocalSolrQueryRequest(null, createReqParams),
- null,
- ccc.getCoreContainer().getCollectionsHandler());
+ // a CollectionOperation reads params and produces a message (Map) that is supposed to be sent
+ // to the Overseer.
+ // Although we could create the Map without it, there are a fair amount of rules we don't want
+ // to reproduce.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/api/collections/AliasCmd.java
##########
@@ -63,22 +70,27 @@ protected AliasCmd(CollectionCommandContext ccc) {
}
}
if (createReqParams.get(COLL_CONF) == null) {
- throw new SolrException(SolrException.ErrorCode.BAD_REQUEST,
- "We require an explicit " + COLL_CONF);
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST, "We require an explicit " + COLL_CONF);
}
createReqParams.set(NAME, createCollName);
- // a CollectionOperation reads params and produces a message (Map) that is supposed to be sent to the Overseer.
- // Although we could create the Map without it, there are a fair amount of rules we don't want to reproduce.
- final Map<String, Object> createMsgMap = CollectionsHandler.CollectionOperation.CREATE_OP.execute(
- new LocalSolrQueryRequest(null, createReqParams),
- null,
- ccc.getCoreContainer().getCollectionsHandler());
+ // a CollectionOperation reads params and produces a message (Map) that is supposed to be sent
+ // to the Overseer.
+ // Although we could create the Map without it, there are a fair amount of rules we don't want
+ // to reproduce.
+ final Map<String, Object> createMsgMap =
+ CollectionsHandler.CollectionOperation.CREATE_OP.execute(
+ new LocalSolrQueryRequest(null, createReqParams),
+ null,
+ ccc.getCoreContainer().getCollectionsHandler());
createMsgMap.put(Overseer.QUEUE_OPERATION, CollectionParams.CollectionAction.CREATE.toLower());
NamedList<Object> results = new NamedList<>();
try {
- // Since we are running in the Overseer here, send the message directly to the Overseer CreateCollectionCmd.
- // note: there's doesn't seem to be any point in locking on the collection name, so we don't. We currently should
+ // Since we are running in the Overseer here, send the message directly to the Overseer
+ // CreateCollectionCmd.
+ // note: there's doesn't seem to be any point in locking on the collection name, so we don't.
+ // We currently should
// already have a lock on the alias name which should be sufficient.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/api/collections/CategoryRoutedAlias.java
##########
@@ -121,11 +116,13 @@ public RoutedAliasTypes getRoutedAliasType() {
@Override
public void validateRouteValue(AddUpdateCommand cmd) throws SolrException {
if (this.aliases == null) {
- updateParsedCollectionAliases(cmd.getReq().getCoreContainer().getZkController().zkStateReader, false);
+ updateParsedCollectionAliases(
+ cmd.getReq().getCoreContainer().getZkController().zkStateReader, false);
}
Object fieldValue = cmd.getSolrInputDocument().getFieldValue(getRouteField());
- // possible future enhancement: allow specification of an "unknown" category name to where we can send
+ // possible future enhancement: allow specification of an "unknown" category name to where we
+ // can send
// docs that are uncategorized.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/ZkController.java
##########
@@ -2198,9 +2440,12 @@ public void rejoinShardLeaderElection(SolrParams params) {
try (SolrCore core = cc.getCore(coreName)) {
Replica.Type replicaType = core.getCoreDescriptor().getCloudDescriptor().getReplicaType();
if (replicaType == Type.TLOG) {
- String leaderUrl = getLeader(core.getCoreDescriptor().getCloudDescriptor(), cloudConfig.getLeaderVoteWait());
+ String leaderUrl =
+ getLeader(
+ core.getCoreDescriptor().getCloudDescriptor(), cloudConfig.getLeaderVoteWait());
if (!leaderUrl.equals(ourUrl)) {
- // restart the replication thread to ensure the replication is running in each new replica
+ // restart the replication thread to ensure the replication is running in each new
+ // replica
// especially if previous role is "leader" (i.e., no replication thread)
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/api/collections/Assign.java
##########
@@ -251,71 +295,92 @@ private static boolean existCoreName(String coreName, Slice slice) {
return liveNodes;
}
} catch (Exception e) {
- throw new SolrException(SolrException.ErrorCode.SERVER_ERROR, "Error fetching roles from Zookeeper", e);
+ throw new SolrException(
+ SolrException.ErrorCode.SERVER_ERROR, "Error fetching roles from Zookeeper", e);
}
}
-
-
// Only called from addReplica (and by extension createShard) (so far).
//
- // Gets a list of candidate nodes to put the required replica(s) on. Throws errors if the AssignStrategy
+ // Gets a list of candidate nodes to put the required replica(s) on. Throws errors if the
+ // AssignStrategy
// can't allocate valid positions.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/ZkController.java
##########
@@ -1911,18 +2121,25 @@ private ZkCoreNodeProps waitForLeaderToSeeDownState(
boolean isLeader = leaderProps.getCoreUrl().equals(ourUrl);
if (!isLeader && !SKIP_AUTO_RECOVERY) {
if (!getShardTerms(collection, shard).canBecomeLeader(myCoreNodeName)) {
- log.debug("Term of replica {} is already less than leader, so not waiting for leader to see down state."
- , myCoreNodeName);
+ log.debug(
+ "Term of replica {} is already less than leader, so not waiting for leader to see down state.",
+ myCoreNodeName);
} else {
if (log.isInfoEnabled()) {
- log.info("replica={} is making a best effort attempt to wait for leader={} to see it's DOWN state.", myCoreNodeName, leaderProps.getCoreUrl());
+ log.info(
+ "replica={} is making a best effort attempt to wait for leader={} to see it's DOWN state.",
+ myCoreNodeName,
+ leaderProps.getCoreUrl());
}
- try (HttpSolrClient client = new Builder(leaderBaseUrl)
- .withConnectionTimeout(8000) // short timeouts, we may be in a storm and this is best effort and maybe we should be the leader now
- .withSocketTimeout(30000)
- .build()) {
+ try (HttpSolrClient client =
+ new Builder(leaderBaseUrl)
+ .withConnectionTimeout(
+ 8000) // short timeouts, we may be in a storm and this is best effort and maybe
+ // we should be the leader now
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/ZkController.java
##########
@@ -2313,20 +2572,25 @@ public static int persistConfigResourceToZooKeeper(ZkSolrResourceLoader zkLoader
try {
try {
Stat stat = zkClient.setData(resourceLocation, content, znodeVersion, true);
- latestVersion = stat.getVersion();// if the set succeeded , it should have incremented the version by one always
+ latestVersion =
+ stat.getVersion(); // if the set succeeded , it should have incremented the version by
+ // one always
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/api/collections/BackupCmd.java
##########
@@ -330,19 +407,29 @@ private void incrementalCopyIndexFiles(URI backupUri, String collectionName, ZkN
return aggRsp;
}
- private ModifiableSolrParams coreBackupParams(URI backupPath, String repoName, Slice slice, String coreName, boolean incremental) {
+ private ModifiableSolrParams coreBackupParams(
+ URI backupPath, String repoName, Slice slice, String coreName, boolean incremental) {
ModifiableSolrParams params = new ModifiableSolrParams();
params.set(CoreAdminParams.ACTION, CoreAdminParams.CoreAdminAction.BACKUPCORE.toString());
params.set(NAME, slice.getName());
params.set(CoreAdminParams.BACKUP_REPOSITORY, repoName);
- params.set(CoreAdminParams.BACKUP_LOCATION, backupPath.toASCIIString()); // note: index dir will be here then the "snapshot." + slice name
+ params.set(
+ CoreAdminParams.BACKUP_LOCATION,
+ backupPath
+ .toASCIIString()); // note: index dir will be here then the "snapshot." + slice name
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/api/collections/Assign.java
##########
@@ -251,71 +295,92 @@ private static boolean existCoreName(String coreName, Slice slice) {
return liveNodes;
}
} catch (Exception e) {
- throw new SolrException(SolrException.ErrorCode.SERVER_ERROR, "Error fetching roles from Zookeeper", e);
+ throw new SolrException(
+ SolrException.ErrorCode.SERVER_ERROR, "Error fetching roles from Zookeeper", e);
}
}
-
-
// Only called from addReplica (and by extension createShard) (so far).
//
- // Gets a list of candidate nodes to put the required replica(s) on. Throws errors if the AssignStrategy
+ // Gets a list of candidate nodes to put the required replica(s) on. Throws errors if the
+ // AssignStrategy
// can't allocate valid positions.
@SuppressWarnings({"unchecked"})
- public static List<ReplicaPosition> getNodesForNewReplicas(ClusterState clusterState, String collectionName,
- String shard, int nrtReplicas, int tlogReplicas, int pullReplicas,
- Object createNodeSet, SolrCloudManager cloudManager,
- CoreContainer coreContainer) throws IOException, InterruptedException, AssignmentException {
- log.debug("getNodesForNewReplicas() shard: {} , nrtReplicas : {} , tlogReplicas: {} , pullReplicas: {} , createNodeSet {}"
- , shard, nrtReplicas, tlogReplicas, pullReplicas, createNodeSet);
+ public static List<ReplicaPosition> getNodesForNewReplicas(
+ ClusterState clusterState,
+ String collectionName,
+ String shard,
+ int nrtReplicas,
+ int tlogReplicas,
+ int pullReplicas,
+ Object createNodeSet,
+ SolrCloudManager cloudManager,
+ CoreContainer coreContainer)
+ throws IOException, InterruptedException, AssignmentException {
+ log.debug(
+ "getNodesForNewReplicas() shard: {} , nrtReplicas : {} , tlogReplicas: {} , pullReplicas: {} , createNodeSet {}",
+ shard,
+ nrtReplicas,
+ tlogReplicas,
+ pullReplicas,
+ createNodeSet);
List<String> createNodeList;
if (createNodeSet instanceof List) {
createNodeList = (List<String>) createNodeSet;
} else {
// deduplicate
- createNodeList = createNodeSet == null ? null : new ArrayList<>(new LinkedHashSet<>(StrUtils.splitSmart((String) createNodeSet, ",", true)));
+ createNodeList =
+ createNodeSet == null
+ ? null
+ : new ArrayList<>(
+ new LinkedHashSet<>(StrUtils.splitSmart((String) createNodeSet, ",", true)));
}
// produces clear message when down nodes are the root cause, without this the user just
- // gets a log message of detail about the nodes that are up, and a message that policies could not
+ // gets a log message of detail about the nodes that are up, and a message that policies could
+ // not
// be satisfied which then requires study to diagnose the issue.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/ZkController.java
##########
@@ -477,37 +507,44 @@ public boolean isClosed() {
}
this.overseerCollectionQueue = overseer.getCollectionQueue(zkClient);
this.overseerConfigSetQueue = overseer.getConfigSetQueue(zkClient);
- this.sysPropsCacher = new NodesSysPropsCacher(getSolrCloudManager().getNodeStateProvider(),
- getNodeName(), zkStateReader);
+ this.sysPropsCacher =
+ new NodesSysPropsCacher(
+ getSolrCloudManager().getNodeStateProvider(), getNodeName(), zkStateReader);
assert ObjectReleaseTracker.track(this);
}
/**
- * <p>Verifies if /clusterstate.json exists in Zookeepeer, and if it does and is not empty, refuses to start and outputs
- * a helpful message regarding collection migration.</p>
+ * Verifies if /clusterstate.json exists in Zookeepeer, and if it does and is not empty, refuses
+ * to start and outputs a helpful message regarding collection migration.
*
- * <p>If /clusterstate.json exists and is empty, it is removed.</p>
+ * <p>If /clusterstate.json exists and is empty, it is removed.
*/
private void checkNoOldClusterstate(final SolrZkClient zkClient) throws InterruptedException {
try {
if (!zkClient.exists(ZkStateReader.UNSUPPORTED_CLUSTER_STATE, true)) {
return;
}
- final byte[] data = zkClient.getData(ZkStateReader.UNSUPPORTED_CLUSTER_STATE, null, null, true);
+ final byte[] data =
+ zkClient.getData(ZkStateReader.UNSUPPORTED_CLUSTER_STATE, null, null, true);
if (Arrays.equals("{}".getBytes(StandardCharsets.UTF_8), data)) {
// Empty json. This log will only occur once.
- log.warn("{} no longer supported starting with Solr 9. Found empty file on Zookeeper, deleting it.", ZkStateReader.UNSUPPORTED_CLUSTER_STATE);
+ log.warn(
+ "{} no longer supported starting with Solr 9. Found empty file on Zookeeper, deleting it.",
+ ZkStateReader.UNSUPPORTED_CLUSTER_STATE);
zkClient.delete(ZkStateReader.UNSUPPORTED_CLUSTER_STATE, -1, true);
} else {
- // /clusterstate.json not empty: refuse to start but do not automatically delete. A bit of a pain but user shouldn't
+ // /clusterstate.json not empty: refuse to start but do not automatically delete. A bit of a
+ // pain but user shouldn't
// have older collections at this stage anyway.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/api/collections/CollectionApiLockFactory.java
##########
@@ -38,68 +38,96 @@
}
/**
- * For the {@link org.apache.solr.common.params.CollectionParams.LockLevel} of the passed {@code action}, obtains the
- * required locks (if any) and returns.<p>
+ * For the {@link org.apache.solr.common.params.CollectionParams.LockLevel} of the passed {@code
+ * action}, obtains the required locks (if any) and returns.
*
- * This method obtains a write lock at the actual level and path of the action, and also obtains read locks on "lower"
- * lock levels. For example for a lock at the shard level, a write lock will be requested at the corresponding shard path
- * and a read lock on the corresponding collection path (in order to prevent an operation locking at the collection level
- * from executing concurrently with an operation on one of the shards of the collection).
- * See documentation linked to SOLR-14840 regarding Collection API locking.
+ * <p>This method obtains a write lock at the actual level and path of the action, and also
+ * obtains read locks on "lower" lock levels. For example for a lock at the shard level, a write
+ * lock will be requested at the corresponding shard path and a read lock on the corresponding
+ * collection path (in order to prevent an operation locking at the collection level from
+ * executing concurrently with an operation on one of the shards of the collection). See
+ * documentation linked to SOLR-14840 regarding Collection API locking.
*
- * @return a lock that once {@link DistributedMultiLock#isAcquired()} guarantees the corresponding Collection
- * API command can execute safely.
- * The returned lock <b>MUST</b> be {@link DistributedMultiLock#release()} no matter what once no longer needed as otherwise it would
- * prevent other threads from locking.
+ * @return a lock that once {@link DistributedMultiLock#isAcquired()} guarantees the corresponding
+ * Collection API command can execute safely. The returned lock <b>MUST</b> be {@link
+ * DistributedMultiLock#release()} no matter what once no longer needed as otherwise it would
+ * prevent other threads from locking.
*/
- DistributedMultiLock createCollectionApiLock(CollectionParams.LockLevel lockLevel, String collName, String shardId, String replicaName) {
+ DistributedMultiLock createCollectionApiLock(
+ CollectionParams.LockLevel lockLevel, String collName, String shardId, String replicaName) {
if (lockLevel == CollectionParams.LockLevel.NONE) {
return new DistributedMultiLock(List.of());
}
if (lockLevel == CollectionParams.LockLevel.CLUSTER) {
- throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "Bug. Not expecting locking at cluster level.");
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST, "Bug. Not expecting locking at cluster level.");
}
if (collName == null) {
throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "Bug. collName can't be null");
}
- // There is a bug in the way locks are managed in the Overseer, it sometimes locks at a given level (for example shard)
- // without knowing on which shard to lock. This can happen for example for SPLITSHARD using a split.key (details below)
+ // There is a bug in the way locks are managed in the Overseer, it sometimes locks at a given
+ // level (for example shard)
+ // without knowing on which shard to lock. This can happen for example for SPLITSHARD using a
+ // split.key (details below)
// or DELETEREPLICA when specifying a count of replicas to delete but no shard.
//
- // When doing a splitshard using a split.key (and not specifying the actual shard name), when the OverseerCollectionMessageHandler
- // grabs a lock for the Collection API it does not have the shard name and the lock is obtained for a null key.
- // This means that this lock does not prevent another operation specifically targeting the shard from executing concurrently
- // (can even be another split if it is specified with a shard name). The null shard lock does prevent another split.key split
- // for another shard of the same collection at the same time (because trying to obtain another lock on the null key below the
+ // When doing a splitshard using a split.key (and not specifying the actual shard name), when
+ // the OverseerCollectionMessageHandler
+ // grabs a lock for the Collection API it does not have the shard name and the lock is
+ // obtained for a null key.
+ // This means that this lock does not prevent another operation specifically targeting the
+ // shard from executing concurrently
+ // (can even be another split if it is specified with a shard name). The null shard lock does
+ // prevent another split.key split
+ // for another shard of the same collection at the same time (because trying to obtain another
+ // lock on the null key below the
// collection will fail).
- // The splitshard request is built in CollectionAdminRequest.splitShard(). The case of concern is when SplitShard.setSplitKey()
- // is then called (see test CollectionsAPISolrJTest.testSplitShard()). CollectionOperation.SPLITSHARD_OP in CollectionsHandler
+ // The splitshard request is built in CollectionAdminRequest.splitShard(). The case of concern
+ // is when SplitShard.setSplitKey()
+ // is then called (see test CollectionsAPISolrJTest.testSplitShard()).
+ // CollectionOperation.SPLITSHARD_OP in CollectionsHandler
// verifies that when the key is specified no shard name is specified.
- // The locking happens from OverseerTaskProcessor.run() calling OverseerCollectionMessageHandler.lockTask().
- // The ArrayList passed to LockTree.Session.lock() contains the collection name (index 0) but not shard name (value null
- // at index 1) yet the action.lockLevel is SHARD. Note that there ends up being a single shard name associated with the
+ // The locking happens from OverseerTaskProcessor.run() calling
+ // OverseerCollectionMessageHandler.lockTask().
+ // The ArrayList passed to LockTree.Session.lock() contains the collection name (index 0) but
+ // not shard name (value null
+ // at index 1) yet the action.lockLevel is SHARD. Note that there ends up being a single shard
+ // name associated with the
// split key. It is computed in SplitShardCmd.getParentSlice().
//
- // We deal with the issue to trigger the same (wrong) behavior for compatibility of distributed Collection API vs Overseer
- // based collection API. We make up a shard name that is highly unlikely to otherwise exist and lock it instead.
- // Another option would be to bump up the locking level from SHARD to COLLECTION when no shard is specified (would be
- // "more correct" from the perspective of preventing competing Collection API commands from running concurrently, but
- // would restrict concurrency a bit more than existing solution, and if some installations depend on current behavior...).
+ // We deal with the issue to trigger the same (wrong) behavior for compatibility of distributed
+ // Collection API vs Overseer
+ // based collection API. We make up a shard name that is highly unlikely to otherwise exist and
+ // lock it instead.
+ // Another option would be to bump up the locking level from SHARD to COLLECTION when no shard
+ // is specified (would be
+ // "more correct" from the perspective of preventing competing Collection API commands from
+ // running concurrently, but
+ // would restrict concurrency a bit more than existing solution, and if some installations
+ // depend on current behavior...).
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/api/collections/CategoryRoutedAlias.java
##########
@@ -36,14 +37,13 @@
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
-import static org.apache.solr.common.SolrException.ErrorCode.BAD_REQUEST;
-
public class CategoryRoutedAlias extends RoutedAlias {
private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
private static final String COLLECTION_INFIX = "__CRA__";
// This constant is terribly annoying but a great many things fall apart if we allow an alias with
- // no collections to be created. So this kludge seems better than reworking every request path that
+ // no collections to be created. So this kludge seems better than reworking every request path
+ // that
// expects a collection but also works with an alias to handle or error out on empty alias. The
// collection with this constant as a suffix is automatically removed after the alias begins to
// receive data.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/api/collections/CollectionApiLockFactory.java
##########
@@ -38,68 +38,96 @@
}
/**
- * For the {@link org.apache.solr.common.params.CollectionParams.LockLevel} of the passed {@code action}, obtains the
- * required locks (if any) and returns.<p>
+ * For the {@link org.apache.solr.common.params.CollectionParams.LockLevel} of the passed {@code
+ * action}, obtains the required locks (if any) and returns.
*
- * This method obtains a write lock at the actual level and path of the action, and also obtains read locks on "lower"
- * lock levels. For example for a lock at the shard level, a write lock will be requested at the corresponding shard path
- * and a read lock on the corresponding collection path (in order to prevent an operation locking at the collection level
- * from executing concurrently with an operation on one of the shards of the collection).
- * See documentation linked to SOLR-14840 regarding Collection API locking.
+ * <p>This method obtains a write lock at the actual level and path of the action, and also
+ * obtains read locks on "lower" lock levels. For example for a lock at the shard level, a write
+ * lock will be requested at the corresponding shard path and a read lock on the corresponding
+ * collection path (in order to prevent an operation locking at the collection level from
+ * executing concurrently with an operation on one of the shards of the collection). See
+ * documentation linked to SOLR-14840 regarding Collection API locking.
*
- * @return a lock that once {@link DistributedMultiLock#isAcquired()} guarantees the corresponding Collection
- * API command can execute safely.
- * The returned lock <b>MUST</b> be {@link DistributedMultiLock#release()} no matter what once no longer needed as otherwise it would
- * prevent other threads from locking.
+ * @return a lock that once {@link DistributedMultiLock#isAcquired()} guarantees the corresponding
+ * Collection API command can execute safely. The returned lock <b>MUST</b> be {@link
+ * DistributedMultiLock#release()} no matter what once no longer needed as otherwise it would
+ * prevent other threads from locking.
*/
- DistributedMultiLock createCollectionApiLock(CollectionParams.LockLevel lockLevel, String collName, String shardId, String replicaName) {
+ DistributedMultiLock createCollectionApiLock(
+ CollectionParams.LockLevel lockLevel, String collName, String shardId, String replicaName) {
if (lockLevel == CollectionParams.LockLevel.NONE) {
return new DistributedMultiLock(List.of());
}
if (lockLevel == CollectionParams.LockLevel.CLUSTER) {
- throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "Bug. Not expecting locking at cluster level.");
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST, "Bug. Not expecting locking at cluster level.");
}
if (collName == null) {
throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "Bug. collName can't be null");
}
- // There is a bug in the way locks are managed in the Overseer, it sometimes locks at a given level (for example shard)
- // without knowing on which shard to lock. This can happen for example for SPLITSHARD using a split.key (details below)
+ // There is a bug in the way locks are managed in the Overseer, it sometimes locks at a given
+ // level (for example shard)
+ // without knowing on which shard to lock. This can happen for example for SPLITSHARD using a
+ // split.key (details below)
// or DELETEREPLICA when specifying a count of replicas to delete but no shard.
//
- // When doing a splitshard using a split.key (and not specifying the actual shard name), when the OverseerCollectionMessageHandler
- // grabs a lock for the Collection API it does not have the shard name and the lock is obtained for a null key.
- // This means that this lock does not prevent another operation specifically targeting the shard from executing concurrently
- // (can even be another split if it is specified with a shard name). The null shard lock does prevent another split.key split
- // for another shard of the same collection at the same time (because trying to obtain another lock on the null key below the
+ // When doing a splitshard using a split.key (and not specifying the actual shard name), when
+ // the OverseerCollectionMessageHandler
+ // grabs a lock for the Collection API it does not have the shard name and the lock is
+ // obtained for a null key.
+ // This means that this lock does not prevent another operation specifically targeting the
+ // shard from executing concurrently
+ // (can even be another split if it is specified with a shard name). The null shard lock does
+ // prevent another split.key split
+ // for another shard of the same collection at the same time (because trying to obtain another
+ // lock on the null key below the
// collection will fail).
- // The splitshard request is built in CollectionAdminRequest.splitShard(). The case of concern is when SplitShard.setSplitKey()
- // is then called (see test CollectionsAPISolrJTest.testSplitShard()). CollectionOperation.SPLITSHARD_OP in CollectionsHandler
+ // The splitshard request is built in CollectionAdminRequest.splitShard(). The case of concern
+ // is when SplitShard.setSplitKey()
+ // is then called (see test CollectionsAPISolrJTest.testSplitShard()).
+ // CollectionOperation.SPLITSHARD_OP in CollectionsHandler
// verifies that when the key is specified no shard name is specified.
- // The locking happens from OverseerTaskProcessor.run() calling OverseerCollectionMessageHandler.lockTask().
- // The ArrayList passed to LockTree.Session.lock() contains the collection name (index 0) but not shard name (value null
- // at index 1) yet the action.lockLevel is SHARD. Note that there ends up being a single shard name associated with the
+ // The locking happens from OverseerTaskProcessor.run() calling
+ // OverseerCollectionMessageHandler.lockTask().
+ // The ArrayList passed to LockTree.Session.lock() contains the collection name (index 0) but
+ // not shard name (value null
+ // at index 1) yet the action.lockLevel is SHARD. Note that there ends up being a single shard
+ // name associated with the
// split key. It is computed in SplitShardCmd.getParentSlice().
//
- // We deal with the issue to trigger the same (wrong) behavior for compatibility of distributed Collection API vs Overseer
- // based collection API. We make up a shard name that is highly unlikely to otherwise exist and lock it instead.
- // Another option would be to bump up the locking level from SHARD to COLLECTION when no shard is specified (would be
- // "more correct" from the perspective of preventing competing Collection API commands from running concurrently, but
- // would restrict concurrency a bit more than existing solution, and if some installations depend on current behavior...).
+ // We deal with the issue to trigger the same (wrong) behavior for compatibility of distributed
+ // Collection API vs Overseer
+ // based collection API. We make up a shard name that is highly unlikely to otherwise exist and
+ // lock it instead.
+ // Another option would be to bump up the locking level from SHARD to COLLECTION when no shard
+ // is specified (would be
+ // "more correct" from the perspective of preventing competing Collection API commands from
+ // running concurrently, but
+ // would restrict concurrency a bit more than existing solution, and if some installations
+ // depend on current behavior...).
if (lockLevel == CollectionParams.LockLevel.SHARD && shardId == null) {
shardId = "MadeUpShardNameWhenNoShardNameWasProvided";
- // Bumping locking level to collection would be instead: lockLevel = CollectionParams.LockLevel.COLLECTION;
+ // Bumping locking level to collection would be instead: lockLevel =
+ // CollectionParams.LockLevel.COLLECTION;
}
- // The first requested lock is a write one (on the target object for the action, depending on lock level), then requesting
- // read locks on "higher" levels (collection > shard > replica here for the level. Note LockLevel "height" is other way around).
+ // The first requested lock is a write one (on the target object for the action, depending on
+ // lock level), then requesting
+ // read locks on "higher" levels (collection > shard > replica here for the level. Note
+ // LockLevel "height" is other way around).
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/ZkDistributedLock.java
##########
@@ -180,16 +184,22 @@ public boolean isAcquired() {
}
/**
- * @return Another lock node (complete path) that must go away for us to acquire the lock, or {@code null} if the lock is ours.
+ * @return Another lock node (complete path) that must go away for us to acquire the lock, or
+ * {@code null} if the lock is ours.
*/
String nodeToWatch() throws KeeperException, InterruptedException {
List<String> locks = zkClient.getChildren(lockDir, null, true);
boolean foundSelf = false; // For finding bugs or ZK bad behavior
- // We deviate from the ZK recipe here: we do not sort the list of nodes, and we stop waiting on the first one we find
- // that blocks us. This is done in O(n), whereas sorting is more expensive. And we iterate only once over the set of children.
- // Might cause more wakeups than needed though (multiple locks watching the same one rather than one another) but lock
- // contention expected low (how many concurrent and conflicting Collection API requests a reasonable user can issue?)
- // and the code below is simpler. Changing to a sorted "optimal" approach implies only changes in this method.
+ // We deviate from the ZK recipe here: we do not sort the list of nodes, and we stop waiting on
+ // the first one we find
+ // that blocks us. This is done in O(n), whereas sorting is more expensive. And we iterate only
+ // once over the set of children.
+ // Might cause more wakeups than needed though (multiple locks watching the same one rather than
+ // one another) but lock
+ // contention expected low (how many concurrent and conflicting Collection API requests a
+ // reasonable user can issue?)
+ // and the code below is simpler. Changing to a sorted "optimal" approach implies only changes
+ // in this method.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/api/collections/CategoryRoutedAlias.java
##########
@@ -237,27 +243,27 @@ public CandidateCollection findCandidateGivenValue(AddUpdateCommand cmd) {
@Override
protected String getHeadCollectionIfOrdered(AddUpdateCommand cmd) {
- return buildCollectionNameFromValue(String.valueOf(cmd.getSolrInputDocument().getFieldValue(getRouteField())));
+ return buildCollectionNameFromValue(
+ String.valueOf(cmd.getSolrInputDocument().getFieldValue(getRouteField())));
}
@Override
protected List<Action> calculateActions(String targetCol) {
List<String> collectionList = getCollectionList(aliases);
if (!collectionList.contains(targetCol)) {
ArrayList<Action> actionList = new ArrayList<>();
- actionList.add(new Action(this,ActionType.ENSURE_EXISTS, targetCol));
+ actionList.add(new Action(this, ActionType.ENSURE_EXISTS, targetCol));
for (String s : collectionList) {
- // can't remove the uninitialized on the first pass otherwise there is a risk of momentarily having
+ // can't remove the uninitialized on the first pass otherwise there is a risk of momentarily
+ // having
// an empty alias if thread scheduling plays tricks on us.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/api/collections/BackupCmd.java
##########
@@ -70,74 +69,97 @@ public BackupCmd(CollectionCommandContext ccc) {
}
@Override
- public void call(ClusterState state, ZkNodeProps message, NamedList<Object> results) throws Exception {
+ public void call(ClusterState state, ZkNodeProps message, NamedList<Object> results)
+ throws Exception {
String extCollectionName = message.getStr(COLLECTION_PROP);
boolean followAliases = message.getBool(FOLLOW_ALIASES, false);
String collectionName;
if (followAliases) {
- collectionName = ccc.getSolrCloudManager().getClusterStateProvider().resolveSimpleAlias(extCollectionName);
+ collectionName =
+ ccc.getSolrCloudManager().getClusterStateProvider().resolveSimpleAlias(extCollectionName);
} else {
collectionName = extCollectionName;
}
String backupName = message.getStr(NAME);
String repo = message.getStr(CoreAdminParams.BACKUP_REPOSITORY);
boolean incremental = message.getBool(CoreAdminParams.BACKUP_INCREMENTAL, true);
- String configName = ccc.getSolrCloudManager().getClusterStateProvider().getCollection(collectionName).getConfigName();
+ String configName =
+ ccc.getSolrCloudManager()
+ .getClusterStateProvider()
+ .getCollection(collectionName)
+ .getConfigName();
- BackupProperties backupProperties = BackupProperties.create(backupName, collectionName,
- extCollectionName, configName);
+ BackupProperties backupProperties =
+ BackupProperties.create(backupName, collectionName, extCollectionName, configName);
CoreContainer cc = ccc.getCoreContainer();
try (BackupRepository repository = cc.newBackupRepository(repo)) {
// Backup location
URI location = repository.createDirectoryURI(message.getStr(CoreAdminParams.BACKUP_LOCATION));
- final URI backupUri = createAndValidateBackupPath(repository, incremental, location, backupName, collectionName);
-
- BackupManager backupMgr = (incremental) ?
- BackupManager.forIncrementalBackup(repository, ccc.getZkStateReader(), backupUri) :
- BackupManager.forBackup(repository, ccc.getZkStateReader(), backupUri);
-
- String strategy = message.getStr(CollectionAdminParams.INDEX_BACKUP_STRATEGY, CollectionAdminParams.COPY_FILES_STRATEGY);
+ final URI backupUri =
+ createAndValidateBackupPath(
+ repository, incremental, location, backupName, collectionName);
+
+ BackupManager backupMgr =
+ (incremental)
+ ? BackupManager.forIncrementalBackup(repository, ccc.getZkStateReader(), backupUri)
+ : BackupManager.forBackup(repository, ccc.getZkStateReader(), backupUri);
+
+ String strategy =
+ message.getStr(
+ CollectionAdminParams.INDEX_BACKUP_STRATEGY,
+ CollectionAdminParams.COPY_FILES_STRATEGY);
switch (strategy) {
- case CollectionAdminParams.COPY_FILES_STRATEGY: {
- if (incremental) {
- try {
- incrementalCopyIndexFiles(backupUri, collectionName, message, results, backupProperties, backupMgr);
- } catch (SolrException e) {
- log.error("Error happened during incremental backup for collection: {}", collectionName, e);
- CollectionHandlingUtils.cleanBackup(repository, backupUri, backupMgr.getBackupId(), ccc);
- throw e;
+ case CollectionAdminParams.COPY_FILES_STRATEGY:
+ {
+ if (incremental) {
+ try {
+ incrementalCopyIndexFiles(
+ backupUri, collectionName, message, results, backupProperties, backupMgr);
+ } catch (SolrException e) {
+ log.error(
+ "Error happened during incremental backup for collection: {}",
+ collectionName,
+ e);
+ CollectionHandlingUtils.cleanBackup(
+ repository, backupUri, backupMgr.getBackupId(), ccc);
+ throw e;
+ }
+ } else {
+ copyIndexFiles(
+ backupUri, collectionName, message, results, backupProperties, backupMgr);
}
- } else {
- copyIndexFiles(backupUri, collectionName, message, results, backupProperties, backupMgr);
+ break;
+ }
+ case CollectionAdminParams.NO_INDEX_BACKUP_STRATEGY:
+ {
+ NamedList<Object> response = new SimpleOrderedMap<>();
+ response.add("collection", collectionName);
+ response.add("startTime", backupProperties.getStartTime());
+ results.add("response", response);
+ break;
}
- break;
- }
- case CollectionAdminParams.NO_INDEX_BACKUP_STRATEGY: {
- NamedList<Object> response = new SimpleOrderedMap<>();
- response.add("collection", collectionName);
- response.add("startTime", backupProperties.getStartTime());
- results.add("response", response);
- break;
- }
}
log.info("Starting to backup ZK data for backupName={}", backupName);
- //Download the configs
+ // Download the configs
backupMgr.downloadConfigDir(configName, cc.getConfigSetService());
- //Save the collection's state. Can be part of the monolithic clusterstate.json or a individual state.json
- //Since we don't want to distinguish we extract the state and back it up as a separate json
- DocCollection collectionState = ccc.getZkStateReader().getClusterState().getCollection(collectionName);
+ // Save the collection's state. Can be part of the monolithic clusterstate.json or a
+ // individual state.json
+ // Since we don't want to distinguish we extract the state and back it up as a separate json
+ DocCollection collectionState =
+ ccc.getZkStateReader().getClusterState().getCollection(collectionName);
backupMgr.writeCollectionState(collectionName, collectionState);
backupMgr.downloadCollectionProperties(collectionName);
- //TODO: Add MD5 of the configset. If during restore the same name configset exists then we can compare checksums to see if they are the same.
- //if they are not the same then we can throw an error or have an 'overwriteConfig' flag
- //TODO save numDocs for the shardLeader. We can use it to sanity check the restore.
+ // TODO: Add MD5 of the configset. If during restore the same name configset exists then we
+ // can compare checksums to see if they are the same.
+ // if they are not the same then we can throw an error or have an 'overwriteConfig' flag
+ // TODO save numDocs for the shardLeader. We can use it to sanity check the restore.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/ZkController.java
##########
@@ -217,20 +215,24 @@ public String toString() {
private volatile boolean isClosed;
- private final ConcurrentHashMap<String, Throwable> replicasMetTragicEvent = new ConcurrentHashMap<>();
+ private final ConcurrentHashMap<String, Throwable> replicasMetTragicEvent =
+ new ConcurrentHashMap<>();
@Deprecated
// keeps track of replicas that have been asked to recover by leaders running on this node
- private final Map<String, String> replicasInLeaderInitiatedRecovery = new HashMap<String, String>();
+ private final Map<String, String> replicasInLeaderInitiatedRecovery =
+ new HashMap<String, String>();
// This is an expert and unsupported development mode that does not create
// an Overseer or register a /live node. This let's you monitor the cluster
// and interact with zookeeper via the Solr admin UI on a node outside the cluster,
// and so one that will not be killed or stopped when testing. See developer cloud-scripts.
private boolean zkRunOnly = Boolean.getBoolean("zkRunOnly"); // expert
- // keeps track of a list of objects that need to know a new ZooKeeper session was created after expiration occurred
- // ref is held as a HashSet since we clone the set before notifying to avoid synchronizing too long
+ // keeps track of a list of objects that need to know a new ZooKeeper session was created after
+ // expiration occurred
+ // ref is held as a HashSet since we clone the set before notifying to avoid synchronizing too
+ // long
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/api/collections/CreateCollectionCmd.java
##########
@@ -128,75 +133,114 @@ public void call(ClusterState clusterState, ZkNodeProps message, NamedList<Objec
ZkStateReader zkStateReader = ccc.getZkStateReader();
message.getProperties().put(COLL_CONF, configName);
- Map<String,String> collectionParams = new HashMap<>();
- Map<String,Object> collectionProps = message.getProperties();
+ Map<String, String> collectionParams = new HashMap<>();
+ Map<String, Object> collectionProps = message.getProperties();
for (Map.Entry<String, Object> entry : collectionProps.entrySet()) {
String propName = entry.getKey();
if (propName.startsWith(ZkController.COLLECTION_PARAM_PREFIX)) {
- collectionParams.put(propName.substring(ZkController.COLLECTION_PARAM_PREFIX.length()), (String) entry.getValue());
+ collectionParams.put(
+ propName.substring(ZkController.COLLECTION_PARAM_PREFIX.length()),
+ (String) entry.getValue());
}
}
- createCollectionZkNode(ccc.getSolrCloudManager().getDistribStateManager(), collectionName, collectionParams, ccc.getCoreContainer().getConfigSetService());
+ createCollectionZkNode(
+ ccc.getSolrCloudManager().getDistribStateManager(),
+ collectionName,
+ collectionParams,
+ ccc.getCoreContainer().getConfigSetService());
- // Note that in code below there are two main execution paths: Overseer based cluster state updates and distributed
+ // Note that in code below there are two main execution paths: Overseer based cluster state
+ // updates and distributed
// cluster state updates (look for isDistributedStateUpdate() conditions).
//
- // PerReplicaStates (PRS) collections follow a hybrid approach. Even when the cluster is Overseer cluster state update based,
- // these collections are created locally then the cluster state updater is notified (look for usage of RefreshCollectionMessage).
- // This explains why PRS collections have less diverging execution paths between distributed or Overseer based cluster state updates.
+ // PerReplicaStates (PRS) collections follow a hybrid approach. Even when the cluster is
+ // Overseer cluster state update based,
+ // these collections are created locally then the cluster state updater is notified (look for
+ // usage of RefreshCollectionMessage).
+ // This explains why PRS collections have less diverging execution paths between distributed
+ // or Overseer based cluster state updates.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/api/collections/CreateCollectionCmd.java
##########
@@ -128,75 +133,114 @@ public void call(ClusterState clusterState, ZkNodeProps message, NamedList<Objec
ZkStateReader zkStateReader = ccc.getZkStateReader();
message.getProperties().put(COLL_CONF, configName);
- Map<String,String> collectionParams = new HashMap<>();
- Map<String,Object> collectionProps = message.getProperties();
+ Map<String, String> collectionParams = new HashMap<>();
+ Map<String, Object> collectionProps = message.getProperties();
for (Map.Entry<String, Object> entry : collectionProps.entrySet()) {
String propName = entry.getKey();
if (propName.startsWith(ZkController.COLLECTION_PARAM_PREFIX)) {
- collectionParams.put(propName.substring(ZkController.COLLECTION_PARAM_PREFIX.length()), (String) entry.getValue());
+ collectionParams.put(
+ propName.substring(ZkController.COLLECTION_PARAM_PREFIX.length()),
+ (String) entry.getValue());
}
}
- createCollectionZkNode(ccc.getSolrCloudManager().getDistribStateManager(), collectionName, collectionParams, ccc.getCoreContainer().getConfigSetService());
+ createCollectionZkNode(
+ ccc.getSolrCloudManager().getDistribStateManager(),
+ collectionName,
+ collectionParams,
+ ccc.getCoreContainer().getConfigSetService());
- // Note that in code below there are two main execution paths: Overseer based cluster state updates and distributed
+ // Note that in code below there are two main execution paths: Overseer based cluster state
+ // updates and distributed
// cluster state updates (look for isDistributedStateUpdate() conditions).
//
- // PerReplicaStates (PRS) collections follow a hybrid approach. Even when the cluster is Overseer cluster state update based,
- // these collections are created locally then the cluster state updater is notified (look for usage of RefreshCollectionMessage).
- // This explains why PRS collections have less diverging execution paths between distributed or Overseer based cluster state updates.
+ // PerReplicaStates (PRS) collections follow a hybrid approach. Even when the cluster is
+ // Overseer cluster state update based,
+ // these collections are created locally then the cluster state updater is notified (look for
+ // usage of RefreshCollectionMessage).
+ // This explains why PRS collections have less diverging execution paths between distributed
+ // or Overseer based cluster state updates.
if (isPRS) {
- // In case of a PRS collection, create the collection structure directly instead of resubmitting
+ // In case of a PRS collection, create the collection structure directly instead of
+ // resubmitting
// to the overseer queue.
// TODO: Consider doing this for all collections, not just the PRS collections.
// TODO comment above achieved by switching the cluster to distributed state updates
- // This code directly updates Zookeeper by creating the collection state.json. It is compatible with both distributed
+ // This code directly updates Zookeeper by creating the collection state.json. It is
+ // compatible with both distributed
// cluster state updates and Overseer based cluster state updates.
- ZkWriteCommand command = new ClusterStateMutator(ccc.getSolrCloudManager()).createCollection(clusterState, message);
+ ZkWriteCommand command =
+ new ClusterStateMutator(ccc.getSolrCloudManager())
+ .createCollection(clusterState, message);
byte[] data = Utils.toJSON(Collections.singletonMap(collectionName, command.collection));
- ccc.getZkStateReader().getZkClient().create(collectionPath, data, CreateMode.PERSISTENT, true);
+ ccc.getZkStateReader()
+ .getZkClient()
+ .create(collectionPath, data, CreateMode.PERSISTENT, true);
clusterState = clusterState.copyWith(collectionName, command.collection);
newColl = command.collection;
- // When cluster state updates are handled by Overseer, ask it to load that collection it doesn't know about.
- // When cluster state updates are distributed, ZK is the source of truth for all nodes so no reload needed.
+ // When cluster state updates are handled by Overseer, ask it to load that collection it
+ // doesn't know about.
+ // When cluster state updates are distributed, ZK is the source of truth for all nodes so no
+ // reload needed.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/api/collections/CollectionHandlingUtils.java
##########
@@ -98,17 +98,27 @@
static final String SKIP_CREATE_REPLICA_IN_CLUSTER_STATE = "skipCreateReplicaInClusterState";
// Immutable Maps are null-hostile, so build our own
- public static final Map<String, Object> COLLECTION_PROPS_AND_DEFAULTS = Collections.unmodifiableMap(Utils.makeMap(
- ROUTER, DocRouter.DEFAULT_NAME,
- ZkStateReader.REPLICATION_FACTOR, "1",
- ZkStateReader.NRT_REPLICAS, "1",
- ZkStateReader.TLOG_REPLICAS, "0",
- DocCollection.PER_REPLICA_STATE, null,
- ZkStateReader.PULL_REPLICAS, "0"));
+ public static final Map<String, Object> COLLECTION_PROPS_AND_DEFAULTS =
+ Collections.unmodifiableMap(
+ Utils.makeMap(
+ ROUTER,
+ DocRouter.DEFAULT_NAME,
+ ZkStateReader.REPLICATION_FACTOR,
+ "1",
+ ZkStateReader.NRT_REPLICAS,
+ "1",
+ ZkStateReader.TLOG_REPLICAS,
+ "0",
+ DocCollection.PER_REPLICA_STATE,
+ null,
+ ZkStateReader.PULL_REPLICAS,
+ "0"));
public static final Random RANDOM;
+
static {
- // We try to make things reproducible in the context of our tests by initializing the random instance
+ // We try to make things reproducible in the context of our tests by initializing the random
+ // instance
// based on the current seed
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/api/collections/CollectionHandlingUtils.java
##########
@@ -533,31 +673,37 @@ void processResponses(NamedList<Object> results, ShardHandler shardHandler, bool
// If request is async wait for the core admin to complete before returning
if (asyncId != null) {
- waitForAsyncCallsToComplete(results); // TODO: Shouldn't we abort with msgOnError exception when failure?
+ waitForAsyncCallsToComplete(
+ results); // TODO: Shouldn't we abort with msgOnError exception when failure?
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/api/collections/CreateCollectionCmd.java
##########
@@ -128,75 +133,114 @@ public void call(ClusterState clusterState, ZkNodeProps message, NamedList<Objec
ZkStateReader zkStateReader = ccc.getZkStateReader();
message.getProperties().put(COLL_CONF, configName);
- Map<String,String> collectionParams = new HashMap<>();
- Map<String,Object> collectionProps = message.getProperties();
+ Map<String, String> collectionParams = new HashMap<>();
+ Map<String, Object> collectionProps = message.getProperties();
for (Map.Entry<String, Object> entry : collectionProps.entrySet()) {
String propName = entry.getKey();
if (propName.startsWith(ZkController.COLLECTION_PARAM_PREFIX)) {
- collectionParams.put(propName.substring(ZkController.COLLECTION_PARAM_PREFIX.length()), (String) entry.getValue());
+ collectionParams.put(
+ propName.substring(ZkController.COLLECTION_PARAM_PREFIX.length()),
+ (String) entry.getValue());
}
}
- createCollectionZkNode(ccc.getSolrCloudManager().getDistribStateManager(), collectionName, collectionParams, ccc.getCoreContainer().getConfigSetService());
+ createCollectionZkNode(
+ ccc.getSolrCloudManager().getDistribStateManager(),
+ collectionName,
+ collectionParams,
+ ccc.getCoreContainer().getConfigSetService());
- // Note that in code below there are two main execution paths: Overseer based cluster state updates and distributed
+ // Note that in code below there are two main execution paths: Overseer based cluster state
+ // updates and distributed
// cluster state updates (look for isDistributedStateUpdate() conditions).
//
- // PerReplicaStates (PRS) collections follow a hybrid approach. Even when the cluster is Overseer cluster state update based,
- // these collections are created locally then the cluster state updater is notified (look for usage of RefreshCollectionMessage).
- // This explains why PRS collections have less diverging execution paths between distributed or Overseer based cluster state updates.
+ // PerReplicaStates (PRS) collections follow a hybrid approach. Even when the cluster is
+ // Overseer cluster state update based,
+ // these collections are created locally then the cluster state updater is notified (look for
+ // usage of RefreshCollectionMessage).
+ // This explains why PRS collections have less diverging execution paths between distributed
+ // or Overseer based cluster state updates.
if (isPRS) {
- // In case of a PRS collection, create the collection structure directly instead of resubmitting
+ // In case of a PRS collection, create the collection structure directly instead of
+ // resubmitting
// to the overseer queue.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/api/collections/CreateCollectionCmd.java
##########
@@ -128,75 +133,114 @@ public void call(ClusterState clusterState, ZkNodeProps message, NamedList<Objec
ZkStateReader zkStateReader = ccc.getZkStateReader();
message.getProperties().put(COLL_CONF, configName);
- Map<String,String> collectionParams = new HashMap<>();
- Map<String,Object> collectionProps = message.getProperties();
+ Map<String, String> collectionParams = new HashMap<>();
+ Map<String, Object> collectionProps = message.getProperties();
for (Map.Entry<String, Object> entry : collectionProps.entrySet()) {
String propName = entry.getKey();
if (propName.startsWith(ZkController.COLLECTION_PARAM_PREFIX)) {
- collectionParams.put(propName.substring(ZkController.COLLECTION_PARAM_PREFIX.length()), (String) entry.getValue());
+ collectionParams.put(
+ propName.substring(ZkController.COLLECTION_PARAM_PREFIX.length()),
+ (String) entry.getValue());
}
}
- createCollectionZkNode(ccc.getSolrCloudManager().getDistribStateManager(), collectionName, collectionParams, ccc.getCoreContainer().getConfigSetService());
+ createCollectionZkNode(
+ ccc.getSolrCloudManager().getDistribStateManager(),
+ collectionName,
+ collectionParams,
+ ccc.getCoreContainer().getConfigSetService());
- // Note that in code below there are two main execution paths: Overseer based cluster state updates and distributed
+ // Note that in code below there are two main execution paths: Overseer based cluster state
+ // updates and distributed
// cluster state updates (look for isDistributedStateUpdate() conditions).
//
- // PerReplicaStates (PRS) collections follow a hybrid approach. Even when the cluster is Overseer cluster state update based,
- // these collections are created locally then the cluster state updater is notified (look for usage of RefreshCollectionMessage).
- // This explains why PRS collections have less diverging execution paths between distributed or Overseer based cluster state updates.
+ // PerReplicaStates (PRS) collections follow a hybrid approach. Even when the cluster is
+ // Overseer cluster state update based,
+ // these collections are created locally then the cluster state updater is notified (look for
+ // usage of RefreshCollectionMessage).
+ // This explains why PRS collections have less diverging execution paths between distributed
+ // or Overseer based cluster state updates.
if (isPRS) {
- // In case of a PRS collection, create the collection structure directly instead of resubmitting
+ // In case of a PRS collection, create the collection structure directly instead of
+ // resubmitting
// to the overseer queue.
// TODO: Consider doing this for all collections, not just the PRS collections.
// TODO comment above achieved by switching the cluster to distributed state updates
- // This code directly updates Zookeeper by creating the collection state.json. It is compatible with both distributed
+ // This code directly updates Zookeeper by creating the collection state.json. It is
+ // compatible with both distributed
// cluster state updates and Overseer based cluster state updates.
- ZkWriteCommand command = new ClusterStateMutator(ccc.getSolrCloudManager()).createCollection(clusterState, message);
+ ZkWriteCommand command =
+ new ClusterStateMutator(ccc.getSolrCloudManager())
+ .createCollection(clusterState, message);
byte[] data = Utils.toJSON(Collections.singletonMap(collectionName, command.collection));
- ccc.getZkStateReader().getZkClient().create(collectionPath, data, CreateMode.PERSISTENT, true);
+ ccc.getZkStateReader()
+ .getZkClient()
+ .create(collectionPath, data, CreateMode.PERSISTENT, true);
clusterState = clusterState.copyWith(collectionName, command.collection);
newColl = command.collection;
- // When cluster state updates are handled by Overseer, ask it to load that collection it doesn't know about.
- // When cluster state updates are distributed, ZK is the source of truth for all nodes so no reload needed.
+ // When cluster state updates are handled by Overseer, ask it to load that collection it
+ // doesn't know about.
+ // When cluster state updates are distributed, ZK is the source of truth for all nodes so no
+ // reload needed.
if (!ccc.getDistributedClusterStateUpdater().isDistributedStateUpdate()) {
- // If cluster state update is not distributed and we execute here, the Collection API is not distributed either
- // and this execution happens on the Overseer node, so direct memory access as done below is ok.
+ // If cluster state update is not distributed and we execute here, the Collection API is
+ // not distributed either
+ // and this execution happens on the Overseer node, so direct memory access as done below
+ // is ok.
ccc.submitIntraProcessMessage(new RefreshCollectionMessage(collectionName));
}
} else {
if (ccc.getDistributedClusterStateUpdater().isDistributedStateUpdate()) {
- // The message has been crafted by CollectionsHandler.CollectionOperation.CREATE_OP and defines the QUEUE_OPERATION
+ // The message has been crafted by CollectionsHandler.CollectionOperation.CREATE_OP and
+ // defines the QUEUE_OPERATION
// to be CollectionParams.CollectionAction.CREATE.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/api/collections/CreateCollectionCmd.java
##########
@@ -209,58 +253,89 @@ public void call(ClusterState clusterState, ZkNodeProps message, NamedList<Objec
return;
}
- final ShardRequestTracker shardRequestTracker = CollectionHandlingUtils.asyncRequestTracker(async, ccc);
+ final ShardRequestTracker shardRequestTracker =
+ CollectionHandlingUtils.asyncRequestTracker(async, ccc);
if (log.isDebugEnabled()) {
- log.debug(formatString("Creating SolrCores for new collection {0}, shardNames {1} , message : {2}",
- collectionName, shardNames, message));
+ log.debug(
+ formatString(
+ "Creating SolrCores for new collection {0}, shardNames {1} , message : {2}",
+ collectionName, shardNames, message));
}
- Map<String,ShardRequest> coresToCreate = new LinkedHashMap<>();
+ Map<String, ShardRequest> coresToCreate = new LinkedHashMap<>();
ShardHandler shardHandler = ccc.newShardHandler();
final DistributedClusterStateUpdater.StateChangeRecorder scr;
// PRS collections update Zookeeper directly, so even if we run in distributed state update,
- // there's nothing to update in state.json for such collection in the loop over replica positions below.
+ // there's nothing to update in state.json for such collection in the loop over replica
+ // positions below.
if (!isPRS && ccc.getDistributedClusterStateUpdater().isDistributedStateUpdate()) {
- // The collection got created. Now we're adding replicas (and will update ZK only once when done adding).
- scr = ccc.getDistributedClusterStateUpdater().createStateChangeRecorder(collectionName, false);;
+ // The collection got created. Now we're adding replicas (and will update ZK only once when
+ // done adding).
+ scr =
+ ccc.getDistributedClusterStateUpdater()
+ .createStateChangeRecorder(collectionName, false);
+ ;
} else {
scr = null;
}
for (ReplicaPosition replicaPosition : replicaPositions) {
String nodeName = replicaPosition.node;
- String coreName = Assign.buildSolrCoreName(ccc.getSolrCloudManager().getDistribStateManager(), collectionName,
- ccc.getSolrCloudManager().getClusterStateProvider().getClusterState().getCollectionOrNull(collectionName),
- replicaPosition.shard, replicaPosition.type, true);
+ String coreName =
+ Assign.buildSolrCoreName(
+ ccc.getSolrCloudManager().getDistribStateManager(),
+ collectionName,
+ ccc.getSolrCloudManager()
+ .getClusterStateProvider()
+ .getClusterState()
+ .getCollectionOrNull(collectionName),
+ replicaPosition.shard,
+ replicaPosition.type,
+ true);
if (log.isDebugEnabled()) {
- log.debug(formatString("Creating core {0} as part of shard {1} of collection {2} on {3}"
- , coreName, replicaPosition.shard, collectionName, nodeName));
+ log.debug(
+ formatString(
+ "Creating core {0} as part of shard {1} of collection {2} on {3}",
+ coreName, replicaPosition.shard, collectionName, nodeName));
}
String baseUrl = zkStateReader.getBaseUrlForNodeName(nodeName);
// create the replica in the collection's state.json in ZK prior to creating the core.
// Otherwise the core creation fails
- ZkNodeProps props = new ZkNodeProps(
- Overseer.QUEUE_OPERATION, ADDREPLICA.toString(),
- ZkStateReader.COLLECTION_PROP, collectionName,
- ZkStateReader.SHARD_ID_PROP, replicaPosition.shard,
- ZkStateReader.CORE_NAME_PROP, coreName,
- ZkStateReader.STATE_PROP, Replica.State.DOWN.toString(),
- ZkStateReader.NODE_NAME_PROP, nodeName,
- ZkStateReader.BASE_URL_PROP, baseUrl,
- ZkStateReader.REPLICA_TYPE, replicaPosition.type.name(),
- CommonAdminParams.WAIT_FOR_FINAL_STATE, Boolean.toString(waitForFinalState));
+ ZkNodeProps props =
+ new ZkNodeProps(
+ Overseer.QUEUE_OPERATION,
+ ADDREPLICA.toString(),
+ ZkStateReader.COLLECTION_PROP,
+ collectionName,
+ ZkStateReader.SHARD_ID_PROP,
+ replicaPosition.shard,
+ ZkStateReader.CORE_NAME_PROP,
+ coreName,
+ ZkStateReader.STATE_PROP,
+ Replica.State.DOWN.toString(),
+ ZkStateReader.NODE_NAME_PROP,
+ nodeName,
+ ZkStateReader.BASE_URL_PROP,
+ baseUrl,
+ ZkStateReader.REPLICA_TYPE,
+ replicaPosition.type.name(),
+ CommonAdminParams.WAIT_FOR_FINAL_STATE,
+ Boolean.toString(waitForFinalState));
if (isPRS) {
// In case of a PRS collection, execute the ADDREPLICA directly instead of resubmitting
// to the overseer queue.
// TODO: Consider doing this for all collections, not just the PRS collections.
- // TODO: consider doing this once after the loop for all replicas rather than writing state.json repeatedly
- // This PRS specific code is compatible with both Overseer and distributed cluster state update strategies
- ZkWriteCommand command = new SliceMutator(ccc.getSolrCloudManager()).addReplica(clusterState, props);
+ // TODO: consider doing this once after the loop for all replicas rather than writing
+ // state.json repeatedly
+ // This PRS specific code is compatible with both Overseer and distributed cluster state
+ // update strategies
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/api/collections/CreateCollectionCmd.java
##########
@@ -358,27 +457,38 @@ public void call(ClusterState clusterState, ZkNodeProps message, NamedList<Objec
}
if (failure) {
// Let's cleanup as we hit an exception
- // We shouldn't be passing 'results' here for the cleanup as the response would then contain 'success'
+ // We shouldn't be passing 'results' here for the cleanup as the response would then contain
+ // 'success'
// element, which may be interpreted by the user as a positive ack
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/api/collections/CreateCollectionCmd.java
##########
@@ -318,12 +397,22 @@ public void call(ClusterState clusterState, ZkNodeProps message, NamedList<Objec
final Map<String, Replica> replicas;
if (isPRS) {
replicas = new ConcurrentHashMap<>();
- newColl.getSlices().stream().flatMap(slice -> slice.getReplicas().stream())
- .filter(r -> coresToCreate.containsKey(r.getCoreName())) // Only the elements that were asked for...
- .forEach(r -> replicas.putIfAbsent(r.getCoreName(), r)); // ...get added to the map
+ newColl.getSlices().stream()
+ .flatMap(slice -> slice.getReplicas().stream())
+ .filter(
+ r ->
+ coresToCreate.containsKey(
+ r.getCoreName())) // Only the elements that were asked for...
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/api/collections/CreateCollectionCmd.java
##########
@@ -295,21 +370,25 @@ public void call(ClusterState clusterState, ZkNodeProps message, NamedList<Objec
sreq.nodeName = nodeName;
params.set("qt", ccc.getAdminPath());
sreq.purpose = ShardRequest.PURPOSE_PRIVATE;
- sreq.shards = new String[]{baseUrl};
+ sreq.shards = new String[] {baseUrl};
sreq.actualShards = sreq.shards;
sreq.params = params;
coresToCreate.put(coreName, sreq);
}
- // PRS collections updated ZK state.json in the loop above. When Overseer is managing cluster state updates, need to
- // tell it to refresh itself to know about the replicas and be able to execute nodes shard requests regarding the replicas.
+ // PRS collections updated ZK state.json in the loop above. When Overseer is managing cluster
+ // state updates, need to
+ // tell it to refresh itself to know about the replicas and be able to execute nodes shard
+ // requests regarding the replicas.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/api/collections/CreateCollectionCmd.java
##########
@@ -128,75 +133,114 @@ public void call(ClusterState clusterState, ZkNodeProps message, NamedList<Objec
ZkStateReader zkStateReader = ccc.getZkStateReader();
message.getProperties().put(COLL_CONF, configName);
- Map<String,String> collectionParams = new HashMap<>();
- Map<String,Object> collectionProps = message.getProperties();
+ Map<String, String> collectionParams = new HashMap<>();
+ Map<String, Object> collectionProps = message.getProperties();
for (Map.Entry<String, Object> entry : collectionProps.entrySet()) {
String propName = entry.getKey();
if (propName.startsWith(ZkController.COLLECTION_PARAM_PREFIX)) {
- collectionParams.put(propName.substring(ZkController.COLLECTION_PARAM_PREFIX.length()), (String) entry.getValue());
+ collectionParams.put(
+ propName.substring(ZkController.COLLECTION_PARAM_PREFIX.length()),
+ (String) entry.getValue());
}
}
- createCollectionZkNode(ccc.getSolrCloudManager().getDistribStateManager(), collectionName, collectionParams, ccc.getCoreContainer().getConfigSetService());
+ createCollectionZkNode(
+ ccc.getSolrCloudManager().getDistribStateManager(),
+ collectionName,
+ collectionParams,
+ ccc.getCoreContainer().getConfigSetService());
- // Note that in code below there are two main execution paths: Overseer based cluster state updates and distributed
+ // Note that in code below there are two main execution paths: Overseer based cluster state
+ // updates and distributed
// cluster state updates (look for isDistributedStateUpdate() conditions).
//
- // PerReplicaStates (PRS) collections follow a hybrid approach. Even when the cluster is Overseer cluster state update based,
- // these collections are created locally then the cluster state updater is notified (look for usage of RefreshCollectionMessage).
- // This explains why PRS collections have less diverging execution paths between distributed or Overseer based cluster state updates.
+ // PerReplicaStates (PRS) collections follow a hybrid approach. Even when the cluster is
+ // Overseer cluster state update based,
+ // these collections are created locally then the cluster state updater is notified (look for
+ // usage of RefreshCollectionMessage).
+ // This explains why PRS collections have less diverging execution paths between distributed
+ // or Overseer based cluster state updates.
if (isPRS) {
- // In case of a PRS collection, create the collection structure directly instead of resubmitting
+ // In case of a PRS collection, create the collection structure directly instead of
+ // resubmitting
// to the overseer queue.
// TODO: Consider doing this for all collections, not just the PRS collections.
// TODO comment above achieved by switching the cluster to distributed state updates
- // This code directly updates Zookeeper by creating the collection state.json. It is compatible with both distributed
+ // This code directly updates Zookeeper by creating the collection state.json. It is
+ // compatible with both distributed
// cluster state updates and Overseer based cluster state updates.
- ZkWriteCommand command = new ClusterStateMutator(ccc.getSolrCloudManager()).createCollection(clusterState, message);
+ ZkWriteCommand command =
+ new ClusterStateMutator(ccc.getSolrCloudManager())
+ .createCollection(clusterState, message);
byte[] data = Utils.toJSON(Collections.singletonMap(collectionName, command.collection));
- ccc.getZkStateReader().getZkClient().create(collectionPath, data, CreateMode.PERSISTENT, true);
+ ccc.getZkStateReader()
+ .getZkClient()
+ .create(collectionPath, data, CreateMode.PERSISTENT, true);
clusterState = clusterState.copyWith(collectionName, command.collection);
newColl = command.collection;
- // When cluster state updates are handled by Overseer, ask it to load that collection it doesn't know about.
- // When cluster state updates are distributed, ZK is the source of truth for all nodes so no reload needed.
+ // When cluster state updates are handled by Overseer, ask it to load that collection it
+ // doesn't know about.
+ // When cluster state updates are distributed, ZK is the source of truth for all nodes so no
+ // reload needed.
if (!ccc.getDistributedClusterStateUpdater().isDistributedStateUpdate()) {
- // If cluster state update is not distributed and we execute here, the Collection API is not distributed either
- // and this execution happens on the Overseer node, so direct memory access as done below is ok.
+ // If cluster state update is not distributed and we execute here, the Collection API is
+ // not distributed either
+ // and this execution happens on the Overseer node, so direct memory access as done below
+ // is ok.
ccc.submitIntraProcessMessage(new RefreshCollectionMessage(collectionName));
}
} else {
if (ccc.getDistributedClusterStateUpdater().isDistributedStateUpdate()) {
- // The message has been crafted by CollectionsHandler.CollectionOperation.CREATE_OP and defines the QUEUE_OPERATION
+ // The message has been crafted by CollectionsHandler.CollectionOperation.CREATE_OP and
+ // defines the QUEUE_OPERATION
// to be CollectionParams.CollectionAction.CREATE.
- ccc.getDistributedClusterStateUpdater().doSingleStateUpdate(DistributedClusterStateUpdater.MutatingCommand.ClusterCreateCollection, message,
- ccc.getSolrCloudManager(), ccc.getZkStateReader());
+ ccc.getDistributedClusterStateUpdater()
+ .doSingleStateUpdate(
+ DistributedClusterStateUpdater.MutatingCommand.ClusterCreateCollection,
+ message,
+ ccc.getSolrCloudManager(),
+ ccc.getZkStateReader());
} else {
ccc.offerStateUpdate(Utils.toJSON(message));
}
// wait for a while until we see the collection
- TimeOut waitUntil = new TimeOut(30, TimeUnit.SECONDS, ccc.getSolrCloudManager().getTimeSource());
+ TimeOut waitUntil =
+ new TimeOut(30, TimeUnit.SECONDS, ccc.getSolrCloudManager().getTimeSource());
boolean created = false;
while (!waitUntil.hasTimedOut()) {
waitUntil.sleep(100);
- created = ccc.getSolrCloudManager().getClusterStateProvider().getClusterState().hasCollection(collectionName);
+ created =
+ ccc.getSolrCloudManager()
+ .getClusterStateProvider()
+ .getClusterState()
+ .hasCollection(collectionName);
if (created) break;
}
if (!created) {
- throw new SolrException(SolrException.ErrorCode.SERVER_ERROR, "Could not fully create collection: " + collectionName);
+ throw new SolrException(
+ SolrException.ErrorCode.SERVER_ERROR,
+ "Could not fully create collection: " + collectionName);
}
- // refresh cluster state (value read below comes from Zookeeper watch firing following the update done previously,
+ // refresh cluster state (value read below comes from Zookeeper watch firing following the
+ // update done previously,
// be it by Overseer or by this thread when updates are distributed)
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/api/collections/CreateCollectionCmd.java
##########
@@ -295,21 +370,25 @@ public void call(ClusterState clusterState, ZkNodeProps message, NamedList<Objec
sreq.nodeName = nodeName;
params.set("qt", ccc.getAdminPath());
sreq.purpose = ShardRequest.PURPOSE_PRIVATE;
- sreq.shards = new String[]{baseUrl};
+ sreq.shards = new String[] {baseUrl};
sreq.actualShards = sreq.shards;
sreq.params = params;
coresToCreate.put(coreName, sreq);
}
- // PRS collections updated ZK state.json in the loop above. When Overseer is managing cluster state updates, need to
- // tell it to refresh itself to know about the replicas and be able to execute nodes shard requests regarding the replicas.
+ // PRS collections updated ZK state.json in the loop above. When Overseer is managing cluster
+ // state updates, need to
+ // tell it to refresh itself to know about the replicas and be able to execute nodes shard
+ // requests regarding the replicas.
if (isPRS && !ccc.getDistributedClusterStateUpdater().isDistributedStateUpdate()) {
ccc.submitIntraProcessMessage(new RefreshCollectionMessage(collectionName));
}
- // Distributed updates don't need to do anything for PRS collections that wrote state.json directly
- // For non PRS collections, distributed updates have to be executed if that's how the cluster is configured
+ // Distributed updates don't need to do anything for PRS collections that wrote state.json
+ // directly
+ // For non PRS collections, distributed updates have to be executed if that's how the cluster
+ // is configured
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/api/collections/CreateCollectionCmd.java
##########
@@ -553,17 +697,22 @@ public static void createCollectionZkNode(DistribStateManager stateManager, Stri
// the conf name should should be the collection name of this core
collectionProps.put(ZkController.CONFIGNAME_PROP, collection);
} else {
- getConfName(stateManager, collection, collectionPath, collectionProps, configSetService);
+ getConfName(
+ stateManager, collection, collectionPath, collectionProps, configSetService);
}
- collectionProps.remove(ZkStateReader.NUM_SHARDS_PROP); // we don't put numShards in the collections properties
- collectionProps.remove(ZkStateReader.CONFIGNAME_PROP); // we don't write configName on a zk collection node
+ collectionProps.remove(
+ ZkStateReader
+ .NUM_SHARDS_PROP); // we don't put numShards in the collections properties
+ collectionProps.remove(
+ ZkStateReader.CONFIGNAME_PROP); // we don't write configName on a zk collection node
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/api/collections/CreateCollectionCmd.java
##########
@@ -128,75 +133,114 @@ public void call(ClusterState clusterState, ZkNodeProps message, NamedList<Objec
ZkStateReader zkStateReader = ccc.getZkStateReader();
message.getProperties().put(COLL_CONF, configName);
- Map<String,String> collectionParams = new HashMap<>();
- Map<String,Object> collectionProps = message.getProperties();
+ Map<String, String> collectionParams = new HashMap<>();
+ Map<String, Object> collectionProps = message.getProperties();
for (Map.Entry<String, Object> entry : collectionProps.entrySet()) {
String propName = entry.getKey();
if (propName.startsWith(ZkController.COLLECTION_PARAM_PREFIX)) {
- collectionParams.put(propName.substring(ZkController.COLLECTION_PARAM_PREFIX.length()), (String) entry.getValue());
+ collectionParams.put(
+ propName.substring(ZkController.COLLECTION_PARAM_PREFIX.length()),
+ (String) entry.getValue());
}
}
- createCollectionZkNode(ccc.getSolrCloudManager().getDistribStateManager(), collectionName, collectionParams, ccc.getCoreContainer().getConfigSetService());
+ createCollectionZkNode(
+ ccc.getSolrCloudManager().getDistribStateManager(),
+ collectionName,
+ collectionParams,
+ ccc.getCoreContainer().getConfigSetService());
- // Note that in code below there are two main execution paths: Overseer based cluster state updates and distributed
+ // Note that in code below there are two main execution paths: Overseer based cluster state
+ // updates and distributed
// cluster state updates (look for isDistributedStateUpdate() conditions).
//
- // PerReplicaStates (PRS) collections follow a hybrid approach. Even when the cluster is Overseer cluster state update based,
- // these collections are created locally then the cluster state updater is notified (look for usage of RefreshCollectionMessage).
- // This explains why PRS collections have less diverging execution paths between distributed or Overseer based cluster state updates.
+ // PerReplicaStates (PRS) collections follow a hybrid approach. Even when the cluster is
+ // Overseer cluster state update based,
+ // these collections are created locally then the cluster state updater is notified (look for
+ // usage of RefreshCollectionMessage).
+ // This explains why PRS collections have less diverging execution paths between distributed
+ // or Overseer based cluster state updates.
if (isPRS) {
- // In case of a PRS collection, create the collection structure directly instead of resubmitting
+ // In case of a PRS collection, create the collection structure directly instead of
+ // resubmitting
// to the overseer queue.
// TODO: Consider doing this for all collections, not just the PRS collections.
// TODO comment above achieved by switching the cluster to distributed state updates
- // This code directly updates Zookeeper by creating the collection state.json. It is compatible with both distributed
+ // This code directly updates Zookeeper by creating the collection state.json. It is
+ // compatible with both distributed
// cluster state updates and Overseer based cluster state updates.
- ZkWriteCommand command = new ClusterStateMutator(ccc.getSolrCloudManager()).createCollection(clusterState, message);
+ ZkWriteCommand command =
+ new ClusterStateMutator(ccc.getSolrCloudManager())
+ .createCollection(clusterState, message);
byte[] data = Utils.toJSON(Collections.singletonMap(collectionName, command.collection));
- ccc.getZkStateReader().getZkClient().create(collectionPath, data, CreateMode.PERSISTENT, true);
+ ccc.getZkStateReader()
+ .getZkClient()
+ .create(collectionPath, data, CreateMode.PERSISTENT, true);
clusterState = clusterState.copyWith(collectionName, command.collection);
newColl = command.collection;
- // When cluster state updates are handled by Overseer, ask it to load that collection it doesn't know about.
- // When cluster state updates are distributed, ZK is the source of truth for all nodes so no reload needed.
+ // When cluster state updates are handled by Overseer, ask it to load that collection it
+ // doesn't know about.
+ // When cluster state updates are distributed, ZK is the source of truth for all nodes so no
+ // reload needed.
if (!ccc.getDistributedClusterStateUpdater().isDistributedStateUpdate()) {
- // If cluster state update is not distributed and we execute here, the Collection API is not distributed either
- // and this execution happens on the Overseer node, so direct memory access as done below is ok.
+ // If cluster state update is not distributed and we execute here, the Collection API is
+ // not distributed either
+ // and this execution happens on the Overseer node, so direct memory access as done below
+ // is ok.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/api/collections/CreateShardCmd.java
##########
@@ -52,93 +50,126 @@ public CreateShardCmd(CollectionCommandContext ccc) {
}
@Override
- public void call(ClusterState clusterState, ZkNodeProps message, NamedList<Object> results) throws Exception {
+ public void call(ClusterState clusterState, ZkNodeProps message, NamedList<Object> results)
+ throws Exception {
String extCollectionName = message.getStr(COLLECTION_PROP);
String sliceName = message.getStr(SHARD_ID_PROP);
boolean waitForFinalState = message.getBool(CommonAdminParams.WAIT_FOR_FINAL_STATE, false);
log.info("Create shard invoked: {}", message);
if (extCollectionName == null || sliceName == null)
- throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "'collection' and 'shard' are required parameters");
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST, "'collection' and 'shard' are required parameters");
boolean followAliases = message.getBool(FOLLOW_ALIASES, false);
String collectionName;
if (followAliases) {
- collectionName = ccc.getSolrCloudManager().getClusterStateProvider().resolveSimpleAlias(extCollectionName);
+ collectionName =
+ ccc.getSolrCloudManager().getClusterStateProvider().resolveSimpleAlias(extCollectionName);
} else {
collectionName = extCollectionName;
}
DocCollection collection = clusterState.getCollection(collectionName);
- int numNrtReplicas = message.getInt(NRT_REPLICAS, message.getInt(REPLICATION_FACTOR, collection.getInt(NRT_REPLICAS, collection.getInt(REPLICATION_FACTOR, 1))));
+ int numNrtReplicas =
+ message.getInt(
+ NRT_REPLICAS,
+ message.getInt(
+ REPLICATION_FACTOR,
+ collection.getInt(NRT_REPLICAS, collection.getInt(REPLICATION_FACTOR, 1))));
int numPullReplicas = message.getInt(PULL_REPLICAS, collection.getInt(PULL_REPLICAS, 0));
int numTlogReplicas = message.getInt(TLOG_REPLICAS, collection.getInt(TLOG_REPLICAS, 0));
if (numNrtReplicas + numTlogReplicas <= 0) {
- throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, NRT_REPLICAS + " + " + TLOG_REPLICAS + " must be greater than 0");
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ NRT_REPLICAS + " + " + TLOG_REPLICAS + " must be greater than 0");
}
if (ccc.getDistributedClusterStateUpdater().isDistributedStateUpdate()) {
- // The message has been crafted by CollectionsHandler.CollectionOperation.CREATESHARD_OP and defines the QUEUE_OPERATION
+ // The message has been crafted by CollectionsHandler.CollectionOperation.CREATESHARD_OP and
+ // defines the QUEUE_OPERATION
// to be CollectionParams.CollectionAction.CREATESHARD.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/api/collections/CreateCollectionCmd.java
##########
@@ -553,17 +697,22 @@ public static void createCollectionZkNode(DistribStateManager stateManager, Stri
// the conf name should should be the collection name of this core
collectionProps.put(ZkController.CONFIGNAME_PROP, collection);
} else {
- getConfName(stateManager, collection, collectionPath, collectionProps, configSetService);
+ getConfName(
+ stateManager, collection, collectionPath, collectionProps, configSetService);
}
- collectionProps.remove(ZkStateReader.NUM_SHARDS_PROP); // we don't put numShards in the collections properties
- collectionProps.remove(ZkStateReader.CONFIGNAME_PROP); // we don't write configName on a zk collection node
+ collectionProps.remove(
+ ZkStateReader
+ .NUM_SHARDS_PROP); // we don't put numShards in the collections properties
+ collectionProps.remove(
+ ZkStateReader.CONFIGNAME_PROP); // we don't write configName on a zk collection node
// create a node
stateManager.makePath(collectionPath);
} catch (KeeperException e) {
- //TODO shouldn't the stateManager ensure this does not happen; should throw AlreadyExistsException
+ // TODO shouldn't the stateManager ensure this does not happen; should throw
+ // AlreadyExistsException
// it's okay if the node already exists
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/api/collections/DeleteCollectionCmd.java
##########
@@ -105,9 +106,13 @@ public void call(ClusterState state, ZkNodeProps message, NamedList<Object> resu
SolrSnapshotManager.cleanupCollectionLevelSnapshots(zkClient, collection);
if (zkStateReader.getClusterState().getCollectionOrNull(collection) == null) {
- if (zkStateReader.getZkClient().exists(ZkStateReader.COLLECTIONS_ZKNODE + "/" + collection, true)) {
- // if the collection is not in the clusterstate, but is listed in zk, do nothing, it will just
- // be removed in the finally - we cannot continue, because the below code will error if the collection
+ if (zkStateReader
+ .getZkClient()
+ .exists(ZkStateReader.COLLECTIONS_ZKNODE + "/" + collection, true)) {
+ // if the collection is not in the clusterstate, but is listed in zk, do nothing, it will
+ // just
+ // be removed in the finally - we cannot continue, because the below code will error if
+ // the collection
// is not in the clusterstate
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/api/collections/DeleteCollectionCmd.java
##########
@@ -66,10 +65,12 @@ public DeleteCollectionCmd(CollectionCommandContext ccc) {
}
@Override
- public void call(ClusterState state, ZkNodeProps message, NamedList<Object> results) throws Exception {
+ public void call(ClusterState state, ZkNodeProps message, NamedList<Object> results)
+ throws Exception {
Object o = message.get(MaintainRoutedAliasCmd.INVOKED_BY_ROUTED_ALIAS);
if (o != null) {
- ((Runnable)o).run(); // this will ensure the collection is removed from the alias before it disappears.
+ ((Runnable) o)
+ .run(); // this will ensure the collection is removed from the alias before it disappears.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/api/collections/DeleteBackupCmd.java
##########
@@ -34,385 +52,418 @@
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
-import java.io.FileNotFoundException;
-import java.io.IOException;
-import java.lang.invoke.MethodHandles;
-import java.net.URI;
-import java.util.ArrayList;
-import java.util.Arrays;
-import java.util.Collections;
-import java.util.HashMap;
-import java.util.HashSet;
-import java.util.List;
-import java.util.Locale;
-import java.util.Map;
-import java.util.Set;
-import java.util.stream.Collectors;
-
-import static org.apache.solr.common.params.CommonParams.NAME;
-import static org.apache.solr.core.backup.BackupManager.COLLECTION_NAME_PROP;
-import static org.apache.solr.core.backup.BackupManager.START_TIME_PROP;
-
/**
* An overseer command used to delete files associated with incremental backups.
*
- * This assumes use of the incremental backup format, and not the (now deprecated) traditional 'full-snapshot' format.
- * The deletion can either delete a specific {@link BackupId}, delete everything except the most recent N backup
- * points, or can be used to trigger a "garbage collection" of unused index files in the backup repository.
+ * <p>This assumes use of the incremental backup format, and not the (now deprecated) traditional
+ * 'full-snapshot' format. The deletion can either delete a specific {@link BackupId}, delete
+ * everything except the most recent N backup points, or can be used to trigger a "garbage
+ * collection" of unused index files in the backup repository.
*/
public class DeleteBackupCmd implements CollApiCmds.CollectionApiCommand {
- private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
-
- private final CollectionCommandContext ccc;
-
- public DeleteBackupCmd(CollectionCommandContext ccc) {
- this.ccc = ccc;
+ private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ private final CollectionCommandContext ccc;
+
+ public DeleteBackupCmd(CollectionCommandContext ccc) {
+ this.ccc = ccc;
+ }
+
+ @Override
+ public void call(ClusterState state, ZkNodeProps message, NamedList<Object> results)
+ throws Exception {
+ String backupLocation = message.getStr(CoreAdminParams.BACKUP_LOCATION);
+ String backupName = message.getStr(NAME);
+ String repo = message.getStr(CoreAdminParams.BACKUP_REPOSITORY);
+ int backupId = message.getInt(CoreAdminParams.BACKUP_ID, -1);
+ int lastNumBackupPointsToKeep = message.getInt(CoreAdminParams.MAX_NUM_BACKUP_POINTS, -1);
+ boolean purge = message.getBool(CoreAdminParams.BACKUP_PURGE_UNUSED, false);
+ if (backupId == -1 && lastNumBackupPointsToKeep == -1 && !purge) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ String.format(
+ Locale.ROOT,
+ "%s, %s or %s param must be provided",
+ CoreAdminParams.BACKUP_ID,
+ CoreAdminParams.MAX_NUM_BACKUP_POINTS,
+ CoreAdminParams.BACKUP_PURGE_UNUSED));
}
-
- @Override
- public void call(ClusterState state, ZkNodeProps message, NamedList<Object> results) throws Exception {
- String backupLocation = message.getStr(CoreAdminParams.BACKUP_LOCATION);
- String backupName = message.getStr(NAME);
- String repo = message.getStr(CoreAdminParams.BACKUP_REPOSITORY);
- int backupId = message.getInt(CoreAdminParams.BACKUP_ID, -1);
- int lastNumBackupPointsToKeep = message.getInt(CoreAdminParams.MAX_NUM_BACKUP_POINTS, -1);
- boolean purge = message.getBool(CoreAdminParams.BACKUP_PURGE_UNUSED, false);
- if (backupId == -1 && lastNumBackupPointsToKeep == -1 && !purge) {
- throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, String.format(Locale.ROOT, "%s, %s or %s param must be provided", CoreAdminParams.BACKUP_ID, CoreAdminParams.MAX_NUM_BACKUP_POINTS,
- CoreAdminParams.BACKUP_PURGE_UNUSED));
- }
- CoreContainer cc = ccc.getCoreContainer();
- try (BackupRepository repository = cc.newBackupRepository(repo)) {
- URI location = repository.createDirectoryURI(backupLocation);
- final URI backupPath = BackupFilePaths.buildExistingBackupLocationURI(repository, location, backupName);
- if (repository.exists(repository.resolve(backupPath, BackupManager.TRADITIONAL_BACKUP_PROPS_FILE))) {
- throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "The backup name [" + backupName + "] at " +
- "location [" + location + "] holds a non-incremental (legacy) backup, but " +
- "backup-deletion is only supported on incremental backups");
- }
-
- if (purge) {
- purge(repository, backupPath, results);
- } else if (backupId != -1){
- deleteBackupId(repository, backupPath, backupId, results);
- } else {
- keepNumberOfBackup(repository, backupPath, lastNumBackupPointsToKeep, results);
- }
- }
+ CoreContainer cc = ccc.getCoreContainer();
+ try (BackupRepository repository = cc.newBackupRepository(repo)) {
+ URI location = repository.createDirectoryURI(backupLocation);
+ final URI backupPath =
+ BackupFilePaths.buildExistingBackupLocationURI(repository, location, backupName);
+ if (repository.exists(
+ repository.resolve(backupPath, BackupManager.TRADITIONAL_BACKUP_PROPS_FILE))) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "The backup name ["
+ + backupName
+ + "] at "
+ + "location ["
+ + location
+ + "] holds a non-incremental (legacy) backup, but "
+ + "backup-deletion is only supported on incremental backups");
+ }
+
+ if (purge) {
+ purge(repository, backupPath, results);
+ } else if (backupId != -1) {
+ deleteBackupId(repository, backupPath, backupId, results);
+ } else {
+ keepNumberOfBackup(repository, backupPath, lastNumBackupPointsToKeep, results);
+ }
}
-
- /**
- * Clean up {@code backupPath} by removing all index files, shard-metadata files, and backup property files that are
- * unreachable, uncompleted or corrupted.
- */
- void purge(BackupRepository repository, URI backupPath, NamedList<Object> result) throws IOException {
- PurgeGraph purgeGraph = new PurgeGraph();
- purgeGraph.build(repository, backupPath);
-
- BackupFilePaths backupPaths = new BackupFilePaths(repository, backupPath);
- repository.delete(backupPaths.getIndexDir(), purgeGraph.indexFileDeletes, true);
- repository.delete(backupPaths.getShardBackupMetadataDir(), purgeGraph.shardBackupMetadataDeletes, true);
- repository.delete(backupPath, purgeGraph.backupIdDeletes, true);
-
- NamedList<Integer> details = new NamedList<>();
- details.add("numBackupIds", purgeGraph.backupIdDeletes.size());
- details.add("numShardBackupIds", purgeGraph.shardBackupMetadataDeletes.size());
- details.add("numIndexFiles", purgeGraph.indexFileDeletes.size());
- result.add("deleted", details);
+ }
+
+ /**
+ * Clean up {@code backupPath} by removing all index files, shard-metadata files, and backup
+ * property files that are unreachable, uncompleted or corrupted.
+ */
+ void purge(BackupRepository repository, URI backupPath, NamedList<Object> result)
+ throws IOException {
+ PurgeGraph purgeGraph = new PurgeGraph();
+ purgeGraph.build(repository, backupPath);
+
+ BackupFilePaths backupPaths = new BackupFilePaths(repository, backupPath);
+ repository.delete(backupPaths.getIndexDir(), purgeGraph.indexFileDeletes, true);
+ repository.delete(
+ backupPaths.getShardBackupMetadataDir(), purgeGraph.shardBackupMetadataDeletes, true);
+ repository.delete(backupPath, purgeGraph.backupIdDeletes, true);
+
+ NamedList<Integer> details = new NamedList<>();
+ details.add("numBackupIds", purgeGraph.backupIdDeletes.size());
+ details.add("numShardBackupIds", purgeGraph.shardBackupMetadataDeletes.size());
+ details.add("numIndexFiles", purgeGraph.indexFileDeletes.size());
+ result.add("deleted", details);
+ }
+
+ /** Keep most recent {@code maxNumBackup} and delete the rest. */
+ void keepNumberOfBackup(
+ BackupRepository repository, URI backupPath, int maxNumBackup, NamedList<Object> results)
+ throws Exception {
+ List<BackupId> backupIds =
+ BackupFilePaths.findAllBackupIdsFromFileListing(repository.listAllOrEmpty(backupPath));
+ if (backupIds.size() <= maxNumBackup) {
+ return;
}
- /**
- * Keep most recent {@code maxNumBackup} and delete the rest.
- */
- void keepNumberOfBackup(BackupRepository repository, URI backupPath,
- int maxNumBackup,
- NamedList<Object> results) throws Exception {
- List<BackupId> backupIds = BackupFilePaths.findAllBackupIdsFromFileListing(repository.listAllOrEmpty(backupPath));
- if (backupIds.size() <= maxNumBackup) {
- return;
- }
-
- Collections.sort(backupIds);
- List<BackupId> backupIdDeletes = backupIds.subList(0, backupIds.size() - maxNumBackup);
- deleteBackupIds(backupPath, repository, new HashSet<>(backupIdDeletes), results);
+ Collections.sort(backupIds);
+ List<BackupId> backupIdDeletes = backupIds.subList(0, backupIds.size() - maxNumBackup);
+ deleteBackupIds(backupPath, repository, new HashSet<>(backupIdDeletes), results);
+ }
+
+ void deleteBackupIds(
+ URI backupUri,
+ BackupRepository repository,
+ Set<BackupId> backupIdsDeletes,
+ NamedList<Object> results)
+ throws IOException {
+ BackupFilePaths incBackupFiles = new BackupFilePaths(repository, backupUri);
+ URI shardBackupMetadataDir = incBackupFiles.getShardBackupMetadataDir();
+
+ Set<String> referencedIndexFiles = new HashSet<>();
+ List<ShardBackupId> shardBackupIdFileDeletes = new ArrayList<>();
+
+ List<ShardBackupId> shardBackupIds =
+ Arrays.stream(repository.listAllOrEmpty(shardBackupMetadataDir))
+ .map(sbi -> ShardBackupId.fromShardMetadataFilename(sbi))
+ .collect(Collectors.toList());
+ for (ShardBackupId shardBackupId : shardBackupIds) {
+ final BackupId backupId = shardBackupId.getContainingBackupId();
+
+ if (backupIdsDeletes.contains(backupId)) {
+ shardBackupIdFileDeletes.add(shardBackupId);
+ } else {
+ ShardBackupMetadata shardBackupMetadata =
+ ShardBackupMetadata.from(repository, shardBackupMetadataDir, shardBackupId);
+ if (shardBackupMetadata != null)
+ referencedIndexFiles.addAll(shardBackupMetadata.listUniqueFileNames());
+ }
}
- void deleteBackupIds(URI backupUri, BackupRepository repository,
- Set<BackupId> backupIdsDeletes,
- NamedList<Object> results) throws IOException {
- BackupFilePaths incBackupFiles = new BackupFilePaths(repository, backupUri);
- URI shardBackupMetadataDir = incBackupFiles.getShardBackupMetadataDir();
-
- Set<String> referencedIndexFiles = new HashSet<>();
- List<ShardBackupId> shardBackupIdFileDeletes = new ArrayList<>();
-
+ Map<BackupId, AggregateBackupStats> backupIdToCollectionBackupPoint = new HashMap<>();
+ List<String> unusedFiles = new ArrayList<>();
+ for (ShardBackupId shardBackupIdToDelete : shardBackupIdFileDeletes) {
+ BackupId backupId = shardBackupIdToDelete.getContainingBackupId();
+ ShardBackupMetadata shardBackupMetadata =
+ ShardBackupMetadata.from(repository, shardBackupMetadataDir, shardBackupIdToDelete);
+ if (shardBackupMetadata == null) continue;
- List<ShardBackupId> shardBackupIds = Arrays.stream(repository.listAllOrEmpty(shardBackupMetadataDir))
- .map(sbi -> ShardBackupId.fromShardMetadataFilename(sbi))
- .collect(Collectors.toList());
- for (ShardBackupId shardBackupId : shardBackupIds) {
- final BackupId backupId = shardBackupId.getContainingBackupId();
+ backupIdToCollectionBackupPoint.putIfAbsent(backupId, new AggregateBackupStats());
+ backupIdToCollectionBackupPoint.get(backupId).add(shardBackupMetadata);
- if (backupIdsDeletes.contains(backupId)) {
- shardBackupIdFileDeletes.add(shardBackupId);
- } else {
- ShardBackupMetadata shardBackupMetadata = ShardBackupMetadata.from(repository, shardBackupMetadataDir, shardBackupId);
- if (shardBackupMetadata != null)
- referencedIndexFiles.addAll(shardBackupMetadata.listUniqueFileNames());
- }
- }
-
-
- Map<BackupId, AggregateBackupStats> backupIdToCollectionBackupPoint = new HashMap<>();
- List<String> unusedFiles = new ArrayList<>();
- for (ShardBackupId shardBackupIdToDelete : shardBackupIdFileDeletes) {
- BackupId backupId = shardBackupIdToDelete.getContainingBackupId();
- ShardBackupMetadata shardBackupMetadata = ShardBackupMetadata.from(repository, shardBackupMetadataDir, shardBackupIdToDelete);
- if (shardBackupMetadata == null)
- continue;
-
- backupIdToCollectionBackupPoint
- .putIfAbsent(backupId, new AggregateBackupStats());
- backupIdToCollectionBackupPoint.get(backupId).add(shardBackupMetadata);
-
- for (String uniqueIndexFile : shardBackupMetadata.listUniqueFileNames()) {
- if (!referencedIndexFiles.contains(uniqueIndexFile)) {
- unusedFiles.add(uniqueIndexFile);
- }
- }
+ for (String uniqueIndexFile : shardBackupMetadata.listUniqueFileNames()) {
+ if (!referencedIndexFiles.contains(uniqueIndexFile)) {
+ unusedFiles.add(uniqueIndexFile);
}
-
- repository.delete(incBackupFiles.getShardBackupMetadataDir(),
- shardBackupIdFileDeletes.stream().map(ShardBackupId::getBackupMetadataFilename).collect(Collectors.toList()), true);
- repository.delete(incBackupFiles.getIndexDir(), unusedFiles, true);
- try {
- for (BackupId backupId : backupIdsDeletes) {
- repository.deleteDirectory(repository.resolveDirectory(backupUri, BackupFilePaths.getZkStateDir(backupId)));
- }
- } catch (FileNotFoundException e) {
- //ignore this
- }
-
- //add details to result before deleting backupPropFiles
- addResult(backupUri, repository, backupIdsDeletes, backupIdToCollectionBackupPoint, results);
- repository.delete(backupUri, backupIdsDeletes.stream().map(id -> BackupFilePaths.getBackupPropsName(id)).collect(Collectors.toList()), true);
+ }
}
- private void addResult(URI backupPath, BackupRepository repository,
- Set<BackupId> backupIdDeletes,
- Map<BackupId, AggregateBackupStats> backupIdToCollectionBackupPoint,
- NamedList<Object> results) {
-
- String collectionName = null;
- List<NamedList<Object>> shardBackupIdDetails = new ArrayList<>();
- results.add("deleted", shardBackupIdDetails);
- for (BackupId backupId : backupIdDeletes) {
- NamedList<Object> backupIdResult = new NamedList<>();
-
- try {
- BackupProperties props = BackupProperties.readFrom(repository, backupPath, BackupFilePaths.getBackupPropsName(backupId));
- backupIdResult.add(START_TIME_PROP, props.getStartTime());
- if (collectionName == null) {
- collectionName = props.getCollection();
- results.add(COLLECTION_NAME_PROP, collectionName);
- }
- } catch (IOException e) {
- //prop file not found
- }
-
- AggregateBackupStats cbp = backupIdToCollectionBackupPoint.getOrDefault(backupId, new AggregateBackupStats());
- backupIdResult.add("backupId", backupId.getId());
- backupIdResult.add("size", cbp.getTotalSize());
- backupIdResult.add("numFiles", cbp.getNumFiles());
- shardBackupIdDetails.add(backupIdResult);
- }
+ repository.delete(
+ incBackupFiles.getShardBackupMetadataDir(),
+ shardBackupIdFileDeletes.stream()
+ .map(ShardBackupId::getBackupMetadataFilename)
+ .collect(Collectors.toList()),
+ true);
+ repository.delete(incBackupFiles.getIndexDir(), unusedFiles, true);
+ try {
+ for (BackupId backupId : backupIdsDeletes) {
+ repository.deleteDirectory(
+ repository.resolveDirectory(backupUri, BackupFilePaths.getZkStateDir(backupId)));
+ }
+ } catch (FileNotFoundException e) {
+ // ignore this
}
- private void deleteBackupId(BackupRepository repository, URI backupPath,
- int bid, NamedList<Object> results) throws Exception {
- BackupId backupId = new BackupId(bid);
- if (!repository.exists(repository.resolve(backupPath, BackupFilePaths.getBackupPropsName(backupId)))) {
- throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "Backup ID [" + bid + "] not found; cannot be deleted");
+ // add details to result before deleting backupPropFiles
+ addResult(backupUri, repository, backupIdsDeletes, backupIdToCollectionBackupPoint, results);
+ repository.delete(
+ backupUri,
+ backupIdsDeletes.stream()
+ .map(id -> BackupFilePaths.getBackupPropsName(id))
+ .collect(Collectors.toList()),
+ true);
+ }
+
+ private void addResult(
+ URI backupPath,
+ BackupRepository repository,
+ Set<BackupId> backupIdDeletes,
+ Map<BackupId, AggregateBackupStats> backupIdToCollectionBackupPoint,
+ NamedList<Object> results) {
+
+ String collectionName = null;
+ List<NamedList<Object>> shardBackupIdDetails = new ArrayList<>();
+ results.add("deleted", shardBackupIdDetails);
+ for (BackupId backupId : backupIdDeletes) {
+ NamedList<Object> backupIdResult = new NamedList<>();
+
+ try {
+ BackupProperties props =
+ BackupProperties.readFrom(
+ repository, backupPath, BackupFilePaths.getBackupPropsName(backupId));
+ backupIdResult.add(START_TIME_PROP, props.getStartTime());
+ if (collectionName == null) {
+ collectionName = props.getCollection();
+ results.add(COLLECTION_NAME_PROP, collectionName);
}
-
- deleteBackupIds(backupPath, repository, Collections.singleton(backupId), results);
+ } catch (IOException e) {
+ // prop file not found
+ }
+
+ AggregateBackupStats cbp =
+ backupIdToCollectionBackupPoint.getOrDefault(backupId, new AggregateBackupStats());
+ backupIdResult.add("backupId", backupId.getId());
+ backupIdResult.add("size", cbp.getTotalSize());
+ backupIdResult.add("numFiles", cbp.getNumFiles());
+ shardBackupIdDetails.add(backupIdResult);
+ }
+ }
+
+ private void deleteBackupId(
+ BackupRepository repository, URI backupPath, int bid, NamedList<Object> results)
+ throws Exception {
+ BackupId backupId = new BackupId(bid);
+ if (!repository.exists(
+ repository.resolve(backupPath, BackupFilePaths.getBackupPropsName(backupId)))) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Backup ID [" + bid + "] not found; cannot be deleted");
}
- final static class PurgeGraph {
- // graph
- Map<String, Node> backupIdNodeMap = new HashMap<>();
- Map<String, Node> shardBackupMetadataNodeMap = new HashMap<>();
- Map<String, Node> indexFileNodeMap = new HashMap<>();
-
- // delete queues
- List<String> backupIdDeletes = new ArrayList<>();
- List<String> shardBackupMetadataDeletes = new ArrayList<>();
- List<String> indexFileDeletes = new ArrayList<>();
-
- public void build(BackupRepository repository, URI backupPath) throws IOException {
- BackupFilePaths backupPaths = new BackupFilePaths(repository, backupPath);
- buildLogicalGraph(repository, backupPaths);
-
- findDeletableNodes(repository, backupPaths);
- }
-
- public void findDeletableNodes(BackupRepository repository, BackupFilePaths backupPaths) {
- // mark nodes as existing
- visitExistingNodes(repository.listAllOrEmpty(backupPaths.getShardBackupMetadataDir()),
- shardBackupMetadataNodeMap, shardBackupMetadataDeletes);
- // this may be a long running commands
- visitExistingNodes(repository.listAllOrEmpty(backupPaths.getIndexDir()),
- indexFileNodeMap, indexFileDeletes);
-
- // TODO Is this propagation logic really necessary?
- // The intention seems to be that if some index files are only referenced by a shard-metadata file that
- // is itself orphaned, then propagating the "orphaned" status down to each of these index files will allow
- // them to be deleted.
- //
- // But that doesn't seem to hold up under closer inspection.
- //
- // The graph is populated by following links out from the set of valid backup-id's. All files (shard-
- // metadata, or index) that trace back to a valid backup-ID will appear in the graph. If a shard-metadata
- // file becomes orphaned, it will be ignored during graph construction and any index files that it alone
- // references will not appear in the graph. Since those index files will be unrepresented in the graph, the
- // 'visitExistingNodes' calls above should be sufficient to detect them as orphaned.
- //
- // This all assumes though that propagation is intended to solve the scenario I think it does. If that
- // assumption is wrong, then this whole comment is wrong.
-
- // for nodes which are not existing, propagate that information to other nodes
- shardBackupMetadataNodeMap.values().forEach(Node::propagateNotExisting);
- indexFileNodeMap.values().forEach(Node::propagateNotExisting);
-
- addDeleteNodesToQueue(backupIdNodeMap, backupIdDeletes);
- addDeleteNodesToQueue(shardBackupMetadataNodeMap, shardBackupMetadataDeletes);
- addDeleteNodesToQueue(indexFileNodeMap, indexFileDeletes);
- }
+ deleteBackupIds(backupPath, repository, Collections.singleton(backupId), results);
+ }
- /**
- * Visiting files (nodes) actually present in physical layer,
- * if it does not present in the {@code nodeMap}, it should be deleted by putting into the {@code deleteQueue}
- */
- private void visitExistingNodes(String[] existingNodeKeys, Map<String, Node> nodeMap, List<String> deleteQueue) {
- for (String nodeKey : existingNodeKeys) {
- Node node = nodeMap.get(nodeKey);
-
- if (node == null) {
- deleteQueue.add(nodeKey);
- } else {
- node.existing = true;
- }
- }
- }
+ static final class PurgeGraph {
+ // graph
+ Map<String, Node> backupIdNodeMap = new HashMap<>();
+ Map<String, Node> shardBackupMetadataNodeMap = new HashMap<>();
+ Map<String, Node> indexFileNodeMap = new HashMap<>();
- private <T> void addDeleteNodesToQueue(Map<T, Node> tNodeMap, List<T> deleteQueue) {
- tNodeMap.forEach((key, value) -> {
- if (value.delete) {
- deleteQueue.add(key);
- }
- });
- }
+ // delete queues
+ List<String> backupIdDeletes = new ArrayList<>();
+ List<String> shardBackupMetadataDeletes = new ArrayList<>();
+ List<String> indexFileDeletes = new ArrayList<>();
- Node getBackupIdNode(String backupPropsName) {
- return backupIdNodeMap.computeIfAbsent(backupPropsName, bid -> {
- Node node = new Node();
- node.existing = true;
- return node;
- });
- }
+ public void build(BackupRepository repository, URI backupPath) throws IOException {
+ BackupFilePaths backupPaths = new BackupFilePaths(repository, backupPath);
+ buildLogicalGraph(repository, backupPaths);
- Node getShardBackupIdNode(String shardBackupId) {
- return shardBackupMetadataNodeMap.computeIfAbsent(shardBackupId, s -> new Node());
- }
+ findDeletableNodes(repository, backupPaths);
+ }
- Node getIndexFileNode(String indexFile) {
- return indexFileNodeMap.computeIfAbsent(indexFile, s -> new IndexFileNode());
- }
+ public void findDeletableNodes(BackupRepository repository, BackupFilePaths backupPaths) {
+ // mark nodes as existing
+ visitExistingNodes(
+ repository.listAllOrEmpty(backupPaths.getShardBackupMetadataDir()),
+ shardBackupMetadataNodeMap,
+ shardBackupMetadataDeletes);
+ // this may be a long running commands
+ visitExistingNodes(
+ repository.listAllOrEmpty(backupPaths.getIndexDir()), indexFileNodeMap, indexFileDeletes);
+
+ // TODO Is this propagation logic really necessary?
+ // The intention seems to be that if some index files are only referenced by a shard-metadata
+ // file that
+ // is itself orphaned, then propagating the "orphaned" status down to each of these index
+ // files will allow
+ // them to be deleted.
+ //
+ // But that doesn't seem to hold up under closer inspection.
+ //
+ // The graph is populated by following links out from the set of valid backup-id's. All files
+ // (shard-
+ // metadata, or index) that trace back to a valid backup-ID will appear in the graph. If a
+ // shard-metadata
+ // file becomes orphaned, it will be ignored during graph construction and any index files
+ // that it alone
+ // references will not appear in the graph. Since those index files will be unrepresented in
+ // the graph, the
Review comment:
Fix this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] risdenk merged pull request #705: SOLR-14920: Spotless formatting for core - non-test only
Posted by GitBox <gi...@apache.org>.
risdenk merged pull request #705:
URL: https://github.com/apache/solr/pull/705
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] ErickErickson commented on a change in pull request #705: SOLR-14920: Spotless formatting for core - non-test only
Posted by GitBox <gi...@apache.org>.
ErickErickson commented on a change in pull request #705:
URL: https://github.com/apache/solr/pull/705#discussion_r819190237
##########
File path: solr/core/src/java/org/apache/solr/uninverting/DocTermOrds.java
##########
@@ -198,104 +189,113 @@ public long ramBytesUsed() {
public DocTermOrds(LeafReader reader, Bits liveDocs, String field) throws IOException {
this(reader, liveDocs, field, null, Integer.MAX_VALUE);
}
-
+
// TODO: instead of all these ctors and options, take termsenum!
/** Inverts only terms starting w/ prefix */
- public DocTermOrds(LeafReader reader, Bits liveDocs, String field, BytesRef termPrefix) throws IOException {
+ public DocTermOrds(LeafReader reader, Bits liveDocs, String field, BytesRef termPrefix)
+ throws IOException {
this(reader, liveDocs, field, termPrefix, Integer.MAX_VALUE);
}
- /** Inverts only terms starting w/ prefix, and only terms
- * whose docFreq (not taking deletions into account) is
- * <= maxTermDocFreq */
- public DocTermOrds(LeafReader reader, Bits liveDocs, String field, BytesRef termPrefix, int maxTermDocFreq) throws IOException {
+ /**
+ * Inverts only terms starting w/ prefix, and only terms whose docFreq (not taking deletions into
+ * account) is <= maxTermDocFreq
+ */
+ public DocTermOrds(
+ LeafReader reader, Bits liveDocs, String field, BytesRef termPrefix, int maxTermDocFreq)
+ throws IOException {
this(reader, liveDocs, field, termPrefix, maxTermDocFreq, DEFAULT_INDEX_INTERVAL_BITS);
}
- /** Inverts only terms starting w/ prefix, and only terms
- * whose docFreq (not taking deletions into account) is
- * <= maxTermDocFreq, with a custom indexing interval
- * (default is every 128nd term). */
- public DocTermOrds(LeafReader reader, Bits liveDocs, String field, BytesRef termPrefix, int maxTermDocFreq, int indexIntervalBits) throws IOException {
+ /**
+ * Inverts only terms starting w/ prefix, and only terms whose docFreq (not taking deletions into
+ * account) is <= maxTermDocFreq, with a custom indexing interval (default is every 128nd
+ * term).
+ */
+ public DocTermOrds(
+ LeafReader reader,
+ Bits liveDocs,
+ String field,
+ BytesRef termPrefix,
+ int maxTermDocFreq,
+ int indexIntervalBits)
+ throws IOException {
this(field, maxTermDocFreq, indexIntervalBits);
uninvert(reader, liveDocs, termPrefix);
}
- /** Subclass inits w/ this, but be sure you then call
- * uninvert, only once */
+ /** Subclass inits w/ this, but be sure you then call uninvert, only once */
protected DocTermOrds(String field, int maxTermDocFreq, int indexIntervalBits) {
- //System.out.println("DTO init field=" + field + " maxTDFreq=" + maxTermDocFreq);
+ // System.out.println("DTO init field=" + field + " maxTDFreq=" + maxTermDocFreq);
this.field = field;
this.maxTermDocFreq = maxTermDocFreq;
this.indexIntervalBits = indexIntervalBits;
- indexIntervalMask = 0xffffffff >>> (32-indexIntervalBits);
+ indexIntervalMask = 0xffffffff >>> (32 - indexIntervalBits);
indexInterval = 1 << indexIntervalBits;
}
- /**
+ /**
* Returns a TermsEnum that implements ord, or null if no terms in field.
- * <p>
- * we build a "private" terms
- * index internally (WARNING: consumes RAM) and use that
- * index to implement ord. This also enables ord on top
- * of a composite reader. The returned TermsEnum is
- * unpositioned. This returns null if there are no terms.
- * </p>
- * <p><b>NOTE</b>: you must pass the same reader that was
- * used when creating this class
+ *
+ * <p>we build a "private" terms index internally (WARNING: consumes RAM) and use that index to
+ * implement ord. This also enables ord on top of a composite reader. The returned TermsEnum is
+ * unpositioned. This returns null if there are no terms.
+ *
+ * <p><b>NOTE</b>: you must pass the same reader that was used when creating this class
*/
public TermsEnum getOrdTermsEnum(LeafReader reader) throws IOException {
// NOTE: see LUCENE-6529 before attempting to optimize this method to
// return a TermsEnum directly from the reader if it already supports ord().
assert null != indexedTermsArray;
-
+
if (0 == indexedTermsArray.length) {
return null;
} else {
return new OrdWrappedTermsEnum(reader);
}
}
- /**
- * Returns the number of terms in this field
- */
+ /** Returns the number of terms in this field */
public int numTerms() {
return numTermsInField;
}
- /**
- * Returns {@code true} if no terms were indexed.
- */
+ /** Returns {@code true} if no terms were indexed. */
public boolean isEmpty() {
return index == null;
}
/** Subclass can override this */
- protected void visitTerm(TermsEnum te, int termNum) throws IOException {
- }
+ protected void visitTerm(TermsEnum te, int termNum) throws IOException {}
- /** Invoked during {@link #uninvert(org.apache.lucene.index.LeafReader,Bits,BytesRef)}
- * to record the document frequency for each uninverted
- * term. */
- protected void setActualDocFreq(int termNum, int df) throws IOException {
- }
+ /**
+ * Invoked during {@link #uninvert(org.apache.lucene.index.LeafReader,Bits,BytesRef)} to record
+ * the document frequency for each uninverted term.
+ */
+ protected void setActualDocFreq(int termNum, int df) throws IOException {}
/** Call this only once (if you subclass!) */
- protected void uninvert(final LeafReader reader, Bits liveDocs, final BytesRef termPrefix) throws IOException {
+ protected void uninvert(final LeafReader reader, Bits liveDocs, final BytesRef termPrefix)
+ throws IOException {
final FieldInfo info = reader.getFieldInfos().fieldInfo(field);
if (checkForDocValues && info != null && info.getDocValuesType() != DocValuesType.NONE) {
- throw new IllegalStateException("Type mismatch: " + field + " was indexed as " + info.getDocValuesType());
+ throw new IllegalStateException(
+ "Type mismatch: " + field + " was indexed as " + info.getDocValuesType());
}
- //System.out.println("DTO uninvert field=" + field + " prefix=" + termPrefix);
+ // System.out.println("DTO uninvert field=" + field + " prefix=" + termPrefix);
final long startTime = System.nanoTime();
prefix = termPrefix == null ? null : BytesRef.deepCopyOf(termPrefix);
final int maxDoc = reader.maxDoc();
- final int[] index = new int[maxDoc]; // immediate term numbers, or the index into the byte[] representing the last number
- final int[] lastTerm = new int[maxDoc]; // last term we saw for this document
- final byte[][] bytes = new byte[maxDoc][]; // list of term numbers for the doc (delta encoded vInts)
+ final int[] index =
Review comment:
These comments got broken weirdly
##########
File path: solr/core/src/java/org/apache/solr/update/processor/AtomicUpdateProcessorFactory.java
##########
@@ -170,11 +173,14 @@ private void processAddWithRetry(AddUpdateCommand cmd, int attempts, SolrInputDo
// if lastVersion is null then we put -1 to assert that document must not exist
lastVersion = lastVersion == null ? -1 : lastVersion;
- // The AtomicUpdateDocumentMerger modifies the AddUpdateCommand.solrDoc to populate the real values of the
- // modified fields. We don't want those absolute values because they are out-of-date due to the conflict
- // so we restore the original document created in processAdd method and set the right version on it
+ // The AtomicUpdateDocumentMerger modifies the AddUpdateCommand.solrDoc to populate the
+ // real values of the modified fields. We don't want those absolute values because they
+ // are out-of-date due to the conflict so we restore the original document created in
+ // processAdd method and set the right version on it
cmd.solrDoc = clonedOriginalDoc;
- clonedOriginalDoc = clonedOriginalDoc.deepCopy(); // copy again because the old cloned ref will be modified during processAdd
+ // copy again because the old cloned ref will be modified during
+ clonedOriginalDoc = clonedOriginalDoc.deepCopy();
+ // processAdd
Review comment:
move to comment above
##########
File path: solr/core/src/java/org/apache/solr/analysis/LowerCaseTokenizer.java
##########
@@ -106,34 +101,38 @@ public final boolean incrementToken() throws IOException {
dataLen = ioBuffer.getLength();
bufferIndex = 0;
}
- // use CharacterUtils here to support < 3.1 UTF-16 code unit behavior if the char based methods are gone
+ // use CharacterUtils here to support < 3.1 UTF-16 code unit behavior if the char based
+ // methods are gone
final int c = Character.codePointAt(ioBuffer.getBuffer(), bufferIndex, ioBuffer.getLength());
final int charCount = Character.charCount(c);
bufferIndex += charCount;
- if (Character.isLetter(c)) { // if it's a token char
- if (length == 0) { // start of token
+ if (Character.isLetter(c)) { // if it's a token char
+ if (length == 0) { // start of token
assert start == -1;
start = offset + bufferIndex - charCount;
end = start;
- } else if (length >= buffer.length-1) { // check if a supplementary could run out of bounds
- buffer = termAtt.resizeBuffer(2+length); // make sure a supplementary fits in the buffer
+ } else if (length
+ >= buffer.length - 1) { // check if a supplementary could run out of bounds
+ buffer = termAtt.resizeBuffer(2 + length); // make sure a supplementary fits in the buffer
}
end += charCount;
- length += Character.toChars(Character.toLowerCase(c), buffer, length); // buffer it, normalized
- if (length >= maxTokenLen) { // buffer overflow! make sure to check for >= surrogate pair could break == test
+ length +=
+ Character.toChars(Character.toLowerCase(c), buffer, length); // buffer it, normalized
+ if (length
+ >= maxTokenLen) { // buffer overflow! make sure to check for >= surrogate pair could
+ // break == test
Review comment:
move entire comment above if?
##########
File path: solr/core/src/java/org/apache/solr/search/JoinQuery.java
##########
@@ -192,18 +197,19 @@ public boolean isCacheable(LeafReaderContext ctx) {
}
// most of these statistics are only used for the enum method
- int fromSetSize; // number of docs in the fromSet (that match the from query)
- long resultListDocs; // total number of docs collected
+ int fromSetSize; // number of docs in the fromSet (that match the from query)
+ long resultListDocs; // total number of docs collected
int fromTermCount;
long fromTermTotalDf;
- int fromTermDirectCount; // number of fromTerms that were too small to use the filter cache
- int fromTermHits; // number of fromTerms that intersected the from query
+ int fromTermDirectCount; // number of fromTerms that were too small to use the filter cache
+ int fromTermHits; // number of fromTerms that intersected the from query
long fromTermHitsTotalDf; // sum of the df of the matching terms
- int toTermHits; // num if intersecting from terms that match a term in the to field
- long toTermHitsTotalDf; // sum of the df for the toTermHits
- int toTermDirectCount; // number of toTerms that we set directly on a bitset rather than doing set intersections
- int smallSetsDeferred; // number of small sets collected to be used later to intersect w/ bitset or create another small set
-
+ int toTermHits; // num if intersecting from terms that match a term in the to field
+ long toTermHitsTotalDf; // sum of the df for the toTermHits
+ int toTermDirectCount; // number of toTerms that we set directly on a bitset rather than doing
+ // set intersections
+ int smallSetsDeferred; // number of small sets collected to be used later to intersect w/ bitset
+ // or create another small set
Review comment:
Move this to above declaration
##########
File path: solr/core/src/java/org/apache/solr/search/facet/FacetMerger.java
##########
@@ -24,35 +26,37 @@
import java.util.IdentityHashMap;
import java.util.Map;
-import static org.apache.solr.search.facet.FacetRequest.RefineMethod.SIMPLE;
-
-
public abstract class FacetMerger {
public abstract void merge(Object facetResult, Context mcontext);
// FIXME
// public abstract Map<String,Object> getRefinement(Context mcontext);
- public Map<String,Object> getRefinement(Context mcontext) {
+ public Map<String, Object> getRefinement(Context mcontext) {
return null;
}
+
public abstract void finish(Context mcontext);
- public abstract Object getMergedResult(); // TODO: we should pass mcontext through here as well
+
+ public abstract Object getMergedResult(); // TODO: we should pass mcontext through here as well
// This class lets mergers know overall context such as what shard is being merged
// and what buckets have been seen by what shard.
public static class Context {
// FacetComponentState state; // todo: is this needed?
final int numShards;
- private final BitSet sawShard = new BitSet(); // [bucket0_shard0, bucket0_shard1, bucket0_shard2, bucket1_shard0, bucket1_shard1, bucket1_shard2]
- private Map<String,Integer> shardmap = new HashMap<>();
+ private final BitSet sawShard =
+ new BitSet(); // [bucket0_shard0, bucket0_shard1, bucket0_shard2, bucket1_shard0,
Review comment:
move entire comment above declaration
##########
File path: solr/core/src/java/org/apache/solr/handler/DocumentAnalysisRequestHandler.java
##########
@@ -250,26 +253,30 @@ DocumentAnalysisRequest resolveAnalysisRequest(SolrQueryRequest req) throws IOEx
/**
* Reads the document from the given xml stream reader. The following document format is expected:
- * <p/>
+ *
+ * <p>
Review comment:
remove
##########
File path: solr/core/src/java/org/apache/solr/util/hll/BitVector.java
##########
@@ -19,242 +19,268 @@
import org.apache.solr.util.LongIterator;
/**
- * A vector (array) of bits that is accessed in units ("registers") of <code>width</code>
- * bits which are stored as 64bit "words" (<code>long</code>s). In this context
- * a register is at most 64bits.
+ * A vector (array) of bits that is accessed in units ("registers") of <code>width</code> bits which
+ * are stored as 64bit "words" (<code>long</code>s). In this context a register is at most 64bits.
*/
class BitVector implements Cloneable {
- // NOTE: in this context, a word is 64bits
-
- // rather than doing division to determine how a bit index fits into 64bit
- // words (i.e. longs), bit shifting is used
- private static final int LOG2_BITS_PER_WORD = 6/*=>64bits*/;
- private static final int BITS_PER_WORD = 1 << LOG2_BITS_PER_WORD;
- private static final int BITS_PER_WORD_MASK = BITS_PER_WORD - 1;
-
- // ditto from above but for bytes (for output)
- private static final int LOG2_BITS_PER_BYTE = 3/*=>8bits*/;
- public static final int BITS_PER_BYTE = 1 << LOG2_BITS_PER_BYTE;
-
- // ========================================================================
- public static final int BYTES_PER_WORD = 8/*8 bytes in a long*/;
-
- // ************************************************************************
- // 64bit words
- private final long[] words;
- public final long[] words() { return words; }
- public final int wordCount() { return words.length; }
- public final int byteCount() { return wordCount() * BYTES_PER_WORD; }
-
- // the width of a register in bits (this cannot be more than 64 (the word size))
- private final int registerWidth;
- public final int registerWidth() { return registerWidth; }
-
- private final long count;
-
- // ------------------------------------------------------------------------
- private final long registerMask;
-
- // ========================================================================
- /**
- * @param width the width of each register. This cannot be negative or
- * zero or greater than 63 (the signed word size).
- * @param count the number of registers. This cannot be negative or zero
- */
- public BitVector(final int width, final long count) {
- // ceil((width * count)/BITS_PER_WORD)
- this.words = new long[(int)(((width * count) + BITS_PER_WORD_MASK) >>> LOG2_BITS_PER_WORD)];
- this.registerWidth = width;
- this.count = count;
-
- this.registerMask = (1L << width) - 1;
- }
+ // NOTE: in this context, a word is 64bits
- // ========================================================================
- /**
- * @param registerIndex the index of the register whose value is to be
- * retrieved. This cannot be negative.
- * @return the value at the specified register index
- * @see #setRegister(long, long)
- * @see #setMaxRegister(long, long)
- */
- // NOTE: if this changes then setMaxRegister() must change
- public long getRegister(final long registerIndex) {
- final long bitIndex = registerIndex * registerWidth;
- final int firstWordIndex = (int)(bitIndex >>> LOG2_BITS_PER_WORD)/*aka (bitIndex / BITS_PER_WORD)*/;
- final int secondWordIndex = (int)((bitIndex + registerWidth - 1) >>> LOG2_BITS_PER_WORD)/*see above*/;
- final int bitRemainder = (int)(bitIndex & BITS_PER_WORD_MASK)/*aka (bitIndex % BITS_PER_WORD)*/;
-
- if(firstWordIndex == secondWordIndex)
- return ((words[firstWordIndex] >>> bitRemainder) & registerMask);
- /* else -- register spans words */
- return (words[firstWordIndex] >>> bitRemainder)/*no need to mask since at top of word*/
- | (words[secondWordIndex] << (BITS_PER_WORD - bitRemainder)) & registerMask;
- }
+ // rather than doing division to determine how a bit index fits into 64bit
+ // words (i.e. longs), bit shifting is used
+ private static final int LOG2_BITS_PER_WORD = 6 /*=>64bits*/;
+ private static final int BITS_PER_WORD = 1 << LOG2_BITS_PER_WORD;
+ private static final int BITS_PER_WORD_MASK = BITS_PER_WORD - 1;
- /**
- * @param registerIndex the index of the register whose value is to be set.
- * This cannot be negative
- * @param value the value to set in the register
- * @see #getRegister(long)
- * @see #setMaxRegister(long, long)
- */
- // NOTE: if this changes then setMaxRegister() must change
- public void setRegister(final long registerIndex, final long value) {
- final long bitIndex = registerIndex * registerWidth;
- final int firstWordIndex = (int)(bitIndex >>> LOG2_BITS_PER_WORD)/*aka (bitIndex / BITS_PER_WORD)*/;
- final int secondWordIndex = (int)((bitIndex + registerWidth - 1) >>> LOG2_BITS_PER_WORD)/*see above*/;
- final int bitRemainder = (int)(bitIndex & BITS_PER_WORD_MASK)/*aka (bitIndex % BITS_PER_WORD)*/;
-
- final long words[] = this.words/*for convenience/performance*/;
- if(firstWordIndex == secondWordIndex) {
- // clear then set
- words[firstWordIndex] &= ~(registerMask << bitRemainder);
- words[firstWordIndex] |= (value << bitRemainder);
- } else {/*register spans words*/
- // clear then set each partial word
- words[firstWordIndex] &= (1L << bitRemainder) - 1;
- words[firstWordIndex] |= (value << bitRemainder);
-
- words[secondWordIndex] &= ~(registerMask >>> (BITS_PER_WORD - bitRemainder));
- words[secondWordIndex] |= (value >>> (BITS_PER_WORD - bitRemainder));
- }
- }
+ // ditto from above but for bytes (for output)
+ private static final int LOG2_BITS_PER_BYTE = 3 /*=>8bits*/;
+ public static final int BITS_PER_BYTE = 1 << LOG2_BITS_PER_BYTE;
- // ------------------------------------------------------------------------
- /**
- * @return a <code>LongIterator</code> for iterating starting at the register
- * with index zero. This will never be <code>null</code>.
- */
- public LongIterator registerIterator() {
- return new LongIterator() {
- final int registerWidth = BitVector.this.registerWidth;
- final long[] words = BitVector.this.words;
- final long registerMask = BitVector.this.registerMask;
-
- // register setup
- long registerIndex = 0;
- int wordIndex = 0;
- int remainingWordBits = BITS_PER_WORD;
- long word = words[wordIndex];
-
- @Override public long next() {
- long register;
- if(remainingWordBits >= registerWidth) {
- register = word & registerMask;
-
- // shift to the next register
- word >>>= registerWidth;
- remainingWordBits -= registerWidth;
- } else { /*insufficient bits remaining in current word*/
- wordIndex++/*move to the next word*/;
-
- register = (word | (words[wordIndex] << remainingWordBits)) & registerMask;
-
- // shift to the next partial register (word)
- word = words[wordIndex] >>> (registerWidth - remainingWordBits);
- remainingWordBits += BITS_PER_WORD - registerWidth;
- }
- registerIndex++;
- return register;
- }
-
- @Override public boolean hasNext() {
- return registerIndex < count;
- }
- };
- }
+ // ========================================================================
+ public static final int BYTES_PER_WORD = 8 /*8 bytes in a long*/;
- // ------------------------------------------------------------------------
- // composite accessors
- /**
- * Sets the value of the specified index register if and only if the specified
- * value is greater than the current value in the register. This is equivalent
- * to but much more performant than:<p/>
- *
- * <pre>vector.setRegister(index, Math.max(vector.getRegister(index), value));</pre>
- *
- * @param registerIndex the index of the register whose value is to be set.
- * This cannot be negative
- * @param value the value to set in the register if and only if this value
- * is greater than the current value in the register
- * @return <code>true</code> if and only if the specified value is greater
- * than or equal to the current register value. <code>false</code>
- * otherwise.
- * @see #getRegister(long)
- * @see #setRegister(long, long)
- * @see java.lang.Math#max(long, long)
- */
- // NOTE: if this changes then setRegister() must change
- public boolean setMaxRegister(final long registerIndex, final long value) {
- final long bitIndex = registerIndex * registerWidth;
- final int firstWordIndex = (int)(bitIndex >>> LOG2_BITS_PER_WORD)/*aka (bitIndex / BITS_PER_WORD)*/;
- final int secondWordIndex = (int)((bitIndex + registerWidth - 1) >>> LOG2_BITS_PER_WORD)/*see above*/;
- final int bitRemainder = (int)(bitIndex & BITS_PER_WORD_MASK)/*aka (bitIndex % BITS_PER_WORD)*/;
-
- // NOTE: matches getRegister()
- final long registerValue;
- final long words[] = this.words/*for convenience/performance*/;
- if(firstWordIndex == secondWordIndex)
- registerValue = ((words[firstWordIndex] >>> bitRemainder) & registerMask);
- else /*register spans words*/
- registerValue = (words[firstWordIndex] >>> bitRemainder)/*no need to mask since at top of word*/
- | (words[secondWordIndex] << (BITS_PER_WORD - bitRemainder)) & registerMask;
-
- // determine which is the larger and update as necessary
- if(value > registerValue) {
- // NOTE: matches setRegister()
- if(firstWordIndex == secondWordIndex) {
- // clear then set
- words[firstWordIndex] &= ~(registerMask << bitRemainder);
- words[firstWordIndex] |= (value << bitRemainder);
- } else {/*register spans words*/
- // clear then set each partial word
- words[firstWordIndex] &= (1L << bitRemainder) - 1;
- words[firstWordIndex] |= (value << bitRemainder);
-
- words[secondWordIndex] &= ~(registerMask >>> (BITS_PER_WORD - bitRemainder));
- words[secondWordIndex] |= (value >>> (BITS_PER_WORD - bitRemainder));
- }
- } /* else -- the register value is greater (or equal) so nothing needs to be done */
-
- return (value >= registerValue);
- }
+ // ************************************************************************
+ // 64bit words
+ private final long[] words;
- // ========================================================================
- /**
- * Fills this bit vector with the specified bit value. This can be used to
- * clear the vector by specifying <code>0</code>.
- *
- * @param value the value to set all bits to (only the lowest bit is used)
- */
- public void fill(final long value) {
- for(long i=0; i<count; i++) {
- setRegister(i, value);
- }
+ public final long[] words() {
+ return words;
+ }
+
+ public final int wordCount() {
+ return words.length;
+ }
+
+ public final int byteCount() {
+ return wordCount() * BYTES_PER_WORD;
+ }
+
+ // the width of a register in bits (this cannot be more than 64 (the word size))
+ private final int registerWidth;
+
+ public final int registerWidth() {
+ return registerWidth;
+ }
+
+ private final long count;
+
+ // ------------------------------------------------------------------------
+ private final long registerMask;
+
+ // ========================================================================
+ /**
+ * @param width the width of each register. This cannot be negative or zero or greater than 63
+ * (the signed word size).
+ * @param count the number of registers. This cannot be negative or zero
+ */
+ public BitVector(final int width, final long count) {
+ // ceil((width * count)/BITS_PER_WORD)
+ this.words = new long[(int) (((width * count) + BITS_PER_WORD_MASK) >>> LOG2_BITS_PER_WORD)];
+ this.registerWidth = width;
+ this.count = count;
+
+ this.registerMask = (1L << width) - 1;
+ }
+
+ // ========================================================================
+ /**
+ * @param registerIndex the index of the register whose value is to be retrieved. This cannot be
+ * negative.
+ * @return the value at the specified register index
+ * @see #setRegister(long, long)
+ * @see #setMaxRegister(long, long)
+ */
+ // NOTE: if this changes then setMaxRegister() must change
+ public long getRegister(final long registerIndex) {
+ final long bitIndex = registerIndex * registerWidth;
+ final int firstWordIndex =
+ (int) (bitIndex >>> LOG2_BITS_PER_WORD) /*aka (bitIndex / BITS_PER_WORD)*/;
+ final int secondWordIndex =
+ (int) ((bitIndex + registerWidth - 1) >>> LOG2_BITS_PER_WORD) /*see above*/;
+ final int bitRemainder =
+ (int) (bitIndex & BITS_PER_WORD_MASK) /*aka (bitIndex % BITS_PER_WORD)*/;
+
+ if (firstWordIndex == secondWordIndex)
+ return ((words[firstWordIndex] >>> bitRemainder) & registerMask);
+ /* else -- register spans words */
+ return (words[firstWordIndex] >>> bitRemainder) /*no need to mask since at top of word*/
+ | (words[secondWordIndex] << (BITS_PER_WORD - bitRemainder)) & registerMask;
+ }
+
+ /**
+ * @param registerIndex the index of the register whose value is to be set. This cannot be
+ * negative
+ * @param value the value to set in the register
+ * @see #getRegister(long)
+ * @see #setMaxRegister(long, long)
+ */
+ // NOTE: if this changes then setMaxRegister() must change
+ public void setRegister(final long registerIndex, final long value) {
+ final long bitIndex = registerIndex * registerWidth;
+ final int firstWordIndex =
+ (int) (bitIndex >>> LOG2_BITS_PER_WORD) /*aka (bitIndex / BITS_PER_WORD)*/;
+ final int secondWordIndex =
+ (int) ((bitIndex + registerWidth - 1) >>> LOG2_BITS_PER_WORD) /*see above*/;
+ final int bitRemainder =
+ (int) (bitIndex & BITS_PER_WORD_MASK) /*aka (bitIndex % BITS_PER_WORD)*/;
+
+ final long words[] = this.words /*for convenience/performance*/;
+ if (firstWordIndex == secondWordIndex) {
+ // clear then set
+ words[firstWordIndex] &= ~(registerMask << bitRemainder);
+ words[firstWordIndex] |= (value << bitRemainder);
+ } else {
+ /*register spans words*/
+ // clear then set each partial word
+ words[firstWordIndex] &= (1L << bitRemainder) - 1;
+ words[firstWordIndex] |= (value << bitRemainder);
+
+ words[secondWordIndex] &= ~(registerMask >>> (BITS_PER_WORD - bitRemainder));
+ words[secondWordIndex] |= (value >>> (BITS_PER_WORD - bitRemainder));
}
+ }
+
+ // ------------------------------------------------------------------------
+ /**
+ * @return a <code>LongIterator</code> for iterating starting at the register with index zero.
+ * This will never be <code>null</code>.
+ */
+ public LongIterator registerIterator() {
+ return new LongIterator() {
+ final int registerWidth = BitVector.this.registerWidth;
+ final long[] words = BitVector.this.words;
+ final long registerMask = BitVector.this.registerMask;
+
+ // register setup
+ long registerIndex = 0;
+ int wordIndex = 0;
+ int remainingWordBits = BITS_PER_WORD;
+ long word = words[wordIndex];
+
+ @Override
+ public long next() {
+ long register;
+ if (remainingWordBits >= registerWidth) {
+ register = word & registerMask;
+
+ // shift to the next register
+ word >>>= registerWidth;
+ remainingWordBits -= registerWidth;
+ } else {
+ /*insufficient bits remaining in current word*/
+ wordIndex++ /*move to the next word*/;
+
+ register = (word | (words[wordIndex] << remainingWordBits)) & registerMask;
- // ------------------------------------------------------------------------
- /**
- * Serializes the registers of the vector using the specified serializer.
- *
- * @param serializer the serializer to use. This cannot be <code>null</code>.
- */
- public void getRegisterContents(final IWordSerializer serializer) {
- for(final LongIterator iter = registerIterator(); iter.hasNext();) {
- serializer.writeWord(iter.next());
+ // shift to the next partial register (word)
+ word = words[wordIndex] >>> (registerWidth - remainingWordBits);
+ remainingWordBits += BITS_PER_WORD - registerWidth;
}
+ registerIndex++;
+ return register;
+ }
+
+ @Override
+ public boolean hasNext() {
+ return registerIndex < count;
+ }
+ };
+ }
+
+ // ------------------------------------------------------------------------
+ // composite accessors
+ /**
+ * Sets the value of the specified index register if and only if the specified value is greater
+ * than the current value in the register. This is equivalent to but much more performant than:
+ *
+ * <p>
Review comment:
remove
##########
File path: solr/core/src/java/org/apache/solr/cloud/api/collections/AliasCmd.java
##########
@@ -51,10 +51,17 @@ protected AliasCmd(CollectionCommandContext ccc) {
/**
* Creates a collection (for use in a routed alias), waiting for it to be ready before returning.
- * If the collection already exists then this is not an error.<p>
+ * If the collection already exists then this is not an error.
+ *
+ * <p>
Review comment:
remove
##########
File path: solr/core/src/java/org/apache/solr/search/facet/FacetMerger.java
##########
@@ -24,35 +26,37 @@
import java.util.IdentityHashMap;
import java.util.Map;
-import static org.apache.solr.search.facet.FacetRequest.RefineMethod.SIMPLE;
-
-
public abstract class FacetMerger {
public abstract void merge(Object facetResult, Context mcontext);
// FIXME
// public abstract Map<String,Object> getRefinement(Context mcontext);
- public Map<String,Object> getRefinement(Context mcontext) {
+ public Map<String, Object> getRefinement(Context mcontext) {
return null;
}
+
public abstract void finish(Context mcontext);
- public abstract Object getMergedResult(); // TODO: we should pass mcontext through here as well
+
+ public abstract Object getMergedResult(); // TODO: we should pass mcontext through here as well
// This class lets mergers know overall context such as what shard is being merged
// and what buckets have been seen by what shard.
public static class Context {
// FacetComponentState state; // todo: is this needed?
final int numShards;
- private final BitSet sawShard = new BitSet(); // [bucket0_shard0, bucket0_shard1, bucket0_shard2, bucket1_shard0, bucket1_shard1, bucket1_shard2]
- private Map<String,Integer> shardmap = new HashMap<>();
+ private final BitSet sawShard =
+ new BitSet(); // [bucket0_shard0, bucket0_shard1, bucket0_shard2, bucket1_shard0,
+ // bucket1_shard1, bucket1_shard2]
+ private Map<String, Integer> shardmap = new HashMap<>();
public Context(int numShards) {
this.numShards = numShards;
}
- Object root; // per-shard response
- int maxBucket; // the current max bucket across all bucket types... incremented as we encounter more
- int shardNum = -1; // TODO: keep same mapping across multiple phases...
+ Object root; // per-shard response
+ int maxBucket; // the current max bucket across all bucket types... incremented as we encounter
+ // more
Review comment:
move entire comment above declaration
##########
File path: solr/core/src/java/org/apache/solr/search/Grouping.java
##########
@@ -464,67 +481,59 @@ public static int getMax(int offset, int len, int max) {
}
/**
- * Returns whether a cache warning should be send to the client.
- * The value <code>true</code> is returned when the cache is emptied because the caching limits where met, otherwise
- * <code>false</code> is returned.
+ * Returns whether a cache warning should be send to the client. The value <code>true</code> is
+ * returned when the cache is emptied because the caching limits where met, otherwise <code>false
+ * </code> is returned.
*
* @return whether a cache warning should be send to the client
*/
public boolean isSignalCacheWarning() {
return signalCacheWarning;
}
- //====================================== Inner classes =============================================================
+ // ===== Inner classes =====
public static enum Format {
- /**
- * Grouped result. Each group has its own result set.
- */
+ /** Grouped result. Each group has its own result set. */
grouped,
- /**
- * Flat result. All documents of all groups are put in one list.
- */
+ /** Flat result. All documents of all groups are put in one list. */
simple
}
public static enum TotalCount {
- /**
- * Computations should be based on groups.
- */
+ /** Computations should be based on groups. */
grouped,
- /**
- * Computations should be based on plain documents, so not taking grouping into account.
- */
+ /** Computations should be based on plain documents, so not taking grouping into account. */
ungrouped
}
/**
- * General group command. A group command is responsible for creating the first and second pass collectors.
- * A group command is also responsible for creating the response structure.
- * <p>
- * Note: Maybe the creating the response structure should be done in something like a ReponseBuilder???
- * Warning NOT thread save!
+ * General group command. A group command is responsible for creating the first and second pass
+ * collectors. A group command is also responsible for creating the response structure.
+ *
+ * <p>Note: Maybe the creating the response structure should be done in something like a
+ * ReponseBuilder??? Warning NOT thread save!
*/
public abstract class Command<T> {
- public String key; // the name to use for this group in the response
- public Sort withinGroupSort; // the sort of the documents *within* a single group.
- public Sort groupSort; // the sort between groups
+ public String key; // the name to use for this group in the response
+ public Sort withinGroupSort; // the sort of the documents *within* a single group.
+ public Sort groupSort; // the sort between groups
public int docsPerGroup; // how many docs in each group - from "group.limit" param, default=1
- public int groupOffset; // the offset within each group (for paging within each group)
- public int numGroups; // how many groups - defaults to the "rows" parameter
- int actualGroupsToFind; // How many groups should actually be found. Based on groupOffset and numGroups.
- public int offset; // offset into the list of groups
+ public int groupOffset; // the offset within each group (for paging within each group)
+ public int numGroups; // how many groups - defaults to the "rows" parameter
+ int actualGroupsToFind; // How many groups should actually be found. Based on groupOffset and
+ // numGroups.
Review comment:
combine with comment above.
##########
File path: solr/core/src/java/org/apache/solr/search/JoinQuery.java
##########
@@ -192,18 +197,19 @@ public boolean isCacheable(LeafReaderContext ctx) {
}
// most of these statistics are only used for the enum method
- int fromSetSize; // number of docs in the fromSet (that match the from query)
- long resultListDocs; // total number of docs collected
+ int fromSetSize; // number of docs in the fromSet (that match the from query)
+ long resultListDocs; // total number of docs collected
int fromTermCount;
long fromTermTotalDf;
- int fromTermDirectCount; // number of fromTerms that were too small to use the filter cache
- int fromTermHits; // number of fromTerms that intersected the from query
+ int fromTermDirectCount; // number of fromTerms that were too small to use the filter cache
+ int fromTermHits; // number of fromTerms that intersected the from query
long fromTermHitsTotalDf; // sum of the df of the matching terms
- int toTermHits; // num if intersecting from terms that match a term in the to field
- long toTermHitsTotalDf; // sum of the df for the toTermHits
- int toTermDirectCount; // number of toTerms that we set directly on a bitset rather than doing set intersections
- int smallSetsDeferred; // number of small sets collected to be used later to intersect w/ bitset or create another small set
-
+ int toTermHits; // num if intersecting from terms that match a term in the to field
+ long toTermHitsTotalDf; // sum of the df for the toTermHits
+ int toTermDirectCount; // number of toTerms that we set directly on a bitset rather than doing
+ // set intersections
Review comment:
Move this to above declaration
##########
File path: solr/core/src/java/org/apache/solr/cloud/OverseerCollectionConfigSetProcessor.java
##########
@@ -87,10 +98,20 @@ private static OverseerMessageHandlerSelector getOverseerMessageHandlerSelector(
Stats stats,
Overseer overseer,
OverseerNodePrioritizer overseerNodePrioritizer) {
- final OverseerCollectionMessageHandler collMessageHandler = new OverseerCollectionMessageHandler(
- zkStateReader, myId, shardHandlerFactory, adminPath, stats, overseer, overseerNodePrioritizer);
- final OverseerConfigSetMessageHandler configMessageHandler = new OverseerConfigSetMessageHandler(
- zkStateReader, overseer.getCoreContainer()); //coreContainer is passed instead of configSetService as configSetService is loaded late
+ final OverseerCollectionMessageHandler collMessageHandler =
+ new OverseerCollectionMessageHandler(
+ zkStateReader,
+ myId,
+ shardHandlerFactory,
+ adminPath,
+ stats,
+ overseer,
+ overseerNodePrioritizer);
+ final OverseerConfigSetMessageHandler configMessageHandler =
+ new OverseerConfigSetMessageHandler(
+ zkStateReader,
+ overseer.getCoreContainer()); // coreContainer is passed instead of configSetService as
+ // configSetService is loaded late
Review comment:
combine these comments above?
##########
File path: solr/core/src/java/org/apache/solr/response/JSONResponseWriter.java
##########
@@ -74,218 +71,226 @@ public String getContentType(SolrQueryRequest request, SolrQueryResponse respons
return contentType;
}
- public static PushWriter getPushWriter(Writer writer, SolrQueryRequest req, SolrQueryResponse rsp) {
+ public static PushWriter getPushWriter(
+ Writer writer, SolrQueryRequest req, SolrQueryResponse rsp) {
return new JSONWriter(writer, req, rsp);
}
-
-
-/**
- * Writes NamedLists directly as an array of NameTypeValue JSON objects...
- * NamedList("a"=1,"b"=null,null=3,null=null) =>
- * [{"name":"a","type":"int","value":1},
- * {"name":"b","type":"null","value":null},
- * {"name":null,"type":"int","value":3},
- * {"name":null,"type":"null","value":null}]
- * NamedList("a"=1,"bar"="foo",null=3.4f) =>
- * [{"name":"a","type":"int","value":1},
- * {"name":"bar","type":"str","value":"foo"},
- * {"name":null,"type":"float","value":3.4}]
- */
-class ArrayOfNameTypeValueJSONWriter extends JSONWriter {
- protected boolean writeTypeAndValueKey = false;
- private final boolean writeNullName;
-
- public ArrayOfNameTypeValueJSONWriter(Writer writer, SolrQueryRequest req, SolrQueryResponse rsp,
- String wrapperFunction, String namedListStyle, boolean writeNullName) {
- super(writer, req, rsp, wrapperFunction, namedListStyle);
- this.writeNullName = writeNullName;
- }
-
- @Override
- public void writeNamedList(String name, NamedList<?> val) throws IOException {
-
- if (val instanceof SimpleOrderedMap) {
- super.writeNamedList(name, val);
- return;
+ /**
+ * Writes NamedLists directly as an array of NameTypeValue JSON objects...
+ * NamedList("a"=1,"b"=null,null=3,null=null) => [{"name":"a","type":"int","value":1},
+ * {"name":"b","type":"null","value":null}, {"name":null,"type":"int","value":3},
+ * {"name":null,"type":"null","value":null}] NamedList("a"=1,"bar"="foo",null=3.4f) =>
+ * [{"name":"a","type":"int","value":1}, {"name":"bar","type":"str","value":"foo"},
+ * {"name":null,"type":"float","value":3.4}]
+ */
+ class ArrayOfNameTypeValueJSONWriter extends JSONWriter {
+ protected boolean writeTypeAndValueKey = false;
+ private final boolean writeNullName;
+
+ public ArrayOfNameTypeValueJSONWriter(
+ Writer writer,
+ SolrQueryRequest req,
+ SolrQueryResponse rsp,
+ String wrapperFunction,
+ String namedListStyle,
+ boolean writeNullName) {
+ super(writer, req, rsp, wrapperFunction, namedListStyle);
+ this.writeNullName = writeNullName;
}
- final int sz = val.size();
- indent();
-
- writeArrayOpener(sz);
- incLevel();
+ @Override
+ public void writeNamedList(String name, NamedList<?> val) throws IOException {
- boolean first = true;
- for (int i=0; i<sz; i++) {
- if (first) {
- first = false;
- } else {
- writeArraySeparator();
+ if (val instanceof SimpleOrderedMap) {
+ super.writeNamedList(name, val);
+ return;
}
+ final int sz = val.size();
indent();
- final String elementName = val.getName(i);
- final Object elementVal = val.getVal(i);
-
- /*
- * JSONWriter's writeNamedListAsArrMap turns NamedList("bar"="foo") into [{"foo":"bar"}]
- * but we here wish to turn it into [ {"name":"bar","type":"str","value":"foo"} ] instead.
- *
- * So first we write the <code>{"name":"bar",</code> portion ...
- */
- writeMapOpener(-1);
- if (elementName != null || writeNullName) {
- writeKey("name", false);
- writeVal("name", elementName);
- writeMapSeparator();
+ writeArrayOpener(sz);
+ incLevel();
+
+ boolean first = true;
+ for (int i = 0; i < sz; i++) {
+ if (first) {
+ first = false;
+ } else {
+ writeArraySeparator();
+ }
+
+ indent();
+
+ final String elementName = val.getName(i);
+ final Object elementVal = val.getVal(i);
+
+ /*
+ * JSONWriter's writeNamedListAsArrMap turns NamedList("bar"="foo") into [{"foo":"bar"}]
+ * but we here wish to turn it into [ {"name":"bar","type":"str","value":"foo"} ] instead.
+ *
+ * So first we write the <code>{"name":"bar",</code> portion ...
+ */
+ writeMapOpener(-1);
+ if (elementName != null || writeNullName) {
+ writeKey("name", false);
+ writeVal("name", elementName);
+ writeMapSeparator();
+ }
+
+ /*
+ * ... and then we write the <code>"type":"str","value":"foo"}</code> portion.
+ */
+ writeTypeAndValueKey = true;
+ writeVal(
+ null,
+ elementVal); // passing null since writeVal doesn't actually use name (and we already
+ // wrote elementName above)
Review comment:
awkward
##########
File path: solr/core/src/java/org/apache/solr/servlet/CoreContainerProvider.java
##########
@@ -147,21 +154,27 @@ private static void waitForCoreContainer(Supplier<CoreContainer> provider, Count
public void close() {
CoreContainer cc = cores;
-// if (cc != null) {
-// ZkController zkController = cc.getZkController();
-// if (zkController != null) {
-//
-// // Mark Miller suggested that we should be publishing that we are down before anything else which makes
-// // good sense, but the following causes test failures, so that improvement can be the subject of another
-// // PR/issue. Also, jetty might already be refusing requests by this point so that's a potential issue too.
-// // Digging slightly I see that there's a whole mess of code looking up collections and calculating state
-// // changes associated with this call, which smells a lot like we're duplicating node state in collection
-// // stuff, but it will take a lot of code reading to figure out if that's really what it is, why we
-// // did it and if there's room for improvement.
-//
-// zkController.publishNodeAsDown(zkController.getNodeName());
-// }
-// }
+ // if (cc != null) {
+ // ZkController zkController = cc.getZkController();
+ // if (zkController != null) {
+ //
+ // // Mark Miller suggested that we should be publishing that we are down before anything
Review comment:
This all got broken up strangely with a lot of dangling lines, tighten them up?
##########
File path: solr/core/src/java/org/apache/solr/schema/ManagedIndexSchemaFactory.java
##########
@@ -93,60 +101,71 @@ public void init(NamedList<?> args) {
@Override
public String getSchemaResourceName(String cdResourceName) {
- return managedSchemaResourceName; // actually a guess; reality depends on the actual files in the config set :-(
+ return managedSchemaResourceName; // actually a guess; reality depends on the actual files in
+ // the config set :-(
Review comment:
combine comment fragments above return
##########
File path: solr/core/src/java/org/apache/solr/cloud/api/collections/SplitShardCmd.java
##########
@@ -805,24 +999,33 @@ public static Slice getParentSlice(ClusterState clusterState, String collectionN
}
if (parentSlice == null) {
- // no chance of the collection being null because ClusterState#getCollection(String) would have thrown
+ // no chance of the collection being null because ClusterState#getCollection(String) would
+ // have thrown
Review comment:
awkward split
##########
File path: solr/core/src/java/org/apache/solr/cloud/overseer/CollectionMutator.java
##########
@@ -109,27 +113,30 @@ public ZkWriteCommand modifyCollection(final ClusterState clusterState, ZkNodePr
if (val == null) continue;
boolean enable = Boolean.parseBoolean(val);
if (enable == coll.isPerReplicaState()) {
- //already enabled
+ // already enabled
log.error("trying to set perReplicaState to {} from {}", val, coll.isPerReplicaState());
continue;
}
- replicaOps = PerReplicaStatesOps.modifyCollection(coll, enable, PerReplicaStates.fetch(coll.getZNode(), zkClient, null));
+ replicaOps =
+ PerReplicaStatesOps.modifyCollection(
+ coll, enable, PerReplicaStates.fetch(coll.getZNode(), zkClient, null));
}
-
if (message.containsKey(prop)) {
hasAnyOps = true;
- if (message.get(prop) == null) {
+ if (message.get(prop) == null) {
props.remove(prop);
- } else {
+ } else {
// rename key from collection.configName to configName
if (prop.equals(COLL_CONF)) {
props.put(CONFIGNAME_PROP, message.get(prop));
} else {
props.put(prop, message.get(prop));
}
}
- if (prop == REPLICATION_FACTOR) { //SOLR-11676 : keep NRT_REPLICAS and REPLICATION_FACTOR in sync
+ // SOLR-11676 : keep NRT_REPLICAS and REPLICATION_FACTOR in
+ if (prop == REPLICATION_FACTOR) {
+ // sync
Review comment:
awward comment split
##########
File path: solr/core/src/java/org/apache/solr/search/facet/RelatednessAgg.java
##########
@@ -184,12 +195,20 @@ public FacetMerger createFacetMerger(Object prototype) {
// we can't get the allBuckets info from the slotContext in collect(), b/c the whole point of
// sweep collection is that the "collect" methods aren't called.
- // So this is the compromise: note in construction either that we're using a processor w/NO_ALL_BUCKETS
+ // So this is the compromise: note in construction either that we're using a processor
+ // w/NO_ALL_BUCKETS
Review comment:
tighten these
##########
File path: solr/core/src/java/org/apache/solr/cloud/RecoveryStrategy.java
##########
@@ -564,9 +581,8 @@ public final void doSyncOrReplicateRecovery(SolrCore core) throws Exception {
final String ourUrl = ZkCoreNodeProps.getCoreUrl(baseUrl, coreName);
Future<RecoveryInfo> replayFuture = null;
- while (!successfulRecovery && !Thread.currentThread().isInterrupted() && !isClosed()) { // don't use interruption or
- // it will close channels
- // though
+ // don't use interruption or it will close channels though
Review comment:
put this above the while.
##########
File path: solr/core/src/java/org/apache/solr/cloud/RecoveryStrategy.java
##########
@@ -332,27 +335,29 @@ final public void doRecovery(SolrCore core) throws Exception {
}
}
- final private void doReplicateOnlyRecovery(SolrCore core) throws InterruptedException {
+ private final void doReplicateOnlyRecovery(SolrCore core) throws InterruptedException {
final RTimer timer = new RTimer();
boolean successfulRecovery = false;
// if (core.getUpdateHandler().getUpdateLog() != null) {
- // SolrException.log(log, "'replicate-only' recovery strategy should only be used if no update logs are present, but
+ // SolrException.log(log, "'replicate-only' recovery strategy should only be used if no update
+ // logs are present, but
// this core has one: "
// + core.getUpdateHandler().getUpdateLog());
// return;
// }
- while (!successfulRecovery && !Thread.currentThread().isInterrupted() && !isClosed()) { // don't use interruption or
- // it will close channels
- // though
+ while (!successfulRecovery && !Thread.currentThread().isInterrupted() && !isClosed()) {
+ // don't use interruption or it will close channels though
Review comment:
put this above the while.
##########
File path: solr/core/src/java/org/apache/solr/update/SolrIndexWriter.java
##########
@@ -254,40 +327,30 @@ protected void merge(MergePolicy.OneMerge merge) throws IOException {
@Override
protected void doAfterFlush() throws IOException {
if (flushMeter != null) { // this is null when writer is used only for snapshot cleanup
- flushMeter.mark(); // or if mergeTotals == false
+ flushMeter.mark(); // or if mergeTotals == false
}
super.doAfterFlush();
}
/**
- * use DocumentBuilder now...
- * private final void addField(Document doc, String name, String val) {
+ * use DocumentBuilder now... private final void addField(Document doc, String name, String val) {
* SchemaField ftype = schema.getField(name);
- * <p/>
- * // we don't check for a null val ourselves because a solr.FieldType
- * // might actually want to map it to something. If createField()
- * // returns null, then we don't store the field.
- * <p/>
- * Field field = ftype.createField(val, boost);
- * if (field != null) doc.add(field);
- * }
- * <p/>
- * <p/>
- * public void addRecord(String[] fieldNames, String[] fieldValues) throws IOException {
- * Document doc = new Document();
- * for (int i=0; i<fieldNames.length; i++) {
- * String name = fieldNames[i];
- * String val = fieldNames[i];
- * <p/>
- * // first null is end of list. client can reuse arrays if they want
- * // and just write a single null if there is unused space.
- * if (name==null) break;
- * <p/>
- * addField(doc,name,val);
- * }
- * addDocument(doc);
- * }
- * ****
+ *
+ * <p>// we don't check for a null val ourselves because a solr.FieldType // might actually want
+ * to map it to something. If createField() // returns null, then we don't store the field.
+ *
+ * <p>Field field = ftype.createField(val, boost); if (field != null) doc.add(field); }
+ *
+ * <p>
Review comment:
remove
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] dsmiley commented on pull request #705: SOLR-14920: Spotless formatting for core
Posted by GitBox <gi...@apache.org>.
dsmiley commented on pull request #705:
URL: https://github.com/apache/solr/pull/705#issuecomment-1050962426
wow; indeed lets split this, as you suggested in JIRA
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] risdenk commented on a change in pull request #705: SOLR-14920: Spotless formatting for core - non-test only
Posted by GitBox <gi...@apache.org>.
risdenk commented on a change in pull request #705:
URL: https://github.com/apache/solr/pull/705#discussion_r818949965
##########
File path: solr/core/src/java/org/apache/solr/parser/QueryParser.java
##########
@@ -8,25 +8,25 @@
import java.util.HashSet;
import java.util.List;
import java.util.Set;
-
import org.apache.lucene.analysis.Analyzer;
+import org.apache.lucene.queryparser.charstream.CharStream;
+import org.apache.lucene.queryparser.charstream.FastCharStream;
import org.apache.lucene.search.BooleanClause;
import org.apache.lucene.search.Query;
-import org.apache.solr.search.SyntaxError;
import org.apache.solr.search.QParser;
-
-import org.apache.lucene.queryparser.charstream.CharStream;
-import org.apache.lucene.queryparser.charstream.FastCharStream;
+import org.apache.solr.search.SyntaxError;
public class QueryParser extends SolrQueryParserBase implements QueryParserConstants {
Review comment:
Yea I did format it. If it were in like `.generated.` or something it would be easy to exclude. I didn't see a downside to formatting the file even though it was machine generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] dsmiley commented on a change in pull request #705: SOLR-14920: Spotless formatting for core - non-test only
Posted by GitBox <gi...@apache.org>.
dsmiley commented on a change in pull request #705:
URL: https://github.com/apache/solr/pull/705#discussion_r818966572
##########
File path: solr/core/src/java/org/apache/solr/parser/QueryParser.java
##########
@@ -8,25 +8,25 @@
import java.util.HashSet;
import java.util.List;
import java.util.Set;
-
import org.apache.lucene.analysis.Analyzer;
+import org.apache.lucene.queryparser.charstream.CharStream;
+import org.apache.lucene.queryparser.charstream.FastCharStream;
import org.apache.lucene.search.BooleanClause;
import org.apache.lucene.search.Query;
-import org.apache.solr.search.SyntaxError;
import org.apache.solr.search.QParser;
-
-import org.apache.lucene.queryparser.charstream.CharStream;
-import org.apache.lucene.queryparser.charstream.FastCharStream;
+import org.apache.solr.search.SyntaxError;
public class QueryParser extends SolrQueryParserBase implements QueryParserConstants {
Review comment:
Okay. Since Lucene has the same concern, we should probably mimick them. See this commit: https://github.com/apache/lucene/commit/2cbf261032dc8aca56c846971c090c991ac594a6 notice that the spotless config expressly excluded these generated files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] risdenk commented on a change in pull request #705: SOLR-14920: Spotless formatting for core - non-test only
Posted by GitBox <gi...@apache.org>.
risdenk commented on a change in pull request #705:
URL: https://github.com/apache/solr/pull/705#discussion_r815150824
##########
File path: solr/core/src/java/org/apache/solr/api/ContainerPluginsRegistry.java
##########
@@ -88,16 +88,16 @@ public boolean onChange(Map<String, Object> properties) {
Phaser localPhaser = phaser; // volatile read
if (localPhaser != null) {
assert localPhaser.getRegisteredParties() == 1;
- localPhaser.arrive(); // we should be the only ones registered, so this will advance phase each time
+ localPhaser
+ .arrive(); // we should be the only ones registered, so this will advance phase each time
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/DistributedApiAsyncTracker.java
##########
@@ -93,48 +95,57 @@ public DistributedApiAsyncTracker(SolrZkClient zkClient, String rootPath) {
persistentIdsPath = rootPath + ZK_ASYNC_PERSISTENT;
inFlightIdsPath = rootPath + ZK_ASYNC_INFLIGHT;
- trackedAsyncTasks = new SizeLimitedDistributedMap(zkClient, persistentIdsPath, maxTrackedTasks, null);
+ trackedAsyncTasks =
+ new SizeLimitedDistributedMap(zkClient, persistentIdsPath, maxTrackedTasks, null);
inFlightAsyncTasks = new InFlightJobs(zkClient, inFlightIdsPath);
}
/**
- * After a successful call to this method, caller MUST eventually call {@link #setTaskCompleted} or {@link #cancelAsyncId}
- * otherwise the task will forever be considered as in progress.
+ * After a successful call to this method, caller MUST eventually call {@link #setTaskCompleted}
+ * or {@link #cancelAsyncId} otherwise the task will forever be considered as in progress.
+ *
* @param asyncId if {@code null} this method will do nothing.
- * @return {@code true} if the asyncId was not already in use (or is {@code null}) and {@code false} if it is already
- * in use and can't be allocated again.
+ * @return {@code true} if the asyncId was not already in use (or is {@code null}) and {@code
+ * false} if it is already in use and can't be allocated again.
*/
public boolean createNewAsyncJobTracker(String asyncId) {
if (asyncId == null) {
return true;
}
try {
- // First create the persistent node, with no content. If that fails, it means the asyncId has been previously used
+ // First create the persistent node, with no content. If that fails, it means the asyncId has
+ // been previously used
// and not yet cleared...
if (!trackedAsyncTasks.putIfAbsent(asyncId, null)) {
return false;
}
- // ...then create the transient node. If the corresponding ephemeral node already exists, it means the persistent node
- // was removed (maybe trackedAsyncTasks grew too large? It has a max size then evicts). We cannot then track the new
+ // ...then create the transient node. If the corresponding ephemeral node already exists, it
+ // means the persistent node
+ // was removed (maybe trackedAsyncTasks grew too large? It has a max size then evicts). We
+ // cannot then track the new
// provided asyncId, and have simply "revived" its persistent node...
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/DistributedApiAsyncTracker.java
##########
@@ -226,15 +244,20 @@ public void cancelAsyncId(String asyncId) {
return new Pair<>(RequestStatusState.RUNNING, null);
}
- // The task has failed, but there are two options: if response is null, it has failed because the node on which it was
- // running has crashed. If it is not null, it has failed because the execution has failed. Because caller expects a non
+ // The task has failed, but there are two options: if response is null, it has failed because
+ // the node on which it was
+ // running has crashed. If it is not null, it has failed because the execution has failed.
+ // Because caller expects a non
// null response in any case, let's make up one if needed...
if (response == null) {
- // Node crash has removed the ephemeral node, but the command did not complete execution (or didn't even start it, who
+ // Node crash has removed the ephemeral node, but the command did not complete execution (or
+ // didn't even start it, who
// knows). We have a failure to report though so let's create a reasonable return response.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/ExclusiveSliceProperty.java
##########
@@ -129,17 +148,21 @@ private boolean collectCurrentPropStats() {
for (Replica replica : slice.getReplicas()) {
if (onlyActiveNodes && isActive(replica) == false) {
if (StringUtils.isNotBlank(replica.getStr(property))) {
- removeProp(slice, replica.getName()); // Note, we won't be committing this to ZK until later.
+ removeProp(
+ slice, replica.getName()); // Note, we won't be committing this to ZK until later.
}
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/ExclusiveSliceProperty.java
##########
@@ -342,12 +387,15 @@ boolean balanceProperty() {
}
}
- // At this point, nodesHostingProp should contain _only_ lists of replicas that belong to slices that do _not_
+ // At this point, nodesHostingProp should contain _only_ lists of replicas that belong to slices
+ // that do _not_
// have any replica hosting the property. So let's assign them.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/OverseerTaskProcessor.java
##########
@@ -283,18 +310,22 @@ public void run() {
continue;
}
- // clear the blocked tasks, may get refilled below. Given blockedTasks can only get entries from heads and heads
- // has at most MAX_BLOCKED_TASKS tasks, blockedTasks will never exceed MAX_BLOCKED_TASKS entries.
- // Note blockedTasks can't be cleared too early as it is used in the excludedTasks Predicate above.
+ // clear the blocked tasks, may get refilled below. Given blockedTasks can only get
+ // entries from heads and heads
+ // has at most MAX_BLOCKED_TASKS tasks, blockedTasks will never exceed MAX_BLOCKED_TASKS
+ // entries.
+ // Note blockedTasks can't be cleared too early as it is used in the excludedTasks
+ // Predicate above.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/ShardLeaderElectionContext.java
##########
@@ -133,8 +157,10 @@ void runLeaderProcess(boolean weAreReplacement, int pauseBeforeStart) throws Kee
}
if (isClosed) {
- // Solr is shutting down or the ZooKeeper session expired while waiting for replicas. If the later,
- // we cannot be sure we are still the leader, so we should bail out. The OnReconnect handler will
+ // Solr is shutting down or the ZooKeeper session expired while waiting for replicas. If the
+ // later,
+ // we cannot be sure we are still the leader, so we should bail out. The OnReconnect handler
+ // will
// re-register the cores and handle a new leadership election.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/RecoveryStrategy.java
##########
@@ -564,9 +588,11 @@ public final void doSyncOrReplicateRecovery(SolrCore core) throws Exception {
final String ourUrl = ZkCoreNodeProps.getCoreUrl(baseUrl, coreName);
Future<RecoveryInfo> replayFuture = null;
- while (!successfulRecovery && !Thread.currentThread().isInterrupted() && !isClosed()) { // don't use interruption or
- // it will close channels
- // though
+ while (!successfulRecovery
+ && !Thread.currentThread().isInterrupted()
+ && !isClosed()) { // don't use interruption or
+ // it will close channels
+ // though
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/DistributedClusterStateUpdater.java
##########
@@ -365,27 +411,37 @@ private void applyUpdate() throws KeeperException, InterruptedException {
// Note we DO NOT track nor use the live nodes in the cluster state.
// That may means the two abstractions (collection metadata vs. nodes) should be separated.
- // For now trying to diverge as little as possible from existing data structures and code given the need to
- // support both the old way (Overseer) and new way (distributed) of handling cluster state update.
+ // For now trying to diverge as little as possible from existing data structures and code
+ // given the need to
+ // support both the old way (Overseer) and new way (distributed) of handling cluster state
+ // update.
final Set<String> liveNodes = Collections.emptySet();
- // Per Replica States updates are done before all other updates and not subject to the number of attempts of CAS
- // made here, given they have their own CAS strategy and implementation (see PerReplicaStatesOps.persist()).
+ // Per Replica States updates are done before all other updates and not subject to the number
+ // of attempts of CAS
+ // made here, given they have their own CAS strategy and implementation (see
+ // PerReplicaStatesOps.persist()).
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/DistributedClusterStateUpdater.java
##########
@@ -421,17 +480,26 @@ private void applyUpdate() throws KeeperException, InterruptedException {
}
// Get the latest version of the collection from the cluster state first.
- // There is no notion of "cached" here (the boolean passed below) as we the updatedState is based on CollectionRef
- DocCollection docCollection = updatedState.getCollectionOrNull(updater.getCollectionName(), true);
-
- // If we did update per replica states and we're also updating state.json, update the content of state.json to reflect
- // the changes made to replica states. Not strictly necessary (the state source of truth is in per replica states), but nice to have...
+ // There is no notion of "cached" here (the boolean passed below) as we the updatedState is
+ // based on CollectionRef
+ DocCollection docCollection =
+ updatedState.getCollectionOrNull(updater.getCollectionName(), true);
+
+ // If we did update per replica states and we're also updating state.json, update the
+ // content of state.json to reflect
+ // the changes made to replica states. Not strictly necessary (the state source of truth is
+ // in per replica states), but nice to have...
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/DistributedClusterStateUpdater.java
##########
@@ -705,36 +850,54 @@ public ClusterState getUpdatedClusterState() {
}
/**
- * Using optimistic locking (and retries when needed) updates Zookeeper with the changes previously recorded by calls
- * to {@link #record(MutatingCommand, ZkNodeProps)}.
+ * Using optimistic locking (and retries when needed) updates Zookeeper with the changes
+ * previously recorded by calls to {@link #record(MutatingCommand, ZkNodeProps)}.
*/
- public void executeStateUpdates(SolrCloudManager scm, ZkStateReader zkStateReader) throws KeeperException, InterruptedException {
+ public void executeStateUpdates(SolrCloudManager scm, ZkStateReader zkStateReader)
+ throws KeeperException, InterruptedException {
if (log.isDebugEnabled()) {
- log.debug("Executing updates for collection " + collectionName + ", is creation=" + isCollectionCreation + ", " + mutations.size() + " recorded mutations.", new Exception("StackTraceOnly")); // nowarn
+ log.debug(
+ "Executing updates for collection "
+ + collectionName
+ + ", is creation="
+ + isCollectionCreation
+ + ", "
+ + mutations.size()
+ + " recorded mutations.",
+ new Exception("StackTraceOnly")); // nowarn
}
if (mutations.isEmpty()) {
- final String err = "Internal bug. Unexpected empty set of mutations to apply for collection " + collectionName;
+ final String err =
+ "Internal bug. Unexpected empty set of mutations to apply for collection "
+ + collectionName;
log.error(err);
throw new SolrException(SolrException.ErrorCode.SERVER_ERROR, err);
}
- RecordedMutationsPlayer mutationPlayer = new RecordedMutationsPlayer(scm, collectionName, isCollectionCreation, mutations);
+ RecordedMutationsPlayer mutationPlayer =
+ new RecordedMutationsPlayer(scm, collectionName, isCollectionCreation, mutations);
ZkUpdateApplicator.applyUpdate(zkStateReader, mutationPlayer);
// TODO update stats here for the various commands executed successfully or not?
- // This would replace the stats about cluster state updates that the Collection API currently makes available using
- // the OVERSEERSTATUS command, but obviously would be per node and will not have stats about queues (since there
- // will be no queues). Would be useful in some tests though, for example TestSkipOverseerOperations.
- // Probably better to rethink what types of stats are expected from a distributed system rather than trying to present
+ // This would replace the stats about cluster state updates that the Collection API currently
+ // makes available using
+ // the OVERSEERSTATUS command, but obviously would be per node and will not have stats about
+ // queues (since there
+ // will be no queues). Would be useful in some tests though, for example
+ // TestSkipOverseerOperations.
+ // Probably better to rethink what types of stats are expected from a distributed system
+ // rather than trying to present
// those previously provided by a central server in the system (the Overseer).
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/DistributedApiAsyncTracker.java
##########
@@ -146,19 +157,21 @@ public void setTaskRunning(String asyncId) {
throw new SolrException(SERVER_ERROR, "Error setting async task as running " + asyncId, ke);
} catch (InterruptedException ie) {
Thread.currentThread().interrupt();
- throw new SolrException(SERVER_ERROR, "Interrupted setting async task as running " + asyncId, ie);
+ throw new SolrException(
+ SERVER_ERROR, "Interrupted setting async task as running " + asyncId, ie);
}
}
/**
- * Mark the completion (success or error) of an async task. The success or error is judged by the contents
- * of the {@link OverseerSolrResponse}.
+ * Mark the completion (success or error) of an async task. The success or error is judged by the
+ * contents of the {@link OverseerSolrResponse}.
*/
public void setTaskCompleted(String asyncId, OverseerSolrResponse solrResponse) {
if (asyncId == null) {
return;
}
- // First update the persistent node with the execution result, only then remove the transient node
+ // First update the persistent node with the execution result, only then remove the transient
+ // node
// (otherwise a status check might report the task in error)
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/DistributedApiAsyncTracker.java
##########
@@ -226,15 +244,20 @@ public void cancelAsyncId(String asyncId) {
return new Pair<>(RequestStatusState.RUNNING, null);
}
- // The task has failed, but there are two options: if response is null, it has failed because the node on which it was
- // running has crashed. If it is not null, it has failed because the execution has failed. Because caller expects a non
+ // The task has failed, but there are two options: if response is null, it has failed because
+ // the node on which it was
+ // running has crashed. If it is not null, it has failed because the execution has failed.
+ // Because caller expects a non
// null response in any case, let's make up one if needed...
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/ExclusiveSliceProperty.java
##########
@@ -56,12 +56,14 @@
private final DocCollection collection;
private final String collectionName;
- // Key structure. For each node, list all replicas on it regardless of whether they have the property or not.
+ // Key structure. For each node, list all replicas on it regardless of whether they have the
+ // property or not.
private final Map<String, List<SliceReplica>> nodesHostingReplicas = new HashMap<>();
// Key structure. For each node, a list of the replicas _currently_ hosting the property.
private final Map<String, List<SliceReplica>> nodesHostingProp = new HashMap<>();
Set<String> shardsNeedingHosts = new HashSet<>();
- Map<String, Slice> changedSlices = new HashMap<>(); // Work on copies rather than the underlying cluster state.
+ Map<String, Slice> changedSlices =
+ new HashMap<>(); // Work on copies rather than the underlying cluster state.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/ExclusiveSliceProperty.java
##########
@@ -325,7 +369,8 @@ boolean balanceProperty() {
// So, remove a replica from the nodes that have too many
removeOverallocatedReplicas();
- // prune replicas belonging to a slice that have the property currently assigned from the list of replicas
+ // prune replicas belonging to a slice that have the property currently assigned from the list
+ // of replicas
// that could host the property.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/DistributedApiAsyncTracker.java
##########
@@ -93,48 +95,57 @@ public DistributedApiAsyncTracker(SolrZkClient zkClient, String rootPath) {
persistentIdsPath = rootPath + ZK_ASYNC_PERSISTENT;
inFlightIdsPath = rootPath + ZK_ASYNC_INFLIGHT;
- trackedAsyncTasks = new SizeLimitedDistributedMap(zkClient, persistentIdsPath, maxTrackedTasks, null);
+ trackedAsyncTasks =
+ new SizeLimitedDistributedMap(zkClient, persistentIdsPath, maxTrackedTasks, null);
inFlightAsyncTasks = new InFlightJobs(zkClient, inFlightIdsPath);
}
/**
- * After a successful call to this method, caller MUST eventually call {@link #setTaskCompleted} or {@link #cancelAsyncId}
- * otherwise the task will forever be considered as in progress.
+ * After a successful call to this method, caller MUST eventually call {@link #setTaskCompleted}
+ * or {@link #cancelAsyncId} otherwise the task will forever be considered as in progress.
+ *
* @param asyncId if {@code null} this method will do nothing.
- * @return {@code true} if the asyncId was not already in use (or is {@code null}) and {@code false} if it is already
- * in use and can't be allocated again.
+ * @return {@code true} if the asyncId was not already in use (or is {@code null}) and {@code
+ * false} if it is already in use and can't be allocated again.
*/
public boolean createNewAsyncJobTracker(String asyncId) {
if (asyncId == null) {
return true;
}
try {
- // First create the persistent node, with no content. If that fails, it means the asyncId has been previously used
+ // First create the persistent node, with no content. If that fails, it means the asyncId has
+ // been previously used
// and not yet cleared...
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/ExclusiveSliceProperty.java
##########
@@ -208,7 +234,8 @@ private void balanceUnassignedReplicas() {
SliceReplica srToChange = null;
for (String slice : shardsNeedingHosts) {
for (Map.Entry<String, List<SliceReplica>> ent : nodesHostingReplicas.entrySet()) {
- // A little tricky. If we don't set this to something below, then it means all possible places to
+ // A little tricky. If we don't set this to something below, then it means all possible
+ // places to
// put this property are full up, so just put it somewhere.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/ExclusiveSliceProperty.java
##########
@@ -157,24 +180,28 @@ private boolean collectCurrentPropStats() {
}
// If the total number of already-hosted properties assigned to nodes
- // that have potential to host leaders is equal to the slice count _AND_ none of the current nodes has more than
+ // that have potential to host leaders is equal to the slice count _AND_ none of the current
+ // nodes has more than
// the max number of properties, there's nothing to do.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/ExclusiveSliceProperty.java
##########
@@ -112,11 +128,14 @@ private boolean isActive(Replica replica) {
return replica.getState() == Replica.State.ACTIVE;
}
- // Collect a list of all the nodes that _can_ host the indicated property. Along the way, also collect any of
- // the replicas on that node that _already_ host the property as well as any slices that do _not_ have the
+ // Collect a list of all the nodes that _can_ host the indicated property. Along the way, also
+ // collect any of
+ // the replicas on that node that _already_ host the property as well as any slices that do _not_
+ // have the
// property hosted.
//
- // Return true if anything node needs it's property reassigned. False if the property is already balanced for
+ // Return true if anything node needs it's property reassigned. False if the property is already
+ // balanced for
// the collection.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/OverseerTaskProcessor.java
##########
@@ -208,11 +217,17 @@ public void run() {
// TODO: Make maxThreads configurable.
- this.tpe = new ExecutorUtil.MDCAwareThreadPoolExecutor(5, MAX_PARALLEL_TASKS, 0L, TimeUnit.MILLISECONDS,
- new SynchronousQueue<>(),
- new SolrNamedThreadFactory("OverseerThreadFactory"));
-
- // In OverseerCollectionMessageHandler, a new Session needs to be created for each new iteration over the tasks in the
+ this.tpe =
+ new ExecutorUtil.MDCAwareThreadPoolExecutor(
+ 5,
+ MAX_PARALLEL_TASKS,
+ 0L,
+ TimeUnit.MILLISECONDS,
+ new SynchronousQueue<>(),
+ new SolrNamedThreadFactory("OverseerThreadFactory"));
+
+ // In OverseerCollectionMessageHandler, a new Session needs to be created for each new iteration
+ // over the tasks in the
// queue. Incrementing this id causes a new session to be created there.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/OverseerTaskProcessor.java
##########
@@ -169,20 +173,24 @@ public void run() {
LeaderStatus isLeader = amILeader();
while (isLeader == LeaderStatus.DONT_KNOW) {
log.debug("am_i_leader unclear {}", isLeader);
- isLeader = amILeader(); // not a no, not a yes, try ask again
+ isLeader = amILeader(); // not a no, not a yes, try ask again
}
String oldestItemInWorkQueue = null;
- // hasLeftOverItems - used for avoiding re-execution of async tasks that were processed by a previous Overseer.
- // This variable is set in case there's any task found on the workQueue when the OCP starts up and
- // the id for the queue tail is used as a marker to check for the task in completed/failed map in zk.
+ // hasLeftOverItems - used for avoiding re-execution of async tasks that were processed by a
+ // previous Overseer.
+ // This variable is set in case there's any task found on the workQueue when the OCP starts up
+ // and
+ // the id for the queue tail is used as a marker to check for the task in completed/failed map
+ // in zk.
// Beyond the marker, all tasks can safely be assumed to have never been executed.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/OverseerTaskProcessor.java
##########
@@ -169,20 +173,24 @@ public void run() {
LeaderStatus isLeader = amILeader();
while (isLeader == LeaderStatus.DONT_KNOW) {
log.debug("am_i_leader unclear {}", isLeader);
- isLeader = amILeader(); // not a no, not a yes, try ask again
+ isLeader = amILeader(); // not a no, not a yes, try ask again
}
String oldestItemInWorkQueue = null;
- // hasLeftOverItems - used for avoiding re-execution of async tasks that were processed by a previous Overseer.
- // This variable is set in case there's any task found on the workQueue when the OCP starts up and
- // the id for the queue tail is used as a marker to check for the task in completed/failed map in zk.
+ // hasLeftOverItems - used for avoiding re-execution of async tasks that were processed by a
+ // previous Overseer.
+ // This variable is set in case there's any task found on the workQueue when the OCP starts up
+ // and
+ // the id for the queue tail is used as a marker to check for the task in completed/failed map
+ // in zk.
// Beyond the marker, all tasks can safely be assumed to have never been executed.
boolean hasLeftOverItems = true;
try {
oldestItemInWorkQueue = workQueue.getTailId();
} catch (KeeperException e) {
- // We don't need to handle this. This is just a fail-safe which comes in handy in skipping already processed
+ // We don't need to handle this. This is just a fail-safe which comes in handy in skipping
+ // already processed
// async calls.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/RecoveryStrategy.java
##########
@@ -332,27 +336,33 @@ final public void doRecovery(SolrCore core) throws Exception {
}
}
- final private void doReplicateOnlyRecovery(SolrCore core) throws InterruptedException {
+ private final void doReplicateOnlyRecovery(SolrCore core) throws InterruptedException {
final RTimer timer = new RTimer();
boolean successfulRecovery = false;
// if (core.getUpdateHandler().getUpdateLog() != null) {
- // SolrException.log(log, "'replicate-only' recovery strategy should only be used if no update logs are present, but
+ // SolrException.log(log, "'replicate-only' recovery strategy should only be used if no update
+ // logs are present, but
// this core has one: "
// + core.getUpdateHandler().getUpdateLog());
// return;
// }
- while (!successfulRecovery && !Thread.currentThread().isInterrupted() && !isClosed()) { // don't use interruption or
- // it will close channels
- // though
+ while (!successfulRecovery
+ && !Thread.currentThread().isInterrupted()
+ && !isClosed()) { // don't use interruption or
+ // it will close channels
+ // though
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/OverseerTaskProcessor.java
##########
@@ -238,41 +256,50 @@ public void run() {
while (runningTasks.size() > MAX_PARALLEL_TASKS) {
synchronized (waitLock) {
- waitLock.wait(100);//wait for 100 ms or till a task is complete
+ waitLock.wait(100); // wait for 100 ms or till a task is complete
}
waited = true;
}
- if (waited)
- cleanUpWorkQueue();
+ if (waited) cleanUpWorkQueue();
ArrayList<QueueEvent> heads = new ArrayList<>(blockedTasks.size() + MAX_PARALLEL_TASKS);
heads.addAll(blockedTasks.values());
- //If we have enough items in the blocked tasks already, it makes
+ // If we have enough items in the blocked tasks already, it makes
// no sense to read more items from the work queue. it makes sense
// to clear out at least a few items in the queue before we read more items
if (heads.size() < MAX_BLOCKED_TASKS) {
- //instead of reading MAX_PARALLEL_TASKS items always, we should only fetch as much as we can execute
- int toFetch = Math.min(MAX_BLOCKED_TASKS - heads.size(), MAX_PARALLEL_TASKS - runningTasks.size());
+ // instead of reading MAX_PARALLEL_TASKS items always, we should only fetch as much as
+ // we can execute
+ int toFetch =
+ Math.min(
+ MAX_BLOCKED_TASKS - heads.size(), MAX_PARALLEL_TASKS - runningTasks.size());
List<QueueEvent> newTasks = workQueue.peekTopN(toFetch, excludedTasks, 2000L);
if (log.isDebugEnabled()) {
log.debug("Got {} tasks from work-queue : [{}]", newTasks.size(), newTasks);
}
// heads has at most MAX_BLOCKED_TASKS tasks.
heads.addAll(newTasks);
} else {
- // The sleep below slows down spinning when heads is full from previous work dispatch attempt below and no new
- // tasks got executed (all executors are busy or all waiting tasks require locks currently held by executors).
+ // The sleep below slows down spinning when heads is full from previous work dispatch
+ // attempt below and no new
+ // tasks got executed (all executors are busy or all waiting tasks require locks
+ // currently held by executors).
//
- // When heads is not full but no progress was made (no new work got dispatched in the for loop below), slowing down
+ // When heads is not full but no progress was made (no new work got dispatched in the
+ // for loop below), slowing down
// of the spinning is done by the wait time in the call to workQueue.peekTopN() above.
- // (at least in theory because the method eventually called from there is ZkDistributedQueue.peekElements()
- // and because it filters out entries that have just completed on a Runner thread in a different way than the
- // predicate based filtering, it can return quickly without waiting the configured delay time. Therefore spinning
+ // (at least in theory because the method eventually called from there is
+ // ZkDistributedQueue.peekElements()
+ // and because it filters out entries that have just completed on a Runner thread in a
+ // different way than the
+ // predicate based filtering, it can return quickly without waiting the configured delay
+ // time. Therefore spinning
// can be observed, likely something to clean up at some point).
//
- // If heads is not empty and new tasks appeared in the queue there's no delay, workQueue.peekTopN() above will
+ // If heads is not empty and new tasks appeared in the queue there's no delay,
+ // workQueue.peekTopN() above will
// return immediately.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/ExclusiveSliceProperty.java
##########
@@ -157,24 +180,28 @@ private boolean collectCurrentPropStats() {
}
// If the total number of already-hosted properties assigned to nodes
- // that have potential to host leaders is equal to the slice count _AND_ none of the current nodes has more than
+ // that have potential to host leaders is equal to the slice count _AND_ none of the current
+ // nodes has more than
// the max number of properties, there's nothing to do.
origMaxPropPerNode = collection.getSlices().size() / allHosts.size();
// Some nodes can have one more of the proeprty if the numbers aren't exactly even.
origModulo = collection.getSlices().size() % allHosts.size();
if (origModulo > 0) {
- origMaxPropPerNode++; // have to have some nodes with 1 more property.
+ origMaxPropPerNode++; // have to have some nodes with 1 more property.
}
- // We can say for sure that we need to rebalance if we don't have as many assigned properties as slices.
+ // We can say for sure that we need to rebalance if we don't have as many assigned properties as
+ // slices.
if (assigned != collection.getSlices().size()) {
return true;
}
// Make sure there are no more slices at the limit than the "leftovers"
- // Let's say there's 7 slices and 3 nodes. We need to distribute the property as 3 on node1, 2 on node2 and 2 on node3
- // (3, 2, 2) We need to be careful to not distribute them as 3, 3, 1. that's what this check is all about.
+ // Let's say there's 7 slices and 3 nodes. We need to distribute the property as 3 on node1, 2
+ // on node2 and 2 on node3
+ // (3, 2, 2) We need to be careful to not distribute them as 3, 3, 1. that's what this check is
+ // all about.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/ReplicateFromLeader.java
##########
@@ -80,29 +81,36 @@ public void startReplication(boolean switchTransactionLog) {
pollIntervalStr = "00:00:01";
}
if (uinfo.autoCommmitMaxTime != -1) {
- pollIntervalStr = toPollIntervalStr(uinfo.autoCommmitMaxTime/2);
+ pollIntervalStr = toPollIntervalStr(uinfo.autoCommmitMaxTime / 2);
} else if (uinfo.autoSoftCommmitMaxTime != -1) {
- pollIntervalStr = toPollIntervalStr(uinfo.autoSoftCommmitMaxTime/2);
+ pollIntervalStr = toPollIntervalStr(uinfo.autoSoftCommmitMaxTime / 2);
}
- log.info("Will start replication from leader with poll interval: {}", pollIntervalStr );
+ log.info("Will start replication from leader with poll interval: {}", pollIntervalStr);
NamedList<Object> followerConfig = new NamedList<>();
followerConfig.add("fetchFromLeader", Boolean.TRUE);
- // don't commit on leader version zero for PULL replicas as PULL should only get its index state from leader
+ // don't commit on leader version zero for PULL replicas as PULL should only get its index
+ // state from leader
boolean skipCommitOnLeaderVersionZero = switchTransactionLog;
if (!skipCommitOnLeaderVersionZero) {
CloudDescriptor cloudDescriptor = core.getCoreDescriptor().getCloudDescriptor();
if (cloudDescriptor != null) {
Replica replica =
- cc.getZkController().getZkStateReader().getCollection(cloudDescriptor.getCollectionName())
- .getSlice(cloudDescriptor.getShardId()).getReplica(cloudDescriptor.getCoreNodeName());
+ cc.getZkController()
+ .getZkStateReader()
+ .getCollection(cloudDescriptor.getCollectionName())
+ .getSlice(cloudDescriptor.getShardId())
+ .getReplica(cloudDescriptor.getCoreNodeName());
if (replica != null && replica.getType() == Replica.Type.PULL) {
- skipCommitOnLeaderVersionZero = true; // only set this to true if we're a PULL replica, otherwise use value of switchTransactionLog
+ skipCommitOnLeaderVersionZero =
+ true; // only set this to true if we're a PULL replica, otherwise use value of
+ // switchTransactionLog
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/api/ApiBag.java
##########
@@ -115,7 +122,8 @@ protected void attachValueToNode(PathTrie<Api>.Node node, Api o) {
return;
}
- // If 'o' and 'node.obj' aren't both AnnotatedApi's then we can't aggregate the commands, so fallback to the
+ // If 'o' and 'node.obj' aren't both AnnotatedApi's then we can't aggregate the commands, so
+ // fallback to the
// default behavior
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/RecoveryStrategy.java
##########
@@ -273,25 +276,23 @@ final private void replicate(String nodeName, SolrCore core, ZkNodeProps leaderp
log.debug("Error in solrcloud_debug block", e);
}
}
-
}
- final private void commitOnLeader(String leaderUrl) throws SolrServerException,
- IOException {
+ private final void commitOnLeader(String leaderUrl) throws SolrServerException, IOException {
try (HttpSolrClient client = recoverySolrClientBuilder(leaderUrl).build()) {
UpdateRequest ureq = new UpdateRequest();
ureq.setParams(new ModifiableSolrParams());
// ureq.getParams().set(DistributedUpdateProcessor.COMMIT_END_POINT, true);
- // ureq.getParams().set(UpdateParams.OPEN_SEARCHER, onlyLeaderIndexes);// Why do we need to open searcher if
+ // ureq.getParams().set(UpdateParams.OPEN_SEARCHER, onlyLeaderIndexes);// Why do we need to
+ // open searcher if
// "onlyLeaderIndexes"?
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/ShardLeaderElectionContextBase.java
##########
@@ -89,10 +95,15 @@ public void cancelElection() throws InterruptedException, KeeperException {
// no problem
try {
// We need to be careful and make sure we *only* delete our own leader registration node.
- // We do this by using a multi and ensuring the parent znode of the leader registration node
- // matches the version we expect - there is a setData call that increments the parent's znode
+ // We do this by using a multi and ensuring the parent znode of the leader registration
+ // node
+ // matches the version we expect - there is a setData call that increments the parent's
+ // znode
// version whenever a leader registers.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/RecoveryStrategy.java
##########
@@ -541,15 +563,17 @@ public final void doSyncOrReplicateRecovery(SolrCore core) throws Exception {
}
if (recoveringAfterStartup) {
- // if we're recovering after startup (i.e. we have been down), then we need to know what the last versions were
+ // if we're recovering after startup (i.e. we have been down), then we need to know what the
+ // last versions were
// when we went down. We may have received updates since then.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/RecoveryStrategy.java
##########
@@ -332,27 +336,33 @@ final public void doRecovery(SolrCore core) throws Exception {
}
}
- final private void doReplicateOnlyRecovery(SolrCore core) throws InterruptedException {
+ private final void doReplicateOnlyRecovery(SolrCore core) throws InterruptedException {
final RTimer timer = new RTimer();
boolean successfulRecovery = false;
// if (core.getUpdateHandler().getUpdateLog() != null) {
- // SolrException.log(log, "'replicate-only' recovery strategy should only be used if no update logs are present, but
+ // SolrException.log(log, "'replicate-only' recovery strategy should only be used if no update
+ // logs are present, but
// this core has one: "
// + core.getUpdateHandler().getUpdateLog());
// return;
// }
- while (!successfulRecovery && !Thread.currentThread().isInterrupted() && !isClosed()) { // don't use interruption or
- // it will close channels
- // though
+ while (!successfulRecovery
+ && !Thread.currentThread().isInterrupted()
+ && !isClosed()) { // don't use interruption or
+ // it will close channels
+ // though
try {
CloudDescriptor cloudDesc = this.coreDescriptor.getCloudDescriptor();
- ZkNodeProps leaderprops = zkStateReader.getLeaderRetry(cloudDesc.getCollectionName(), cloudDesc.getShardId());
+ ZkNodeProps leaderprops =
+ zkStateReader.getLeaderRetry(cloudDesc.getCollectionName(), cloudDesc.getShardId());
final String leaderUrl = ZkCoreNodeProps.getCoreUrl(leaderprops);
final String ourUrl = ZkCoreNodeProps.getCoreUrl(baseUrl, coreName);
- boolean isLeader = ourUrl.equals(leaderUrl); // TODO: We can probably delete most of this code if we say this
- // strategy can only be used for pull replicas
+ boolean isLeader =
+ ourUrl.equals(
+ leaderUrl); // TODO: We can probably delete most of this code if we say this
+ // strategy can only be used for pull replicas
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/Overseer.java
##########
@@ -310,30 +336,44 @@ else if (LeaderStatus.YES != isLeader) {
byte[] data = head.second();
final ZkNodeProps message = ZkNodeProps.load(data);
if (log.isDebugEnabled()) {
- log.debug("processMessage: queueSize: {}, message = {}", stateUpdateQueue.getZkStats().getQueueLength(), message);
+ log.debug(
+ "processMessage: queueSize: {}, message = {}",
+ stateUpdateQueue.getZkStats().getQueueLength(),
+ message);
}
processedNodes.add(head.first());
fallbackQueueSize = processedNodes.size();
- // force flush to ZK after each message because there is no fallback if workQueue items
+ // force flush to ZK after each message because there is no fallback if workQueue
+ // items
// are removed from workQueue but fail to be written to ZK
while (unprocessedMessages.size() > 0) {
clusterState = zkStateWriter.writePendingUpdates();
Message m = unprocessedMessages.remove(0);
clusterState = m.run(clusterState, Overseer.this);
}
// The callback always be called on this thread
- clusterState = processQueueItem(message, clusterState, zkStateWriter, true, () -> {
- stateUpdateQueue.remove(processedNodes);
- processedNodes.clear();
- });
+ clusterState =
+ processQueueItem(
+ message,
+ clusterState,
+ zkStateWriter,
+ true,
+ () -> {
+ stateUpdateQueue.remove(processedNodes);
+ processedNodes.clear();
+ });
}
if (isClosed) break;
// if an event comes in the next 100ms batch it together
- queue = new LinkedList<>(stateUpdateQueue.peekElements(1000, 100, node -> !processedNodes.contains(node)));
+ queue =
+ new LinkedList<>(
+ stateUpdateQueue.peekElements(
+ 1000, 100, node -> !processedNodes.contains(node)));
}
fallbackQueueSize = processedNodes.size();
- // we should force write all pending updates because the next iteration might sleep until there
+ // we should force write all pending updates because the next iteration might sleep
+ // until there
// are more items in the main queue
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/Overseer.java
##########
@@ -627,49 +707,73 @@ public Overseer(HttpShardHandler shardHandler,
this.zkController = zkController;
this.stats = new Stats();
this.config = config;
- this.distributedClusterStateUpdater = new DistributedClusterStateUpdater(config.getDistributedClusterStateUpdates());
-
- this.solrMetricsContext = new SolrMetricsContext(zkController.getCoreContainer().getMetricManager(), SolrInfoBean.Group.overseer.toString(), metricTag);
+ this.distributedClusterStateUpdater =
+ new DistributedClusterStateUpdater(config.getDistributedClusterStateUpdates());
+
+ this.solrMetricsContext =
+ new SolrMetricsContext(
+ zkController.getCoreContainer().getMetricManager(),
+ SolrInfoBean.Group.overseer.toString(),
+ metricTag);
}
public synchronized void start(String id) {
- MDCLoggingContext.setNode(zkController == null ?
- null :
- zkController.getNodeName());
+ MDCLoggingContext.setNode(zkController == null ? null : zkController.getNodeName());
this.id = id;
closed = false;
doClose();
stats = new Stats();
log.info("Overseer (id={}) starting", id);
createOverseerNode(reader.getZkClient());
- //launch cluster state updater thread
+ // launch cluster state updater thread
ThreadGroup tg = new ThreadGroup("Overseer state updater.");
- updaterThread = new OverseerThread(tg, new ClusterStateUpdater(reader, id, stats), "OverseerStateUpdate-" + id);
+ updaterThread =
+ new OverseerThread(
+ tg, new ClusterStateUpdater(reader, id, stats), "OverseerStateUpdate-" + id);
updaterThread.setDaemon(true);
ThreadGroup ccTg = new ThreadGroup("Overseer collection creation process.");
- // Below is the only non test usage of the "cluster state update" queue even when distributed cluster state updates are enabled.
- // That queue is used to tell the Overseer to quit. As long as we have an Overseer, we need to support this.
- OverseerNodePrioritizer overseerPrioritizer = new OverseerNodePrioritizer(reader, this, adminPath, shardHandler.getShardHandlerFactory());
- overseerCollectionConfigSetProcessor = new OverseerCollectionConfigSetProcessor(reader, id, shardHandler, adminPath, stats, Overseer.this, overseerPrioritizer, solrMetricsContext);
- ccThread = new OverseerThread(ccTg, overseerCollectionConfigSetProcessor, "OverseerCollectionConfigSetProcessor-" + id);
+ // Below is the only non test usage of the "cluster state update" queue even when distributed
+ // cluster state updates are enabled.
+ // That queue is used to tell the Overseer to quit. As long as we have an Overseer, we need to
+ // support this.
+ OverseerNodePrioritizer overseerPrioritizer =
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/DistributedClusterStateUpdater.java
##########
@@ -365,27 +411,37 @@ private void applyUpdate() throws KeeperException, InterruptedException {
// Note we DO NOT track nor use the live nodes in the cluster state.
// That may means the two abstractions (collection metadata vs. nodes) should be separated.
- // For now trying to diverge as little as possible from existing data structures and code given the need to
- // support both the old way (Overseer) and new way (distributed) of handling cluster state update.
+ // For now trying to diverge as little as possible from existing data structures and code
+ // given the need to
+ // support both the old way (Overseer) and new way (distributed) of handling cluster state
+ // update.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/Overseer.java
##########
@@ -1047,19 +1172,26 @@ private void createOverseerNode(final SolrZkClient zkClient) {
throw new RuntimeException(e);
}
}
-
+
public ZkStateReader getZkStateReader() {
return reader;
}
public void offerStateUpdate(byte[] data) throws KeeperException, InterruptedException {
- // When cluster state update is distributed, the Overseer cluster state update queue should only ever receive QUIT messages.
+ // When cluster state update is distributed, the Overseer cluster state update queue should only
+ // ever receive QUIT messages.
// These go to sendQuitToOverseer for execution path clarity.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/DistributedClusterStateUpdater.java
##########
@@ -365,27 +411,37 @@ private void applyUpdate() throws KeeperException, InterruptedException {
// Note we DO NOT track nor use the live nodes in the cluster state.
// That may means the two abstractions (collection metadata vs. nodes) should be separated.
- // For now trying to diverge as little as possible from existing data structures and code given the need to
- // support both the old way (Overseer) and new way (distributed) of handling cluster state update.
+ // For now trying to diverge as little as possible from existing data structures and code
+ // given the need to
+ // support both the old way (Overseer) and new way (distributed) of handling cluster state
+ // update.
final Set<String> liveNodes = Collections.emptySet();
- // Per Replica States updates are done before all other updates and not subject to the number of attempts of CAS
- // made here, given they have their own CAS strategy and implementation (see PerReplicaStatesOps.persist()).
+ // Per Replica States updates are done before all other updates and not subject to the number
+ // of attempts of CAS
+ // made here, given they have their own CAS strategy and implementation (see
+ // PerReplicaStatesOps.persist()).
boolean firstAttempt = true;
- // When there are multiple retries of state.json write and the cluster state gets updated over and over again with
+ // When there are multiple retries of state.json write and the cluster state gets updated over
+ // and over again with
// the changes done in the per replica states, we avoid refetching those multiple times.
PerReplicaStates fetchedPerReplicaStates = null;
- // Later on (when Collection API commands are distributed) we will have to rely on the version of state.json
- // to implement the replacement of Collection API locking. Then we should not blindly retry cluster state updates
- // as we do here but instead intelligently fail (or retry completely) the Collection API call when seeing that
+ // Later on (when Collection API commands are distributed) we will have to rely on the version
+ // of state.json
+ // to implement the replacement of Collection API locking. Then we should not blindly retry
+ // cluster state updates
+ // as we do here but instead intelligently fail (or retry completely) the Collection API call
+ // when seeing that
// state.json was changed by a concurrent command execution.
- // The loop below is ok for distributing cluster state updates from Overseer to all nodes while Collection API
+ // The loop below is ok for distributing cluster state updates from Overseer to all nodes
+ // while Collection API
// commands are still executed on the Overseer and manage their locking the old fashioned way.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/DistributedClusterStateUpdater.java
##########
@@ -394,9 +450,12 @@ private void applyUpdate() throws KeeperException, InterruptedException {
initialClusterState = fetchStateForCollection();
}
- // Apply the desired changes. Note that the cluster state passed to the chain of mutators is totally up to date
- // (it's read from ZK just above). So assumptions made in the mutators (like SliceMutator.removeReplica() deleting
- // the whole collection if it's not found) are ok. Actually in the removeReplica case, the collection will always
+ // Apply the desired changes. Note that the cluster state passed to the chain of mutators is
+ // totally up to date
+ // (it's read from ZK just above). So assumptions made in the mutators (like
+ // SliceMutator.removeReplica() deleting
+ // the whole collection if it's not found) are ok. Actually in the removeReplica case, the
+ // collection will always
// exist otherwise the call to fetchStateForCollection() above would have failed.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/DistributedClusterStateUpdater.java
##########
@@ -365,27 +411,37 @@ private void applyUpdate() throws KeeperException, InterruptedException {
// Note we DO NOT track nor use the live nodes in the cluster state.
// That may means the two abstractions (collection metadata vs. nodes) should be separated.
- // For now trying to diverge as little as possible from existing data structures and code given the need to
- // support both the old way (Overseer) and new way (distributed) of handling cluster state update.
+ // For now trying to diverge as little as possible from existing data structures and code
+ // given the need to
+ // support both the old way (Overseer) and new way (distributed) of handling cluster state
+ // update.
final Set<String> liveNodes = Collections.emptySet();
- // Per Replica States updates are done before all other updates and not subject to the number of attempts of CAS
- // made here, given they have their own CAS strategy and implementation (see PerReplicaStatesOps.persist()).
+ // Per Replica States updates are done before all other updates and not subject to the number
+ // of attempts of CAS
+ // made here, given they have their own CAS strategy and implementation (see
+ // PerReplicaStatesOps.persist()).
boolean firstAttempt = true;
- // When there are multiple retries of state.json write and the cluster state gets updated over and over again with
+ // When there are multiple retries of state.json write and the cluster state gets updated over
+ // and over again with
// the changes done in the per replica states, we avoid refetching those multiple times.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/DistributedClusterStateUpdater.java
##########
@@ -441,58 +509,83 @@ private void applyUpdate() throws KeeperException, InterruptedException {
return; // state.json updated successfully.
} catch (KeeperException.BadVersionException bve) {
if (updater.isCollectionCreation()) {
- // Not expecting to see this exception when creating new state.json fails, so throwing it up the food chain.
+ // Not expecting to see this exception when creating new state.json fails, so throwing
+ // it up the food chain.
throw bve;
}
}
- // We've tried to update an existing state.json and got a BadVersionException. We'll try again a few times.
- // When only two threads compete, no point in waiting: if we lost this time we'll get it next time right away.
- // But if more threads compete, then waiting a bit (random delay) can improve our chances. The delay should in
- // theory grow as the number of concurrent threads attempting updates increase, but we don't know that number, so
+ // We've tried to update an existing state.json and got a BadVersionException. We'll try
+ // again a few times.
+ // When only two threads compete, no point in waiting: if we lost this time we'll get it
+ // next time right away.
+ // But if more threads compete, then waiting a bit (random delay) can improve our chances.
+ // The delay should in
+ // theory grow as the number of concurrent threads attempting updates increase, but we don't
+ // know that number, so
// doing exponential backoff instead.
- // With "per replica states" collections, concurrent attempts of even just two threads are expected to be extremely rare.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/DistributedClusterStateUpdater.java
##########
@@ -441,58 +509,83 @@ private void applyUpdate() throws KeeperException, InterruptedException {
return; // state.json updated successfully.
} catch (KeeperException.BadVersionException bve) {
if (updater.isCollectionCreation()) {
- // Not expecting to see this exception when creating new state.json fails, so throwing it up the food chain.
+ // Not expecting to see this exception when creating new state.json fails, so throwing
+ // it up the food chain.
throw bve;
}
}
- // We've tried to update an existing state.json and got a BadVersionException. We'll try again a few times.
- // When only two threads compete, no point in waiting: if we lost this time we'll get it next time right away.
- // But if more threads compete, then waiting a bit (random delay) can improve our chances. The delay should in
- // theory grow as the number of concurrent threads attempting updates increase, but we don't know that number, so
+ // We've tried to update an existing state.json and got a BadVersionException. We'll try
+ // again a few times.
+ // When only two threads compete, no point in waiting: if we lost this time we'll get it
+ // next time right away.
+ // But if more threads compete, then waiting a bit (random delay) can improve our chances.
+ // The delay should in
+ // theory grow as the number of concurrent threads attempting updates increase, but we don't
+ // know that number, so
// doing exponential backoff instead.
- // With "per replica states" collections, concurrent attempts of even just two threads are expected to be extremely rare.
- Thread.sleep(CollectionHandlingUtils.RANDOM.nextInt(attempt < 13 ? 1 << attempt : 1 << 13)); // max wait 2^13ms=8.192 sec
+ // With "per replica states" collections, concurrent attempts of even just two threads are
+ // expected to be extremely rare.
+ Thread.sleep(
+ CollectionHandlingUtils.RANDOM.nextInt(
+ attempt < 13 ? 1 << attempt : 1 << 13)); // max wait 2^13ms=8.192 sec
}
- // We made quite a few attempts but failed repeatedly. This is pretty bad but we can't loop trying forever.
- // Offering a job to the Overseer wouldn't usually fail if the ZK queue can be written to (but the Overseer can then
+ // We made quite a few attempts but failed repeatedly. This is pretty bad but we can't loop
+ // trying forever.
+ // Offering a job to the Overseer wouldn't usually fail if the ZK queue can be written to (but
+ // the Overseer can then
// loop forever attempting the update).
- // We do want whoever called us to fail right away rather than to wait for a cluster change and timeout because it
- // didn't happen. Likely need to review call by call what is the appropriate behaviour, especially once Collection
- // API is distributed (because then the Collection API call will fail if the underlying cluster state update cannot
+ // We do want whoever called us to fail right away rather than to wait for a cluster change
+ // and timeout because it
+ // didn't happen. Likely need to review call by call what is the appropriate behaviour,
+ // especially once Collection
+ // API is distributed (because then the Collection API call will fail if the underlying
+ // cluster state update cannot
// be done, and that's a desirable thing).
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/DistributedClusterStateUpdater.java
##########
@@ -681,11 +820,17 @@ public void computeUpdates(ClusterState clusterState, SolrZkClient client) {
perReplicaStateOps.add(zkcmd.ops);
}
} catch (Exception e) {
- // Seems weird to skip rather than fail, but that's what Overseer is doing (see ClusterStateUpdater.processQueueItem()).
- // Maybe in the new distributed update world we should make the caller fail? (something Overseer cluster state updater can't do)
- // To be reconsidered once Collection API commands are distributed because then cluster updates are done synchronously and
+ // Seems weird to skip rather than fail, but that's what Overseer is doing (see
+ // ClusterStateUpdater.processQueueItem()).
+ // Maybe in the new distributed update world we should make the caller fail? (something
+ // Overseer cluster state updater can't do)
+ // To be reconsidered once Collection API commands are distributed because then cluster
+ // updates are done synchronously and
// have the opportunity to make the Collection API call fail directly.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/DistributedClusterStateUpdater.java
##########
@@ -747,36 +910,47 @@ public void executeStateUpdates(SolrCloudManager scm, ZkStateReader zkStateReade
private List<PerReplicaStatesOps> replicaOpsList = null;
/**
- * Entry point to mark all replicas of all collections present on a single node as being DOWN (because the node is down)
+ * Entry point to mark all replicas of all collections present on a single node as being DOWN
+ * (because the node is down)
*/
public static void executeNodeDownStateUpdate(String nodeName, ZkStateReader zkStateReader) {
- // This code does a version of what NodeMutator.downNode() is doing. We can't assume we have a cache of the collections,
- // so we're going to read all of them from ZK, fetch the state.json for each and if it has any replicas on the
+ // This code does a version of what NodeMutator.downNode() is doing. We can't assume we have a
+ // cache of the collections,
+ // so we're going to read all of them from ZK, fetch the state.json for each and if it has any
+ // replicas on the
// failed node, do an update (conditional of course) of the state.json
- // For Per Replica States collections there is still a need to read state.json, but the update of state.json is replaced
- // by a few znode deletions and creations. Might be faster or slower overall, depending on the number of impacted
+ // For Per Replica States collections there is still a need to read state.json, but the update
+ // of state.json is replaced
+ // by a few znode deletions and creations. Might be faster or slower overall, depending on the
+ // number of impacted
// replicas of such a collection and the total size of that collection's state.json.
- // Note code here also has to duplicate some of the work done in ZkStateReader because ZkStateReader couples reading of
- // the cluster state and maintaining a cached copy of the cluster state. Something likely to be refactored later (once
+ // Note code here also has to duplicate some of the work done in ZkStateReader because
+ // ZkStateReader couples reading of
+ // the cluster state and maintaining a cached copy of the cluster state. Something likely to
+ // be refactored later (once
// Overseer is totally removed and Zookeeper access patterns become clearer).
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/DistributedClusterStateUpdater.java
##########
@@ -441,58 +509,83 @@ private void applyUpdate() throws KeeperException, InterruptedException {
return; // state.json updated successfully.
} catch (KeeperException.BadVersionException bve) {
if (updater.isCollectionCreation()) {
- // Not expecting to see this exception when creating new state.json fails, so throwing it up the food chain.
+ // Not expecting to see this exception when creating new state.json fails, so throwing
+ // it up the food chain.
throw bve;
}
}
- // We've tried to update an existing state.json and got a BadVersionException. We'll try again a few times.
- // When only two threads compete, no point in waiting: if we lost this time we'll get it next time right away.
- // But if more threads compete, then waiting a bit (random delay) can improve our chances. The delay should in
- // theory grow as the number of concurrent threads attempting updates increase, but we don't know that number, so
+ // We've tried to update an existing state.json and got a BadVersionException. We'll try
+ // again a few times.
+ // When only two threads compete, no point in waiting: if we lost this time we'll get it
+ // next time right away.
+ // But if more threads compete, then waiting a bit (random delay) can improve our chances.
+ // The delay should in
+ // theory grow as the number of concurrent threads attempting updates increase, but we don't
+ // know that number, so
// doing exponential backoff instead.
- // With "per replica states" collections, concurrent attempts of even just two threads are expected to be extremely rare.
- Thread.sleep(CollectionHandlingUtils.RANDOM.nextInt(attempt < 13 ? 1 << attempt : 1 << 13)); // max wait 2^13ms=8.192 sec
+ // With "per replica states" collections, concurrent attempts of even just two threads are
+ // expected to be extremely rare.
+ Thread.sleep(
+ CollectionHandlingUtils.RANDOM.nextInt(
+ attempt < 13 ? 1 << attempt : 1 << 13)); // max wait 2^13ms=8.192 sec
}
- // We made quite a few attempts but failed repeatedly. This is pretty bad but we can't loop trying forever.
- // Offering a job to the Overseer wouldn't usually fail if the ZK queue can be written to (but the Overseer can then
+ // We made quite a few attempts but failed repeatedly. This is pretty bad but we can't loop
+ // trying forever.
+ // Offering a job to the Overseer wouldn't usually fail if the ZK queue can be written to (but
+ // the Overseer can then
// loop forever attempting the update).
- // We do want whoever called us to fail right away rather than to wait for a cluster change and timeout because it
- // didn't happen. Likely need to review call by call what is the appropriate behaviour, especially once Collection
- // API is distributed (because then the Collection API call will fail if the underlying cluster state update cannot
+ // We do want whoever called us to fail right away rather than to wait for a cluster change
+ // and timeout because it
+ // didn't happen. Likely need to review call by call what is the appropriate behaviour,
+ // especially once Collection
+ // API is distributed (because then the Collection API call will fail if the underlying
+ // cluster state update cannot
// be done, and that's a desirable thing).
- throw new KeeperException.BadVersionException(ZkStateReader.getCollectionPath(updater.getCollectionName()));
+ throw new KeeperException.BadVersionException(
+ ZkStateReader.getCollectionPath(updater.getCollectionName()));
}
/**
- * After the computing of the new {@link ClusterState} containing all needed updates to the collection based on what the
- * {@link StateChangeCalculator} computed, this method does an update in ZK to the collection's {@code state.json}. It is the
- * equivalent of Overseer's {@link ZkStateWriter#writePendingUpdates} (in its actions related to {@code state.json}
- * as opposed to the per replica states).
- * <p>
- * Note that in a similar way to what happens in {@link ZkStateWriter#writePendingUpdates}, collection delete is handled
- * as a special case. (see comment on {@link DistributedClusterStateUpdater.StateChangeRecorder.RecordedMutationsPlayer}
- * on why the code has to be duplicated)<p>
+ * After the computing of the new {@link ClusterState} containing all needed updates to the
+ * collection based on what the {@link StateChangeCalculator} computed, this method does an
+ * update in ZK to the collection's {@code state.json}. It is the equivalent of Overseer's
+ * {@link ZkStateWriter#writePendingUpdates} (in its actions related to {@code state.json} as
+ * opposed to the per replica states).
+ *
+ * <p>Note that in a similar way to what happens in {@link ZkStateWriter#writePendingUpdates},
+ * collection delete is handled as a special case. (see comment on {@link
+ * DistributedClusterStateUpdater.StateChangeRecorder.RecordedMutationsPlayer} on why the code
+ * has to be duplicated)
*
- * <b>Note for the future:</b> Given this method is where the actually write to ZK is done, that's the place where we
- * can rebuild a DocCollection with updated zk version. Eventually if we maintain a cache of recently used collections,
- * we want to capture the updated collection and put it in the cache to avoid reading it again (unless it changed,
- * the CAS will fail and we will refresh).<p>
+ * <p><b>Note for the future:</b> Given this method is where the actually write to ZK is done,
+ * that's the place where we can rebuild a DocCollection with updated zk version. Eventually if
+ * we maintain a cache of recently used collections, we want to capture the updated collection
+ * and put it in the cache to avoid reading it again (unless it changed, the CAS will fail and
+ * we will refresh).
*
- * This could serve as the basis for a strategy where each node does not need any view of all collections in the cluster
- * but only a cache of recently used collections (possibly not even needing watches on them, but we'll discuss this later).
+ * <p>This could serve as the basis for a strategy where each node does not need any view of all
+ * collections in the cluster but only a cache of recently used collections (possibly not even
+ * needing watches on them, but we'll discuss this later).
*/
- private void doStateDotJsonCasUpdate(ClusterState updatedState) throws KeeperException, InterruptedException {
+ private void doStateDotJsonCasUpdate(ClusterState updatedState)
+ throws KeeperException, InterruptedException {
String jsonPath = ZkStateReader.getCollectionPath(updater.getCollectionName());
// Collection delete
if (!updatedState.hasCollection(updater.getCollectionName())) {
- // We do not have a collection znode version to test we delete the right version of state.json. But this doesn't really matter:
- // if we had one, and the delete failed (because state.json got updated in the meantime), we would re-read the collection
- // state, update our version, run the CAS delete again and it will pass. Which means that one way or another, deletes are final.
- // I hope nobody deletes a collection then creates a new one with the same name immediately (although the creation should fail
- // if the znode still exists, so the creation would only succeed after the delete made it, and we're ok).
- // With Overseer based updates the same behavior can be observed: a collection update is enqueued followed by the
+ // We do not have a collection znode version to test we delete the right version of
+ // state.json. But this doesn't really matter:
+ // if we had one, and the delete failed (because state.json got updated in the meantime), we
+ // would re-read the collection
+ // state, update our version, run the CAS delete again and it will pass. Which means that
+ // one way or another, deletes are final.
+ // I hope nobody deletes a collection then creates a new one with the same name immediately
+ // (although the creation should fail
+ // if the znode still exists, so the creation would only succeed after the delete made it,
+ // and we're ok).
+ // With Overseer based updates the same behavior can be observed: a collection update is
+ // enqueued followed by the
// collection delete before the update was executed.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/ZkDistributedLockFactory.java
##########
@@ -35,11 +35,15 @@
protected DistributedLock doCreateLock(boolean isWriteLock, String lockPath) {
try {
- // TODO optimize by first attempting to create the ZkDistributedLock without calling makeLockPath() and only call it
- // if the lock creation fails. This will be less costly on high contention (and slightly more on low contention)
+ // TODO optimize by first attempting to create the ZkDistributedLock without calling
+ // makeLockPath() and only call it
+ // if the lock creation fails. This will be less costly on high contention (and slightly more
+ // on low contention)
Review comment:
Fix this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] risdenk commented on pull request #705: SOLR-14920: Spotless formatting for core - non-test only
Posted by GitBox <gi...@apache.org>.
risdenk commented on pull request #705:
URL: https://github.com/apache/solr/pull/705#issuecomment-1055586456
Halfway through fixing the things found in the review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] risdenk commented on pull request #705: SOLR-14920: Spotless formatting for core - non-test only
Posted by GitBox <gi...@apache.org>.
risdenk commented on pull request #705:
URL: https://github.com/apache/solr/pull/705#issuecomment-1056952459
@dsmiley any other comments/thoughts/concerns?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] risdenk commented on pull request #705: SOLR-14920: Spotless formatting for core - non-test only
Posted by GitBox <gi...@apache.org>.
risdenk commented on pull request #705:
URL: https://github.com/apache/solr/pull/705#issuecomment-1059377874
Thanks @ErickErickson! Addressed your comments in a60dd59f7ac4f58cdf73fbdbe225426806a6e3b8
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] risdenk commented on a change in pull request #705: SOLR-14920: Spotless formatting for core - non-test only
Posted by GitBox <gi...@apache.org>.
risdenk commented on a change in pull request #705:
URL: https://github.com/apache/solr/pull/705#discussion_r816289962
##########
File path: solr/core/src/java/org/apache/solr/cloud/api/collections/DeleteReplicaCmd.java
##########
@@ -150,107 +157,141 @@ void deleteReplicaBasedOnCount(ClusterState clusterState,
Slice shardSlice = entry.getKey();
String shardId = shardSlice.getName();
Set<String> replicaNames = entry.getValue();
- Set<Replica> replicas = replicaNames.stream()
- .map(name -> shardSlice.getReplica(name)).collect(Collectors.toSet());
+ Set<Replica> replicas =
+ replicaNames.stream()
+ .map(name -> shardSlice.getReplica(name))
+ .collect(Collectors.toSet());
assignStrategy.verifyDeleteReplicas(ccc.getSolrCloudManager(), coll, shardId, replicas);
}
for (Map.Entry<Slice, Set<String>> entry : shardToReplicasMapping.entrySet()) {
Slice shardSlice = entry.getKey();
String shardId = shardSlice.getName();
Set<String> replicas = entry.getValue();
- //callDeleteReplica on all replicas
- for (String replica: replicas) {
+ // callDeleteReplica on all replicas
+ for (String replica : replicas) {
log.debug("Deleting replica {} for shard {} based on count {}", replica, shardId, count);
// don't verify with the placement plugin - we already did it
deleteCore(coll, shardId, replica, message, results, onComplete, parallel, false);
}
results.add("shard_id", shardId);
results.add("replicas_deleted", replicas);
}
-
}
-
- /**
- * Pick replicas to be deleted. Avoid picking the leader.
- */
- private Set<String> pickReplicasTobeDeleted(Slice slice, String shard, String collectionName, int count) {
+ /** Pick replicas to be deleted. Avoid picking the leader. */
+ private Set<String> pickReplicasTobeDeleted(
+ Slice slice, String shard, String collectionName, int count) {
validateReplicaAvailability(slice, shard, collectionName, count);
Collection<Replica> allReplicas = slice.getReplicas();
Set<String> replicasToBeRemoved = new HashSet<String>();
Replica leader = slice.getLeader();
- for (Replica replica: allReplicas) {
+ for (Replica replica : allReplicas) {
if (count == 0) {
break;
}
- //Try avoiding to pick up the leader to minimize activity on the cluster.
+ // Try avoiding to pick up the leader to minimize activity on the cluster.
if (leader.getCoreName().equals(replica.getCoreName())) {
continue;
}
replicasToBeRemoved.add(replica.getName());
- count --;
+ count--;
}
return replicasToBeRemoved;
}
/**
- * Validate if there is less replicas than requested to remove. Also error out if there is
- * only one replica available
+ * Validate if there is less replicas than requested to remove. Also error out if there is only
+ * one replica available
*/
- private void validateReplicaAvailability(Slice slice, String shard, String collectionName, int count) {
- //If there is a specific shard passed, validate if there any or just 1 replica left
+ private void validateReplicaAvailability(
+ Slice slice, String shard, String collectionName, int count) {
+ // If there is a specific shard passed, validate if there any or just 1 replica left
if (slice != null) {
Collection<Replica> allReplicasForShard = slice.getReplicas();
if (allReplicasForShard == null) {
- throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "No replicas found in shard/collection: " +
- shard + "/" + collectionName);
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "No replicas found in shard/collection: " + shard + "/" + collectionName);
}
-
if (allReplicasForShard.size() == 1) {
- throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "There is only one replica available in shard/collection: " +
- shard + "/" + collectionName + ". Cannot delete that.");
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "There is only one replica available in shard/collection: "
+ + shard
+ + "/"
+ + collectionName
+ + ". Cannot delete that.");
}
if (allReplicasForShard.size() <= count) {
- throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "There are lesser num replicas requested to be deleted than are available in shard/collection : " +
- shard + "/" + collectionName + " Requested: " + count + " Available: " + allReplicasForShard.size() + ".");
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "There are lesser num replicas requested to be deleted than are available in shard/collection : "
+ + shard
+ + "/"
+ + collectionName
+ + " Requested: "
+ + count
+ + " Available: "
+ + allReplicasForShard.size()
+ + ".");
}
}
}
- void deleteCore(DocCollection coll,
- String shardId,
- String replicaName,
- ZkNodeProps message,
- NamedList<Object> results,
- Runnable onComplete,
- boolean parallel,
- boolean verifyPlacement) throws KeeperException, IOException, InterruptedException {
+ void deleteCore(
+ DocCollection coll,
+ String shardId,
+ String replicaName,
+ ZkNodeProps message,
+ NamedList<Object> results,
+ Runnable onComplete,
+ boolean parallel,
+ boolean verifyPlacement)
+ throws KeeperException, IOException, InterruptedException {
Slice slice = coll.getSlice(shardId);
Replica replica = slice.getReplica(replicaName);
if (replica == null) {
ArrayList<String> l = new ArrayList<>();
- for (Replica r : slice.getReplicas())
- l.add(r.getName());
- throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "Invalid replica : " + replicaName + " in shard/collection : " +
- shardId + "/" + coll.getName() + " available replicas are " + StrUtils.join(l, ','));
+ for (Replica r : slice.getReplicas()) l.add(r.getName());
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Invalid replica : "
+ + replicaName
+ + " in shard/collection : "
+ + shardId
+ + "/"
+ + coll.getName()
+ + " available replicas are "
+ + StrUtils.join(l, ','));
}
- // If users are being safe and only want to remove a shard if it is down, they can specify onlyIfDown=true
+ // If users are being safe and only want to remove a shard if it is down, they can specify
+ // onlyIfDown=true
// on the command.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/api/collections/DistributedCollectionConfigSetCommandRunner.java
##########
@@ -110,132 +109,176 @@ public DistributedCollectionConfigSetCommandRunner(CoreContainer coreContainer)
this.coreContainer = coreContainer;
if (log.isInfoEnabled()) {
- // Note is it hard to print a log when Collection API is handled by Overseer because Overseer is started regardless
+ // Note is it hard to print a log when Collection API is handled by Overseer because Overseer
+ // is started regardless
// of how Collection API is handled, so it doesn't really know...
- log.info("Creating DistributedCollectionConfigSetCommandRunner. Collection and ConfigSet APIs are running distributed (not Overseer based)");
+ log.info(
+ "Creating DistributedCollectionConfigSetCommandRunner. Collection and ConfigSet APIs are running distributed (not Overseer based)");
}
// TODO we should look at how everything is getting closed when the node is shutdown.
- // But it seems that CollectionsHandler (that creates instances of this class) is not really closed, so maybe it doesn't matter?
- // With distributed Collection API execution, each node will have such an executor but given how thread pools work,
+ // But it seems that CollectionsHandler (that creates instances of this class) is not really
+ // closed, so maybe it doesn't matter?
+ // With distributed Collection API execution, each node will have such an executor but given how
+ // thread pools work,
// threads will only be created if needed (including the corePoolSize threads).
- distributedCollectionApiExecutorService = new ExecutorUtil.MDCAwareThreadPoolExecutor(5, 10, 0L, TimeUnit.MILLISECONDS,
- new SynchronousQueue<>(),
- new SolrNamedThreadFactory("DistributedCollectionApiExecutorService"));
-
- commandsExecutor = new ExecutorUtil.MDCAwareThreadPoolExecutor(5, 20, 0L, TimeUnit.MILLISECONDS,
- new SynchronousQueue<>(),
- new SolrNamedThreadFactory("DistributedCollectionApiCommandExecutor"));
-
- ccc = new DistributedCollectionCommandContext(this.coreContainer, this.distributedCollectionApiExecutorService);
+ distributedCollectionApiExecutorService =
+ new ExecutorUtil.MDCAwareThreadPoolExecutor(
+ 5,
+ 10,
+ 0L,
+ TimeUnit.MILLISECONDS,
+ new SynchronousQueue<>(),
+ new SolrNamedThreadFactory("DistributedCollectionApiExecutorService"));
+
+ commandsExecutor =
+ new ExecutorUtil.MDCAwareThreadPoolExecutor(
+ 5,
+ 20,
+ 0L,
+ TimeUnit.MILLISECONDS,
+ new SynchronousQueue<>(),
+ new SolrNamedThreadFactory("DistributedCollectionApiCommandExecutor"));
+
+ ccc =
+ new DistributedCollectionCommandContext(
+ this.coreContainer, this.distributedCollectionApiExecutorService);
commandMapper = new CollApiCmds.CommandMap(ccc);
- asyncTaskTracker = new DistributedApiAsyncTracker(ccc.getZkStateReader().getZkClient(), ZK_ASYNC_ROOT);
+ asyncTaskTracker =
+ new DistributedApiAsyncTracker(ccc.getZkStateReader().getZkClient(), ZK_ASYNC_ROOT);
}
- /**
- * See {@link DistributedApiAsyncTracker#getAsyncTaskRequestStatus(String)}
- */
- public Pair<RequestStatusState, OverseerSolrResponse> getAsyncTaskRequestStatus(String asyncId) throws Exception {
+ /** See {@link DistributedApiAsyncTracker#getAsyncTaskRequestStatus(String)} */
+ public Pair<RequestStatusState, OverseerSolrResponse> getAsyncTaskRequestStatus(String asyncId)
+ throws Exception {
return asyncTaskTracker.getAsyncTaskRequestStatus(asyncId);
}
- /**
- * See {@link DistributedApiAsyncTracker#deleteSingleAsyncId(String)}
- */
+ /** See {@link DistributedApiAsyncTracker#deleteSingleAsyncId(String)} */
public boolean deleteSingleAsyncId(String asyncId) throws Exception {
return asyncTaskTracker.deleteSingleAsyncId(asyncId);
}
- /**
- * See {@link DistributedApiAsyncTracker#deleteAllAsyncIds()}
- */
+ /** See {@link DistributedApiAsyncTracker#deleteAllAsyncIds()} */
public void deleteAllAsyncIds() throws Exception {
asyncTaskTracker.deleteAllAsyncIds();
}
-
/**
- * When {@link org.apache.solr.handler.admin.CollectionsHandler#invokeAction} does not enqueue to overseer queue and
- * instead calls this method, this method is expected to do the equivalent of what Overseer does in
- * {@link org.apache.solr.cloud.OverseerConfigSetMessageHandler#processMessage}.
- * <p>
- * The steps leading to that call in the Overseer execution path are (and the equivalent is done here):
+ * When {@link org.apache.solr.handler.admin.CollectionsHandler#invokeAction} does not enqueue to
+ * overseer queue and instead calls this method, this method is expected to do the equivalent of
+ * what Overseer does in {@link
+ * org.apache.solr.cloud.OverseerConfigSetMessageHandler#processMessage}.
+ *
+ * <p>The steps leading to that call in the Overseer execution path are (and the equivalent is
+ * done here):
+ *
* <ul>
- * <li>{@link org.apache.solr.cloud.OverseerTaskProcessor#run()} gets the message from the ZK queue, grabs the
- * corresponding locks (write lock on the config set target of the API command and a read lock on the base config set
- * if any - the case for config set creation) then executes the command using an executor service (it also checks the
- * asyncId if any is specified but async calls are not supported for Config Set API calls).</li>
- * <li>In {@link org.apache.solr.cloud.OverseerTaskProcessor}.{@code Runner.run()} (run on an executor thread) a call is made to
- * {@link org.apache.solr.cloud.OverseerConfigSetMessageHandler#processMessage} which does a few checks and calls the
- * appropriate Config Set method.
+ * <li>{@link org.apache.solr.cloud.OverseerTaskProcessor#run()} gets the message from the ZK
+ * queue, grabs the corresponding locks (write lock on the config set target of the API
+ * command and a read lock on the base config set if any - the case for config set creation)
+ * then executes the command using an executor service (it also checks the asyncId if any is
+ * specified but async calls are not supported for Config Set API calls).
+ * <li>In {@link org.apache.solr.cloud.OverseerTaskProcessor}.{@code Runner.run()} (run on an
+ * executor thread) a call is made to {@link
+ * org.apache.solr.cloud.OverseerConfigSetMessageHandler#processMessage} which does a few
+ * checks and calls the appropriate Config Set method.
* </ul>
*/
- public void runConfigSetCommand(SolrQueryResponse rsp, ConfigSetsHandler.ConfigSetOperation operation, Map<String, Object> result, long timeoutMs) throws Exception {
- // We refuse new tasks, but will wait for already submitted ones (i.e. those that made it through this method earlier).
+ public void runConfigSetCommand(
+ SolrQueryResponse rsp,
+ ConfigSetsHandler.ConfigSetOperation operation,
+ Map<String, Object> result,
+ long timeoutMs)
+ throws Exception {
+ // We refuse new tasks, but will wait for already submitted ones (i.e. those that made it
+ // through this method earlier).
// See stopAndWaitForPendingTasksToComplete() below
if (shuttingDown) {
- throw new SolrException(SolrException.ErrorCode.CONFLICT, "Solr is shutting down, no more Config Set API tasks may be executed");
+ throw new SolrException(
+ SolrException.ErrorCode.CONFLICT,
+ "Solr is shutting down, no more Config Set API tasks may be executed");
}
ConfigSetParams.ConfigSetAction action = operation.getAction();
// never null
String configSetName = (String) result.get(NAME);
// baseConfigSetName will be null if we're not creating a new config set
- String baseConfigSetName = ConfigSetCmds.getBaseConfigSetName(action, (String) result.get(ConfigSetCmds.BASE_CONFIGSET));
+ String baseConfigSetName =
+ ConfigSetCmds.getBaseConfigSetName(
+ action, (String) result.get(ConfigSetCmds.BASE_CONFIGSET));
if (log.isInfoEnabled()) {
log.info("Running Config Set API locally for " + action + " " + configSetName); // nowarn
}
- ConfigSetCommandRunner commandRunner = new ConfigSetCommandRunner(new ZkNodeProps(result), action, configSetName, baseConfigSetName);
+ ConfigSetCommandRunner commandRunner =
+ new ConfigSetCommandRunner(
+ new ZkNodeProps(result), action, configSetName, baseConfigSetName);
final Future<Void> taskFuture;
try {
taskFuture = commandsExecutor.submit(commandRunner);
} catch (RejectedExecutionException ree) {
- throw new SolrException(SolrException.ErrorCode.SERVICE_UNAVAILABLE, "Too many executing commands", ree);
+ throw new SolrException(
+ SolrException.ErrorCode.SERVICE_UNAVAILABLE, "Too many executing commands", ree);
}
- // Wait for a while... Just like Overseer based Config Set API (wait can timeout but actual command execution does not)
+ // Wait for a while... Just like Overseer based Config Set API (wait can timeout but actual
+ // command execution does not)
try {
taskFuture.get(timeoutMs, TimeUnit.MILLISECONDS);
} catch (TimeoutException te) {
- throw new SolrException(SolrException.ErrorCode.SERVER_ERROR, action + " " + configSetName + " timed out after " + timeoutMs + "ms");
+ throw new SolrException(
+ SolrException.ErrorCode.SERVER_ERROR,
+ action + " " + configSetName + " timed out after " + timeoutMs + "ms");
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
- throw new SolrException(SolrException.ErrorCode.SERVER_ERROR, action + " " + configSetName + " interrupted", e);
+ throw new SolrException(
+ SolrException.ErrorCode.SERVER_ERROR, action + " " + configSetName + " interrupted", e);
}
}
/**
- * When {@link org.apache.solr.handler.admin.CollectionsHandler#invokeAction} does not enqueue to overseer queue and
- * instead calls this method, this method is expected to do the equivalent of what Overseer does in
- * {@link org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler#processMessage}.
- * <p>
- * The steps leading to that call in the Overseer execution path are (and the equivalent is done here):
+ * When {@link org.apache.solr.handler.admin.CollectionsHandler#invokeAction} does not enqueue to
+ * overseer queue and instead calls this method, this method is expected to do the equivalent of
+ * what Overseer does in {@link
+ * org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler#processMessage}.
+ *
+ * <p>The steps leading to that call in the Overseer execution path are (and the equivalent is
+ * done here):
+ *
* <ul>
- * <li>{@link org.apache.solr.cloud.OverseerTaskProcessor#run()} gets the message from the ZK queue, grabs the
- * corresponding lock (Collection API calls do locking to prevent non compatible concurrent modifications of a collection),
- * marks the async id of the task as running then executes the command using an executor service</li>
- * <li>In {@link org.apache.solr.cloud.OverseerTaskProcessor}.{@code Runner.run()} (run on an executor thread) a call is made to
- * {@link org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler#processMessage} which sets the logging
- * context, calls {@link CollApiCmds.CollectionApiCommand#call}
+ * <li>{@link org.apache.solr.cloud.OverseerTaskProcessor#run()} gets the message from the ZK
+ * queue, grabs the corresponding lock (Collection API calls do locking to prevent non
+ * compatible concurrent modifications of a collection), marks the async id of the task as
+ * running then executes the command using an executor service
+ * <li>In {@link org.apache.solr.cloud.OverseerTaskProcessor}.{@code Runner.run()} (run on an
+ * executor thread) a call is made to {@link
+ * org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler#processMessage}
+ * which sets the logging context, calls {@link CollApiCmds.CollectionApiCommand#call}
* </ul>
*/
- public OverseerSolrResponse runCollectionCommand(ZkNodeProps message, CollectionParams.CollectionAction action, long timeoutMs) {
- // We refuse new tasks, but will wait for already submitted ones (i.e. those that made it through this method earlier).
+ public OverseerSolrResponse runCollectionCommand(
+ ZkNodeProps message, CollectionParams.CollectionAction action, long timeoutMs) {
+ // We refuse new tasks, but will wait for already submitted ones (i.e. those that made it
+ // through this method earlier).
// See stopAndWaitForPendingTasksToComplete() below
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/api/collections/DistributedCollectionConfigSetCommandRunner.java
##########
@@ -376,12 +451,15 @@ public OverseerSolrResponse call() {
}
} finally {
try {
- // TODO If the Collection API command failed because the collection does not exist, we've just created some lock directory
- // structure for a non existent collection... Maybe try to remove it here? No big deal for now as leftover nodes in the
+ // TODO If the Collection API command failed because the collection does not exist,
+ // we've just created some lock directory
+ // structure for a non existent collection... Maybe try to remove it here? No big deal
+ // for now as leftover nodes in the
// lock hierarchy do no harm, and there shouldn't be too many of those.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/api/collections/RoutedAlias.java
##########
@@ -340,56 +389,79 @@ private String createAllRequiredCollections(AddUpdateCommand cmd, CandidateColle
} while (true);
}
- private CandidateCollection doSynchronous(AddUpdateCommand cmd, CandidateCollection targetCollectionDesc, CoreContainer coreContainer) {
- ensureCollection(targetCollectionDesc.getCreationCollection(), coreContainer); // *should* throw if fails for some reason but...
+ private CandidateCollection doSynchronous(
+ AddUpdateCommand cmd, CandidateCollection targetCollectionDesc, CoreContainer coreContainer) {
+ ensureCollection(
+ targetCollectionDesc.getCreationCollection(),
+ coreContainer); // *should* throw if fails for some reason but...
ZkController zkController = coreContainer.getZkController();
updateParsedCollectionAliases(zkController.zkStateReader, true);
- List<String> observedCols = zkController.zkStateReader.aliasesManager.getAliases().getCollectionAliasListMap().get(getAliasName());
+ List<String> observedCols =
+ zkController
+ .zkStateReader
+ .aliasesManager
+ .getAliases()
+ .getCollectionAliasListMap()
+ .get(getAliasName());
if (!observedCols.contains(targetCollectionDesc.creationCollection)) {
// if collection creation did not occur we've failed. Bail out.
- throw new SolrException(SERVER_ERROR, "After we attempted to create " + targetCollectionDesc.creationCollection + " it did not exist");
+ throw new SolrException(
+ SERVER_ERROR,
+ "After we attempted to create "
+ + targetCollectionDesc.creationCollection
+ + " it did not exist");
}
- // then recalculate the candiate, which may result in continuation or termination the loop calling this method
+ // then recalculate the candiate, which may result in continuation or termination the loop
+ // calling this method
targetCollectionDesc = findCandidateGivenValue(cmd);
return targetCollectionDesc;
}
- private String doPreemptive(CandidateCollection targetCollectionDesc, SolrCore core, CoreContainer coreContainer) {
+ private String doPreemptive(
+ CandidateCollection targetCollectionDesc, SolrCore core, CoreContainer coreContainer) {
if (!this.preemptiveCreateOnceAlready) {
- preemptiveAsync(() -> {
- try {
- ensureCollection(targetCollectionDesc.creationCollection, coreContainer);
- } catch (Exception e) {
- log.error("Async creation of a collection for routed Alias {} failed!", this.getAliasName(), e);
- }
- }, core);
+ preemptiveAsync(
+ () -> {
+ try {
+ ensureCollection(targetCollectionDesc.creationCollection, coreContainer);
+ } catch (Exception e) {
+ log.error(
+ "Async creation of a collection for routed Alias {} failed!",
+ this.getAliasName(),
+ e);
+ }
+ },
+ core);
}
return targetCollectionDesc.destinationCollection;
}
/**
* Calculate the head collection (i.e. the most recent one for a TRA) if this routed alias has an
- * implicit order, or if the collection is unordered return the appropriate collection name
- * for the value in the current document. This method should never return null.
+ * implicit order, or if the collection is unordered return the appropriate collection name for
+ * the value in the current document. This method should never return null.
*/
- abstract protected String getHeadCollectionIfOrdered(AddUpdateCommand cmd);
+ protected abstract String getHeadCollectionIfOrdered(AddUpdateCommand cmd);
private void preemptiveAsync(Runnable r, SolrCore core) {
preemptiveCreateOnceAlready = true;
core.runAsync(r);
}
private SolrException unknownCreateType() {
- return new SolrException(SolrException.ErrorCode.SERVER_ERROR, "Unknown creation type while adding " +
- "document to a Time Routed Alias! This is a bug caused when a creation type has been added but " +
- "not all code has been updated to handle it.");
+ return new SolrException(
+ SolrException.ErrorCode.SERVER_ERROR,
+ "Unknown creation type while adding "
+ + "document to a Time Routed Alias! This is a bug caused when a creation type has been added but "
+ + "not all code has been updated to handle it.");
}
void ensureCollection(String targetCollection, CoreContainer coreContainer) {
CollectionsHandler collectionsHandler = coreContainer.getCollectionsHandler();
- // Invoke MANINTAIN_ROUTED_ALIAS (in the Overseer, locked by alias name). It will create the collection
+ // Invoke MANINTAIN_ROUTED_ALIAS (in the Overseer, locked by alias name). It will create the
+ // collection
// and update the alias contingent on the requested collection name not already existing.
// otherwise it will return (without error).
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/api/collections/SplitShardCmd.java
##########
@@ -548,8 +676,10 @@ public boolean split(ClusterState clusterState, ZkNodeProps message, NamedList<O
if (repFactor == 1) {
// A commit is needed so that documents are visible when the sub-shard replicas come up
- // (Note: This commit used to be after the state switch, but was brought here before the state switch
- // as per SOLR-13945 so that sub shards don't come up empty, momentarily, after being marked active)
+ // (Note: This commit used to be after the state switch, but was brought here before the
+ // state switch
+ // as per SOLR-13945 so that sub shards don't come up empty, momentarily, after being
+ // marked active)
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/api/collections/DistributedCollectionConfigSetCommandRunner.java
##########
@@ -110,132 +109,176 @@ public DistributedCollectionConfigSetCommandRunner(CoreContainer coreContainer)
this.coreContainer = coreContainer;
if (log.isInfoEnabled()) {
- // Note is it hard to print a log when Collection API is handled by Overseer because Overseer is started regardless
+ // Note is it hard to print a log when Collection API is handled by Overseer because Overseer
+ // is started regardless
// of how Collection API is handled, so it doesn't really know...
- log.info("Creating DistributedCollectionConfigSetCommandRunner. Collection and ConfigSet APIs are running distributed (not Overseer based)");
+ log.info(
+ "Creating DistributedCollectionConfigSetCommandRunner. Collection and ConfigSet APIs are running distributed (not Overseer based)");
}
// TODO we should look at how everything is getting closed when the node is shutdown.
- // But it seems that CollectionsHandler (that creates instances of this class) is not really closed, so maybe it doesn't matter?
- // With distributed Collection API execution, each node will have such an executor but given how thread pools work,
+ // But it seems that CollectionsHandler (that creates instances of this class) is not really
+ // closed, so maybe it doesn't matter?
+ // With distributed Collection API execution, each node will have such an executor but given how
+ // thread pools work,
// threads will only be created if needed (including the corePoolSize threads).
- distributedCollectionApiExecutorService = new ExecutorUtil.MDCAwareThreadPoolExecutor(5, 10, 0L, TimeUnit.MILLISECONDS,
- new SynchronousQueue<>(),
- new SolrNamedThreadFactory("DistributedCollectionApiExecutorService"));
-
- commandsExecutor = new ExecutorUtil.MDCAwareThreadPoolExecutor(5, 20, 0L, TimeUnit.MILLISECONDS,
- new SynchronousQueue<>(),
- new SolrNamedThreadFactory("DistributedCollectionApiCommandExecutor"));
-
- ccc = new DistributedCollectionCommandContext(this.coreContainer, this.distributedCollectionApiExecutorService);
+ distributedCollectionApiExecutorService =
+ new ExecutorUtil.MDCAwareThreadPoolExecutor(
+ 5,
+ 10,
+ 0L,
+ TimeUnit.MILLISECONDS,
+ new SynchronousQueue<>(),
+ new SolrNamedThreadFactory("DistributedCollectionApiExecutorService"));
+
+ commandsExecutor =
+ new ExecutorUtil.MDCAwareThreadPoolExecutor(
+ 5,
+ 20,
+ 0L,
+ TimeUnit.MILLISECONDS,
+ new SynchronousQueue<>(),
+ new SolrNamedThreadFactory("DistributedCollectionApiCommandExecutor"));
+
+ ccc =
+ new DistributedCollectionCommandContext(
+ this.coreContainer, this.distributedCollectionApiExecutorService);
commandMapper = new CollApiCmds.CommandMap(ccc);
- asyncTaskTracker = new DistributedApiAsyncTracker(ccc.getZkStateReader().getZkClient(), ZK_ASYNC_ROOT);
+ asyncTaskTracker =
+ new DistributedApiAsyncTracker(ccc.getZkStateReader().getZkClient(), ZK_ASYNC_ROOT);
}
- /**
- * See {@link DistributedApiAsyncTracker#getAsyncTaskRequestStatus(String)}
- */
- public Pair<RequestStatusState, OverseerSolrResponse> getAsyncTaskRequestStatus(String asyncId) throws Exception {
+ /** See {@link DistributedApiAsyncTracker#getAsyncTaskRequestStatus(String)} */
+ public Pair<RequestStatusState, OverseerSolrResponse> getAsyncTaskRequestStatus(String asyncId)
+ throws Exception {
return asyncTaskTracker.getAsyncTaskRequestStatus(asyncId);
}
- /**
- * See {@link DistributedApiAsyncTracker#deleteSingleAsyncId(String)}
- */
+ /** See {@link DistributedApiAsyncTracker#deleteSingleAsyncId(String)} */
public boolean deleteSingleAsyncId(String asyncId) throws Exception {
return asyncTaskTracker.deleteSingleAsyncId(asyncId);
}
- /**
- * See {@link DistributedApiAsyncTracker#deleteAllAsyncIds()}
- */
+ /** See {@link DistributedApiAsyncTracker#deleteAllAsyncIds()} */
public void deleteAllAsyncIds() throws Exception {
asyncTaskTracker.deleteAllAsyncIds();
}
-
/**
- * When {@link org.apache.solr.handler.admin.CollectionsHandler#invokeAction} does not enqueue to overseer queue and
- * instead calls this method, this method is expected to do the equivalent of what Overseer does in
- * {@link org.apache.solr.cloud.OverseerConfigSetMessageHandler#processMessage}.
- * <p>
- * The steps leading to that call in the Overseer execution path are (and the equivalent is done here):
+ * When {@link org.apache.solr.handler.admin.CollectionsHandler#invokeAction} does not enqueue to
+ * overseer queue and instead calls this method, this method is expected to do the equivalent of
+ * what Overseer does in {@link
+ * org.apache.solr.cloud.OverseerConfigSetMessageHandler#processMessage}.
+ *
+ * <p>The steps leading to that call in the Overseer execution path are (and the equivalent is
+ * done here):
+ *
* <ul>
- * <li>{@link org.apache.solr.cloud.OverseerTaskProcessor#run()} gets the message from the ZK queue, grabs the
- * corresponding locks (write lock on the config set target of the API command and a read lock on the base config set
- * if any - the case for config set creation) then executes the command using an executor service (it also checks the
- * asyncId if any is specified but async calls are not supported for Config Set API calls).</li>
- * <li>In {@link org.apache.solr.cloud.OverseerTaskProcessor}.{@code Runner.run()} (run on an executor thread) a call is made to
- * {@link org.apache.solr.cloud.OverseerConfigSetMessageHandler#processMessage} which does a few checks and calls the
- * appropriate Config Set method.
+ * <li>{@link org.apache.solr.cloud.OverseerTaskProcessor#run()} gets the message from the ZK
+ * queue, grabs the corresponding locks (write lock on the config set target of the API
+ * command and a read lock on the base config set if any - the case for config set creation)
+ * then executes the command using an executor service (it also checks the asyncId if any is
+ * specified but async calls are not supported for Config Set API calls).
+ * <li>In {@link org.apache.solr.cloud.OverseerTaskProcessor}.{@code Runner.run()} (run on an
+ * executor thread) a call is made to {@link
+ * org.apache.solr.cloud.OverseerConfigSetMessageHandler#processMessage} which does a few
+ * checks and calls the appropriate Config Set method.
* </ul>
*/
- public void runConfigSetCommand(SolrQueryResponse rsp, ConfigSetsHandler.ConfigSetOperation operation, Map<String, Object> result, long timeoutMs) throws Exception {
- // We refuse new tasks, but will wait for already submitted ones (i.e. those that made it through this method earlier).
+ public void runConfigSetCommand(
+ SolrQueryResponse rsp,
+ ConfigSetsHandler.ConfigSetOperation operation,
+ Map<String, Object> result,
+ long timeoutMs)
+ throws Exception {
+ // We refuse new tasks, but will wait for already submitted ones (i.e. those that made it
+ // through this method earlier).
// See stopAndWaitForPendingTasksToComplete() below
if (shuttingDown) {
- throw new SolrException(SolrException.ErrorCode.CONFLICT, "Solr is shutting down, no more Config Set API tasks may be executed");
+ throw new SolrException(
+ SolrException.ErrorCode.CONFLICT,
+ "Solr is shutting down, no more Config Set API tasks may be executed");
}
ConfigSetParams.ConfigSetAction action = operation.getAction();
// never null
String configSetName = (String) result.get(NAME);
// baseConfigSetName will be null if we're not creating a new config set
- String baseConfigSetName = ConfigSetCmds.getBaseConfigSetName(action, (String) result.get(ConfigSetCmds.BASE_CONFIGSET));
+ String baseConfigSetName =
+ ConfigSetCmds.getBaseConfigSetName(
+ action, (String) result.get(ConfigSetCmds.BASE_CONFIGSET));
if (log.isInfoEnabled()) {
log.info("Running Config Set API locally for " + action + " " + configSetName); // nowarn
}
- ConfigSetCommandRunner commandRunner = new ConfigSetCommandRunner(new ZkNodeProps(result), action, configSetName, baseConfigSetName);
+ ConfigSetCommandRunner commandRunner =
+ new ConfigSetCommandRunner(
+ new ZkNodeProps(result), action, configSetName, baseConfigSetName);
final Future<Void> taskFuture;
try {
taskFuture = commandsExecutor.submit(commandRunner);
} catch (RejectedExecutionException ree) {
- throw new SolrException(SolrException.ErrorCode.SERVICE_UNAVAILABLE, "Too many executing commands", ree);
+ throw new SolrException(
+ SolrException.ErrorCode.SERVICE_UNAVAILABLE, "Too many executing commands", ree);
}
- // Wait for a while... Just like Overseer based Config Set API (wait can timeout but actual command execution does not)
+ // Wait for a while... Just like Overseer based Config Set API (wait can timeout but actual
+ // command execution does not)
try {
taskFuture.get(timeoutMs, TimeUnit.MILLISECONDS);
} catch (TimeoutException te) {
- throw new SolrException(SolrException.ErrorCode.SERVER_ERROR, action + " " + configSetName + " timed out after " + timeoutMs + "ms");
+ throw new SolrException(
+ SolrException.ErrorCode.SERVER_ERROR,
+ action + " " + configSetName + " timed out after " + timeoutMs + "ms");
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
- throw new SolrException(SolrException.ErrorCode.SERVER_ERROR, action + " " + configSetName + " interrupted", e);
+ throw new SolrException(
+ SolrException.ErrorCode.SERVER_ERROR, action + " " + configSetName + " interrupted", e);
}
}
/**
- * When {@link org.apache.solr.handler.admin.CollectionsHandler#invokeAction} does not enqueue to overseer queue and
- * instead calls this method, this method is expected to do the equivalent of what Overseer does in
- * {@link org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler#processMessage}.
- * <p>
- * The steps leading to that call in the Overseer execution path are (and the equivalent is done here):
+ * When {@link org.apache.solr.handler.admin.CollectionsHandler#invokeAction} does not enqueue to
+ * overseer queue and instead calls this method, this method is expected to do the equivalent of
+ * what Overseer does in {@link
+ * org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler#processMessage}.
+ *
+ * <p>The steps leading to that call in the Overseer execution path are (and the equivalent is
+ * done here):
+ *
* <ul>
- * <li>{@link org.apache.solr.cloud.OverseerTaskProcessor#run()} gets the message from the ZK queue, grabs the
- * corresponding lock (Collection API calls do locking to prevent non compatible concurrent modifications of a collection),
- * marks the async id of the task as running then executes the command using an executor service</li>
- * <li>In {@link org.apache.solr.cloud.OverseerTaskProcessor}.{@code Runner.run()} (run on an executor thread) a call is made to
- * {@link org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler#processMessage} which sets the logging
- * context, calls {@link CollApiCmds.CollectionApiCommand#call}
+ * <li>{@link org.apache.solr.cloud.OverseerTaskProcessor#run()} gets the message from the ZK
+ * queue, grabs the corresponding lock (Collection API calls do locking to prevent non
+ * compatible concurrent modifications of a collection), marks the async id of the task as
+ * running then executes the command using an executor service
+ * <li>In {@link org.apache.solr.cloud.OverseerTaskProcessor}.{@code Runner.run()} (run on an
+ * executor thread) a call is made to {@link
+ * org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler#processMessage}
+ * which sets the logging context, calls {@link CollApiCmds.CollectionApiCommand#call}
* </ul>
*/
- public OverseerSolrResponse runCollectionCommand(ZkNodeProps message, CollectionParams.CollectionAction action, long timeoutMs) {
- // We refuse new tasks, but will wait for already submitted ones (i.e. those that made it through this method earlier).
+ public OverseerSolrResponse runCollectionCommand(
+ ZkNodeProps message, CollectionParams.CollectionAction action, long timeoutMs) {
+ // We refuse new tasks, but will wait for already submitted ones (i.e. those that made it
+ // through this method earlier).
// See stopAndWaitForPendingTasksToComplete() below
if (shuttingDown) {
- throw new SolrException(SolrException.ErrorCode.CONFLICT, "Solr is shutting down, no more Collection API tasks may be executed");
+ throw new SolrException(
+ SolrException.ErrorCode.CONFLICT,
+ "Solr is shutting down, no more Collection API tasks may be executed");
}
final String asyncId = message.getStr(ASYNC);
if (log.isInfoEnabled()) {
- log.info("Running Collection API locally for " + action.name() + " asyncId=" + asyncId); // nowarn
+ log.info(
+ "Running Collection API locally for " + action.name() + " asyncId=" + asyncId); // nowarn
}
- // Following the call below returning true, we must eventually cancel or complete the task. Happens either in the
+ // Following the call below returning true, we must eventually cancel or complete the task.
+ // Happens either in the
// CollectionCommandRunner below or in the catch when the runner would not execute.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/overseer/ZkStateWriter.java
##########
@@ -238,26 +252,37 @@ public ClusterState writePendingUpdates(Map<String, ZkWriteCommand> updates) thr
log.debug("going to update_collection {} version: {}", path, c.getZNodeVersion());
}
Stat stat = reader.getZkClient().setData(path, data, c.getZNodeVersion(), true);
- DocCollection newCollection = new DocCollection(name, c.getSlicesMap(), c.getProperties(), c.getRouter(), stat.getVersion());
+ DocCollection newCollection =
+ new DocCollection(
+ name, c.getSlicesMap(), c.getProperties(), c.getRouter(), stat.getVersion());
clusterState = clusterState.copyWith(name, newCollection);
} else {
log.debug("going to create_collection {}", path);
reader.getZkClient().create(path, data, CreateMode.PERSISTENT, true);
- DocCollection newCollection = new DocCollection(name, c.getSlicesMap(), c.getProperties(), c.getRouter(), 0);
+ DocCollection newCollection =
+ new DocCollection(name, c.getSlicesMap(), c.getProperties(), c.getRouter(), 0);
clusterState = clusterState.copyWith(name, newCollection);
}
}
- // When dealing with a per replica collection that did not do any update to the per replica states znodes but did
- // update state.json, we add then remove a dummy node to change the cversion of the parent znode.
- // This is not needed by Solr, there's no code watching the children and not watching the state.json node itself.
- // It would be useful for external code watching the collection's Zookeeper state.json node children but not the node itself.
+ // When dealing with a per replica collection that did not do any update to the per
+ // replica states znodes but did
+ // update state.json, we add then remove a dummy node to change the cversion of the parent
+ // znode.
+ // This is not needed by Solr, there's no code watching the children and not watching the
+ // state.json node itself.
+ // It would be useful for external code watching the collection's Zookeeper state.json
+ // node children but not the node itself.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/api/collections/DimensionalRoutedAlias.java
##########
@@ -98,11 +105,15 @@ public boolean updateParsedCollectionAliases(ZkStateReader zkStateReader, boolea
public String computeInitialCollectionName() {
StringBuilder sb = new StringBuilder(getAliasName());
for (RoutedAlias dimension : dimensions) {
- // N. B. getAliasName is generally safe as a regex because it must conform to collection naming rules
- // and those rules exclude regex special characters. A malicious request might do something expensive, but
- // if you have malicious users able to run admin commands and create aliases, it is very likely that you have
+ // N. B. getAliasName is generally safe as a regex because it must conform to collection
+ // naming rules
+ // and those rules exclude regex special characters. A malicious request might do something
+ // expensive, but
+ // if you have malicious users able to run admin commands and create aliases, it is very
+ // likely that you have
// much bigger problems than an expensive regex.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/api/collections/DeleteSnapshotCmd.java
##########
@@ -149,22 +158,38 @@ public void call(ClusterState state, ZkNodeProps message, NamedList<Object> resu
}
}
- // Update the ZK meta-data to include only cores with the snapshot. This will enable users to figure out
+ // Update the ZK meta-data to include only cores with the snapshot. This will enable users to
+ // figure out
// which cores still contain the named snapshot.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/api/collections/DistributedCollectionConfigSetCommandRunner.java
##########
@@ -110,132 +109,176 @@ public DistributedCollectionConfigSetCommandRunner(CoreContainer coreContainer)
this.coreContainer = coreContainer;
if (log.isInfoEnabled()) {
- // Note is it hard to print a log when Collection API is handled by Overseer because Overseer is started regardless
+ // Note is it hard to print a log when Collection API is handled by Overseer because Overseer
+ // is started regardless
// of how Collection API is handled, so it doesn't really know...
- log.info("Creating DistributedCollectionConfigSetCommandRunner. Collection and ConfigSet APIs are running distributed (not Overseer based)");
+ log.info(
+ "Creating DistributedCollectionConfigSetCommandRunner. Collection and ConfigSet APIs are running distributed (not Overseer based)");
}
// TODO we should look at how everything is getting closed when the node is shutdown.
- // But it seems that CollectionsHandler (that creates instances of this class) is not really closed, so maybe it doesn't matter?
- // With distributed Collection API execution, each node will have such an executor but given how thread pools work,
+ // But it seems that CollectionsHandler (that creates instances of this class) is not really
+ // closed, so maybe it doesn't matter?
+ // With distributed Collection API execution, each node will have such an executor but given how
+ // thread pools work,
// threads will only be created if needed (including the corePoolSize threads).
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/api/collections/DistributedCollectionConfigSetCommandRunner.java
##########
@@ -342,31 +397,51 @@ public OverseerSolrResponse call() {
NamedList<Object> results = new NamedList<>();
try {
- // Create API lock for executing the command. This call is non blocking (not blocked on waiting for a lock to be acquired anyway,
+ // Create API lock for executing the command. This call is non blocking (not blocked on
+ // waiting for a lock to be acquired anyway,
// might be blocked on access to ZK etc)
- // We create a new CollectionApiLockFactory using a new ZkDistributedCollectionLockFactory because earlier (in the constructor of this class)
- // the ZkStateReader was not yet available, due to how CoreContainer is built in part in the constructor and in part in its
+ // We create a new CollectionApiLockFactory using a new ZkDistributedCollectionLockFactory
+ // because earlier (in the constructor of this class)
+ // the ZkStateReader was not yet available, due to how CoreContainer is built in part in the
+ // constructor and in part in its
// load() method. And this class is built from the CoreContainer constructor...
- // The cost of these creations is low, and these classes do not hold state but only serve as an interface to Zookeeper.
+ // The cost of these creations is low, and these classes do not hold state but only serve as
+ // an interface to Zookeeper.
// Note that after this call, we MUST execute the lock.release(); in the finally below
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/api/collections/DistributedCollectionConfigSetCommandRunner.java
##########
@@ -249,35 +292,42 @@ public OverseerSolrResponse runCollectionCommand(ZkNodeProps message, Collection
try {
taskFuture = commandsExecutor.submit(commandRunner);
} catch (RejectedExecutionException ree) {
- // The command will not run, need to cancel the async ID so it can be reused on a subsequent attempt by the client
+ // The command will not run, need to cancel the async ID so it can be reused on a subsequent
+ // attempt by the client
asyncTaskTracker.cancelAsyncId(asyncId);
- throw new SolrException(SolrException.ErrorCode.SERVICE_UNAVAILABLE, "Too many executing commands", ree);
+ throw new SolrException(
+ SolrException.ErrorCode.SERVICE_UNAVAILABLE, "Too many executing commands", ree);
}
if (asyncId == null) {
- // Non async calls wait for a while in case the command completes. If they time out, there's no way to track the
- // job progress (improvement suggestion: decorrelate having a task ID from the fact of waiting for the job to complete)
+ // Non async calls wait for a while in case the command completes. If they time out, there's
+ // no way to track the
+ // job progress (improvement suggestion: decorrelate having a task ID from the fact of waiting
+ // for the job to complete)
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/api/collections/DeleteShardCmd.java
##########
@@ -95,15 +105,25 @@ public void call(ClusterState clusterState, ZkNodeProps message, NamedList<Objec
propMap.put(ZkStateReader.COLLECTION_PROP, collectionName);
ZkNodeProps m = new ZkNodeProps(propMap);
if (ccc.getDistributedClusterStateUpdater().isDistributedStateUpdate()) {
- // In this DeleteShardCmd.call() method there are potentially two cluster state updates. This is the first one.
- // Even though the code of this method does not wait for it to complete, it does call the Collection API before
- // it issues the second state change below. The collection API will be doing its own state change(s), and these will
- // happen after this one (given it's for the same collection). Therefore we persist this state change
+ // In this DeleteShardCmd.call() method there are potentially two cluster state updates.
+ // This is the first one.
+ // Even though the code of this method does not wait for it to complete, it does call the
+ // Collection API before
+ // it issues the second state change below. The collection API will be doing its own state
+ // change(s), and these will
+ // happen after this one (given it's for the same collection). Therefore we persist this
+ // state change
// immediately and do not group it with the one done further down.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/api/collections/DistributedCollectionConfigSetCommandRunner.java
##########
@@ -110,132 +109,176 @@ public DistributedCollectionConfigSetCommandRunner(CoreContainer coreContainer)
this.coreContainer = coreContainer;
if (log.isInfoEnabled()) {
- // Note is it hard to print a log when Collection API is handled by Overseer because Overseer is started regardless
+ // Note is it hard to print a log when Collection API is handled by Overseer because Overseer
+ // is started regardless
// of how Collection API is handled, so it doesn't really know...
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/api/collections/DistributedCollectionConfigSetCommandRunner.java
##########
@@ -342,31 +397,51 @@ public OverseerSolrResponse call() {
NamedList<Object> results = new NamedList<>();
try {
- // Create API lock for executing the command. This call is non blocking (not blocked on waiting for a lock to be acquired anyway,
+ // Create API lock for executing the command. This call is non blocking (not blocked on
+ // waiting for a lock to be acquired anyway,
// might be blocked on access to ZK etc)
- // We create a new CollectionApiLockFactory using a new ZkDistributedCollectionLockFactory because earlier (in the constructor of this class)
- // the ZkStateReader was not yet available, due to how CoreContainer is built in part in the constructor and in part in its
+ // We create a new CollectionApiLockFactory using a new ZkDistributedCollectionLockFactory
+ // because earlier (in the constructor of this class)
+ // the ZkStateReader was not yet available, due to how CoreContainer is built in part in the
+ // constructor and in part in its
// load() method. And this class is built from the CoreContainer constructor...
- // The cost of these creations is low, and these classes do not hold state but only serve as an interface to Zookeeper.
+ // The cost of these creations is low, and these classes do not hold state but only serve as
+ // an interface to Zookeeper.
// Note that after this call, we MUST execute the lock.release(); in the finally below
- DistributedMultiLock lock = new CollectionApiLockFactory(new ZkDistributedCollectionLockFactory(ccc.getZkStateReader().getZkClient(), ZK_COLLECTION_LOCKS)).createCollectionApiLock(action.lockLevel, collName, shardId, replicaName);
+ DistributedMultiLock lock =
+ new CollectionApiLockFactory(
+ new ZkDistributedCollectionLockFactory(
+ ccc.getZkStateReader().getZkClient(), ZK_COLLECTION_LOCKS))
+ .createCollectionApiLock(action.lockLevel, collName, shardId, replicaName);
try {
- log.debug("CollectionCommandRunner about to acquire lock for action {} lock level {}. {}/{}/{}",
- action, action.lockLevel, collName, shardId, replicaName);
+ log.debug(
+ "CollectionCommandRunner about to acquire lock for action {} lock level {}. {}/{}/{}",
+ action,
+ action.lockLevel,
+ collName,
+ shardId,
+ replicaName);
// Block this thread until all required locks are acquired.
lock.waitUntilAcquired();
- // Got the lock so moving from submitted to running if we run for an async task (if asyncId is null the asyncTaskTracker
+ // Got the lock so moving from submitted to running if we run for an async task (if
+ // asyncId is null the asyncTaskTracker
// calls do nothing).
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/api/collections/MaintainRoutedAliasCmd.java
##########
@@ -51,107 +50,139 @@
}
/**
- * Invokes this command from the client. If there's a problem it will throw an exception.
- * Please note that is important to never add async to this invocation. This method must
- * block (up to the standard OCP timeout) to prevent large batches of add's from sending a message
- * to the overseer for every document added in RoutedAliasUpdateProcessor.
+ * Invokes this command from the client. If there's a problem it will throw an exception. Please
+ * note that is important to never add async to this invocation. This method must block (up to the
+ * standard OCP timeout) to prevent large batches of add's from sending a message to the overseer
+ * for every document added in RoutedAliasUpdateProcessor.
*/
static void remoteInvoke(CollectionsHandler collHandler, String aliasName, String targetCol)
throws Exception {
- final CollectionParams.CollectionAction maintainroutedalias = CollectionParams.CollectionAction.MAINTAINROUTEDALIAS;
+ final CollectionParams.CollectionAction maintainroutedalias =
+ CollectionParams.CollectionAction.MAINTAINROUTEDALIAS;
Map<String, Object> msg = new HashMap<>();
msg.put(Overseer.QUEUE_OPERATION, maintainroutedalias.toLower());
msg.put(CollectionParams.NAME, aliasName);
msg.put(MaintainRoutedAliasCmd.ROUTED_ALIAS_TARGET_COL, targetCol);
- final SolrResponse rsp = collHandler.submitCollectionApiCommand(new ZkNodeProps(msg), maintainroutedalias);
+ final SolrResponse rsp =
+ collHandler.submitCollectionApiCommand(new ZkNodeProps(msg), maintainroutedalias);
if (rsp.getException() != null) {
throw rsp.getException();
}
}
- void addCollectionToAlias(String aliasName, ZkStateReader.AliasesManager aliasesManager, String createCollName) {
- aliasesManager.applyModificationAndExportToZk(curAliases -> {
- final List<String> curTargetCollections = curAliases.getCollectionAliasListMap().get(aliasName);
- if (curTargetCollections.contains(createCollName)) {
- return curAliases;
- } else {
- List<String> newTargetCollections = new ArrayList<>(curTargetCollections.size() + 1);
- // prepend it on purpose (thus reverse sorted). Solr alias resolution defaults to the first collection in a list
- newTargetCollections.add(createCollName);
- newTargetCollections.addAll(curTargetCollections);
- return curAliases.cloneWithCollectionAlias(aliasName, StrUtils.join(newTargetCollections, ','));
- }
- });
+ void addCollectionToAlias(
+ String aliasName, ZkStateReader.AliasesManager aliasesManager, String createCollName) {
+ aliasesManager.applyModificationAndExportToZk(
+ curAliases -> {
+ final List<String> curTargetCollections =
+ curAliases.getCollectionAliasListMap().get(aliasName);
+ if (curTargetCollections.contains(createCollName)) {
+ return curAliases;
+ } else {
+ List<String> newTargetCollections = new ArrayList<>(curTargetCollections.size() + 1);
+ // prepend it on purpose (thus reverse sorted). Solr alias resolution defaults to the
+ // first collection in a list
+ newTargetCollections.add(createCollName);
+ newTargetCollections.addAll(curTargetCollections);
+ return curAliases.cloneWithCollectionAlias(
+ aliasName, StrUtils.join(newTargetCollections, ','));
+ }
+ });
}
- private void removeCollectionFromAlias(String aliasName, ZkStateReader.AliasesManager aliasesManager, String createCollName) {
- aliasesManager.applyModificationAndExportToZk(curAliases -> {
- final List<String> curTargetCollections = curAliases.getCollectionAliasListMap().get(aliasName);
- if (curTargetCollections.contains(createCollName)) {
- List<String> newTargetCollections = new ArrayList<>(curTargetCollections.size());
- newTargetCollections.addAll(curTargetCollections);
- newTargetCollections.remove(createCollName);
- return curAliases.cloneWithCollectionAlias(aliasName, StrUtils.join(newTargetCollections, ','));
- } else {
- return curAliases;
- }
- });
+ private void removeCollectionFromAlias(
+ String aliasName, ZkStateReader.AliasesManager aliasesManager, String createCollName) {
+ aliasesManager.applyModificationAndExportToZk(
+ curAliases -> {
+ final List<String> curTargetCollections =
+ curAliases.getCollectionAliasListMap().get(aliasName);
+ if (curTargetCollections.contains(createCollName)) {
+ List<String> newTargetCollections = new ArrayList<>(curTargetCollections.size());
+ newTargetCollections.addAll(curTargetCollections);
+ newTargetCollections.remove(createCollName);
+ return curAliases.cloneWithCollectionAlias(
+ aliasName, StrUtils.join(newTargetCollections, ','));
+ } else {
+ return curAliases;
+ }
+ });
}
@Override
- public void call(ClusterState clusterState, ZkNodeProps message, NamedList<Object> results) throws Exception {
- //---- PARSE PRIMARY MESSAGE PARAMS
- // important that we use NAME for the alias as that is what the Overseer will get a lock on before calling us
+ public void call(ClusterState clusterState, ZkNodeProps message, NamedList<Object> results)
+ throws Exception {
+ // ---- PARSE PRIMARY MESSAGE PARAMS
+ // important that we use NAME for the alias as that is what the Overseer will get a lock on
+ // before calling us
final String aliasName = message.getStr(NAME);
final String routeValue = message.getStr(ROUTED_ALIAS_TARGET_COL);
final ZkStateReader.AliasesManager aliasesManager = ccc.getZkStateReader().aliasesManager;
final Aliases aliases = aliasesManager.getAliases();
final Map<String, String> aliasMetadata = aliases.getCollectionAliasProperties(aliasName);
if (aliasMetadata.isEmpty()) {
- throw new SolrException(SolrException.ErrorCode.BAD_REQUEST,
- "Alias " + aliasName + " does not exist or is not a routed alias."); // if it did exist, we'd have a non-null map
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Alias "
+ + aliasName
+ + " does not exist or is not a routed alias."); // if it did exist, we'd have a
+ // non-null map
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/api/collections/RestoreCmd.java
##########
@@ -163,26 +168,38 @@ private RestoreContext(ZkNodeProps message, CollectionCommandContext ccc) throws
this.container = ccc.getCoreContainer();
this.repository = this.container.newBackupRepository(repo);
- this.location = repository.createDirectoryURI(message.getStr(CoreAdminParams.BACKUP_LOCATION));
+ this.location =
+ repository.createDirectoryURI(message.getStr(CoreAdminParams.BACKUP_LOCATION));
final URI backupNameUri = repository.resolveDirectory(location, backupName);
final String[] entries = repository.listAll(backupNameUri);
- final boolean incremental = ! Arrays.stream(entries).anyMatch(entry -> entry.equals(BackupManager.TRADITIONAL_BACKUP_PROPS_FILE));
- this.backupPath = (incremental) ?
- repository.resolveDirectory(backupNameUri, entries[0]) : // incremental backups have an extra path component representing the backed up collection
+ final boolean incremental =
+ !Arrays.stream(entries)
+ .anyMatch(entry -> entry.equals(BackupManager.TRADITIONAL_BACKUP_PROPS_FILE));
+ this.backupPath =
+ (incremental)
+ ? repository.resolveDirectory(backupNameUri, entries[0])
+ : // incremental backups have an extra path component representing the backed up
+ // collection
backupNameUri;
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cluster/placement/plugins/AffinityPlacementFactory.java
##########
@@ -219,7 +239,8 @@ private AffinityPlacementPlugin(long minimalFreeDiskGB, long prioritizedFreeDisk
.requestNodeMetric(NodeMetricImpl.FREE_DISK_GB);
attributeFetcher.fetchFrom(allNodes);
final AttributeValues attrValues = attributeFetcher.fetchAttributes();
- // Get the number of currently existing cores per node, so we can update as we place new cores to not end up
+ // Get the number of currently existing cores per node, so we can update as we place new cores
+ // to not end up
// always selecting the same node(s). This is used across placement requests
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/api/collections/RestoreCmd.java
##########
@@ -274,44 +309,56 @@ private void validate(DocCollection backupCollectionState, int availableNodeCoun
assert totalReplicasPerShard > 0;
}
- private void uploadConfig(String configName, String restoreConfigName, BackupManager backupMgr, ConfigSetService configSetService) throws IOException {
+ private void uploadConfig(
+ String configName,
+ String restoreConfigName,
+ BackupManager backupMgr,
+ ConfigSetService configSetService)
+ throws IOException {
if (configSetService.checkConfigExists(restoreConfigName)) {
log.info("Using existing config {}", restoreConfigName);
- //TODO add overwrite option?
+ // TODO add overwrite option?
} else {
log.info("Uploading config {}", restoreConfigName);
backupMgr.uploadConfigDir(configName, restoreConfigName, configSetService);
}
}
- private void createCoreLessCollection(String restoreCollectionName,
- String restoreConfigName,
- DocCollection backupCollectionState,
- ClusterState clusterState) throws Exception {
+ private void createCoreLessCollection(
+ String restoreCollectionName,
+ String restoreConfigName,
+ DocCollection backupCollectionState,
+ ClusterState clusterState)
+ throws Exception {
Map<String, Object> propMap = new HashMap<>();
propMap.put(Overseer.QUEUE_OPERATION, CREATE.toString());
- propMap.put("fromApi", "true"); // mostly true. Prevents autoCreated=true in the collection state.
+ propMap.put(
+ "fromApi", "true"); // mostly true. Prevents autoCreated=true in the collection state.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/api/collections/OverseerStatusCmd.java
##########
@@ -147,10 +162,13 @@ public OverseerStatusCmd(CollectionCommandContext ccc) {
}
@Override
- public void call(ClusterState state, ZkNodeProps message, NamedList<Object> results) throws Exception {
- // If Collection API execution is distributed, we're not running on the Overseer node so can't return any Overseer stats.
+ public void call(ClusterState state, ZkNodeProps message, NamedList<Object> results)
+ throws Exception {
+ // If Collection API execution is distributed, we're not running on the Overseer node so can't
+ // return any Overseer stats.
if (ccc.getCoreContainer().getDistributedCollectionCommandRunner().isPresent()) {
- // TODO: introduce a per node status command allowing insight into how Cluster state updates, Collection API and
+ // TODO: introduce a per node status command allowing insight into how Cluster state updates,
+ // Collection API and
// config set API execution went on that node...
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/api/collections/RestoreCmd.java
##########
@@ -321,13 +368,19 @@ private void createCoreLessCollection(String restoreCollectionName,
propMap.put(CollectionHandlingUtils.SHARDS_PROP, StrUtils.join(sliceNames, ','));
} else {
propMap.put(CollectionHandlingUtils.NUM_SLICES, sliceNames.size());
- // ClusterStateMutator.createCollection detects that "slices" is in fact a slice structure instead of a
+ // ClusterStateMutator.createCollection detects that "slices" is in fact a slice structure
+ // instead of a
// list of names, and if so uses this instead of building it. We clear the replica list.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/api/collections/MaintainRoutedAliasCmd.java
##########
@@ -51,107 +50,139 @@
}
/**
- * Invokes this command from the client. If there's a problem it will throw an exception.
- * Please note that is important to never add async to this invocation. This method must
- * block (up to the standard OCP timeout) to prevent large batches of add's from sending a message
- * to the overseer for every document added in RoutedAliasUpdateProcessor.
+ * Invokes this command from the client. If there's a problem it will throw an exception. Please
+ * note that is important to never add async to this invocation. This method must block (up to the
+ * standard OCP timeout) to prevent large batches of add's from sending a message to the overseer
+ * for every document added in RoutedAliasUpdateProcessor.
*/
static void remoteInvoke(CollectionsHandler collHandler, String aliasName, String targetCol)
throws Exception {
- final CollectionParams.CollectionAction maintainroutedalias = CollectionParams.CollectionAction.MAINTAINROUTEDALIAS;
+ final CollectionParams.CollectionAction maintainroutedalias =
+ CollectionParams.CollectionAction.MAINTAINROUTEDALIAS;
Map<String, Object> msg = new HashMap<>();
msg.put(Overseer.QUEUE_OPERATION, maintainroutedalias.toLower());
msg.put(CollectionParams.NAME, aliasName);
msg.put(MaintainRoutedAliasCmd.ROUTED_ALIAS_TARGET_COL, targetCol);
- final SolrResponse rsp = collHandler.submitCollectionApiCommand(new ZkNodeProps(msg), maintainroutedalias);
+ final SolrResponse rsp =
+ collHandler.submitCollectionApiCommand(new ZkNodeProps(msg), maintainroutedalias);
if (rsp.getException() != null) {
throw rsp.getException();
}
}
- void addCollectionToAlias(String aliasName, ZkStateReader.AliasesManager aliasesManager, String createCollName) {
- aliasesManager.applyModificationAndExportToZk(curAliases -> {
- final List<String> curTargetCollections = curAliases.getCollectionAliasListMap().get(aliasName);
- if (curTargetCollections.contains(createCollName)) {
- return curAliases;
- } else {
- List<String> newTargetCollections = new ArrayList<>(curTargetCollections.size() + 1);
- // prepend it on purpose (thus reverse sorted). Solr alias resolution defaults to the first collection in a list
- newTargetCollections.add(createCollName);
- newTargetCollections.addAll(curTargetCollections);
- return curAliases.cloneWithCollectionAlias(aliasName, StrUtils.join(newTargetCollections, ','));
- }
- });
+ void addCollectionToAlias(
+ String aliasName, ZkStateReader.AliasesManager aliasesManager, String createCollName) {
+ aliasesManager.applyModificationAndExportToZk(
+ curAliases -> {
+ final List<String> curTargetCollections =
+ curAliases.getCollectionAliasListMap().get(aliasName);
+ if (curTargetCollections.contains(createCollName)) {
+ return curAliases;
+ } else {
+ List<String> newTargetCollections = new ArrayList<>(curTargetCollections.size() + 1);
+ // prepend it on purpose (thus reverse sorted). Solr alias resolution defaults to the
+ // first collection in a list
+ newTargetCollections.add(createCollName);
+ newTargetCollections.addAll(curTargetCollections);
+ return curAliases.cloneWithCollectionAlias(
+ aliasName, StrUtils.join(newTargetCollections, ','));
+ }
+ });
}
- private void removeCollectionFromAlias(String aliasName, ZkStateReader.AliasesManager aliasesManager, String createCollName) {
- aliasesManager.applyModificationAndExportToZk(curAliases -> {
- final List<String> curTargetCollections = curAliases.getCollectionAliasListMap().get(aliasName);
- if (curTargetCollections.contains(createCollName)) {
- List<String> newTargetCollections = new ArrayList<>(curTargetCollections.size());
- newTargetCollections.addAll(curTargetCollections);
- newTargetCollections.remove(createCollName);
- return curAliases.cloneWithCollectionAlias(aliasName, StrUtils.join(newTargetCollections, ','));
- } else {
- return curAliases;
- }
- });
+ private void removeCollectionFromAlias(
+ String aliasName, ZkStateReader.AliasesManager aliasesManager, String createCollName) {
+ aliasesManager.applyModificationAndExportToZk(
+ curAliases -> {
+ final List<String> curTargetCollections =
+ curAliases.getCollectionAliasListMap().get(aliasName);
+ if (curTargetCollections.contains(createCollName)) {
+ List<String> newTargetCollections = new ArrayList<>(curTargetCollections.size());
+ newTargetCollections.addAll(curTargetCollections);
+ newTargetCollections.remove(createCollName);
+ return curAliases.cloneWithCollectionAlias(
+ aliasName, StrUtils.join(newTargetCollections, ','));
+ } else {
+ return curAliases;
+ }
+ });
}
@Override
- public void call(ClusterState clusterState, ZkNodeProps message, NamedList<Object> results) throws Exception {
- //---- PARSE PRIMARY MESSAGE PARAMS
- // important that we use NAME for the alias as that is what the Overseer will get a lock on before calling us
+ public void call(ClusterState clusterState, ZkNodeProps message, NamedList<Object> results)
+ throws Exception {
+ // ---- PARSE PRIMARY MESSAGE PARAMS
+ // important that we use NAME for the alias as that is what the Overseer will get a lock on
+ // before calling us
final String aliasName = message.getStr(NAME);
final String routeValue = message.getStr(ROUTED_ALIAS_TARGET_COL);
final ZkStateReader.AliasesManager aliasesManager = ccc.getZkStateReader().aliasesManager;
final Aliases aliases = aliasesManager.getAliases();
final Map<String, String> aliasMetadata = aliases.getCollectionAliasProperties(aliasName);
if (aliasMetadata.isEmpty()) {
- throw new SolrException(SolrException.ErrorCode.BAD_REQUEST,
- "Alias " + aliasName + " does not exist or is not a routed alias."); // if it did exist, we'd have a non-null map
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Alias "
+ + aliasName
+ + " does not exist or is not a routed alias."); // if it did exist, we'd have a
+ // non-null map
}
final RoutedAlias ra = RoutedAlias.fromProps(aliasName, aliasMetadata);
if (ra == null) {
- throw new SolrException(SolrException.ErrorCode.SERVER_ERROR, "MaintainRoutedAlias called on non-routed alias");
+ throw new SolrException(
+ SolrException.ErrorCode.SERVER_ERROR, "MaintainRoutedAlias called on non-routed alias");
}
ra.updateParsedCollectionAliases(ccc.getZkStateReader(), true);
List<RoutedAlias.Action> actions = ra.calculateActions(routeValue);
for (RoutedAlias.Action action : actions) {
- boolean exists = ccc.getZkStateReader().getClusterState().getCollectionOrNull(action.targetCollection) != null;
+ boolean exists =
+ ccc.getZkStateReader().getClusterState().getCollectionOrNull(action.targetCollection)
+ != null;
switch (action.actionType) {
case ENSURE_REMOVED:
if (exists) {
- ccc.getExecutorService().submit(() -> {
- try {
- deleteTargetCollection(clusterState, results, aliasName, aliasesManager, action);
- } catch (Exception e) {
- log.warn("Deletion of {} by {} {} failed (this might be ok if two clients were"
- , action.targetCollection, ra.getAliasName()
- , " writing to a routed alias at the same time and both caused a deletion)");
- log.debug("Exception for last message:", e);
- }
- });
+ ccc.getExecutorService()
+ .submit(
+ () -> {
+ try {
+ deleteTargetCollection(
+ clusterState, results, aliasName, aliasesManager, action);
+ } catch (Exception e) {
+ log.warn(
+ "Deletion of {} by {} {} failed (this might be ok if two clients were",
+ action.targetCollection,
+ ra.getAliasName(),
+ " writing to a routed alias at the same time and both caused a deletion)");
+ log.debug("Exception for last message:", e);
+ }
+ });
}
break;
case ENSURE_EXISTS:
if (!exists) {
- addTargetCollection(clusterState, results, aliasName, aliasesManager, aliasMetadata, action);
+ addTargetCollection(
+ clusterState, results, aliasName, aliasesManager, aliasMetadata, action);
} else {
// check that the collection is properly integrated into the alias (see
- // TimeRoutedAliasUpdateProcessorTest.java:141). Presently we need to ensure inclusion in the alias
- // and the presence of the appropriate collection property. Note that this only works if the collection
- // happens to fall where we would have created one already. Support for un-even collection sizes will
- // take additional work (though presently they might work if the below book keeping is done by hand)
+ // TimeRoutedAliasUpdateProcessorTest.java:141). Presently we need to ensure inclusion
+ // in the alias
+ // and the presence of the appropriate collection property. Note that this only works if
+ // the collection
+ // happens to fall where we would have created one already. Support for un-even
+ // collection sizes will
+ // take additional work (though presently they might work if the below book keeping is
+ // done by hand)
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cluster/placement/impl/ReplicaPlacementImpl.java
##########
@@ -66,21 +66,37 @@ public String toString() {
}
/**
- * Translates a set of {@link ReplicaPlacement} returned by a plugin into a list of {@link ReplicaPosition} expected
- * by {@link org.apache.solr.cloud.api.collections.Assign.AssignStrategy}
+ * Translates a set of {@link ReplicaPlacement} returned by a plugin into a list of {@link
+ * ReplicaPosition} expected by {@link
+ * org.apache.solr.cloud.api.collections.Assign.AssignStrategy}
*/
- static List<ReplicaPosition> toReplicaPositions(String collection, Set<ReplicaPlacement> replicaPlacementSet) {
- // The replica index in ReplicaPosition is not as strict a concept as it might seem. It is used in rules
- // based placement (for sorting replicas) but its presence in ReplicaPosition is not justified (and when the code
+ static List<ReplicaPosition> toReplicaPositions(
+ String collection, Set<ReplicaPlacement> replicaPlacementSet) {
+ // The replica index in ReplicaPosition is not as strict a concept as it might seem. It is used
+ // in rules
+ // based placement (for sorting replicas) but its presence in ReplicaPosition is not justified
+ // (and when the code
// is executing here, it means rules based placement is not used).
- // Looking at ReplicaAssigner.tryAllPermutations, it is well possible to create replicas with same index
- // living on a given node for the same shard. This likely never happens because of the way replicas are
- // placed on nodes (never two on the same node for same shard). Adopting the same shortcut/bad design here,
+ // Looking at ReplicaAssigner.tryAllPermutations, it is well possible to create replicas with
+ // same index
+ // living on a given node for the same shard. This likely never happens because of the way
+ // replicas are
+ // placed on nodes (never two on the same node for same shard). Adopting the same shortcut/bad
+ // design here,
// but index should be removed at some point from ReplicaPosition.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/api/collections/RoutedAlias.java
##########
@@ -96,28 +105,33 @@ public static RoutedAlias fromProps(String aliasName, Map<String, String> props)
String[] types = typeStr.split(",");
java.util.List<String> fields = new ArrayList<>();
if (types.length > 2) {
- throw new SolrException(BAD_REQUEST,"More than 2 dimensions is not supported yet. " +
- "Please monitor SOLR-13628 for progress");
+ throw new SolrException(
+ BAD_REQUEST,
+ "More than 2 dimensions is not supported yet. "
+ + "Please monitor SOLR-13628 for progress");
}
for (int i = 0; i < types.length; i++) {
String type = types[i];
addRouterTypeOf(type, routerTypes);
- // v2 api case - the v2 -> v1 mapping mechanisms can't handle this conversion because they expect
+ // v2 api case - the v2 -> v1 mapping mechanisms can't handle this conversion because they
+ // expect
// strings or arrays of strings, not lists of objects.
// TODO: The generic types for props are a lie
if (props.containsKey("router.routerList")) {
HashMap<String, Object> tmp = new HashMap<>(props);
@SuppressWarnings({"unchecked"})
- List<Map<String, Object>> v2RouterList = (List<Map<String, Object>>) tmp.get("router.routerList");
+ List<Map<String, Object>> v2RouterList =
+ (List<Map<String, Object>>) tmp.get("router.routerList");
Map<String, Object> o = v2RouterList.get(i);
for (Map.Entry<String, Object> entry : o.entrySet()) {
props.put(ROUTER_PREFIX + i + "." + entry.getKey(), String.valueOf(entry.getValue()));
}
}
// Here we need to push the type into each dimension's params. We could have eschewed the
// "Dimensional[dim1,dim2]" style notation, to simplify this case but I think it's nice
- // to be able to understand the dimensionality at a glance without having to hunt for name properties
+ // to be able to understand the dimensionality at a glance without having to hunt for name
+ // properties
// in the list of properties for each dimension.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/api/collections/RoutedAlias.java
##########
@@ -45,44 +48,50 @@
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
-import static org.apache.solr.common.SolrException.ErrorCode.BAD_REQUEST;
-import static org.apache.solr.common.SolrException.ErrorCode.SERVER_ERROR;
-import static org.apache.solr.common.params.CollectionAdminParams.ROUTER_PREFIX;
-
public abstract class RoutedAlias {
private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
-
@SuppressWarnings("WeakerAccess")
public static final String ROUTER_TYPE_NAME = ROUTER_PREFIX + "name";
+
@SuppressWarnings("WeakerAccess")
public static final String ROUTER_FIELD = ROUTER_PREFIX + "field";
+
public static final String CREATE_COLLECTION_PREFIX = "create-collection.";
+
@SuppressWarnings("WeakerAccess")
- public static final Set<String> MINIMAL_REQUIRED_PARAMS = Sets.newHashSet(ROUTER_TYPE_NAME, ROUTER_FIELD);
+ public static final Set<String> MINIMAL_REQUIRED_PARAMS =
+ Sets.newHashSet(ROUTER_TYPE_NAME, ROUTER_FIELD);
+
public static final String ROUTED_ALIAS_NAME_CORE_PROP = "routedAliasName"; // core prop
private static final String DIMENSIONAL = "Dimensional[";
- // This class is created once per request and the overseer methods prevent duplicate create requests
- // from creating extra copies via locking on the alias name. All we need to track here is that we don't
+ // This class is created once per request and the overseer methods prevent duplicate create
+ // requests
+ // from creating extra copies via locking on the alias name. All we need to track here is that we
+ // don't
// spam preemptive creates to the overseer multiple times from *this* request.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cluster/placement/impl/ReplicaPlacementImpl.java
##########
@@ -66,21 +66,37 @@ public String toString() {
}
/**
- * Translates a set of {@link ReplicaPlacement} returned by a plugin into a list of {@link ReplicaPosition} expected
- * by {@link org.apache.solr.cloud.api.collections.Assign.AssignStrategy}
+ * Translates a set of {@link ReplicaPlacement} returned by a plugin into a list of {@link
+ * ReplicaPosition} expected by {@link
+ * org.apache.solr.cloud.api.collections.Assign.AssignStrategy}
*/
- static List<ReplicaPosition> toReplicaPositions(String collection, Set<ReplicaPlacement> replicaPlacementSet) {
- // The replica index in ReplicaPosition is not as strict a concept as it might seem. It is used in rules
- // based placement (for sorting replicas) but its presence in ReplicaPosition is not justified (and when the code
+ static List<ReplicaPosition> toReplicaPositions(
+ String collection, Set<ReplicaPlacement> replicaPlacementSet) {
+ // The replica index in ReplicaPosition is not as strict a concept as it might seem. It is used
+ // in rules
+ // based placement (for sorting replicas) but its presence in ReplicaPosition is not justified
+ // (and when the code
// is executing here, it means rules based placement is not used).
- // Looking at ReplicaAssigner.tryAllPermutations, it is well possible to create replicas with same index
- // living on a given node for the same shard. This likely never happens because of the way replicas are
- // placed on nodes (never two on the same node for same shard). Adopting the same shortcut/bad design here,
+ // Looking at ReplicaAssigner.tryAllPermutations, it is well possible to create replicas with
+ // same index
+ // living on a given node for the same shard. This likely never happens because of the way
+ // replicas are
+ // placed on nodes (never two on the same node for same shard). Adopting the same shortcut/bad
+ // design here,
// but index should be removed at some point from ReplicaPosition.
List<ReplicaPosition> replicaPositions = new ArrayList<>(replicaPlacementSet.size());
- int index = 0; // This really an arbitrary value when adding replicas and a possible source of core name collisions
+ int index =
+ 0; // This really an arbitrary value when adding replicas and a possible source of core name
+ // collisions
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/api/collections/RoutedAlias.java
##########
@@ -127,8 +141,10 @@ public static RoutedAlias fromProps(String aliasName, Map<String, String> props)
}
fields.add(props.get(ROUTER_PREFIX + i + ".field"));
}
- // this next remove is checked for key because when we build from aliases.json's data it we get an
- // immutable map which would cause UnsupportedOperationException to be thrown. This remove is here
+ // this next remove is checked for key because when we build from aliases.json's data it we
+ // get an
+ // immutable map which would cause UnsupportedOperationException to be thrown. This remove is
+ // here
// to prevent this property from making it into aliases.json
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/api/collections/RoutedAlias.java
##########
@@ -96,28 +105,33 @@ public static RoutedAlias fromProps(String aliasName, Map<String, String> props)
String[] types = typeStr.split(",");
java.util.List<String> fields = new ArrayList<>();
if (types.length > 2) {
- throw new SolrException(BAD_REQUEST,"More than 2 dimensions is not supported yet. " +
- "Please monitor SOLR-13628 for progress");
+ throw new SolrException(
+ BAD_REQUEST,
+ "More than 2 dimensions is not supported yet. "
+ + "Please monitor SOLR-13628 for progress");
}
for (int i = 0; i < types.length; i++) {
String type = types[i];
addRouterTypeOf(type, routerTypes);
- // v2 api case - the v2 -> v1 mapping mechanisms can't handle this conversion because they expect
+ // v2 api case - the v2 -> v1 mapping mechanisms can't handle this conversion because they
+ // expect
// strings or arrays of strings, not lists of objects.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/api/collections/RoutedAlias.java
##########
@@ -233,33 +267,45 @@ private static RoutedAlias routedAliasForType(String aliasName, Map<String, Stri
public abstract void validateRouteValue(AddUpdateCommand cmd) throws SolrException;
/**
- * Create any required collections and return the name of the collection to which the current document should be sent.
+ * Create any required collections and return the name of the collection to which the current
+ * document should be sent.
*
* @param cmd The command that might cause collection creation
- * @return The name of the proper destination collection for the document which may or may not be a
- * newly created collection
+ * @return The name of the proper destination collection for the document which may or may not be
+ * a newly created collection
*/
public String createCollectionsIfRequired(AddUpdateCommand cmd) {
// Even though it is possible that multiple requests hit this code in the 1-2 sec that
- // it takes to create a collection, it's an established anti-pattern to feed data with a very large number
- // of client connections. This in mind, we only guard against spamming the overseer within a batch of
- // updates. We are intentionally tolerating a low level of redundant requests in favor of simpler code. Most
- // super-sized installations with many update clients will likely be multi-tenant and multiple tenants
- // probably don't write to the same alias. As such, we have deferred any solution to the "many clients causing
- // collection creation simultaneously" problem until such time as someone actually has that problem in a
+ // it takes to create a collection, it's an established anti-pattern to feed data with a very
+ // large number
+ // of client connections. This in mind, we only guard against spamming the overseer within a
+ // batch of
+ // updates. We are intentionally tolerating a low level of redundant requests in favor of
+ // simpler code. Most
+ // super-sized installations with many update clients will likely be multi-tenant and multiple
+ // tenants
+ // probably don't write to the same alias. As such, we have deferred any solution to the "many
+ // clients causing
+ // collection creation simultaneously" problem until such time as someone actually has that
+ // problem in a
// real world use case that isn't just an anti-pattern.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/api/collections/RoutedAlias.java
##########
@@ -398,10 +470,11 @@ void ensureCollection(String targetCollection, CoreContainer coreContainer) {
// of a race and that's okay... we'll ultimately retry any way.
// Ensure our view of the aliases has updated. If we didn't do this, our zkStateReader might
- // not yet know about the new alias (thus won't see the newly added collection to it), and we might think
+ // not yet know about the new alias (thus won't see the newly added collection to it), and we
+ // might think
// we failed.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/api/collections/SplitShardCmd.java
##########
@@ -430,47 +509,71 @@ public boolean split(ClusterState clusterState, ZkNodeProps message, NamedList<O
}
t = timings.sub("identifyNodesForReplicas");
- Assign.AssignRequest assignRequest = new Assign.AssignRequestBuilder()
- .forCollection(collectionName)
- .forShard(subSlices)
- .assignNrtReplicas(numNrt.get())
- .assignTlogReplicas(numTlog.get())
- .assignPullReplicas(numPull.get())
- .onNodes(new ArrayList<>(clusterState.getLiveNodes()))
- .build();
+ Assign.AssignRequest assignRequest =
+ new Assign.AssignRequestBuilder()
+ .forCollection(collectionName)
+ .forShard(subSlices)
+ .assignNrtReplicas(numNrt.get())
+ .assignTlogReplicas(numTlog.get())
+ .assignPullReplicas(numPull.get())
+ .onNodes(new ArrayList<>(clusterState.getLiveNodes()))
+ .build();
Assign.AssignStrategy assignStrategy = Assign.createAssignStrategy(ccc.getCoreContainer());
- List<ReplicaPosition> replicaPositions = assignStrategy.assign(ccc.getSolrCloudManager(), assignRequest);
+ List<ReplicaPosition> replicaPositions =
+ assignStrategy.assign(ccc.getSolrCloudManager(), assignRequest);
t.stop();
t = timings.sub("createReplicaPlaceholders");
final DistributedClusterStateUpdater.StateChangeRecorder scr;
boolean hasRecordedDistributedUpdate = false;
if (ccc.getDistributedClusterStateUpdater().isDistributedStateUpdate()) {
- scr = ccc.getDistributedClusterStateUpdater().createStateChangeRecorder(collectionName, false);
+ scr =
+ ccc.getDistributedClusterStateUpdater()
+ .createStateChangeRecorder(collectionName, false);
} else {
scr = null;
}
for (ReplicaPosition replicaPosition : replicaPositions) {
String sliceName = replicaPosition.shard;
String subShardNodeName = replicaPosition.node;
- String solrCoreName = Assign.buildSolrCoreName(collectionName, sliceName, replicaPosition.type, replicaPosition.index);
-
- log.debug("Creating replica shard {} as part of slice {} of collection {} on {}"
- , solrCoreName, sliceName, collectionName, subShardNodeName);
-
- // we first create all replicas in DOWN state without actually creating their cores in order to
- // avoid a race condition where Overseer may prematurely activate the new sub-slices (and deactivate
- // the parent slice) before all new replicas are added. This situation may lead to a loss of performance
+ String solrCoreName =
+ Assign.buildSolrCoreName(
+ collectionName, sliceName, replicaPosition.type, replicaPosition.index);
+
+ log.debug(
+ "Creating replica shard {} as part of slice {} of collection {} on {}",
+ solrCoreName,
+ sliceName,
+ collectionName,
+ subShardNodeName);
+
+ // we first create all replicas in DOWN state without actually creating their cores in order
+ // to
+ // avoid a race condition where Overseer may prematurely activate the new sub-slices (and
+ // deactivate
+ // the parent slice) before all new replicas are added. This situation may lead to a loss of
+ // performance
// because the new shards will be activated with possibly many fewer replicas.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/api/collections/SplitShardCmd.java
##########
@@ -495,25 +598,36 @@ public boolean split(ClusterState clusterState, ZkNodeProps message, NamedList<O
if (asyncId != null) {
propMap.put(ASYNC, asyncId);
}
- // special flag param to instruct addReplica not to create the replica in cluster state again
+ // special flag param to instruct addReplica not to create the replica in cluster state
+ // again
propMap.put(CollectionHandlingUtils.SKIP_CREATE_REPLICA_IN_CLUSTER_STATE, "true");
propMap.put(CommonAdminParams.WAIT_FOR_FINAL_STATE, Boolean.toString(waitForFinalState));
replicas.add(propMap);
}
- if (hasRecordedDistributedUpdate && ccc.getDistributedClusterStateUpdater().isDistributedStateUpdate()) {
- // Actually add the replicas to the collection state. Note that when Overseer takes care of the state,
- // there is no wait here for the state update to be visible, but with distributed state update done synchronously
- // we wait (we could in theory create a thread and have it do the work if we REALLY needed, but we likely don't).
+ if (hasRecordedDistributedUpdate
+ && ccc.getDistributedClusterStateUpdater().isDistributedStateUpdate()) {
+ // Actually add the replicas to the collection state. Note that when Overseer takes care of
+ // the state,
+ // there is no wait here for the state update to be visible, but with distributed state
+ // update done synchronously
+ // we wait (we could in theory create a thread and have it do the work if we REALLY needed,
+ // but we likely don't).
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/api/collections/TimeRoutedAlias.java
##########
@@ -352,18 +373,28 @@ public void validateRouteValue(AddUpdateCommand cmd) throws SolrException {
// FUTURE: maybe in some cases the user would want to ignore/warn instead?
if (docTimestamp.isAfter(Instant.now().plusMillis(getMaxFutureMs()))) {
- throw new SolrException(BAD_REQUEST,
- "The document's time routed key of " + docTimestamp + " is too far in the future given " +
- ROUTER_MAX_FUTURE + "=" + getMaxFutureMs());
+ throw new SolrException(
+ BAD_REQUEST,
+ "The document's time routed key of "
+ + docTimestamp
+ + " is too far in the future given "
+ + ROUTER_MAX_FUTURE
+ + "="
+ + getMaxFutureMs());
}
- // Although this is also checked later, we need to check it here too to handle the case in Dimensional Routed
- // aliases where one can legally have zero collections for a newly encountered category and thus the loop later
+ // Although this is also checked later, we need to check it here too to handle the case in
+ // Dimensional Routed
+ // aliases where one can legally have zero collections for a newly encountered category and thus
+ // the loop later
// can't catch this.
- // SOLR-13760 - we need to fix the date math to a specific instant when the first document arrives.
- // If we don't do this DRA's with a time dimension have variable start times across the other dimensions
- // and logic gets much to complicated, and depends too much on queries to zookeeper. This keeps life simpler.
+ // SOLR-13760 - we need to fix the date math to a specific instant when the first document
+ // arrives.
+ // If we don't do this DRA's with a time dimension have variable start times across the other
+ // dimensions
+ // and logic gets much to complicated, and depends too much on queries to zookeeper. This keeps
+ // life simpler.
// I have to admit I'm not terribly fond of the mutation during a validate method however.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/api/collections/DistributedCollectionConfigSetCommandRunner.java
##########
@@ -110,132 +109,176 @@ public DistributedCollectionConfigSetCommandRunner(CoreContainer coreContainer)
this.coreContainer = coreContainer;
if (log.isInfoEnabled()) {
- // Note is it hard to print a log when Collection API is handled by Overseer because Overseer is started regardless
+ // Note is it hard to print a log when Collection API is handled by Overseer because Overseer
+ // is started regardless
// of how Collection API is handled, so it doesn't really know...
- log.info("Creating DistributedCollectionConfigSetCommandRunner. Collection and ConfigSet APIs are running distributed (not Overseer based)");
+ log.info(
+ "Creating DistributedCollectionConfigSetCommandRunner. Collection and ConfigSet APIs are running distributed (not Overseer based)");
}
// TODO we should look at how everything is getting closed when the node is shutdown.
- // But it seems that CollectionsHandler (that creates instances of this class) is not really closed, so maybe it doesn't matter?
- // With distributed Collection API execution, each node will have such an executor but given how thread pools work,
+ // But it seems that CollectionsHandler (that creates instances of this class) is not really
+ // closed, so maybe it doesn't matter?
+ // With distributed Collection API execution, each node will have such an executor but given how
+ // thread pools work,
// threads will only be created if needed (including the corePoolSize threads).
- distributedCollectionApiExecutorService = new ExecutorUtil.MDCAwareThreadPoolExecutor(5, 10, 0L, TimeUnit.MILLISECONDS,
- new SynchronousQueue<>(),
- new SolrNamedThreadFactory("DistributedCollectionApiExecutorService"));
-
- commandsExecutor = new ExecutorUtil.MDCAwareThreadPoolExecutor(5, 20, 0L, TimeUnit.MILLISECONDS,
- new SynchronousQueue<>(),
- new SolrNamedThreadFactory("DistributedCollectionApiCommandExecutor"));
-
- ccc = new DistributedCollectionCommandContext(this.coreContainer, this.distributedCollectionApiExecutorService);
+ distributedCollectionApiExecutorService =
+ new ExecutorUtil.MDCAwareThreadPoolExecutor(
+ 5,
+ 10,
+ 0L,
+ TimeUnit.MILLISECONDS,
+ new SynchronousQueue<>(),
+ new SolrNamedThreadFactory("DistributedCollectionApiExecutorService"));
+
+ commandsExecutor =
+ new ExecutorUtil.MDCAwareThreadPoolExecutor(
+ 5,
+ 20,
+ 0L,
+ TimeUnit.MILLISECONDS,
+ new SynchronousQueue<>(),
+ new SolrNamedThreadFactory("DistributedCollectionApiCommandExecutor"));
+
+ ccc =
+ new DistributedCollectionCommandContext(
+ this.coreContainer, this.distributedCollectionApiExecutorService);
commandMapper = new CollApiCmds.CommandMap(ccc);
- asyncTaskTracker = new DistributedApiAsyncTracker(ccc.getZkStateReader().getZkClient(), ZK_ASYNC_ROOT);
+ asyncTaskTracker =
+ new DistributedApiAsyncTracker(ccc.getZkStateReader().getZkClient(), ZK_ASYNC_ROOT);
}
- /**
- * See {@link DistributedApiAsyncTracker#getAsyncTaskRequestStatus(String)}
- */
- public Pair<RequestStatusState, OverseerSolrResponse> getAsyncTaskRequestStatus(String asyncId) throws Exception {
+ /** See {@link DistributedApiAsyncTracker#getAsyncTaskRequestStatus(String)} */
+ public Pair<RequestStatusState, OverseerSolrResponse> getAsyncTaskRequestStatus(String asyncId)
+ throws Exception {
return asyncTaskTracker.getAsyncTaskRequestStatus(asyncId);
}
- /**
- * See {@link DistributedApiAsyncTracker#deleteSingleAsyncId(String)}
- */
+ /** See {@link DistributedApiAsyncTracker#deleteSingleAsyncId(String)} */
public boolean deleteSingleAsyncId(String asyncId) throws Exception {
return asyncTaskTracker.deleteSingleAsyncId(asyncId);
}
- /**
- * See {@link DistributedApiAsyncTracker#deleteAllAsyncIds()}
- */
+ /** See {@link DistributedApiAsyncTracker#deleteAllAsyncIds()} */
public void deleteAllAsyncIds() throws Exception {
asyncTaskTracker.deleteAllAsyncIds();
}
-
/**
- * When {@link org.apache.solr.handler.admin.CollectionsHandler#invokeAction} does not enqueue to overseer queue and
- * instead calls this method, this method is expected to do the equivalent of what Overseer does in
- * {@link org.apache.solr.cloud.OverseerConfigSetMessageHandler#processMessage}.
- * <p>
- * The steps leading to that call in the Overseer execution path are (and the equivalent is done here):
+ * When {@link org.apache.solr.handler.admin.CollectionsHandler#invokeAction} does not enqueue to
+ * overseer queue and instead calls this method, this method is expected to do the equivalent of
+ * what Overseer does in {@link
+ * org.apache.solr.cloud.OverseerConfigSetMessageHandler#processMessage}.
+ *
+ * <p>The steps leading to that call in the Overseer execution path are (and the equivalent is
+ * done here):
+ *
* <ul>
- * <li>{@link org.apache.solr.cloud.OverseerTaskProcessor#run()} gets the message from the ZK queue, grabs the
- * corresponding locks (write lock on the config set target of the API command and a read lock on the base config set
- * if any - the case for config set creation) then executes the command using an executor service (it also checks the
- * asyncId if any is specified but async calls are not supported for Config Set API calls).</li>
- * <li>In {@link org.apache.solr.cloud.OverseerTaskProcessor}.{@code Runner.run()} (run on an executor thread) a call is made to
- * {@link org.apache.solr.cloud.OverseerConfigSetMessageHandler#processMessage} which does a few checks and calls the
- * appropriate Config Set method.
+ * <li>{@link org.apache.solr.cloud.OverseerTaskProcessor#run()} gets the message from the ZK
+ * queue, grabs the corresponding locks (write lock on the config set target of the API
+ * command and a read lock on the base config set if any - the case for config set creation)
+ * then executes the command using an executor service (it also checks the asyncId if any is
+ * specified but async calls are not supported for Config Set API calls).
+ * <li>In {@link org.apache.solr.cloud.OverseerTaskProcessor}.{@code Runner.run()} (run on an
+ * executor thread) a call is made to {@link
+ * org.apache.solr.cloud.OverseerConfigSetMessageHandler#processMessage} which does a few
+ * checks and calls the appropriate Config Set method.
* </ul>
*/
- public void runConfigSetCommand(SolrQueryResponse rsp, ConfigSetsHandler.ConfigSetOperation operation, Map<String, Object> result, long timeoutMs) throws Exception {
- // We refuse new tasks, but will wait for already submitted ones (i.e. those that made it through this method earlier).
+ public void runConfigSetCommand(
+ SolrQueryResponse rsp,
+ ConfigSetsHandler.ConfigSetOperation operation,
+ Map<String, Object> result,
+ long timeoutMs)
+ throws Exception {
+ // We refuse new tasks, but will wait for already submitted ones (i.e. those that made it
+ // through this method earlier).
// See stopAndWaitForPendingTasksToComplete() below
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/api/collections/RoutedAlias.java
##########
@@ -233,33 +267,45 @@ private static RoutedAlias routedAliasForType(String aliasName, Map<String, Stri
public abstract void validateRouteValue(AddUpdateCommand cmd) throws SolrException;
/**
- * Create any required collections and return the name of the collection to which the current document should be sent.
+ * Create any required collections and return the name of the collection to which the current
+ * document should be sent.
*
* @param cmd The command that might cause collection creation
- * @return The name of the proper destination collection for the document which may or may not be a
- * newly created collection
+ * @return The name of the proper destination collection for the document which may or may not be
+ * a newly created collection
*/
public String createCollectionsIfRequired(AddUpdateCommand cmd) {
// Even though it is possible that multiple requests hit this code in the 1-2 sec that
- // it takes to create a collection, it's an established anti-pattern to feed data with a very large number
- // of client connections. This in mind, we only guard against spamming the overseer within a batch of
- // updates. We are intentionally tolerating a low level of redundant requests in favor of simpler code. Most
- // super-sized installations with many update clients will likely be multi-tenant and multiple tenants
- // probably don't write to the same alias. As such, we have deferred any solution to the "many clients causing
- // collection creation simultaneously" problem until such time as someone actually has that problem in a
+ // it takes to create a collection, it's an established anti-pattern to feed data with a very
+ // large number
+ // of client connections. This in mind, we only guard against spamming the overseer within a
+ // batch of
+ // updates. We are intentionally tolerating a low level of redundant requests in favor of
+ // simpler code. Most
+ // super-sized installations with many update clients will likely be multi-tenant and multiple
+ // tenants
+ // probably don't write to the same alias. As such, we have deferred any solution to the "many
+ // clients causing
+ // collection creation simultaneously" problem until such time as someone actually has that
+ // problem in a
// real world use case that isn't just an anti-pattern.
CandidateCollection candidateCollectionDesc = findCandidateGivenValue(cmd);
try {
// It's important not to add code between here and the prior call to findCandidateGivenValue()
- // in processAdd() that invokes updateParsedCollectionAliases(). Doing so would update parsedCollectionsDesc
- // and create a race condition. When Routed aliases have an implicit sort for their collections we
- // are relying on the fact that collectionList.get(0) is returning the head of the parsed collections that
- // existed when the collection list was consulted for the candidate value. If this class updates it's notion
- // of the list of collections since candidateCollectionDesc was chosen, we could create collection n+2
+ // in processAdd() that invokes updateParsedCollectionAliases(). Doing so would update
+ // parsedCollectionsDesc
+ // and create a race condition. When Routed aliases have an implicit sort for their
+ // collections we
+ // are relying on the fact that collectionList.get(0) is returning the head of the parsed
+ // collections that
+ // existed when the collection list was consulted for the candidate value. If this class
+ // updates it's notion
+ // of the list of collections since candidateCollectionDesc was chosen, we could create
+ // collection n+2
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/api/collections/TimeRoutedAlias.java
##########
@@ -352,18 +373,28 @@ public void validateRouteValue(AddUpdateCommand cmd) throws SolrException {
// FUTURE: maybe in some cases the user would want to ignore/warn instead?
if (docTimestamp.isAfter(Instant.now().plusMillis(getMaxFutureMs()))) {
- throw new SolrException(BAD_REQUEST,
- "The document's time routed key of " + docTimestamp + " is too far in the future given " +
- ROUTER_MAX_FUTURE + "=" + getMaxFutureMs());
+ throw new SolrException(
+ BAD_REQUEST,
+ "The document's time routed key of "
+ + docTimestamp
+ + " is too far in the future given "
+ + ROUTER_MAX_FUTURE
+ + "="
+ + getMaxFutureMs());
}
- // Although this is also checked later, we need to check it here too to handle the case in Dimensional Routed
- // aliases where one can legally have zero collections for a newly encountered category and thus the loop later
+ // Although this is also checked later, we need to check it here too to handle the case in
+ // Dimensional Routed
+ // aliases where one can legally have zero collections for a newly encountered category and thus
+ // the loop later
// can't catch this.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/api/collections/TimeRoutedAlias.java
##########
@@ -378,20 +409,26 @@ public void validateRouteValue(AddUpdateCommand cmd) throws SolrException {
props.put(ROUTER_START, start);
// This could race, but it only occurs when the alias is first used and the values produced
- // should all be identical and who wins won't matter (baring cases of Date Math involving seconds,
+ // should all be identical and who wins won't matter (baring cases of Date Math involving
+ // seconds,
// which is pretty far fetched). Putting this in a separate thread to ensure that any failed
// races don't cause documents to get rejected.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/api/collections/TimeRoutedAlias.java
##########
@@ -78,43 +77,53 @@
// These two fields may be updated within the calling thread during processing but should
// never be updated by any async creation thread.
- private List<Map.Entry<Instant, String>> parsedCollectionsDesc; // k=timestamp (start), v=collection. Sorted descending
- private Aliases parsedCollectionsAliases; // a cached reference to the source of what we parse into parsedCollectionsDesc
+ private List<Map.Entry<Instant, String>>
+ parsedCollectionsDesc; // k=timestamp (start), v=collection. Sorted descending
+ private Aliases
+ parsedCollectionsAliases; // a cached reference to the source of what we parse into
+ // parsedCollectionsDesc
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/api/collections/TimeRoutedAlias.java
##########
@@ -455,14 +495,17 @@ public CandidateCollection findCandidateGivenValue(AddUpdateCommand cmd) {
final Instant docTimestamp = parseRouteKey(value);
- // reparse explicitly such that if we are a dimension in a DRA, the list gets culled by our context
- // This does not normally happen with the above updateParsedCollectionAliases, because at that point the aliases
+ // reparse explicitly such that if we are a dimension in a DRA, the list gets culled by our
+ // context
+ // This does not normally happen with the above updateParsedCollectionAliases, because at that
+ // point the aliases
// should be up to date and updateParsedCollectionAliases will short circuit
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/api/collections/TimeRoutedAlias.java
##########
@@ -527,33 +580,43 @@ private CandidateCollection calcCandidateCollection(Instant docTimestamp) {
}
/**
- * Deletes some of the oldest collection(s) based on {@link TimeRoutedAlias#getAutoDeleteAgeMath()}. If
- * getAutoDelteAgemath is not present then this method does nothing. Per documentation is relative to a
- * collection being created. Therefore if nothing is being created, nothing is deleted.
- * @param actions The previously calculated add action(s). This collection should not be modified within
- * this method.
+ * Deletes some of the oldest collection(s) based on {@link
+ * TimeRoutedAlias#getAutoDeleteAgeMath()}. If getAutoDelteAgemath is not present then this method
+ * does nothing. Per documentation is relative to a collection being created. Therefore if nothing
+ * is being created, nothing is deleted.
+ *
+ * @param actions The previously calculated add action(s). This collection should not be modified
+ * within this method.
*/
private List<Action> calcDeletes(List<Action> actions) {
final String autoDeleteAgeMathStr = this.getAutoDeleteAgeMath();
- if (autoDeleteAgeMathStr == null || actions .size() == 0) {
+ if (autoDeleteAgeMathStr == null || actions.size() == 0) {
return Collections.emptyList();
}
if (actions.size() > 1) {
- throw new IllegalStateException("We are not supposed to be creating more than one collection at a time");
+ throw new IllegalStateException(
+ "We are not supposed to be creating more than one collection at a time");
}
String deletionReferenceCollection = actions.get(0).targetCollection;
- Instant deletionReferenceInstant = parseInstantFromCollectionName(getAliasName(), deletionReferenceCollection);
+ Instant deletionReferenceInstant =
+ parseInstantFromCollectionName(getAliasName(), deletionReferenceCollection);
final Instant delBefore;
try {
- delBefore = new DateMathParser(Date.from(computeNextCollTimestamp(deletionReferenceInstant)), this.getTimeZone()).parseMath(autoDeleteAgeMathStr).toInstant();
+ delBefore =
+ new DateMathParser(
+ Date.from(computeNextCollTimestamp(deletionReferenceInstant)), this.getTimeZone())
+ .parseMath(autoDeleteAgeMathStr)
+ .toInstant();
} catch (ParseException e) {
- throw new SolrException(SolrException.ErrorCode.SERVER_ERROR, e); // note: should not happen by this point
+ throw new SolrException(
+ SolrException.ErrorCode.SERVER_ERROR, e); // note: should not happen by this point
}
List<Action> collectionsToDelete = new ArrayList<>();
- //iterating from newest to oldest, find the first collection that has a time <= "before". We keep this collection
+ // iterating from newest to oldest, find the first collection that has a time <= "before". We
+ // keep this collection
// (and all newer to left) but we delete older collections, which are the ones that follow.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/overseer/ReplicaMutator.java
##########
@@ -409,65 +482,84 @@ private DocCollection checkAndCompleteShardSplit(ClusterState prevState, DocColl
}
if (allActive) {
if (log.isInfoEnabled()) {
- log.info("Shard: {} - all {} replicas are active. Finding status of fellow sub-shards", sliceName, slice.getReplicasMap().size());
+ log.info(
+ "Shard: {} - all {} replicas are active. Finding status of fellow sub-shards",
+ sliceName,
+ slice.getReplicasMap().size());
}
// find out about other sub shards
Map<String, Slice> allSlicesCopy = new HashMap<>(collection.getSlicesMap());
List<Slice> subShardSlices = new ArrayList<>();
outer:
for (Map.Entry<String, Slice> entry : allSlicesCopy.entrySet()) {
- if (sliceName.equals(entry.getKey()))
- continue;
+ if (sliceName.equals(entry.getKey())) continue;
Slice otherSlice = entry.getValue();
if (otherSlice.getState() == Slice.State.RECOVERY) {
if (slice.getParent() != null && slice.getParent().equals(otherSlice.getParent())) {
if (log.isInfoEnabled()) {
- log.info("Shard: {} - Fellow sub-shard: {} found", sliceName, otherSlice.getName());
+ log.info(
+ "Shard: {} - Fellow sub-shard: {} found", sliceName, otherSlice.getName());
}
// this is a fellow sub shard so check if all replicas are active
- for (Map.Entry<String, Replica> sliceEntry : otherSlice.getReplicasMap().entrySet()) {
+ for (Map.Entry<String, Replica> sliceEntry :
+ otherSlice.getReplicasMap().entrySet()) {
if (sliceEntry.getValue().getState() != Replica.State.ACTIVE) {
allActive = false;
break outer;
}
}
if (log.isInfoEnabled()) {
- log.info("Shard: {} - Fellow sub-shard: {} has all {} replicas active", sliceName, otherSlice.getName(), otherSlice.getReplicasMap().size());
+ log.info(
+ "Shard: {} - Fellow sub-shard: {} has all {} replicas active",
+ sliceName,
+ otherSlice.getName(),
+ otherSlice.getReplicasMap().size());
}
subShardSlices.add(otherSlice);
}
}
}
if (allActive) {
// hurray, all sub shard replicas are active
- log.info("Shard: {} - All replicas across all fellow sub-shards are now ACTIVE.", sliceName);
+ log.info(
+ "Shard: {} - All replicas across all fellow sub-shards are now ACTIVE.", sliceName);
String parentSliceName = (String) sliceProps.remove(Slice.PARENT);
- // now lets see if the parent leader is still the same or else there's a chance of data loss
+ // now lets see if the parent leader is still the same or else there's a chance of data
+ // loss
// see SOLR-9438 for details
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/overseer/ClusterStateMutator.java
##########
@@ -63,14 +62,16 @@ public ZkWriteCommand createCollection(ClusterState clusterState, ZkNodeProps me
}
Map<String, Object> routerSpec = DocRouter.getRouterSpec(message);
- String routerName = routerSpec.get(NAME) == null ? DocRouter.DEFAULT_NAME : (String) routerSpec.get(NAME);
+ String routerName =
+ routerSpec.get(NAME) == null ? DocRouter.DEFAULT_NAME : (String) routerSpec.get(NAME);
DocRouter router = DocRouter.getDocRouter(routerName);
Object messageShardsObj = message.get("shards");
Map<String, Slice> slices;
- if (messageShardsObj instanceof Map) { // we are being explicitly told the slice data (e.g. coll restore)
- slices = Slice.loadAllFromMap(cName, (Map<String, Object>)messageShardsObj);
+ if (messageShardsObj
+ instanceof Map) { // we are being explicitly told the slice data (e.g. coll restore)
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/api/collections/TimeRoutedAlias.java
##########
@@ -484,37 +527,47 @@ private CandidateCollection calcCandidateCollection(Instant docTimestamp) {
if (i == 0) {
next = computeNextCollTimestamp(colStartTime);
}
- if (!docTimestamp.isBefore(colStartTime)) { // (inclusive lower bound)
+ if (!docTimestamp.isBefore(colStartTime)) { // (inclusive lower bound)
CandidateCollection candidate;
if (i == 0) {
- if (docTimestamp.isBefore(next)) { // (exclusive upper bound)
- candidate = new CandidateCollection(NONE, entry.getValue()); //found it
+ if (docTimestamp.isBefore(next)) { // (exclusive upper bound)
+ candidate = new CandidateCollection(NONE, entry.getValue()); // found it
// simply goes to head collection no action required
} else {
- // Although we create collections one at a time, this calculation of the ultimate destination is
+ // Although we create collections one at a time, this calculation of the ultimate
+ // destination is
// useful for contextualizing TRA's used as dimensions in DRA's
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/overseer/CollectionMutator.java
##########
@@ -109,27 +113,30 @@ public ZkWriteCommand modifyCollection(final ClusterState clusterState, ZkNodePr
if (val == null) continue;
boolean enable = Boolean.parseBoolean(val);
if (enable == coll.isPerReplicaState()) {
- //already enabled
+ // already enabled
log.error("trying to set perReplicaState to {} from {}", val, coll.isPerReplicaState());
continue;
}
- replicaOps = PerReplicaStatesOps.modifyCollection(coll, enable, PerReplicaStates.fetch(coll.getZNode(), zkClient, null));
+ replicaOps =
+ PerReplicaStatesOps.modifyCollection(
+ coll, enable, PerReplicaStates.fetch(coll.getZNode(), zkClient, null));
}
-
if (message.containsKey(prop)) {
hasAnyOps = true;
- if (message.get(prop) == null) {
+ if (message.get(prop) == null) {
props.remove(prop);
- } else {
+ } else {
// rename key from collection.configName to configName
if (prop.equals(COLL_CONF)) {
props.put(CONFIGNAME_PROP, message.get(prop));
} else {
props.put(prop, message.get(prop));
}
}
- if (prop == REPLICATION_FACTOR) { //SOLR-11676 : keep NRT_REPLICAS and REPLICATION_FACTOR in sync
+ if (prop
+ == REPLICATION_FACTOR) { // SOLR-11676 : keep NRT_REPLICAS and REPLICATION_FACTOR in
+ // sync
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cluster/placement/impl/AttributeFetcherImpl.java
##########
@@ -92,41 +92,60 @@ public AttributeValues fetchAttributes() {
Map<NodeMetric<?>, Map<Node, Object>> metricSnitchToNodeToValue = new HashMap<>();
Map<String, CollectionMetricsBuilder> collectionMetricsBuilders = new HashMap<>();
Map<Node, Set<String>> nodeToReplicaInternalTags = new HashMap<>();
- Map<String, Set<ReplicaMetric<?>>> requestedCollectionNamesMetrics = requestedCollectionMetrics.entrySet().stream()
- .collect(Collectors.toMap(e -> e.getKey().getName(), e -> e.getValue()));
-
- // In order to match the returned values for the various snitches, we need to keep track of where each
- // received value goes. Given the target maps are of different types (the maps from Node to whatever defined
- // above) we instead pass a function taking two arguments, the node and the (non null) returned value,
- // that will cast the value into the appropriate type for the snitch tag and insert it into the appropriate map
+ Map<String, Set<ReplicaMetric<?>>> requestedCollectionNamesMetrics =
+ requestedCollectionMetrics.entrySet().stream()
+ .collect(Collectors.toMap(e -> e.getKey().getName(), e -> e.getValue()));
+
+ // In order to match the returned values for the various snitches, we need to keep track of
+ // where each
+ // received value goes. Given the target maps are of different types (the maps from Node to
+ // whatever defined
+ // above) we instead pass a function taking two arguments, the node and the (non null) returned
+ // value,
+ // that will cast the value into the appropriate type for the snitch tag and insert it into the
+ // appropriate map
// with the node as the key.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cluster/placement/impl/DelegatingPlacementPluginFactory.java
##########
@@ -55,7 +53,8 @@ public void setDelegate(PlacementPluginFactory<? extends PlacementPluginConfig>
Phaser localPhaser = phaser; // volatile read
if (localPhaser != null) {
assert localPhaser.getRegisteredParties() == 1;
- localPhaser.arrive(); // we should be the only ones registered, so this will advance phase each time
+ localPhaser
+ .arrive(); // we should be the only ones registered, so this will advance phase each time
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cluster/events/impl/DelegatingClusterEventProducer.java
##########
@@ -104,27 +106,33 @@ public void setDelegate(ClusterEventProducer newDelegate) {
Phaser localPhaser = phaser; // volatile read
if (localPhaser != null) {
assert localPhaser.getRegisteredParties() == 1;
- localPhaser.arrive(); // we should be the only ones registered, so this will advance phase each time
+ localPhaser
+ .arrive(); // we should be the only ones registered, so this will advance phase each time
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cloud/overseer/ZkWriteCommand.java
##########
@@ -33,12 +31,19 @@
public final boolean persistJsonState;
public final PerReplicaStatesOps ops;
- public ZkWriteCommand(String name, DocCollection collection, PerReplicaStatesOps replicaOps, boolean persistJsonState) {
+ public ZkWriteCommand(
+ String name,
+ DocCollection collection,
+ PerReplicaStatesOps replicaOps,
+ boolean persistJsonState) {
isPerReplicaStateCollection = collection != null && collection.isPerReplicaState();
this.name = name;
this.collection = collection;
this.ops = replicaOps;
- this.persistJsonState = persistJsonState || !isPerReplicaStateCollection; // Always persist for non "per replica state" collections
+ this.persistJsonState =
+ persistJsonState
+ || !isPerReplicaStateCollection; // Always persist for non "per replica state"
+ // collections
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cluster/placement/plugins/AffinityPlacementFactory.java
##########
@@ -230,61 +251,90 @@ private AffinityPlacementPlugin(long minimalFreeDiskGB, long prioritizedFreeDisk
SolrCollection solrCollection = request.getCollection();
// filter out nodes that don't meet the `withCollection` constraint
- nodes = filterNodesWithCollection(placementContext.getCluster(), request, attrValues, nodes);
+ nodes =
+ filterNodesWithCollection(placementContext.getCluster(), request, attrValues, nodes);
// filter out nodes that don't match the "node types" specified in the collection props
nodes = filterNodesByNodeType(placementContext.getCluster(), request, attrValues, nodes);
- // Split the set of nodes into 3 sets of nodes accepting each replica type (sets can overlap if nodes accept multiple replica types)
- // These subsets sets are actually maps, because we capture the number of cores (of any replica type) present on each node.
- EnumMap<Replica.ReplicaType, Set<Node>> replicaTypeToNodes = getAvailableNodesForReplicaTypes(nodes, attrValues);
-
- // All available zones of live nodes. Due to some nodes not being candidates for placement, and some existing replicas
- // being one availability zones that might be offline (i.e. their nodes are not live), this set might contain zones
+ // Split the set of nodes into 3 sets of nodes accepting each replica type (sets can overlap
+ // if nodes accept multiple replica types)
+ // These subsets sets are actually maps, because we capture the number of cores (of any
+ // replica type) present on each node.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cluster/placement/plugins/AffinityPlacementFactory.java
##########
@@ -230,61 +251,90 @@ private AffinityPlacementPlugin(long minimalFreeDiskGB, long prioritizedFreeDisk
SolrCollection solrCollection = request.getCollection();
// filter out nodes that don't meet the `withCollection` constraint
- nodes = filterNodesWithCollection(placementContext.getCluster(), request, attrValues, nodes);
+ nodes =
+ filterNodesWithCollection(placementContext.getCluster(), request, attrValues, nodes);
// filter out nodes that don't match the "node types" specified in the collection props
nodes = filterNodesByNodeType(placementContext.getCluster(), request, attrValues, nodes);
- // Split the set of nodes into 3 sets of nodes accepting each replica type (sets can overlap if nodes accept multiple replica types)
- // These subsets sets are actually maps, because we capture the number of cores (of any replica type) present on each node.
- EnumMap<Replica.ReplicaType, Set<Node>> replicaTypeToNodes = getAvailableNodesForReplicaTypes(nodes, attrValues);
-
- // All available zones of live nodes. Due to some nodes not being candidates for placement, and some existing replicas
- // being one availability zones that might be offline (i.e. their nodes are not live), this set might contain zones
+ // Split the set of nodes into 3 sets of nodes accepting each replica type (sets can overlap
+ // if nodes accept multiple replica types)
+ // These subsets sets are actually maps, because we capture the number of cores (of any
+ // replica type) present on each node.
+ EnumMap<Replica.ReplicaType, Set<Node>> replicaTypeToNodes =
+ getAvailableNodesForReplicaTypes(nodes, attrValues);
+
+ // All available zones of live nodes. Due to some nodes not being candidates for placement,
+ // and some existing replicas
+ // being one availability zones that might be offline (i.e. their nodes are not live), this
+ // set might contain zones
// on which it is impossible to place replicas. That's ok.
Set<String> availabilityZones = getZonesFromNodes(nodes, attrValues);
// Build the replica placement decisions here
Set<ReplicaPlacement> replicaPlacements = new HashSet<>();
- // Let's now iterate on all shards to create replicas for and start finding home sweet homes for the replicas
+ // Let's now iterate on all shards to create replicas for and start finding home sweet homes
+ // for the replicas
for (String shardName : request.getShardNames()) {
- // Inventory nodes (if any) that already have a replica of any type for the shard, because we can't be placing
- // additional replicas on these. This data structure is updated after each replica to node assign and is used to
- // make sure different replica types are not allocated to the same nodes (protecting same node assignments within
+ // Inventory nodes (if any) that already have a replica of any type for the shard, because
+ // we can't be placing
+ // additional replicas on these. This data structure is updated after each replica to node
+ // assign and is used to
+ // make sure different replica types are not allocated to the same nodes (protecting same
+ // node assignments within
// a given replica type is done "by construction" in makePlacementDecisions()).
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cluster/placement/plugins/AffinityPlacementFactory.java
##########
@@ -230,61 +251,90 @@ private AffinityPlacementPlugin(long minimalFreeDiskGB, long prioritizedFreeDisk
SolrCollection solrCollection = request.getCollection();
// filter out nodes that don't meet the `withCollection` constraint
- nodes = filterNodesWithCollection(placementContext.getCluster(), request, attrValues, nodes);
+ nodes =
+ filterNodesWithCollection(placementContext.getCluster(), request, attrValues, nodes);
// filter out nodes that don't match the "node types" specified in the collection props
nodes = filterNodesByNodeType(placementContext.getCluster(), request, attrValues, nodes);
- // Split the set of nodes into 3 sets of nodes accepting each replica type (sets can overlap if nodes accept multiple replica types)
- // These subsets sets are actually maps, because we capture the number of cores (of any replica type) present on each node.
- EnumMap<Replica.ReplicaType, Set<Node>> replicaTypeToNodes = getAvailableNodesForReplicaTypes(nodes, attrValues);
-
- // All available zones of live nodes. Due to some nodes not being candidates for placement, and some existing replicas
- // being one availability zones that might be offline (i.e. their nodes are not live), this set might contain zones
+ // Split the set of nodes into 3 sets of nodes accepting each replica type (sets can overlap
+ // if nodes accept multiple replica types)
+ // These subsets sets are actually maps, because we capture the number of cores (of any
+ // replica type) present on each node.
+ EnumMap<Replica.ReplicaType, Set<Node>> replicaTypeToNodes =
+ getAvailableNodesForReplicaTypes(nodes, attrValues);
+
+ // All available zones of live nodes. Due to some nodes not being candidates for placement,
+ // and some existing replicas
+ // being one availability zones that might be offline (i.e. their nodes are not live), this
+ // set might contain zones
// on which it is impossible to place replicas. That's ok.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cluster/placement/plugins/AffinityPlacementFactory.java
##########
@@ -402,44 +484,52 @@ private String getNodeAZ(Node n, final AttributeValues attrValues) {
AzWithNodes(String azName, List<Node> availableNodesForPlacement) {
this.azName = azName;
this.availableNodesForPlacement = availableNodesForPlacement;
- // Once the list is sorted to an order we're happy with, this flag is set to true to avoid sorting multiple times
+ // Once the list is sorted to an order we're happy with, this flag is set to true to avoid
+ // sorting multiple times
// unnecessarily.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cluster/placement/plugins/AffinityPlacementFactory.java
##########
@@ -488,96 +600,139 @@ private String getNodeAZ(Node n, final AttributeValues attrValues) {
}
/**
- * <p>Picks nodes from {@code targetNodes} for placing {@code numReplicas} replicas.
+ * Picks nodes from {@code targetNodes} for placing {@code numReplicas} replicas.
*
* <p>The criteria used in this method are, in this order:
+ *
* <ol>
- * <li>No more than one replica of a given shard on a given node (strictly enforced)</li>
- * <li>Balance as much as possible replicas of a given {@link org.apache.solr.cluster.Replica.ReplicaType} over available AZ's.
- * This balancing takes into account existing replicas <b>of the corresponding replica type</b>, if any.</li>
- * <li>Place replicas if possible on nodes having more than a certain amount of free disk space (note that nodes with a too small
- * amount of free disk space were eliminated as placement targets earlier, in {@link #getAvailableNodesForReplicaTypes(Set, AttributeValues)}). There's
- * a threshold here rather than sorting on the amount of free disk space, because sorting on that value would in
- * practice lead to never considering the number of cores on a node.</li>
- * <li>Place replicas on nodes having a smaller number of cores (the number of cores considered
- * for this decision includes previous placement decisions made during the processing of the placement request)</li>
+ * <li>No more than one replica of a given shard on a given node (strictly enforced)
+ * <li>Balance as much as possible replicas of a given {@link
+ * org.apache.solr.cluster.Replica.ReplicaType} over available AZ's. This balancing takes
+ * into account existing replicas <b>of the corresponding replica type</b>, if any.
+ * <li>Place replicas if possible on nodes having more than a certain amount of free disk
+ * space (note that nodes with a too small amount of free disk space were eliminated as
+ * placement targets earlier, in {@link #getAvailableNodesForReplicaTypes(Set,
+ * AttributeValues)}). There's a threshold here rather than sorting on the amount of free
+ * disk space, because sorting on that value would in practice lead to never considering
+ * the number of cores on a node.
+ * <li>Place replicas on nodes having a smaller number of cores (the number of cores
+ * considered for this decision includes previous placement decisions made during the
+ * processing of the placement request)
* </ol>
*/
- @SuppressForbidden(reason = "Ordering.arbitrary() has no equivalent in Comparator class. Rather reuse than copy.")
- private void makePlacementDecisions(SolrCollection solrCollection, String shardName, Set<String> availabilityZones,
- Replica.ReplicaType replicaType, int numReplicas, final AttributeValues attrValues,
- EnumMap<Replica.ReplicaType, Set<Node>> replicaTypeToNodes, Set<Node> nodesWithReplicas,
- Map<Node, Integer> coresOnNodes, PlacementPlanFactory placementPlanFactory,
- Set<ReplicaPlacement> replicaPlacements) throws PlacementException {
- // Count existing replicas per AZ. We count only instances of the type of replica for which we need to do placement.
- // If we ever want to balance replicas of any type across AZ's (and not each replica type balanced independently),
- // we'd have to move this data structure to the caller of this method so it can be reused across different replica
- // type placements for a given shard. Note then that this change would be risky. For example all NRT's and PULL
- // replicas for a shard my be correctly balanced over three AZ's, but then all NRT can end up in the same AZ...
+ @SuppressForbidden(
+ reason =
+ "Ordering.arbitrary() has no equivalent in Comparator class. Rather reuse than copy.")
+ private void makePlacementDecisions(
+ SolrCollection solrCollection,
+ String shardName,
+ Set<String> availabilityZones,
+ Replica.ReplicaType replicaType,
+ int numReplicas,
+ final AttributeValues attrValues,
+ EnumMap<Replica.ReplicaType, Set<Node>> replicaTypeToNodes,
+ Set<Node> nodesWithReplicas,
+ Map<Node, Integer> coresOnNodes,
+ PlacementPlanFactory placementPlanFactory,
+ Set<ReplicaPlacement> replicaPlacements)
+ throws PlacementException {
+ // Count existing replicas per AZ. We count only instances of the type of replica for which we
+ // need to do placement.
+ // If we ever want to balance replicas of any type across AZ's (and not each replica type
+ // balanced independently),
+ // we'd have to move this data structure to the caller of this method so it can be reused
+ // across different replica
+ // type placements for a given shard. Note then that this change would be risky. For example
+ // all NRT's and PULL
+ // replicas for a shard my be correctly balanced over three AZ's, but then all NRT can end up
+ // in the same AZ...
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cluster/placement/plugins/AffinityPlacementFactory.java
##########
@@ -230,61 +251,90 @@ private AffinityPlacementPlugin(long minimalFreeDiskGB, long prioritizedFreeDisk
SolrCollection solrCollection = request.getCollection();
// filter out nodes that don't meet the `withCollection` constraint
- nodes = filterNodesWithCollection(placementContext.getCluster(), request, attrValues, nodes);
+ nodes =
+ filterNodesWithCollection(placementContext.getCluster(), request, attrValues, nodes);
// filter out nodes that don't match the "node types" specified in the collection props
nodes = filterNodesByNodeType(placementContext.getCluster(), request, attrValues, nodes);
- // Split the set of nodes into 3 sets of nodes accepting each replica type (sets can overlap if nodes accept multiple replica types)
- // These subsets sets are actually maps, because we capture the number of cores (of any replica type) present on each node.
- EnumMap<Replica.ReplicaType, Set<Node>> replicaTypeToNodes = getAvailableNodesForReplicaTypes(nodes, attrValues);
-
- // All available zones of live nodes. Due to some nodes not being candidates for placement, and some existing replicas
- // being one availability zones that might be offline (i.e. their nodes are not live), this set might contain zones
+ // Split the set of nodes into 3 sets of nodes accepting each replica type (sets can overlap
+ // if nodes accept multiple replica types)
+ // These subsets sets are actually maps, because we capture the number of cores (of any
+ // replica type) present on each node.
+ EnumMap<Replica.ReplicaType, Set<Node>> replicaTypeToNodes =
+ getAvailableNodesForReplicaTypes(nodes, attrValues);
+
+ // All available zones of live nodes. Due to some nodes not being candidates for placement,
+ // and some existing replicas
+ // being one availability zones that might be offline (i.e. their nodes are not live), this
+ // set might contain zones
// on which it is impossible to place replicas. That's ok.
Set<String> availabilityZones = getZonesFromNodes(nodes, attrValues);
// Build the replica placement decisions here
Set<ReplicaPlacement> replicaPlacements = new HashSet<>();
- // Let's now iterate on all shards to create replicas for and start finding home sweet homes for the replicas
+ // Let's now iterate on all shards to create replicas for and start finding home sweet homes
+ // for the replicas
for (String shardName : request.getShardNames()) {
- // Inventory nodes (if any) that already have a replica of any type for the shard, because we can't be placing
- // additional replicas on these. This data structure is updated after each replica to node assign and is used to
- // make sure different replica types are not allocated to the same nodes (protecting same node assignments within
+ // Inventory nodes (if any) that already have a replica of any type for the shard, because
+ // we can't be placing
+ // additional replicas on these. This data structure is updated after each replica to node
+ // assign and is used to
+ // make sure different replica types are not allocated to the same nodes (protecting same
+ // node assignments within
// a given replica type is done "by construction" in makePlacementDecisions()).
Set<Node> nodesWithReplicas =
allNodesWithReplicas
.computeIfAbsent(solrCollection.getName(), col -> new HashMap<>())
- .computeIfAbsent(shardName, s -> {
- Set<Node> newNodeSet = new HashSet<>();
- Shard shard = solrCollection.getShard(s);
- if (shard != null) {
- // Prefill the set with the existing replicas
- for (Replica r : shard.replicas()) {
- newNodeSet.add(r.getNode());
- }
- }
- return newNodeSet;
- });
-
+ .computeIfAbsent(
+ shardName,
+ s -> {
+ Set<Node> newNodeSet = new HashSet<>();
+ Shard shard = solrCollection.getShard(s);
+ if (shard != null) {
+ // Prefill the set with the existing replicas
+ for (Replica r : shard.replicas()) {
+ newNodeSet.add(r.getNode());
+ }
+ }
+ return newNodeSet;
+ });
// Iterate on the replica types in the enum order. We place more strategic replicas first
- // (NRT is more strategic than TLOG more strategic than PULL). This is in case we eventually decide that less
- // strategic replica placement impossibility is not a problem that should lead to replica placement computation
- // failure. Current code does fail if placement is impossible (constraint is at most one replica of a shard on any node).
+ // (NRT is more strategic than TLOG more strategic than PULL). This is in case we
+ // eventually decide that less
+ // strategic replica placement impossibility is not a problem that should lead to replica
+ // placement computation
+ // failure. Current code does fail if placement is impossible (constraint is at most one
+ // replica of a shard on any node).
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cluster/placement/plugins/AffinityPlacementFactory.java
##########
@@ -593,44 +748,62 @@ private void makePlacementDecisions(SolrCollection solrCollection, String shardN
break;
}
candidateAzEntries.add(entry);
- // We remove all entries that are candidates: the "winner" will be modified, all entries might also be sorted,
+ // We remove all entries that are candidates: the "winner" will be modified, all entries
+ // might also be sorted,
// so we'll insert back the updated versions later.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cluster/placement/plugins/AffinityPlacementFactory.java
##########
@@ -488,96 +600,139 @@ private String getNodeAZ(Node n, final AttributeValues attrValues) {
}
/**
- * <p>Picks nodes from {@code targetNodes} for placing {@code numReplicas} replicas.
+ * Picks nodes from {@code targetNodes} for placing {@code numReplicas} replicas.
*
* <p>The criteria used in this method are, in this order:
+ *
* <ol>
- * <li>No more than one replica of a given shard on a given node (strictly enforced)</li>
- * <li>Balance as much as possible replicas of a given {@link org.apache.solr.cluster.Replica.ReplicaType} over available AZ's.
- * This balancing takes into account existing replicas <b>of the corresponding replica type</b>, if any.</li>
- * <li>Place replicas if possible on nodes having more than a certain amount of free disk space (note that nodes with a too small
- * amount of free disk space were eliminated as placement targets earlier, in {@link #getAvailableNodesForReplicaTypes(Set, AttributeValues)}). There's
- * a threshold here rather than sorting on the amount of free disk space, because sorting on that value would in
- * practice lead to never considering the number of cores on a node.</li>
- * <li>Place replicas on nodes having a smaller number of cores (the number of cores considered
- * for this decision includes previous placement decisions made during the processing of the placement request)</li>
+ * <li>No more than one replica of a given shard on a given node (strictly enforced)
+ * <li>Balance as much as possible replicas of a given {@link
+ * org.apache.solr.cluster.Replica.ReplicaType} over available AZ's. This balancing takes
+ * into account existing replicas <b>of the corresponding replica type</b>, if any.
+ * <li>Place replicas if possible on nodes having more than a certain amount of free disk
+ * space (note that nodes with a too small amount of free disk space were eliminated as
+ * placement targets earlier, in {@link #getAvailableNodesForReplicaTypes(Set,
+ * AttributeValues)}). There's a threshold here rather than sorting on the amount of free
+ * disk space, because sorting on that value would in practice lead to never considering
+ * the number of cores on a node.
+ * <li>Place replicas on nodes having a smaller number of cores (the number of cores
+ * considered for this decision includes previous placement decisions made during the
+ * processing of the placement request)
* </ol>
*/
- @SuppressForbidden(reason = "Ordering.arbitrary() has no equivalent in Comparator class. Rather reuse than copy.")
- private void makePlacementDecisions(SolrCollection solrCollection, String shardName, Set<String> availabilityZones,
- Replica.ReplicaType replicaType, int numReplicas, final AttributeValues attrValues,
- EnumMap<Replica.ReplicaType, Set<Node>> replicaTypeToNodes, Set<Node> nodesWithReplicas,
- Map<Node, Integer> coresOnNodes, PlacementPlanFactory placementPlanFactory,
- Set<ReplicaPlacement> replicaPlacements) throws PlacementException {
- // Count existing replicas per AZ. We count only instances of the type of replica for which we need to do placement.
- // If we ever want to balance replicas of any type across AZ's (and not each replica type balanced independently),
- // we'd have to move this data structure to the caller of this method so it can be reused across different replica
- // type placements for a given shard. Note then that this change would be risky. For example all NRT's and PULL
- // replicas for a shard my be correctly balanced over three AZ's, but then all NRT can end up in the same AZ...
+ @SuppressForbidden(
+ reason =
+ "Ordering.arbitrary() has no equivalent in Comparator class. Rather reuse than copy.")
+ private void makePlacementDecisions(
+ SolrCollection solrCollection,
+ String shardName,
+ Set<String> availabilityZones,
+ Replica.ReplicaType replicaType,
+ int numReplicas,
+ final AttributeValues attrValues,
+ EnumMap<Replica.ReplicaType, Set<Node>> replicaTypeToNodes,
+ Set<Node> nodesWithReplicas,
+ Map<Node, Integer> coresOnNodes,
+ PlacementPlanFactory placementPlanFactory,
+ Set<ReplicaPlacement> replicaPlacements)
+ throws PlacementException {
+ // Count existing replicas per AZ. We count only instances of the type of replica for which we
+ // need to do placement.
+ // If we ever want to balance replicas of any type across AZ's (and not each replica type
+ // balanced independently),
+ // we'd have to move this data structure to the caller of this method so it can be reused
+ // across different replica
+ // type placements for a given shard. Note then that this change would be risky. For example
+ // all NRT's and PULL
+ // replicas for a shard my be correctly balanced over three AZ's, but then all NRT can end up
+ // in the same AZ...
Map<String, Integer> azToNumReplicas = new HashMap<>();
for (String az : availabilityZones) {
azToNumReplicas.put(az, 0);
}
- // Build the set of candidate nodes for the placement, i.e. nodes that can accept the replica type
+ // Build the set of candidate nodes for the placement, i.e. nodes that can accept the replica
+ // type
Set<Node> candidateNodes = new HashSet<>(replicaTypeToNodes.get(replicaType));
- // Remove nodes that already have a replica for the shard (no two replicas of same shard can be put on same node)
+ // Remove nodes that already have a replica for the shard (no two replicas of same shard can
+ // be put on same node)
candidateNodes.removeAll(nodesWithReplicas);
Shard shard = solrCollection.getShard(shardName);
if (shard != null) {
// shard is non null if we're adding replicas to an already existing collection.
// If we're creating the collection, the shards do not exist yet.
for (Replica replica : shard.replicas()) {
- // The node's AZ is counted as having a replica if it has a replica of the same type as the one we need
+ // The node's AZ is counted as having a replica if it has a replica of the same type as
+ // the one we need
// to place here.
if (replica.getType() == replicaType) {
final String az = getNodeAZ(replica.getNode(), attrValues);
if (azToNumReplicas.containsKey(az)) {
- // We do not count replicas on AZ's for which we don't have any node to place on because it would not help
- // the placement decision. If we did want to do that, note the dereferencing below can't be assumed as the
+ // We do not count replicas on AZ's for which we don't have any node to place on
+ // because it would not help
+ // the placement decision. If we did want to do that, note the dereferencing below
+ // can't be assumed as the
// entry will not exist in the map.
azToNumReplicas.put(az, azToNumReplicas.get(az) + 1);
}
}
}
}
- // We now have the set of real candidate nodes, we've enforced "No more than one replica of a given shard on a given node".
- // We also counted for the shard and replica type under consideration how many replicas were per AZ, so we can place
+ // We now have the set of real candidate nodes, we've enforced "No more than one replica of a
+ // given shard on a given node".
+ // We also counted for the shard and replica type under consideration how many replicas were
+ // per AZ, so we can place
// (or try to place) replicas on AZ's that have fewer replicas
- // Get the candidate nodes per AZ in order to build (further down) a mapping of AZ to placement candidates.
+ // Get the candidate nodes per AZ in order to build (further down) a mapping of AZ to
+ // placement candidates.
Map<String, List<Node>> nodesPerAz = new HashMap<>();
for (Node node : candidateNodes) {
String nodeAz = getNodeAZ(node, attrValues);
List<Node> nodesForAz = nodesPerAz.computeIfAbsent(nodeAz, k -> new ArrayList<>());
nodesForAz.add(node);
}
- // Build a treeMap sorted by the number of replicas per AZ and including candidates nodes suitable for placement on the
- // AZ, so we can easily select the next AZ to get a replica assignment and quickly (constant time) decide if placement
+ // Build a treeMap sorted by the number of replicas per AZ and including candidates nodes
+ // suitable for placement on the
+ // AZ, so we can easily select the next AZ to get a replica assignment and quickly (constant
+ // time) decide if placement
// on this AZ is possible or not.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cluster/placement/plugins/AffinityPlacementFactory.java
##########
@@ -488,96 +600,139 @@ private String getNodeAZ(Node n, final AttributeValues attrValues) {
}
/**
- * <p>Picks nodes from {@code targetNodes} for placing {@code numReplicas} replicas.
+ * Picks nodes from {@code targetNodes} for placing {@code numReplicas} replicas.
*
* <p>The criteria used in this method are, in this order:
+ *
* <ol>
- * <li>No more than one replica of a given shard on a given node (strictly enforced)</li>
- * <li>Balance as much as possible replicas of a given {@link org.apache.solr.cluster.Replica.ReplicaType} over available AZ's.
- * This balancing takes into account existing replicas <b>of the corresponding replica type</b>, if any.</li>
- * <li>Place replicas if possible on nodes having more than a certain amount of free disk space (note that nodes with a too small
- * amount of free disk space were eliminated as placement targets earlier, in {@link #getAvailableNodesForReplicaTypes(Set, AttributeValues)}). There's
- * a threshold here rather than sorting on the amount of free disk space, because sorting on that value would in
- * practice lead to never considering the number of cores on a node.</li>
- * <li>Place replicas on nodes having a smaller number of cores (the number of cores considered
- * for this decision includes previous placement decisions made during the processing of the placement request)</li>
+ * <li>No more than one replica of a given shard on a given node (strictly enforced)
+ * <li>Balance as much as possible replicas of a given {@link
+ * org.apache.solr.cluster.Replica.ReplicaType} over available AZ's. This balancing takes
+ * into account existing replicas <b>of the corresponding replica type</b>, if any.
+ * <li>Place replicas if possible on nodes having more than a certain amount of free disk
+ * space (note that nodes with a too small amount of free disk space were eliminated as
+ * placement targets earlier, in {@link #getAvailableNodesForReplicaTypes(Set,
+ * AttributeValues)}). There's a threshold here rather than sorting on the amount of free
+ * disk space, because sorting on that value would in practice lead to never considering
+ * the number of cores on a node.
+ * <li>Place replicas on nodes having a smaller number of cores (the number of cores
+ * considered for this decision includes previous placement decisions made during the
+ * processing of the placement request)
* </ol>
*/
- @SuppressForbidden(reason = "Ordering.arbitrary() has no equivalent in Comparator class. Rather reuse than copy.")
- private void makePlacementDecisions(SolrCollection solrCollection, String shardName, Set<String> availabilityZones,
- Replica.ReplicaType replicaType, int numReplicas, final AttributeValues attrValues,
- EnumMap<Replica.ReplicaType, Set<Node>> replicaTypeToNodes, Set<Node> nodesWithReplicas,
- Map<Node, Integer> coresOnNodes, PlacementPlanFactory placementPlanFactory,
- Set<ReplicaPlacement> replicaPlacements) throws PlacementException {
- // Count existing replicas per AZ. We count only instances of the type of replica for which we need to do placement.
- // If we ever want to balance replicas of any type across AZ's (and not each replica type balanced independently),
- // we'd have to move this data structure to the caller of this method so it can be reused across different replica
- // type placements for a given shard. Note then that this change would be risky. For example all NRT's and PULL
- // replicas for a shard my be correctly balanced over three AZ's, but then all NRT can end up in the same AZ...
+ @SuppressForbidden(
+ reason =
+ "Ordering.arbitrary() has no equivalent in Comparator class. Rather reuse than copy.")
+ private void makePlacementDecisions(
+ SolrCollection solrCollection,
+ String shardName,
+ Set<String> availabilityZones,
+ Replica.ReplicaType replicaType,
+ int numReplicas,
+ final AttributeValues attrValues,
+ EnumMap<Replica.ReplicaType, Set<Node>> replicaTypeToNodes,
+ Set<Node> nodesWithReplicas,
+ Map<Node, Integer> coresOnNodes,
+ PlacementPlanFactory placementPlanFactory,
+ Set<ReplicaPlacement> replicaPlacements)
+ throws PlacementException {
+ // Count existing replicas per AZ. We count only instances of the type of replica for which we
+ // need to do placement.
+ // If we ever want to balance replicas of any type across AZ's (and not each replica type
+ // balanced independently),
+ // we'd have to move this data structure to the caller of this method so it can be reused
+ // across different replica
+ // type placements for a given shard. Note then that this change would be risky. For example
+ // all NRT's and PULL
+ // replicas for a shard my be correctly balanced over three AZ's, but then all NRT can end up
+ // in the same AZ...
Map<String, Integer> azToNumReplicas = new HashMap<>();
for (String az : availabilityZones) {
azToNumReplicas.put(az, 0);
}
- // Build the set of candidate nodes for the placement, i.e. nodes that can accept the replica type
+ // Build the set of candidate nodes for the placement, i.e. nodes that can accept the replica
+ // type
Set<Node> candidateNodes = new HashSet<>(replicaTypeToNodes.get(replicaType));
- // Remove nodes that already have a replica for the shard (no two replicas of same shard can be put on same node)
+ // Remove nodes that already have a replica for the shard (no two replicas of same shard can
+ // be put on same node)
candidateNodes.removeAll(nodesWithReplicas);
Shard shard = solrCollection.getShard(shardName);
if (shard != null) {
// shard is non null if we're adding replicas to an already existing collection.
// If we're creating the collection, the shards do not exist yet.
for (Replica replica : shard.replicas()) {
- // The node's AZ is counted as having a replica if it has a replica of the same type as the one we need
+ // The node's AZ is counted as having a replica if it has a replica of the same type as
+ // the one we need
// to place here.
if (replica.getType() == replicaType) {
final String az = getNodeAZ(replica.getNode(), attrValues);
if (azToNumReplicas.containsKey(az)) {
- // We do not count replicas on AZ's for which we don't have any node to place on because it would not help
- // the placement decision. If we did want to do that, note the dereferencing below can't be assumed as the
+ // We do not count replicas on AZ's for which we don't have any node to place on
+ // because it would not help
+ // the placement decision. If we did want to do that, note the dereferencing below
+ // can't be assumed as the
// entry will not exist in the map.
azToNumReplicas.put(az, azToNumReplicas.get(az) + 1);
}
}
}
}
- // We now have the set of real candidate nodes, we've enforced "No more than one replica of a given shard on a given node".
- // We also counted for the shard and replica type under consideration how many replicas were per AZ, so we can place
+ // We now have the set of real candidate nodes, we've enforced "No more than one replica of a
+ // given shard on a given node".
+ // We also counted for the shard and replica type under consideration how many replicas were
+ // per AZ, so we can place
// (or try to place) replicas on AZ's that have fewer replicas
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cluster/placement/plugins/AffinityPlacementFactory.java
##########
@@ -488,96 +600,139 @@ private String getNodeAZ(Node n, final AttributeValues attrValues) {
}
/**
- * <p>Picks nodes from {@code targetNodes} for placing {@code numReplicas} replicas.
+ * Picks nodes from {@code targetNodes} for placing {@code numReplicas} replicas.
*
* <p>The criteria used in this method are, in this order:
+ *
* <ol>
- * <li>No more than one replica of a given shard on a given node (strictly enforced)</li>
- * <li>Balance as much as possible replicas of a given {@link org.apache.solr.cluster.Replica.ReplicaType} over available AZ's.
- * This balancing takes into account existing replicas <b>of the corresponding replica type</b>, if any.</li>
- * <li>Place replicas if possible on nodes having more than a certain amount of free disk space (note that nodes with a too small
- * amount of free disk space were eliminated as placement targets earlier, in {@link #getAvailableNodesForReplicaTypes(Set, AttributeValues)}). There's
- * a threshold here rather than sorting on the amount of free disk space, because sorting on that value would in
- * practice lead to never considering the number of cores on a node.</li>
- * <li>Place replicas on nodes having a smaller number of cores (the number of cores considered
- * for this decision includes previous placement decisions made during the processing of the placement request)</li>
+ * <li>No more than one replica of a given shard on a given node (strictly enforced)
+ * <li>Balance as much as possible replicas of a given {@link
+ * org.apache.solr.cluster.Replica.ReplicaType} over available AZ's. This balancing takes
+ * into account existing replicas <b>of the corresponding replica type</b>, if any.
+ * <li>Place replicas if possible on nodes having more than a certain amount of free disk
+ * space (note that nodes with a too small amount of free disk space were eliminated as
+ * placement targets earlier, in {@link #getAvailableNodesForReplicaTypes(Set,
+ * AttributeValues)}). There's a threshold here rather than sorting on the amount of free
+ * disk space, because sorting on that value would in practice lead to never considering
+ * the number of cores on a node.
+ * <li>Place replicas on nodes having a smaller number of cores (the number of cores
+ * considered for this decision includes previous placement decisions made during the
+ * processing of the placement request)
* </ol>
*/
- @SuppressForbidden(reason = "Ordering.arbitrary() has no equivalent in Comparator class. Rather reuse than copy.")
- private void makePlacementDecisions(SolrCollection solrCollection, String shardName, Set<String> availabilityZones,
- Replica.ReplicaType replicaType, int numReplicas, final AttributeValues attrValues,
- EnumMap<Replica.ReplicaType, Set<Node>> replicaTypeToNodes, Set<Node> nodesWithReplicas,
- Map<Node, Integer> coresOnNodes, PlacementPlanFactory placementPlanFactory,
- Set<ReplicaPlacement> replicaPlacements) throws PlacementException {
- // Count existing replicas per AZ. We count only instances of the type of replica for which we need to do placement.
- // If we ever want to balance replicas of any type across AZ's (and not each replica type balanced independently),
- // we'd have to move this data structure to the caller of this method so it can be reused across different replica
- // type placements for a given shard. Note then that this change would be risky. For example all NRT's and PULL
- // replicas for a shard my be correctly balanced over three AZ's, but then all NRT can end up in the same AZ...
+ @SuppressForbidden(
+ reason =
+ "Ordering.arbitrary() has no equivalent in Comparator class. Rather reuse than copy.")
+ private void makePlacementDecisions(
+ SolrCollection solrCollection,
+ String shardName,
+ Set<String> availabilityZones,
+ Replica.ReplicaType replicaType,
+ int numReplicas,
+ final AttributeValues attrValues,
+ EnumMap<Replica.ReplicaType, Set<Node>> replicaTypeToNodes,
+ Set<Node> nodesWithReplicas,
+ Map<Node, Integer> coresOnNodes,
+ PlacementPlanFactory placementPlanFactory,
+ Set<ReplicaPlacement> replicaPlacements)
+ throws PlacementException {
+ // Count existing replicas per AZ. We count only instances of the type of replica for which we
+ // need to do placement.
+ // If we ever want to balance replicas of any type across AZ's (and not each replica type
+ // balanced independently),
+ // we'd have to move this data structure to the caller of this method so it can be reused
+ // across different replica
+ // type placements for a given shard. Note then that this change would be risky. For example
+ // all NRT's and PULL
+ // replicas for a shard my be correctly balanced over three AZ's, but then all NRT can end up
+ // in the same AZ...
Map<String, Integer> azToNumReplicas = new HashMap<>();
for (String az : availabilityZones) {
azToNumReplicas.put(az, 0);
}
- // Build the set of candidate nodes for the placement, i.e. nodes that can accept the replica type
+ // Build the set of candidate nodes for the placement, i.e. nodes that can accept the replica
+ // type
Set<Node> candidateNodes = new HashSet<>(replicaTypeToNodes.get(replicaType));
- // Remove nodes that already have a replica for the shard (no two replicas of same shard can be put on same node)
+ // Remove nodes that already have a replica for the shard (no two replicas of same shard can
+ // be put on same node)
candidateNodes.removeAll(nodesWithReplicas);
Shard shard = solrCollection.getShard(shardName);
if (shard != null) {
// shard is non null if we're adding replicas to an already existing collection.
// If we're creating the collection, the shards do not exist yet.
for (Replica replica : shard.replicas()) {
- // The node's AZ is counted as having a replica if it has a replica of the same type as the one we need
+ // The node's AZ is counted as having a replica if it has a replica of the same type as
+ // the one we need
// to place here.
if (replica.getType() == replicaType) {
final String az = getNodeAZ(replica.getNode(), attrValues);
if (azToNumReplicas.containsKey(az)) {
- // We do not count replicas on AZ's for which we don't have any node to place on because it would not help
- // the placement decision. If we did want to do that, note the dereferencing below can't be assumed as the
+ // We do not count replicas on AZ's for which we don't have any node to place on
+ // because it would not help
+ // the placement decision. If we did want to do that, note the dereferencing below
+ // can't be assumed as the
// entry will not exist in the map.
azToNumReplicas.put(az, azToNumReplicas.get(az) + 1);
}
}
}
}
- // We now have the set of real candidate nodes, we've enforced "No more than one replica of a given shard on a given node".
- // We also counted for the shard and replica type under consideration how many replicas were per AZ, so we can place
+ // We now have the set of real candidate nodes, we've enforced "No more than one replica of a
+ // given shard on a given node".
+ // We also counted for the shard and replica type under consideration how many replicas were
+ // per AZ, so we can place
// (or try to place) replicas on AZ's that have fewer replicas
- // Get the candidate nodes per AZ in order to build (further down) a mapping of AZ to placement candidates.
+ // Get the candidate nodes per AZ in order to build (further down) a mapping of AZ to
+ // placement candidates.
Map<String, List<Node>> nodesPerAz = new HashMap<>();
for (Node node : candidateNodes) {
String nodeAz = getNodeAZ(node, attrValues);
List<Node> nodesForAz = nodesPerAz.computeIfAbsent(nodeAz, k -> new ArrayList<>());
nodesForAz.add(node);
}
- // Build a treeMap sorted by the number of replicas per AZ and including candidates nodes suitable for placement on the
- // AZ, so we can easily select the next AZ to get a replica assignment and quickly (constant time) decide if placement
+ // Build a treeMap sorted by the number of replicas per AZ and including candidates nodes
+ // suitable for placement on the
+ // AZ, so we can easily select the next AZ to get a replica assignment and quickly (constant
+ // time) decide if placement
// on this AZ is possible or not.
- TreeMultimap<Integer, AzWithNodes> azByExistingReplicas = TreeMultimap.create(Comparator.naturalOrder(), Ordering.arbitrary());
+ TreeMultimap<Integer, AzWithNodes> azByExistingReplicas =
+ TreeMultimap.create(Comparator.naturalOrder(), Ordering.arbitrary());
for (Map.Entry<String, List<Node>> e : nodesPerAz.entrySet()) {
- azByExistingReplicas.put(azToNumReplicas.get(e.getKey()), new AzWithNodes(e.getKey(), e.getValue()));
+ azByExistingReplicas.put(
+ azToNumReplicas.get(e.getKey()), new AzWithNodes(e.getKey(), e.getValue()));
}
- CoresAndDiskComparator coresAndDiskComparator = new CoresAndDiskComparator(attrValues, coresOnNodes, prioritizedFreeDiskGB);
+ CoresAndDiskComparator coresAndDiskComparator =
+ new CoresAndDiskComparator(attrValues, coresOnNodes, prioritizedFreeDiskGB);
for (int i = 0; i < numReplicas; i++) {
- // We have for each AZ on which we might have a chance of placing a replica, the list of candidate nodes for replicas
- // (candidate: does not already have a replica of this shard and is in the corresponding AZ).
- // Among the AZ's with the minimal number of replicas of the given replica type for the shard, we must pick the AZ that
- // offers the best placement (based on number of cores and free disk space). In order to do so, for these "minimal" AZ's
- // we sort the nodes from best to worst placement candidate (based on the number of cores and free disk space) then pick
- // the AZ that has the best best node. We don't sort all AZ's because that will not necessarily be needed.
+ // We have for each AZ on which we might have a chance of placing a replica, the list of
+ // candidate nodes for replicas
+ // (candidate: does not already have a replica of this shard and is in the corresponding
+ // AZ).
+ // Among the AZ's with the minimal number of replicas of the given replica type for the
+ // shard, we must pick the AZ that
+ // offers the best placement (based on number of cores and free disk space). In order to do
+ // so, for these "minimal" AZ's
+ // we sort the nodes from best to worst placement candidate (based on the number of cores
+ // and free disk space) then pick
+ // the AZ that has the best best node. We don't sort all AZ's because that will not
+ // necessarily be needed.
int minNumberOfReplicasPerAz = 0; // This value never observed but compiler can't tell
Set<Map.Entry<Integer, AzWithNodes>> candidateAzEntries = null;
- // Iterate over AZ's (in the order of increasing number of replicas on that AZ) and do two things: 1. remove those AZ's that
- // have no nodes, no use iterating over these again and again (as we compute placement for more replicas), and 2. collect
+ // Iterate over AZ's (in the order of increasing number of replicas on that AZ) and do two
+ // things: 1. remove those AZ's that
+ // have no nodes, no use iterating over these again and again (as we compute placement for
+ // more replicas), and 2. collect
// all those AZ with a minimal number of replicas.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cluster/placement/plugins/AffinityPlacementFactory.java
##########
@@ -593,44 +748,62 @@ private void makePlacementDecisions(SolrCollection solrCollection, String shardN
break;
}
candidateAzEntries.add(entry);
- // We remove all entries that are candidates: the "winner" will be modified, all entries might also be sorted,
+ // We remove all entries that are candidates: the "winner" will be modified, all entries
+ // might also be sorted,
// so we'll insert back the updated versions later.
it.remove();
}
}
if (candidateAzEntries == null) {
- // This can happen because not enough nodes for the placement request or already too many nodes with replicas of
- // the shard that can't accept new replicas or not enough nodes with enough free disk space.
- throw new PlacementException("Not enough eligible nodes to place " + numReplicas + " replica(s) of type " + replicaType +
- " for shard " + shardName + " of collection " + solrCollection.getName());
+ // This can happen because not enough nodes for the placement request or already too many
+ // nodes with replicas of
+ // the shard that can't accept new replicas or not enough nodes with enough free disk
+ // space.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cluster/placement/plugins/AffinityPlacementFactory.java
##########
@@ -593,44 +748,62 @@ private void makePlacementDecisions(SolrCollection solrCollection, String shardN
break;
}
candidateAzEntries.add(entry);
- // We remove all entries that are candidates: the "winner" will be modified, all entries might also be sorted,
+ // We remove all entries that are candidates: the "winner" will be modified, all entries
+ // might also be sorted,
// so we'll insert back the updated versions later.
it.remove();
}
}
if (candidateAzEntries == null) {
- // This can happen because not enough nodes for the placement request or already too many nodes with replicas of
- // the shard that can't accept new replicas or not enough nodes with enough free disk space.
- throw new PlacementException("Not enough eligible nodes to place " + numReplicas + " replica(s) of type " + replicaType +
- " for shard " + shardName + " of collection " + solrCollection.getName());
+ // This can happen because not enough nodes for the placement request or already too many
+ // nodes with replicas of
+ // the shard that can't accept new replicas or not enough nodes with enough free disk
+ // space.
+ throw new PlacementException(
+ "Not enough eligible nodes to place "
+ + numReplicas
+ + " replica(s) of type "
+ + replicaType
+ + " for shard "
+ + shardName
+ + " of collection "
+ + solrCollection.getName());
}
- // Iterate over all candidate AZ's, sort them if needed and find the best one to use for this placement
+ // Iterate over all candidate AZ's, sort them if needed and find the best one to use for
+ // this placement
Map.Entry<Integer, AzWithNodes> selectedAz = null;
Node selectedAzBestNode = null;
for (Map.Entry<Integer, AzWithNodes> candidateAzEntry : candidateAzEntries) {
AzWithNodes azWithNodes = candidateAzEntry.getValue();
List<Node> nodes = azWithNodes.availableNodesForPlacement;
if (!azWithNodes.hasBeenSorted) {
- // Make sure we do not tend to use always the same nodes (within an AZ) if all conditions are identical (well, this
- // likely is not the case since after having added a replica to a node its number of cores increases for the next
- // placement decision, but let's be defensive here, given that multiple concurrent placement decisions might see
- // the same initial cluster state, and we want placement to be reasonable even in that case without creating an
+ // Make sure we do not tend to use always the same nodes (within an AZ) if all
+ // conditions are identical (well, this
+ // likely is not the case since after having added a replica to a node its number of
+ // cores increases for the next
+ // placement decision, but let's be defensive here, given that multiple concurrent
+ // placement decisions might see
+ // the same initial cluster state, and we want placement to be reasonable even in that
+ // case without creating an
// unnecessary imbalance).
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cluster/placement/plugins/AffinityPlacementFactory.java
##########
@@ -488,96 +600,139 @@ private String getNodeAZ(Node n, final AttributeValues attrValues) {
}
/**
- * <p>Picks nodes from {@code targetNodes} for placing {@code numReplicas} replicas.
+ * Picks nodes from {@code targetNodes} for placing {@code numReplicas} replicas.
*
* <p>The criteria used in this method are, in this order:
+ *
* <ol>
- * <li>No more than one replica of a given shard on a given node (strictly enforced)</li>
- * <li>Balance as much as possible replicas of a given {@link org.apache.solr.cluster.Replica.ReplicaType} over available AZ's.
- * This balancing takes into account existing replicas <b>of the corresponding replica type</b>, if any.</li>
- * <li>Place replicas if possible on nodes having more than a certain amount of free disk space (note that nodes with a too small
- * amount of free disk space were eliminated as placement targets earlier, in {@link #getAvailableNodesForReplicaTypes(Set, AttributeValues)}). There's
- * a threshold here rather than sorting on the amount of free disk space, because sorting on that value would in
- * practice lead to never considering the number of cores on a node.</li>
- * <li>Place replicas on nodes having a smaller number of cores (the number of cores considered
- * for this decision includes previous placement decisions made during the processing of the placement request)</li>
+ * <li>No more than one replica of a given shard on a given node (strictly enforced)
+ * <li>Balance as much as possible replicas of a given {@link
+ * org.apache.solr.cluster.Replica.ReplicaType} over available AZ's. This balancing takes
+ * into account existing replicas <b>of the corresponding replica type</b>, if any.
+ * <li>Place replicas if possible on nodes having more than a certain amount of free disk
+ * space (note that nodes with a too small amount of free disk space were eliminated as
+ * placement targets earlier, in {@link #getAvailableNodesForReplicaTypes(Set,
+ * AttributeValues)}). There's a threshold here rather than sorting on the amount of free
+ * disk space, because sorting on that value would in practice lead to never considering
+ * the number of cores on a node.
+ * <li>Place replicas on nodes having a smaller number of cores (the number of cores
+ * considered for this decision includes previous placement decisions made during the
+ * processing of the placement request)
* </ol>
*/
- @SuppressForbidden(reason = "Ordering.arbitrary() has no equivalent in Comparator class. Rather reuse than copy.")
- private void makePlacementDecisions(SolrCollection solrCollection, String shardName, Set<String> availabilityZones,
- Replica.ReplicaType replicaType, int numReplicas, final AttributeValues attrValues,
- EnumMap<Replica.ReplicaType, Set<Node>> replicaTypeToNodes, Set<Node> nodesWithReplicas,
- Map<Node, Integer> coresOnNodes, PlacementPlanFactory placementPlanFactory,
- Set<ReplicaPlacement> replicaPlacements) throws PlacementException {
- // Count existing replicas per AZ. We count only instances of the type of replica for which we need to do placement.
- // If we ever want to balance replicas of any type across AZ's (and not each replica type balanced independently),
- // we'd have to move this data structure to the caller of this method so it can be reused across different replica
- // type placements for a given shard. Note then that this change would be risky. For example all NRT's and PULL
- // replicas for a shard my be correctly balanced over three AZ's, but then all NRT can end up in the same AZ...
+ @SuppressForbidden(
+ reason =
+ "Ordering.arbitrary() has no equivalent in Comparator class. Rather reuse than copy.")
+ private void makePlacementDecisions(
+ SolrCollection solrCollection,
+ String shardName,
+ Set<String> availabilityZones,
+ Replica.ReplicaType replicaType,
+ int numReplicas,
+ final AttributeValues attrValues,
+ EnumMap<Replica.ReplicaType, Set<Node>> replicaTypeToNodes,
+ Set<Node> nodesWithReplicas,
+ Map<Node, Integer> coresOnNodes,
+ PlacementPlanFactory placementPlanFactory,
+ Set<ReplicaPlacement> replicaPlacements)
+ throws PlacementException {
+ // Count existing replicas per AZ. We count only instances of the type of replica for which we
+ // need to do placement.
+ // If we ever want to balance replicas of any type across AZ's (and not each replica type
+ // balanced independently),
+ // we'd have to move this data structure to the caller of this method so it can be reused
+ // across different replica
+ // type placements for a given shard. Note then that this change would be risky. For example
+ // all NRT's and PULL
+ // replicas for a shard my be correctly balanced over three AZ's, but then all NRT can end up
+ // in the same AZ...
Map<String, Integer> azToNumReplicas = new HashMap<>();
for (String az : availabilityZones) {
azToNumReplicas.put(az, 0);
}
- // Build the set of candidate nodes for the placement, i.e. nodes that can accept the replica type
+ // Build the set of candidate nodes for the placement, i.e. nodes that can accept the replica
+ // type
Set<Node> candidateNodes = new HashSet<>(replicaTypeToNodes.get(replicaType));
- // Remove nodes that already have a replica for the shard (no two replicas of same shard can be put on same node)
+ // Remove nodes that already have a replica for the shard (no two replicas of same shard can
+ // be put on same node)
candidateNodes.removeAll(nodesWithReplicas);
Shard shard = solrCollection.getShard(shardName);
if (shard != null) {
// shard is non null if we're adding replicas to an already existing collection.
// If we're creating the collection, the shards do not exist yet.
for (Replica replica : shard.replicas()) {
- // The node's AZ is counted as having a replica if it has a replica of the same type as the one we need
+ // The node's AZ is counted as having a replica if it has a replica of the same type as
+ // the one we need
// to place here.
if (replica.getType() == replicaType) {
final String az = getNodeAZ(replica.getNode(), attrValues);
if (azToNumReplicas.containsKey(az)) {
- // We do not count replicas on AZ's for which we don't have any node to place on because it would not help
- // the placement decision. If we did want to do that, note the dereferencing below can't be assumed as the
+ // We do not count replicas on AZ's for which we don't have any node to place on
+ // because it would not help
+ // the placement decision. If we did want to do that, note the dereferencing below
+ // can't be assumed as the
// entry will not exist in the map.
azToNumReplicas.put(az, azToNumReplicas.get(az) + 1);
}
}
}
}
- // We now have the set of real candidate nodes, we've enforced "No more than one replica of a given shard on a given node".
- // We also counted for the shard and replica type under consideration how many replicas were per AZ, so we can place
+ // We now have the set of real candidate nodes, we've enforced "No more than one replica of a
+ // given shard on a given node".
+ // We also counted for the shard and replica type under consideration how many replicas were
+ // per AZ, so we can place
// (or try to place) replicas on AZ's that have fewer replicas
- // Get the candidate nodes per AZ in order to build (further down) a mapping of AZ to placement candidates.
+ // Get the candidate nodes per AZ in order to build (further down) a mapping of AZ to
+ // placement candidates.
Map<String, List<Node>> nodesPerAz = new HashMap<>();
for (Node node : candidateNodes) {
String nodeAz = getNodeAZ(node, attrValues);
List<Node> nodesForAz = nodesPerAz.computeIfAbsent(nodeAz, k -> new ArrayList<>());
nodesForAz.add(node);
}
- // Build a treeMap sorted by the number of replicas per AZ and including candidates nodes suitable for placement on the
- // AZ, so we can easily select the next AZ to get a replica assignment and quickly (constant time) decide if placement
+ // Build a treeMap sorted by the number of replicas per AZ and including candidates nodes
+ // suitable for placement on the
+ // AZ, so we can easily select the next AZ to get a replica assignment and quickly (constant
+ // time) decide if placement
// on this AZ is possible or not.
- TreeMultimap<Integer, AzWithNodes> azByExistingReplicas = TreeMultimap.create(Comparator.naturalOrder(), Ordering.arbitrary());
+ TreeMultimap<Integer, AzWithNodes> azByExistingReplicas =
+ TreeMultimap.create(Comparator.naturalOrder(), Ordering.arbitrary());
for (Map.Entry<String, List<Node>> e : nodesPerAz.entrySet()) {
- azByExistingReplicas.put(azToNumReplicas.get(e.getKey()), new AzWithNodes(e.getKey(), e.getValue()));
+ azByExistingReplicas.put(
+ azToNumReplicas.get(e.getKey()), new AzWithNodes(e.getKey(), e.getValue()));
}
- CoresAndDiskComparator coresAndDiskComparator = new CoresAndDiskComparator(attrValues, coresOnNodes, prioritizedFreeDiskGB);
+ CoresAndDiskComparator coresAndDiskComparator =
+ new CoresAndDiskComparator(attrValues, coresOnNodes, prioritizedFreeDiskGB);
for (int i = 0; i < numReplicas; i++) {
- // We have for each AZ on which we might have a chance of placing a replica, the list of candidate nodes for replicas
- // (candidate: does not already have a replica of this shard and is in the corresponding AZ).
- // Among the AZ's with the minimal number of replicas of the given replica type for the shard, we must pick the AZ that
- // offers the best placement (based on number of cores and free disk space). In order to do so, for these "minimal" AZ's
- // we sort the nodes from best to worst placement candidate (based on the number of cores and free disk space) then pick
- // the AZ that has the best best node. We don't sort all AZ's because that will not necessarily be needed.
+ // We have for each AZ on which we might have a chance of placing a replica, the list of
+ // candidate nodes for replicas
+ // (candidate: does not already have a replica of this shard and is in the corresponding
+ // AZ).
+ // Among the AZ's with the minimal number of replicas of the given replica type for the
+ // shard, we must pick the AZ that
+ // offers the best placement (based on number of cores and free disk space). In order to do
+ // so, for these "minimal" AZ's
+ // we sort the nodes from best to worst placement candidate (based on the number of cores
+ // and free disk space) then pick
+ // the AZ that has the best best node. We don't sort all AZ's because that will not
+ // necessarily be needed.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cluster/placement/plugins/MinimizeCoresPlacementFactory.java
##########
@@ -94,23 +101,31 @@ public PlacementPlugin createPluginInstance() {
nodesByCores.put(coresPerNodeTotal.get(node.getName()), node);
}
- Set<ReplicaPlacement> replicaPlacements = new HashSet<>(totalReplicasPerShard * request.getShardNames().size());
+ Set<ReplicaPlacement> replicaPlacements =
+ new HashSet<>(totalReplicasPerShard * request.getShardNames().size());
- // Now place all replicas of all shards on nodes, by placing on nodes with the smallest number of cores and taking
- // into account replicas placed during this computation. Note that for each shard we must place replicas on different
- // nodes, when moving to the next shard we use the nodes sorted by their updated number of cores (due to replica
+ // Now place all replicas of all shards on nodes, by placing on nodes with the smallest
+ // number of cores and taking
+ // into account replicas placed during this computation. Note that for each shard we must
+ // place replicas on different
+ // nodes, when moving to the next shard we use the nodes sorted by their updated number of
+ // cores (due to replica
// placements for previous shards).
for (String shardName : request.getShardNames()) {
- // Assign replicas based on the sort order of the nodesByCores tree multimap to put replicas on nodes with less
- // cores first. We only need totalReplicasPerShard nodes given that's the number of replicas to place.
+ // Assign replicas based on the sort order of the nodesByCores tree multimap to put
+ // replicas on nodes with less
+ // cores first. We only need totalReplicasPerShard nodes given that's the number of
+ // replicas to place.
// We assign based on the passed nodeEntriesToAssign list so the right nodes get replicas.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/core/BlobRepository.java
##########
@@ -52,22 +54,21 @@
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
-import static org.apache.solr.common.SolrException.ErrorCode.SERVER_ERROR;
-import static org.apache.solr.common.SolrException.ErrorCode.SERVICE_UNAVAILABLE;
-
/**
- * The purpose of this class is to store the Jars loaded in memory and to keep only one copy of the Jar in a single node.
+ * The purpose of this class is to store the Jars loaded in memory and to keep only one copy of the
+ * Jar in a single node.
*/
public class BlobRepository {
- private static final long MAX_JAR_SIZE = Long.parseLong(
- System.getProperty("runtime.lib.size", String.valueOf(5 * 1024 * 1024)));
+ private static final long MAX_JAR_SIZE =
+ Long.parseLong(System.getProperty("runtime.lib.size", String.valueOf(5 * 1024 * 1024)));
private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
public static final Random RANDOM;
static final Pattern BLOB_KEY_PATTERN_CHECKER = Pattern.compile(".*/\\d+");
static {
- // We try to make things reproducible in the context of our tests by initializing the random instance
+ // We try to make things reproducible in the context of our tests by initializing the random
+ // instance
// based on the current seed
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cluster/placement/plugins/MinimizeCoresPlacementFactory.java
##########
@@ -94,23 +101,31 @@ public PlacementPlugin createPluginInstance() {
nodesByCores.put(coresPerNodeTotal.get(node.getName()), node);
}
- Set<ReplicaPlacement> replicaPlacements = new HashSet<>(totalReplicasPerShard * request.getShardNames().size());
+ Set<ReplicaPlacement> replicaPlacements =
+ new HashSet<>(totalReplicasPerShard * request.getShardNames().size());
- // Now place all replicas of all shards on nodes, by placing on nodes with the smallest number of cores and taking
- // into account replicas placed during this computation. Note that for each shard we must place replicas on different
- // nodes, when moving to the next shard we use the nodes sorted by their updated number of cores (due to replica
+ // Now place all replicas of all shards on nodes, by placing on nodes with the smallest
+ // number of cores and taking
+ // into account replicas placed during this computation. Note that for each shard we must
+ // place replicas on different
+ // nodes, when moving to the next shard we use the nodes sorted by their updated number of
+ // cores (due to replica
// placements for previous shards).
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cluster/placement/plugins/MinimizeCoresPlacementFactory.java
##########
@@ -94,23 +101,31 @@ public PlacementPlugin createPluginInstance() {
nodesByCores.put(coresPerNodeTotal.get(node.getName()), node);
}
- Set<ReplicaPlacement> replicaPlacements = new HashSet<>(totalReplicasPerShard * request.getShardNames().size());
+ Set<ReplicaPlacement> replicaPlacements =
+ new HashSet<>(totalReplicasPerShard * request.getShardNames().size());
- // Now place all replicas of all shards on nodes, by placing on nodes with the smallest number of cores and taking
- // into account replicas placed during this computation. Note that for each shard we must place replicas on different
- // nodes, when moving to the next shard we use the nodes sorted by their updated number of cores (due to replica
+ // Now place all replicas of all shards on nodes, by placing on nodes with the smallest
+ // number of cores and taking
+ // into account replicas placed during this computation. Note that for each shard we must
+ // place replicas on different
+ // nodes, when moving to the next shard we use the nodes sorted by their updated number of
+ // cores (due to replica
// placements for previous shards).
for (String shardName : request.getShardNames()) {
- // Assign replicas based on the sort order of the nodesByCores tree multimap to put replicas on nodes with less
- // cores first. We only need totalReplicasPerShard nodes given that's the number of replicas to place.
+ // Assign replicas based on the sort order of the nodesByCores tree multimap to put
+ // replicas on nodes with less
+ // cores first. We only need totalReplicasPerShard nodes given that's the number of
+ // replicas to place.
// We assign based on the passed nodeEntriesToAssign list so the right nodes get replicas.
- ArrayList<Map.Entry<Integer, Node>> nodeEntriesToAssign = new ArrayList<>(totalReplicasPerShard);
+ ArrayList<Map.Entry<Integer, Node>> nodeEntriesToAssign =
+ new ArrayList<>(totalReplicasPerShard);
Iterator<Map.Entry<Integer, Node>> treeIterator = nodesByCores.entries().iterator();
for (int i = 0; i < totalReplicasPerShard; i++) {
nodeEntriesToAssign.add(treeIterator.next());
}
- // Update the number of cores each node will have once the assignments below got executed so the next shard picks the
+ // Update the number of cores each node will have once the assignments below got executed
+ // so the next shard picks the
// lowest loaded nodes for its replicas.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/core/CoreContainer.java
##########
@@ -516,56 +538,62 @@ private synchronized void initializeAuthenticationPlugin(Map<String, Object> aut
} catch (Exception e) {
log.error("Exception while attempting to close old authentication plugin", e);
}
-
}
private void setupHttpClientForAuthPlugin(Object authcPlugin) {
if (authcPlugin instanceof HttpClientBuilderPlugin) {
// Setup HttpClient for internode communication
HttpClientBuilderPlugin builderPlugin = ((HttpClientBuilderPlugin) authcPlugin);
- SolrHttpClientBuilder builder = builderPlugin.getHttpClientBuilder(HttpClientUtil.getHttpClientBuilder());
+ SolrHttpClientBuilder builder =
+ builderPlugin.getHttpClientBuilder(HttpClientUtil.getHttpClientBuilder());
-
- // this caused plugins like KerberosPlugin to register it's intercepts, but this intercept logic is also
- // handled by the pki authentication code when it decideds to let the plugin handle auth via it's intercept
+ // this caused plugins like KerberosPlugin to register it's intercepts, but this intercept
+ // logic is also
+ // handled by the pki authentication code when it decideds to let the plugin handle auth via
+ // it's intercept
// - so you would end up with two intercepts
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/core/CoreContainer.java
##########
@@ -516,56 +538,62 @@ private synchronized void initializeAuthenticationPlugin(Map<String, Object> aut
} catch (Exception e) {
log.error("Exception while attempting to close old authentication plugin", e);
}
-
}
private void setupHttpClientForAuthPlugin(Object authcPlugin) {
if (authcPlugin instanceof HttpClientBuilderPlugin) {
// Setup HttpClient for internode communication
HttpClientBuilderPlugin builderPlugin = ((HttpClientBuilderPlugin) authcPlugin);
- SolrHttpClientBuilder builder = builderPlugin.getHttpClientBuilder(HttpClientUtil.getHttpClientBuilder());
+ SolrHttpClientBuilder builder =
+ builderPlugin.getHttpClientBuilder(HttpClientUtil.getHttpClientBuilder());
-
- // this caused plugins like KerberosPlugin to register it's intercepts, but this intercept logic is also
- // handled by the pki authentication code when it decideds to let the plugin handle auth via it's intercept
+ // this caused plugins like KerberosPlugin to register it's intercepts, but this intercept
+ // logic is also
+ // handled by the pki authentication code when it decideds to let the plugin handle auth via
+ // it's intercept
// - so you would end up with two intercepts
// -->
// shardHandlerFactory.setSecurityBuilder(builderPlugin); // calls setup for the authcPlugin
// updateShardHandler.setSecurityBuilder(builderPlugin);
// <--
- // This should not happen here at all - it's only currently required due to its affect on http1 clients
+ // This should not happen here at all - it's only currently required due to its affect on
+ // http1 clients
// in a test or two incorrectly counting on it for their configuration.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/core/ConfigSetService.java
##########
@@ -275,22 +292,27 @@ public ConfigSetService(SolrResourceLoader loader, boolean shareSchema) {
* @param isTrusted is the configset trusted?
* @return a SolrConfig object
*/
- protected SolrConfig createSolrConfig(CoreDescriptor cd, SolrResourceLoader loader, boolean isTrusted) {
- return SolrConfig.readFromResourceLoader(loader, cd.getConfigName(), isTrusted, cd.getSubstitutableProperties());
+ protected SolrConfig createSolrConfig(
+ CoreDescriptor cd, SolrResourceLoader loader, boolean isTrusted) {
+ return SolrConfig.readFromResourceLoader(
+ loader, cd.getConfigName(), isTrusted, cd.getSubstitutableProperties());
}
/**
- * Create an IndexSchema object for a core. It might be a cached lookup.
+ * Create an IndexSchema object for a core. It might be a cached lookup.
*
* @param cd the core's CoreDescriptor
* @param solrConfig the core's SolrConfig
* @return an IndexSchema
*/
- protected IndexSchema createIndexSchema(CoreDescriptor cd, SolrConfig solrConfig, boolean forceFetch) throws IOException {
- // This is the schema name from the core descriptor. Sometimes users specify a custom schema file.
+ protected IndexSchema createIndexSchema(
+ CoreDescriptor cd, SolrConfig solrConfig, boolean forceFetch) throws IOException {
+ // This is the schema name from the core descriptor. Sometimes users specify a custom schema
+ // file.
// Important: indexSchemaFactory.create wants this!
String cdSchemaName = cd.getSchemaName();
- // This is the schema name that we think will actually be used. In the case of a managed schema,
+ // This is the schema name that we think will actually be used. In the case of a managed
+ // schema,
// we don't know for sure without examining what files exists in the configSet, and we don't
// want to pay the overhead of that at this juncture. If we guess wrong, no schema sharing.
// The fix is usually to name your schema managed-schema.xml instead of schema.xml.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cluster/placement/plugins/AffinityPlacementFactory.java
##########
@@ -488,96 +600,139 @@ private String getNodeAZ(Node n, final AttributeValues attrValues) {
}
/**
- * <p>Picks nodes from {@code targetNodes} for placing {@code numReplicas} replicas.
+ * Picks nodes from {@code targetNodes} for placing {@code numReplicas} replicas.
*
* <p>The criteria used in this method are, in this order:
+ *
* <ol>
- * <li>No more than one replica of a given shard on a given node (strictly enforced)</li>
- * <li>Balance as much as possible replicas of a given {@link org.apache.solr.cluster.Replica.ReplicaType} over available AZ's.
- * This balancing takes into account existing replicas <b>of the corresponding replica type</b>, if any.</li>
- * <li>Place replicas if possible on nodes having more than a certain amount of free disk space (note that nodes with a too small
- * amount of free disk space were eliminated as placement targets earlier, in {@link #getAvailableNodesForReplicaTypes(Set, AttributeValues)}). There's
- * a threshold here rather than sorting on the amount of free disk space, because sorting on that value would in
- * practice lead to never considering the number of cores on a node.</li>
- * <li>Place replicas on nodes having a smaller number of cores (the number of cores considered
- * for this decision includes previous placement decisions made during the processing of the placement request)</li>
+ * <li>No more than one replica of a given shard on a given node (strictly enforced)
+ * <li>Balance as much as possible replicas of a given {@link
+ * org.apache.solr.cluster.Replica.ReplicaType} over available AZ's. This balancing takes
+ * into account existing replicas <b>of the corresponding replica type</b>, if any.
+ * <li>Place replicas if possible on nodes having more than a certain amount of free disk
+ * space (note that nodes with a too small amount of free disk space were eliminated as
+ * placement targets earlier, in {@link #getAvailableNodesForReplicaTypes(Set,
+ * AttributeValues)}). There's a threshold here rather than sorting on the amount of free
+ * disk space, because sorting on that value would in practice lead to never considering
+ * the number of cores on a node.
+ * <li>Place replicas on nodes having a smaller number of cores (the number of cores
+ * considered for this decision includes previous placement decisions made during the
+ * processing of the placement request)
* </ol>
*/
- @SuppressForbidden(reason = "Ordering.arbitrary() has no equivalent in Comparator class. Rather reuse than copy.")
- private void makePlacementDecisions(SolrCollection solrCollection, String shardName, Set<String> availabilityZones,
- Replica.ReplicaType replicaType, int numReplicas, final AttributeValues attrValues,
- EnumMap<Replica.ReplicaType, Set<Node>> replicaTypeToNodes, Set<Node> nodesWithReplicas,
- Map<Node, Integer> coresOnNodes, PlacementPlanFactory placementPlanFactory,
- Set<ReplicaPlacement> replicaPlacements) throws PlacementException {
- // Count existing replicas per AZ. We count only instances of the type of replica for which we need to do placement.
- // If we ever want to balance replicas of any type across AZ's (and not each replica type balanced independently),
- // we'd have to move this data structure to the caller of this method so it can be reused across different replica
- // type placements for a given shard. Note then that this change would be risky. For example all NRT's and PULL
- // replicas for a shard my be correctly balanced over three AZ's, but then all NRT can end up in the same AZ...
+ @SuppressForbidden(
+ reason =
+ "Ordering.arbitrary() has no equivalent in Comparator class. Rather reuse than copy.")
+ private void makePlacementDecisions(
+ SolrCollection solrCollection,
+ String shardName,
+ Set<String> availabilityZones,
+ Replica.ReplicaType replicaType,
+ int numReplicas,
+ final AttributeValues attrValues,
+ EnumMap<Replica.ReplicaType, Set<Node>> replicaTypeToNodes,
+ Set<Node> nodesWithReplicas,
+ Map<Node, Integer> coresOnNodes,
+ PlacementPlanFactory placementPlanFactory,
+ Set<ReplicaPlacement> replicaPlacements)
+ throws PlacementException {
+ // Count existing replicas per AZ. We count only instances of the type of replica for which we
+ // need to do placement.
+ // If we ever want to balance replicas of any type across AZ's (and not each replica type
+ // balanced independently),
+ // we'd have to move this data structure to the caller of this method so it can be reused
+ // across different replica
+ // type placements for a given shard. Note then that this change would be risky. For example
+ // all NRT's and PULL
+ // replicas for a shard my be correctly balanced over three AZ's, but then all NRT can end up
+ // in the same AZ...
Map<String, Integer> azToNumReplicas = new HashMap<>();
for (String az : availabilityZones) {
azToNumReplicas.put(az, 0);
}
- // Build the set of candidate nodes for the placement, i.e. nodes that can accept the replica type
+ // Build the set of candidate nodes for the placement, i.e. nodes that can accept the replica
+ // type
Set<Node> candidateNodes = new HashSet<>(replicaTypeToNodes.get(replicaType));
- // Remove nodes that already have a replica for the shard (no two replicas of same shard can be put on same node)
+ // Remove nodes that already have a replica for the shard (no two replicas of same shard can
+ // be put on same node)
candidateNodes.removeAll(nodesWithReplicas);
Shard shard = solrCollection.getShard(shardName);
if (shard != null) {
// shard is non null if we're adding replicas to an already existing collection.
// If we're creating the collection, the shards do not exist yet.
for (Replica replica : shard.replicas()) {
- // The node's AZ is counted as having a replica if it has a replica of the same type as the one we need
+ // The node's AZ is counted as having a replica if it has a replica of the same type as
+ // the one we need
// to place here.
if (replica.getType() == replicaType) {
final String az = getNodeAZ(replica.getNode(), attrValues);
if (azToNumReplicas.containsKey(az)) {
- // We do not count replicas on AZ's for which we don't have any node to place on because it would not help
- // the placement decision. If we did want to do that, note the dereferencing below can't be assumed as the
+ // We do not count replicas on AZ's for which we don't have any node to place on
+ // because it would not help
+ // the placement decision. If we did want to do that, note the dereferencing below
+ // can't be assumed as the
// entry will not exist in the map.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/core/CoreContainer.java
##########
@@ -1278,27 +1418,33 @@ public SolrCore create(String coreName, Path instancePath, Map<String, String> p
try {
if (getZkController() != null) {
if (cd.getCloudDescriptor().getCoreNodeName() == null) {
- throw new SolrException(ErrorCode.SERVER_ERROR, "coreNodeName missing " + parameters.toString());
+ throw new SolrException(
+ ErrorCode.SERVER_ERROR, "coreNodeName missing " + parameters.toString());
}
preExisitingZkEntry = getZkController().checkIfCoreNodeNameAlreadyExists(cd);
}
- // Much of the logic in core handling pre-supposes that the core.properties file already exists, so create it
+ // Much of the logic in core handling pre-supposes that the core.properties file already
+ // exists, so create it
// first and clean it up if there's an error.
coresLocator.create(this, cd);
SolrCore core;
try {
solrCores.waitAddPendingCoreOps(cd.getName());
core = createFromDescriptor(cd, true, newCollection);
- coresLocator.persist(this, cd); // Write out the current core properties in case anything changed when the core was created
+ coresLocator.persist(
+ this,
+ cd); // Write out the current core properties in case anything changed when the core
+ // was created
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/core/CoreContainer.java
##########
@@ -516,56 +538,62 @@ private synchronized void initializeAuthenticationPlugin(Map<String, Object> aut
} catch (Exception e) {
log.error("Exception while attempting to close old authentication plugin", e);
}
-
}
private void setupHttpClientForAuthPlugin(Object authcPlugin) {
if (authcPlugin instanceof HttpClientBuilderPlugin) {
// Setup HttpClient for internode communication
HttpClientBuilderPlugin builderPlugin = ((HttpClientBuilderPlugin) authcPlugin);
- SolrHttpClientBuilder builder = builderPlugin.getHttpClientBuilder(HttpClientUtil.getHttpClientBuilder());
+ SolrHttpClientBuilder builder =
+ builderPlugin.getHttpClientBuilder(HttpClientUtil.getHttpClientBuilder());
-
- // this caused plugins like KerberosPlugin to register it's intercepts, but this intercept logic is also
- // handled by the pki authentication code when it decideds to let the plugin handle auth via it's intercept
+ // this caused plugins like KerberosPlugin to register it's intercepts, but this intercept
+ // logic is also
+ // handled by the pki authentication code when it decideds to let the plugin handle auth via
+ // it's intercept
// - so you would end up with two intercepts
// -->
// shardHandlerFactory.setSecurityBuilder(builderPlugin); // calls setup for the authcPlugin
// updateShardHandler.setSecurityBuilder(builderPlugin);
// <--
- // This should not happen here at all - it's only currently required due to its affect on http1 clients
+ // This should not happen here at all - it's only currently required due to its affect on
+ // http1 clients
// in a test or two incorrectly counting on it for their configuration.
// -->
SolrHttpClientContextBuilder httpClientBuilder = new SolrHttpClientContextBuilder();
if (builder.getCredentialsProviderProvider() != null) {
- httpClientBuilder.setDefaultCredentialsProvider(new CredentialsProviderProvider() {
+ httpClientBuilder.setDefaultCredentialsProvider(
+ new CredentialsProviderProvider() {
- @Override
- public CredentialsProvider getCredentialsProvider() {
- return builder.getCredentialsProviderProvider().getCredentialsProvider();
- }
- });
+ @Override
+ public CredentialsProvider getCredentialsProvider() {
+ return builder.getCredentialsProviderProvider().getCredentialsProvider();
+ }
+ });
}
if (builder.getAuthSchemeRegistryProvider() != null) {
- httpClientBuilder.setAuthSchemeRegistryProvider(new AuthSchemeRegistryProvider() {
+ httpClientBuilder.setAuthSchemeRegistryProvider(
+ new AuthSchemeRegistryProvider() {
- @Override
- public Lookup<AuthSchemeProvider> getAuthSchemeRegistry() {
- return builder.getAuthSchemeRegistryProvider().getAuthSchemeRegistry();
- }
- });
+ @Override
+ public Lookup<AuthSchemeProvider> getAuthSchemeRegistry() {
+ return builder.getAuthSchemeRegistryProvider().getAuthSchemeRegistry();
+ }
+ });
}
HttpClientUtil.setHttpClientRequestContextBuilder(httpClientBuilder);
// <--
}
- // Always register PKI auth interceptor, which will then delegate the decision of who should secure
+ // Always register PKI auth interceptor, which will then delegate the decision of who should
+ // secure
// each request to the configured authentication plugin.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/core/CoreContainer.java
##########
@@ -1049,39 +1167,46 @@ public void shutdown() {
objectCache.clear();
- // It's still possible that one of the pending dynamic load operation is waiting, so wake it up if so.
+ // It's still possible that one of the pending dynamic load operation is waiting, so wake it
+ // up if so.
// Since all the pending operations queues have been drained, there should be nothing to do.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/core/CoreContainer.java
##########
@@ -1278,27 +1418,33 @@ public SolrCore create(String coreName, Path instancePath, Map<String, String> p
try {
if (getZkController() != null) {
if (cd.getCloudDescriptor().getCoreNodeName() == null) {
- throw new SolrException(ErrorCode.SERVER_ERROR, "coreNodeName missing " + parameters.toString());
+ throw new SolrException(
+ ErrorCode.SERVER_ERROR, "coreNodeName missing " + parameters.toString());
}
preExisitingZkEntry = getZkController().checkIfCoreNodeNameAlreadyExists(cd);
}
- // Much of the logic in core handling pre-supposes that the core.properties file already exists, so create it
+ // Much of the logic in core handling pre-supposes that the core.properties file already
+ // exists, so create it
// first and clean it up if there's an error.
coresLocator.create(this, cd);
SolrCore core;
try {
solrCores.waitAddPendingCoreOps(cd.getName());
core = createFromDescriptor(cd, true, newCollection);
- coresLocator.persist(this, cd); // Write out the current core properties in case anything changed when the core was created
+ coresLocator.persist(
+ this,
+ cd); // Write out the current core properties in case anything changed when the core
+ // was created
} finally {
solrCores.removeFromPendingOps(cd.getName());
}
return core;
} catch (Exception ex) {
- // First clean up any core descriptor, there should never be an existing core.properties file for any core that
+ // First clean up any core descriptor, there should never be an existing core.properties
+ // file for any core that
// failed to be created on-the-fly.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/core/BlobRepository.java
##########
@@ -289,14 +311,16 @@ public void decrementBlobRefCount(BlobContentRef<?> ref) {
public static class BlobContent<T> {
public final String key;
- private final T content; // holds byte buffer or cached object, holding both is a waste of memory
+ private final T
+ content; // holds byte buffer or cached object, holding both is a waste of memory
// ref counting mechanism
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/core/CoreContainer.java
##########
@@ -1278,27 +1418,33 @@ public SolrCore create(String coreName, Path instancePath, Map<String, String> p
try {
if (getZkController() != null) {
if (cd.getCloudDescriptor().getCoreNodeName() == null) {
- throw new SolrException(ErrorCode.SERVER_ERROR, "coreNodeName missing " + parameters.toString());
+ throw new SolrException(
+ ErrorCode.SERVER_ERROR, "coreNodeName missing " + parameters.toString());
}
preExisitingZkEntry = getZkController().checkIfCoreNodeNameAlreadyExists(cd);
}
- // Much of the logic in core handling pre-supposes that the core.properties file already exists, so create it
+ // Much of the logic in core handling pre-supposes that the core.properties file already
+ // exists, so create it
// first and clean it up if there's an error.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/core/CoreContainer.java
##########
@@ -731,31 +766,44 @@ public void load() {
MDCLoggingContext.setNode(this);
- securityConfHandler = isZooKeeperAware() ? new SecurityConfHandlerZk(this) : new SecurityConfHandlerLocal(this);
+ securityConfHandler =
+ isZooKeeperAware() ? new SecurityConfHandlerZk(this) : new SecurityConfHandlerLocal(this);
reloadSecurityProperties();
warnUsersOfInsecureSettings();
this.backupRepoFactory = new BackupRepositoryFactory(cfg.getBackupRepositoryPlugins());
coreConfigService = ConfigSetService.createConfigSetService(this);
createHandler(ZK_PATH, ZookeeperInfoHandler.class.getName(), ZookeeperInfoHandler.class);
- createHandler(ZK_STATUS_PATH, ZookeeperStatusHandler.class.getName(), ZookeeperStatusHandler.class);
-
- // CoreContainer is initialized enough at this stage so we can set distributedCollectionCommandRunner (the
- // construction of DistributedCollectionConfigSetCommandRunner uses Zookeeper so can't be done from the CoreContainer constructor
- // because there Zookeeper is not yet ready). Given this is used in the CollectionsHandler created next line, this is
- // the latest point where distributedCollectionCommandRunner can be initialized without refactoring this method...
- // TODO: manage to completely build CoreContainer in the constructor and not in the load() method... Requires some test refactoring.
- this.distributedCollectionCommandRunner = isZooKeeperAware() && cfg.getCloudConfig().getDistributedCollectionConfigSetExecution() ?
- Optional.of(new DistributedCollectionConfigSetCommandRunner(this)) : Optional.empty();
-
- collectionsHandler = createHandler(COLLECTIONS_HANDLER_PATH, cfg.getCollectionsHandlerClass(), CollectionsHandler.class);
+ createHandler(
+ ZK_STATUS_PATH, ZookeeperStatusHandler.class.getName(), ZookeeperStatusHandler.class);
+
+ // CoreContainer is initialized enough at this stage so we can set
+ // distributedCollectionCommandRunner (the
+ // construction of DistributedCollectionConfigSetCommandRunner uses Zookeeper so can't be done
+ // from the CoreContainer constructor
+ // because there Zookeeper is not yet ready). Given this is used in the CollectionsHandler
+ // created next line, this is
+ // the latest point where distributedCollectionCommandRunner can be initialized without
+ // refactoring this method...
+ // TODO: manage to completely build CoreContainer in the constructor and not in the load()
+ // method... Requires some test refactoring.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/core/CoreContainer.java
##########
@@ -1625,20 +1803,27 @@ private CoreDescriptor reloadCoreDescriptor(CoreDescriptor oldDesc) {
}
CorePropertiesLocator cpl = new CorePropertiesLocator(null);
- CoreDescriptor ret = cpl.buildCoreDescriptor(oldDesc.getInstanceDir().resolve(PROPERTIES_FILENAME), this);
+ CoreDescriptor ret =
+ cpl.buildCoreDescriptor(oldDesc.getInstanceDir().resolve(PROPERTIES_FILENAME), this);
- // Ok, this little jewel is all because we still create core descriptors on the fly from lists of properties
- // in tests particularly. Theoretically, there should be _no_ way to create a CoreDescriptor in the new world
+ // Ok, this little jewel is all because we still create core descriptors on the fly from lists
+ // of properties
+ // in tests particularly. Theoretically, there should be _no_ way to create a CoreDescriptor in
+ // the new world
// of core discovery without writing the core.properties file out first.
//
- // TODO: remove core.properties from the conf directory in test files, it's in a bad place there anyway.
+ // TODO: remove core.properties from the conf directory in test files, it's in a bad place there
+ // anyway.
if (ret == null) {
- oldDesc.loadExtraProperties(); // there may be changes to extra properties that we need to pick up.
+ oldDesc
+ .loadExtraProperties(); // there may be changes to extra properties that we need to pick
+ // up.
return oldDesc;
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/core/CoreContainer.java
##########
@@ -1672,7 +1853,8 @@ public void reload(String name, UUID coreId) {
SolrCore newCore = null;
SolrCore core = solrCores.getCoreFromAnyList(name, false, coreId);
if (core != null) {
- // The underlying core properties files may have changed, we don't really know. So we have a (perhaps) stale
+ // The underlying core properties files may have changed, we don't really know. So we have a
+ // (perhaps) stale
// CoreDescriptor and we need to reload it from the disk files
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/core/CoreContainer.java
##########
@@ -1625,20 +1803,27 @@ private CoreDescriptor reloadCoreDescriptor(CoreDescriptor oldDesc) {
}
CorePropertiesLocator cpl = new CorePropertiesLocator(null);
- CoreDescriptor ret = cpl.buildCoreDescriptor(oldDesc.getInstanceDir().resolve(PROPERTIES_FILENAME), this);
+ CoreDescriptor ret =
+ cpl.buildCoreDescriptor(oldDesc.getInstanceDir().resolve(PROPERTIES_FILENAME), this);
- // Ok, this little jewel is all because we still create core descriptors on the fly from lists of properties
- // in tests particularly. Theoretically, there should be _no_ way to create a CoreDescriptor in the new world
+ // Ok, this little jewel is all because we still create core descriptors on the fly from lists
+ // of properties
+ // in tests particularly. Theoretically, there should be _no_ way to create a CoreDescriptor in
+ // the new world
// of core discovery without writing the core.properties file out first.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/core/CoreContainer.java
##########
@@ -1932,18 +2121,24 @@ public SolrCore getCore(String name, UUID id) {
if (null != loadFailure) {
throw new SolrCoreInitializationException(name, loadFailure.exception);
}
- // This is a bit of awkwardness where SolrCloud and transient cores don't play nice together. For transient cores,
- // we have to allow them to be created at any time there hasn't been a core load failure (use reload to cure that).
- // But for TestConfigSetsAPI.testUploadWithScriptUpdateProcessor, this needs to _not_ try to load the core if
- // the core is null and there was an error. If you change this, be sure to run both TestConfiSetsAPI and
+ // This is a bit of awkwardness where SolrCloud and transient cores don't play nice together.
+ // For transient cores,
+ // we have to allow them to be created at any time there hasn't been a core load failure (use
+ // reload to cure that).
+ // But for TestConfigSetsAPI.testUploadWithScriptUpdateProcessor, this needs to _not_ try to
+ // load the core if
+ // the core is null and there was an error. If you change this, be sure to run both
+ // TestConfiSetsAPI and
// TestLazyCores
if (desc == null || zkSys.getZkController() != null) return null;
// This will put an entry in pending core ops if the core isn't loaded. Here's where moving the
// waitAddPendingCoreOps to createFromDescriptor would introduce a race condition.
core = solrCores.waitAddPendingCoreOps(name);
- if (isShutDown) return null; // We're quitting, so stop. This needs to be after the wait above since we may come off
+ if (isShutDown)
+ return null; // We're quitting, so stop. This needs to be after the wait above since we may
+ // come off
// the wait as a consequence of shutting down.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/core/CoreContainer.java
##########
@@ -1932,18 +2121,24 @@ public SolrCore getCore(String name, UUID id) {
if (null != loadFailure) {
throw new SolrCoreInitializationException(name, loadFailure.exception);
}
- // This is a bit of awkwardness where SolrCloud and transient cores don't play nice together. For transient cores,
- // we have to allow them to be created at any time there hasn't been a core load failure (use reload to cure that).
- // But for TestConfigSetsAPI.testUploadWithScriptUpdateProcessor, this needs to _not_ try to load the core if
- // the core is null and there was an error. If you change this, be sure to run both TestConfiSetsAPI and
+ // This is a bit of awkwardness where SolrCloud and transient cores don't play nice together.
+ // For transient cores,
+ // we have to allow them to be created at any time there hasn't been a core load failure (use
+ // reload to cure that).
+ // But for TestConfigSetsAPI.testUploadWithScriptUpdateProcessor, this needs to _not_ try to
+ // load the core if
+ // the core is null and there was an error. If you change this, be sure to run both
+ // TestConfiSetsAPI and
// TestLazyCores
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/core/CoreContainer.java
##########
@@ -2228,7 +2426,8 @@ public void run() {
try {
solrCores.getModifyLock().wait();
} catch (InterruptedException e) {
- // Well, if we've been told to stop, we will. Otherwise, continue on and check to see if there are
+ // Well, if we've been told to stop, we will. Otherwise, continue on and check to see if
+ // there are
// any cores to close.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/core/CoreContainer.java
##########
@@ -1625,20 +1803,27 @@ private CoreDescriptor reloadCoreDescriptor(CoreDescriptor oldDesc) {
}
CorePropertiesLocator cpl = new CorePropertiesLocator(null);
- CoreDescriptor ret = cpl.buildCoreDescriptor(oldDesc.getInstanceDir().resolve(PROPERTIES_FILENAME), this);
+ CoreDescriptor ret =
+ cpl.buildCoreDescriptor(oldDesc.getInstanceDir().resolve(PROPERTIES_FILENAME), this);
- // Ok, this little jewel is all because we still create core descriptors on the fly from lists of properties
- // in tests particularly. Theoretically, there should be _no_ way to create a CoreDescriptor in the new world
+ // Ok, this little jewel is all because we still create core descriptors on the fly from lists
+ // of properties
+ // in tests particularly. Theoretically, there should be _no_ way to create a CoreDescriptor in
+ // the new world
// of core discovery without writing the core.properties file out first.
//
- // TODO: remove core.properties from the conf directory in test files, it's in a bad place there anyway.
+ // TODO: remove core.properties from the conf directory in test files, it's in a bad place there
+ // anyway.
if (ret == null) {
- oldDesc.loadExtraProperties(); // there may be changes to extra properties that we need to pick up.
+ oldDesc
+ .loadExtraProperties(); // there may be changes to extra properties that we need to pick
+ // up.
return oldDesc;
-
}
- // The CloudDescriptor bit here is created in a very convoluted way, requiring access to private methods
- // in ZkController. When reloading, this behavior is identical to what used to happen where a copy of the old
+ // The CloudDescriptor bit here is created in a very convoluted way, requiring access to private
+ // methods
+ // in ZkController. When reloading, this behavior is identical to what used to happen where a
+ // copy of the old
// CoreDescriptor was just re-used.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/core/IndexDeletionPolicyWrapper.java
##########
@@ -172,42 +162,47 @@ public synchronized IndexCommit getAndSaveCommitPoint(Long generation) {
throw new NullPointerException("generation to get and save must not be null");
}
final IndexCommit commit = knownCommits.get(generation);
- if ( (null != commit && false != commit.isDeleted())
- || (null == commit && null != latestCommit && generation < latestCommit.getGeneration()) ) {
- throw new IllegalStateException
- ("Specified index generation is too old to be saved: " + generation);
+ if ((null != commit && false != commit.isDeleted())
+ || (null == commit && null != latestCommit && generation < latestCommit.getGeneration())) {
+ throw new IllegalStateException(
+ "Specified index generation is too old to be saved: " + generation);
}
- final AtomicInteger refCount
- = savedCommits.computeIfAbsent(generation, s -> { return new AtomicInteger(); });
+ final AtomicInteger refCount =
+ savedCommits.computeIfAbsent(
+ generation,
+ s -> {
+ return new AtomicInteger();
+ });
final int currentCount = refCount.incrementAndGet();
log.debug("Saving generation={}, refCount={}", generation, currentCount);
return commit;
}
-
+
public IndexDeletionPolicy getWrappedDeletionPolicy() {
return deletionPolicy;
}
/**
* Set the duration for which commit point is to be reserved by the deletion policy.
- * <p>
- * <b>NOTE:</b> This method does not make any garuntees that the specified index generation exists,
- * or that the specified generation has not already ben deleted. The only garuntee is that
- * <em>if</em> the specified generation exists now, or is created at some point in the future, then
- * it will be resered for <em>at least</em> the specified <code>reserveTime</code>.
- * </p>
+ *
+ * <p><b>NOTE:</b> This method does not make any garuntees that the specified index generation
+ * exists, or that the specified generation has not already ben deleted. The only garuntee is that
+ * <em>if</em> the specified generation exists now, or is created at some point in the future,
+ * then it will be resered for <em>at least</em> the specified <code>reserveTime</code>.
*
* @param indexGen gen of the commit point to be reserved
- * @param reserveTime durration in milliseconds (relative to 'now') for which the commit point is to be reserved
+ * @param reserveTime durration in milliseconds (relative to 'now') for which the commit point is
+ * to be reserved
*/
public void setReserveDuration(Long indexGen, long reserveTime) {
- // since 'reserves' is a concurrent HashMap, we don't need to synchronize this method as long as all
+ // since 'reserves' is a concurrent HashMap, we don't need to synchronize this method as long as
+ // all
// operations on 'reserves' are done atomically.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/core/IndexDeletionPolicyWrapper.java
##########
@@ -450,42 +444,41 @@ private synchronized void updateLatestCommit(final List<IndexCommitWrapper> list
// The upside of this current approach, and not completley synchornizing onInit/onCommit
// is that we have no control over what delegate is used, or how long those calls might take.
//
- // If the hypotehtical situation above ever becomes problematic, then an alternative approach might be
- // to *add* to the Set/Map of all known commits *before* delegating, then *remove* everything except
+ // If the hypotehtical situation above ever becomes problematic, then an alternative approach
+ // might be
+ // to *add* to the Set/Map of all known commits *before* delegating, then *remove* everything
+ // except
// the new (non-deleted) commits *after* delegating.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/core/SolrConfig.java
##########
@@ -817,14 +944,16 @@ public PluginInfo getPluginInfo(String type) {
return result.get(0);
}
- throw new SolrException(SolrException.ErrorCode.SERVER_ERROR,
- "Multiple plugins configured for type: " + type);
+ throw new SolrException(
+ SolrException.ErrorCode.SERVER_ERROR, "Multiple plugins configured for type: " + type);
}
private void initLibs(SolrResourceLoader loader, boolean isConfigsetTrusted) {
// TODO Want to remove SolrResourceLoader.getInstancePath; it can be on a Standalone subclass.
- // For Zk subclass, it's needed for the time being as well. We could remove that one if we remove two things
- // in SolrCloud: (1) instancePath/lib and (2) solrconfig lib directives with relative paths. Can wait till 9.0.
+ // For Zk subclass, it's needed for the time being as well. We could remove that one if we
+ // remove two things
+ // in SolrCloud: (1) instancePath/lib and (2) solrconfig lib directives with relative paths.
+ // Can wait till 9.0.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/core/NodeConfig.java
##########
@@ -107,24 +105,47 @@
private final PluginInfo tracerConfig;
- // Track if this config was loaded from zookeeper so that we can skip validating the zookeeper connection later
- // If it becomes necessary to track multiple potential sources in the future, replace this with an Enum
+ // Track if this config was loaded from zookeeper so that we can skip validating the zookeeper
+ // connection later
+ // If it becomes necessary to track multiple potential sources in the future, replace this with an
+ // Enum
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/core/SolrCores.java
##########
@@ -16,38 +16,42 @@
*/
package org.apache.solr.core;
+import java.lang.invoke.MethodHandles;
+import java.util.*;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.TimeUnit;
import org.apache.solr.common.SolrException;
import org.apache.solr.common.util.ExecutorUtil;
import org.apache.solr.common.util.SolrNamedThreadFactory;
import org.apache.solr.logging.MDCLoggingContext;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
-import java.lang.invoke.MethodHandles;
-import java.util.*;
-import java.util.concurrent.ConcurrentHashMap;
-import java.util.concurrent.ExecutorService;
-import java.util.concurrent.TimeUnit;
-
class SolrCores {
- private static final Object modifyLock = new Object(); // for locking around manipulating any of the core maps.
+ private static final Object modifyLock =
+ new Object(); // for locking around manipulating any of the core maps.
private final Map<String, SolrCore> cores = new LinkedHashMap<>(); // For "permanent" cores
// These descriptors, once loaded, will _not_ be unloaded, i.e. they are not "transient".
private final Map<String, CoreDescriptor> residentDescriptors = new LinkedHashMap<>();
private final CoreContainer container;
-
- private Set<String> currentlyLoadingCores = Collections.newSetFromMap(new ConcurrentHashMap<String,Boolean>());
+
+ private Set<String> currentlyLoadingCores =
+ Collections.newSetFromMap(new ConcurrentHashMap<String, Boolean>());
private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
- // This map will hold objects that are being currently operated on. The core (value) may be null in the case of
- // initial load. The rule is, never to any operation on a core that is currently being operated upon.
+ // This map will hold objects that are being currently operated on. The core (value) may be null
+ // in the case of
+ // initial load. The rule is, never to any operation on a core that is currently being operated
+ // upon.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/core/SolrCores.java
##########
@@ -16,38 +16,42 @@
*/
package org.apache.solr.core;
+import java.lang.invoke.MethodHandles;
+import java.util.*;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.TimeUnit;
import org.apache.solr.common.SolrException;
import org.apache.solr.common.util.ExecutorUtil;
import org.apache.solr.common.util.SolrNamedThreadFactory;
import org.apache.solr.logging.MDCLoggingContext;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
-import java.lang.invoke.MethodHandles;
-import java.util.*;
-import java.util.concurrent.ConcurrentHashMap;
-import java.util.concurrent.ExecutorService;
-import java.util.concurrent.TimeUnit;
-
class SolrCores {
- private static final Object modifyLock = new Object(); // for locking around manipulating any of the core maps.
+ private static final Object modifyLock =
+ new Object(); // for locking around manipulating any of the core maps.
private final Map<String, SolrCore> cores = new LinkedHashMap<>(); // For "permanent" cores
// These descriptors, once loaded, will _not_ be unloaded, i.e. they are not "transient".
private final Map<String, CoreDescriptor> residentDescriptors = new LinkedHashMap<>();
private final CoreContainer container;
-
- private Set<String> currentlyLoadingCores = Collections.newSetFromMap(new ConcurrentHashMap<String,Boolean>());
+
+ private Set<String> currentlyLoadingCores =
+ Collections.newSetFromMap(new ConcurrentHashMap<String, Boolean>());
private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
- // This map will hold objects that are being currently operated on. The core (value) may be null in the case of
- // initial load. The rule is, never to any operation on a core that is currently being operated upon.
+ // This map will hold objects that are being currently operated on. The core (value) may be null
+ // in the case of
+ // initial load. The rule is, never to any operation on a core that is currently being operated
+ // upon.
private static final Set<String> pendingCoreOps = new HashSet<>();
- // Due to the fact that closes happen potentially whenever anything is _added_ to the transient core list, we need
+ // Due to the fact that closes happen potentially whenever anything is _added_ to the transient
+ // core list, we need
// to essentially queue them up to be handled via pendingCoreOps.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/core/SolrCores.java
##########
@@ -404,21 +421,24 @@ protected SolrCore waitAddPendingCoreOps(String name) {
}
} while (pending);
// We _really_ need to do this within the synchronized block!
- if (! container.isShutDown()) {
- if (! pendingCoreOps.add(name)) {
+ if (!container.isShutDown()) {
+ if (!pendingCoreOps.add(name)) {
log.warn("Replaced an entry in pendingCoreOps {}, we should not be doing this", name);
}
- return getCoreFromAnyList(name, false); // we might have been _unloading_ the core, so return the core if it was loaded.
+ return getCoreFromAnyList(
+ name,
+ false); // we might have been _unloading_ the core, so return the core if it was loaded.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/core/SolrCores.java
##########
@@ -328,8 +343,10 @@ SolrCore getCoreFromAnyList(String name, boolean incRefCount, UUID coreId) {
}
}
- // See SOLR-5366 for why the UNLOAD command needs to know whether a core is actually loaded or not, it might have
- // to close the core. However, there's a race condition. If the core happens to be in the pending "to close" queue,
+ // See SOLR-5366 for why the UNLOAD command needs to know whether a core is actually loaded or
+ // not, it might have
+ // to close the core. However, there's a race condition. If the core happens to be in the pending
+ // "to close" queue,
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/core/SolrCores.java
##########
@@ -277,48 +288,52 @@ protected void swap(String n0, String n1) {
throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "No such core: " + n1);
}
}
- // When we swap the cores, we also need to swap the associated core descriptors. Note, this changes the
+ // When we swap the cores, we also need to swap the associated core descriptors. Note, this
+ // changes the
// name of the coreDescriptor by virtue of the c-tor
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/core/CoreContainer.java
##########
@@ -2218,8 +2414,10 @@ public void runAsync(Runnable r) {
}
// It's important that this be the _only_ thread removing things from pendingDynamicCloses!
- // This is single-threaded, but I tried a multi-threaded approach and didn't see any performance gains, so
- // there's no good justification for the complexity. I suspect that the locking on things like DefaultSolrCoreState
+ // This is single-threaded, but I tried a multi-threaded approach and didn't see any performance
+ // gains, so
+ // there's no good justification for the complexity. I suspect that the locking on things like
+ // DefaultSolrCoreState
// essentially create a single-threaded process anyway.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/core/SolrCores.java
##########
@@ -83,10 +87,11 @@ public void load(SolrResourceLoader loader) {
}
}
- // We are shutting down. You can't hold the lock on the various lists of cores while they shut down, so we need to
+ // We are shutting down. You can't hold the lock on the various lists of cores while they shut
+ // down, so we need to
// make a temporary copy of the names and shut them down outside the lock.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/core/SolrCores.java
##########
@@ -96,13 +101,16 @@ protected void close() {
}
}
- // It might be possible for one of the cores to move from one list to another while we're closing them. So
- // loop through the lists until they're all empty. In particular, the core could have moved from the transient
+ // It might be possible for one of the cores to move from one list to another while we're
+ // closing them. So
+ // loop through the lists until they're all empty. In particular, the core could have moved from
+ // the transient
// list to the pendingCloses list.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/core/SolrCores.java
##########
@@ -429,14 +449,17 @@ protected Object getModifyLock() {
return modifyLock;
}
- // Be a little careful. We don't want to either open or close a core unless it's _not_ being opened or closed by
- // another thread. So within this lock we'll walk along the list of pending closes until we find something NOT in
- // the list of threads currently being loaded or reloaded. The "usual" case will probably return the very first
+ // Be a little careful. We don't want to either open or close a core unless it's _not_ being
+ // opened or closed by
+ // another thread. So within this lock we'll walk along the list of pending closes until we find
+ // something NOT in
+ // the list of threads currently being loaded or reloaded. The "usual" case will probably return
+ // the very first
// one anyway..
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/core/SolrResourceLoader.java
##########
@@ -80,44 +79,71 @@
/**
* @since solr 1.3
*/
-public class SolrResourceLoader implements ResourceLoader, Closeable, SolrClassLoader, SolrCoreAware {
+public class SolrResourceLoader
+ implements ResourceLoader, Closeable, SolrClassLoader, SolrCoreAware {
private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
private static final String base = "org.apache.solr";
private static final String[] packages = {
- "", "analysis.", "schema.", "handler.", "handler.tagger.", "search.", "update.", "core.", "response.", "request.",
- "update.processor.", "util.", "spelling.", "handler.component.",
- "spelling.suggest.", "spelling.suggest.fst.", "rest.schema.analysis.", "security.", "handler.admin.",
- "security.jwt.", "security.hadoop.", "handler.sql.", "hdfs.", "hdfs.update."
+ "",
+ "analysis.",
+ "schema.",
+ "handler.",
+ "handler.tagger.",
+ "search.",
+ "update.",
+ "core.",
+ "response.",
+ "request.",
+ "update.processor.",
+ "util.",
+ "spelling.",
+ "handler.component.",
+ "spelling.suggest.",
+ "spelling.suggest.fst.",
+ "rest.schema.analysis.",
+ "security.",
+ "handler.admin.",
+ "security.jwt.",
+ "security.hadoop.",
+ "handler.sql.",
+ "hdfs.",
+ "hdfs.update."
};
private static final Charset UTF_8 = StandardCharsets.UTF_8;
- public static final String SOLR_ALLOW_UNSAFE_RESOURCELOADING_PARAM = "solr.allow.unsafe.resourceloading";
+ public static final String SOLR_ALLOW_UNSAFE_RESOURCELOADING_PARAM =
+ "solr.allow.unsafe.resourceloading";
private final boolean allowUnsafeResourceloading;
-
private String name = "";
protected URLClassLoader classLoader;
private final Path instanceDir;
private String coreName;
private UUID coreId;
private SolrConfig config;
private CoreContainer coreContainer;
- private PackageListeningClassLoader schemaLoader ;
+ private PackageListeningClassLoader schemaLoader;
- private PackageListeningClassLoader coreReloadingClassLoader ;
- private final List<SolrCoreAware> waitingForCore = Collections.synchronizedList(new ArrayList<>());
+ private PackageListeningClassLoader coreReloadingClassLoader;
+ private final List<SolrCoreAware> waitingForCore =
+ Collections.synchronizedList(new ArrayList<>());
private final List<SolrInfoBean> infoMBeans = Collections.synchronizedList(new ArrayList<>());
- private final List<ResourceLoaderAware> waitingForResources = Collections.synchronizedList(new ArrayList<>());
+ private final List<ResourceLoaderAware> waitingForResources =
+ Collections.synchronizedList(new ArrayList<>());
private volatile boolean live;
- // Provide a registry so that managed resources can register themselves while the XML configuration
+ // Provide a registry so that managed resources can register themselves while the XML
+ // configuration
// documents are being parsed ... after all are registered, they are asked by the RestManager to
- // initialize themselves. This two-step process is required because not all resources are available
+ // initialize themselves. This two-step process is required because not all resources are
+ // available
// (such as the SolrZkClient) when XML docs are being parsed.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/core/SolrCores.java
##########
@@ -404,21 +421,24 @@ protected SolrCore waitAddPendingCoreOps(String name) {
}
} while (pending);
// We _really_ need to do this within the synchronized block!
- if (! container.isShutDown()) {
- if (! pendingCoreOps.add(name)) {
+ if (!container.isShutDown()) {
+ if (!pendingCoreOps.add(name)) {
log.warn("Replaced an entry in pendingCoreOps {}, we should not be doing this", name);
}
- return getCoreFromAnyList(name, false); // we might have been _unloading_ the core, so return the core if it was loaded.
+ return getCoreFromAnyList(
+ name,
+ false); // we might have been _unloading_ the core, so return the core if it was loaded.
}
}
return null;
}
- // We should always be removing the first thing in the list with our name! The idea here is to NOT do anything n
+ // We should always be removing the first thing in the list with our name! The idea here is to NOT
+ // do anything n
// any core while some other operation is working on that core.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/core/TransientSolrCoreCacheFactory.java
##########
@@ -63,20 +66,32 @@ public static TransientSolrCoreCacheFactory newInstance(SolrResourceLoader loade
} catch (Exception e) {
// Many things could cause this, bad solrconfig, mis-typed class name, whatever.
// Throw an exception to stop loading here; never return null.
- throw new SolrException(SolrException.ErrorCode.SERVER_ERROR, "Error instantiating "
- + TransientSolrCoreCacheFactory.class.getName() + " class [" + info.className + "]", e);
+ throw new SolrException(
+ SolrException.ErrorCode.SERVER_ERROR,
+ "Error instantiating "
+ + TransientSolrCoreCacheFactory.class.getName()
+ + " class ["
+ + info.className
+ + "]",
+ e);
}
}
public static final PluginInfo DEFAULT_TRANSIENT_SOLR_CACHE_INFO =
- new PluginInfo("transientSolrCoreCacheFactory",
- ImmutableMap.of("class", TransientSolrCoreCacheFactoryDefault.class.getName(),
- "name", TransientSolrCoreCacheFactory.class.getName()),
- null, Collections.<PluginInfo>emptyList());
-
+ new PluginInfo(
+ "transientSolrCoreCacheFactory",
+ ImmutableMap.of(
+ "class",
+ TransientSolrCoreCacheFactoryDefault.class.getName(),
+ "name",
+ TransientSolrCoreCacheFactory.class.getName()),
+ null,
+ Collections.<PluginInfo>emptyList());
- // Need this because the plugin framework doesn't require a PluginINfo in the init method, don't see a way to
- // pass additional parameters and we need this when we create the transient core cache, it's _really_ important.
+ // Need this because the plugin framework doesn't require a PluginINfo in the init method, don't
+ // see a way to
+ // pass additional parameters and we need this when we create the transient core cache, it's
+ // _really_ important.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/core/SolrXmlConfig.java
##########
@@ -36,124 +58,117 @@
import org.w3c.dom.NodeList;
import org.xml.sax.InputSource;
-import javax.management.MBeanServer;
-import javax.xml.xpath.XPath;
-import javax.xml.xpath.XPathConstants;
-import javax.xml.xpath.XPathExpressionException;
-import java.io.ByteArrayInputStream;
-import java.io.InputStream;
-import java.lang.invoke.MethodHandles;
-import java.nio.charset.StandardCharsets;
-import java.nio.file.Files;
-import java.nio.file.Path;
-import java.util.ArrayList;
-import java.util.Arrays;
-import java.util.Collections;
-import java.util.HashMap;
-import java.util.HashSet;
-import java.util.List;
-import java.util.Map;
-import java.util.Properties;
-import java.util.Set;
-import java.util.regex.Pattern;
-
-import static org.apache.solr.common.params.CommonParams.NAME;
-
-
-/**
- * Loads {@code solr.xml}.
- */
+/** Loads {@code solr.xml}. */
public class SolrXmlConfig {
- // TODO should these from* methods return a NodeConfigBuilder so that the caller (a test) can make further
+ // TODO should these from* methods return a NodeConfigBuilder so that the caller (a test) can make
+ // further
// manipulations like add properties and set the CorePropertiesLocator and "async" mode?
- public final static String ZK_HOST = "zkHost";
- public final static String SOLR_XML_FILE = "solr.xml";
- public final static String SOLR_DATA_HOME = "solr.data.home";
+ public static final String ZK_HOST = "zkHost";
+ public static final String SOLR_XML_FILE = "solr.xml";
+ public static final String SOLR_DATA_HOME = "solr.data.home";
private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
private static final Pattern COMMA_SEPARATED_PATTERN = Pattern.compile("\\s*,\\s*");
/**
- * Given some node Properties, checks if non-null and a 'zkHost' is alread included. If so, the Properties are
- * returned as is. If not, then the returned value will be a new Properties, wrapping the original Properties,
- * with the 'zkHost' value set based on the value of the corispond System property (if set)
+ * Given some node Properties, checks if non-null and a 'zkHost' is alread included. If so, the
+ * Properties are returned as is. If not, then the returned value will be a new Properties,
+ * wrapping the original Properties, with the 'zkHost' value set based on the value of the
+ * corispond System property (if set)
*
- * In theory we only need this logic once, ideally in SolrDispatchFilter, but we put it here to re-use
- * redundently because of how much surface area our API has for various tests to poke at us.
+ * <p>In theory we only need this logic once, ideally in SolrDispatchFilter, but we put it here to
+ * re-use redundently because of how much surface area our API has for various tests to poke at
+ * us.
*/
public static Properties wrapAndSetZkHostFromSysPropIfNeeded(final Properties props) {
- if (null != props && ! StringUtils.isEmpty(props.getProperty(ZK_HOST))) {
+ if (null != props && !StringUtils.isEmpty(props.getProperty(ZK_HOST))) {
// nothing to do...
return props;
}
// we always wrap if we might set a property -- never mutate the original props
final Properties results = (null == props ? new Properties() : new Properties(props));
final String sysprop = System.getProperty(ZK_HOST);
- if (! StringUtils.isEmpty(sysprop)) {
+ if (!StringUtils.isEmpty(sysprop)) {
results.setProperty(ZK_HOST, sysprop);
}
return results;
}
-
public static NodeConfig fromConfig(Path solrHome, XmlConfigFile config, boolean fromZookeeper) {
checkForIllegalConfig(config);
- // sanity check: if our config came from zookeeper, then there *MUST* be Node Properties that tell us
- // what zkHost was used to read it (either via webapp context attribute, or that SolrDispatchFilter
+ // sanity check: if our config came from zookeeper, then there *MUST* be Node Properties that
+ // tell us
+ // what zkHost was used to read it (either via webapp context attribute, or that
+ // SolrDispatchFilter
// filled in for us from system properties)
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/core/SolrXmlConfig.java
##########
@@ -36,124 +58,117 @@
import org.w3c.dom.NodeList;
import org.xml.sax.InputSource;
-import javax.management.MBeanServer;
-import javax.xml.xpath.XPath;
-import javax.xml.xpath.XPathConstants;
-import javax.xml.xpath.XPathExpressionException;
-import java.io.ByteArrayInputStream;
-import java.io.InputStream;
-import java.lang.invoke.MethodHandles;
-import java.nio.charset.StandardCharsets;
-import java.nio.file.Files;
-import java.nio.file.Path;
-import java.util.ArrayList;
-import java.util.Arrays;
-import java.util.Collections;
-import java.util.HashMap;
-import java.util.HashSet;
-import java.util.List;
-import java.util.Map;
-import java.util.Properties;
-import java.util.Set;
-import java.util.regex.Pattern;
-
-import static org.apache.solr.common.params.CommonParams.NAME;
-
-
-/**
- * Loads {@code solr.xml}.
- */
+/** Loads {@code solr.xml}. */
public class SolrXmlConfig {
- // TODO should these from* methods return a NodeConfigBuilder so that the caller (a test) can make further
+ // TODO should these from* methods return a NodeConfigBuilder so that the caller (a test) can make
+ // further
// manipulations like add properties and set the CorePropertiesLocator and "async" mode?
- public final static String ZK_HOST = "zkHost";
- public final static String SOLR_XML_FILE = "solr.xml";
- public final static String SOLR_DATA_HOME = "solr.data.home";
+ public static final String ZK_HOST = "zkHost";
+ public static final String SOLR_XML_FILE = "solr.xml";
+ public static final String SOLR_DATA_HOME = "solr.data.home";
private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
private static final Pattern COMMA_SEPARATED_PATTERN = Pattern.compile("\\s*,\\s*");
/**
- * Given some node Properties, checks if non-null and a 'zkHost' is alread included. If so, the Properties are
- * returned as is. If not, then the returned value will be a new Properties, wrapping the original Properties,
- * with the 'zkHost' value set based on the value of the corispond System property (if set)
+ * Given some node Properties, checks if non-null and a 'zkHost' is alread included. If so, the
+ * Properties are returned as is. If not, then the returned value will be a new Properties,
+ * wrapping the original Properties, with the 'zkHost' value set based on the value of the
+ * corispond System property (if set)
*
- * In theory we only need this logic once, ideally in SolrDispatchFilter, but we put it here to re-use
- * redundently because of how much surface area our API has for various tests to poke at us.
+ * <p>In theory we only need this logic once, ideally in SolrDispatchFilter, but we put it here to
+ * re-use redundently because of how much surface area our API has for various tests to poke at
+ * us.
*/
public static Properties wrapAndSetZkHostFromSysPropIfNeeded(final Properties props) {
- if (null != props && ! StringUtils.isEmpty(props.getProperty(ZK_HOST))) {
+ if (null != props && !StringUtils.isEmpty(props.getProperty(ZK_HOST))) {
// nothing to do...
return props;
}
// we always wrap if we might set a property -- never mutate the original props
final Properties results = (null == props ? new Properties() : new Properties(props));
final String sysprop = System.getProperty(ZK_HOST);
- if (! StringUtils.isEmpty(sysprop)) {
+ if (!StringUtils.isEmpty(sysprop)) {
results.setProperty(ZK_HOST, sysprop);
}
return results;
}
-
public static NodeConfig fromConfig(Path solrHome, XmlConfigFile config, boolean fromZookeeper) {
checkForIllegalConfig(config);
- // sanity check: if our config came from zookeeper, then there *MUST* be Node Properties that tell us
- // what zkHost was used to read it (either via webapp context attribute, or that SolrDispatchFilter
+ // sanity check: if our config came from zookeeper, then there *MUST* be Node Properties that
+ // tell us
+ // what zkHost was used to read it (either via webapp context attribute, or that
+ // SolrDispatchFilter
// filled in for us from system properties)
- assert ( (! fromZookeeper) || (null != config.getSubstituteProperties()
- && null != config.getSubstituteProperties().getProperty(ZK_HOST)));
-
- // Regardless of where/how we this XmlConfigFile was loaded from, if it contains a zkHost property,
- // we're going to use that as our "default" and only *directly* check the system property if it's not specified.
+ assert ((!fromZookeeper)
+ || (null != config.getSubstituteProperties()
+ && null != config.getSubstituteProperties().getProperty(ZK_HOST)));
+
+ // Regardless of where/how we this XmlConfigFile was loaded from, if it contains a zkHost
+ // property,
+ // we're going to use that as our "default" and only *directly* check the system property if
+ // it's not specified.
//
- // (checking the sys prop here is really just for tests that by-pass SolrDispatchFilter. In non-test situations,
- // SolrDispatchFilter will check the system property if needed in order to try and load solr.xml from ZK, and
+ // (checking the sys prop here is really just for tests that by-pass SolrDispatchFilter. In
+ // non-test situations,
+ // SolrDispatchFilter will check the system property if needed in order to try and load solr.xml
+ // from ZK, and
// should have put the sys prop value in the node properties for us)
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/core/TransientSolrCoreCacheDefault.java
##########
@@ -196,8 +204,10 @@ public CoreDescriptor removeTransientDescriptor(String name) {
}
@Override
- public int getStatus(String coreName) { return 0; } //no_op for default handler.
+ public int getStatus(String coreName) {
+ return 0;
+ } // no_op for default handler.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/core/SolrXmlConfig.java
##########
@@ -36,124 +58,117 @@
import org.w3c.dom.NodeList;
import org.xml.sax.InputSource;
-import javax.management.MBeanServer;
-import javax.xml.xpath.XPath;
-import javax.xml.xpath.XPathConstants;
-import javax.xml.xpath.XPathExpressionException;
-import java.io.ByteArrayInputStream;
-import java.io.InputStream;
-import java.lang.invoke.MethodHandles;
-import java.nio.charset.StandardCharsets;
-import java.nio.file.Files;
-import java.nio.file.Path;
-import java.util.ArrayList;
-import java.util.Arrays;
-import java.util.Collections;
-import java.util.HashMap;
-import java.util.HashSet;
-import java.util.List;
-import java.util.Map;
-import java.util.Properties;
-import java.util.Set;
-import java.util.regex.Pattern;
-
-import static org.apache.solr.common.params.CommonParams.NAME;
-
-
-/**
- * Loads {@code solr.xml}.
- */
+/** Loads {@code solr.xml}. */
public class SolrXmlConfig {
- // TODO should these from* methods return a NodeConfigBuilder so that the caller (a test) can make further
+ // TODO should these from* methods return a NodeConfigBuilder so that the caller (a test) can make
+ // further
// manipulations like add properties and set the CorePropertiesLocator and "async" mode?
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/core/backup/repository/BackupRepositoryFactory.java
##########
@@ -59,21 +60,25 @@ public BackupRepositoryFactory(PluginInfo[] backupRepoPlugins) {
}
if (this.defaultBackupRepoPlugin != null) {
- log.info("Default configuration for backup repository is with configuration params {}",
- defaultBackupRepoPlugin);
+ log.info(
+ "Default configuration for backup repository is with configuration params {}",
+ defaultBackupRepoPlugin);
}
}
}
public BackupRepository newInstance(SolrResourceLoader loader, String name) {
Objects.requireNonNull(loader);
Objects.requireNonNull(name);
- PluginInfo repo = Objects.requireNonNull(backupRepoPluginByName.get(name),
+ PluginInfo repo =
+ Objects.requireNonNull(
+ backupRepoPluginByName.get(name),
"Could not find a backup repository with name " + name);
BackupRepository result = loader.newInstance(repo.className, BackupRepository.class);
if ("trackingBackupRepository".equals(name)) {
- // newInstance can be called by multiple threads, synchronization prevents simultaneous multi-threaded 'adds' from
+ // newInstance can be called by multiple threads, synchronization prevents simultaneous
+ // multi-threaded 'adds' from
// corrupting the namedlist
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/AnalysisRequestHandlerBase.java
##########
@@ -289,63 +294,62 @@ private int arrayCompare(int[] a, int[] b) {
tokenNamedList.add("match", true);
}
- token.reflectWith(new AttributeReflector() {
- @Override
- public void reflect(Class<? extends Attribute> attClass, String key, Object value) {
- // leave out position and bytes term
- if (TermToBytesRefAttribute.class.isAssignableFrom(attClass))
- return;
- if (CharTermAttribute.class.isAssignableFrom(attClass))
- return;
- if (PositionIncrementAttribute.class.isAssignableFrom(attClass))
- return;
-
- String k = attClass.getName() + '#' + key;
-
- // map keys for "standard attributes":
- if (ATTRIBUTE_MAPPING.containsKey(k)) {
- k = ATTRIBUTE_MAPPING.get(k);
- }
-
- if (value instanceof BytesRef) {
- final BytesRef p = (BytesRef) value;
- value = p.toString();
- }
+ token.reflectWith(
+ new AttributeReflector() {
+ @Override
+ public void reflect(Class<? extends Attribute> attClass, String key, Object value) {
+ // leave out position and bytes term
+ if (TermToBytesRefAttribute.class.isAssignableFrom(attClass)) return;
+ if (CharTermAttribute.class.isAssignableFrom(attClass)) return;
+ if (PositionIncrementAttribute.class.isAssignableFrom(attClass)) return;
- tokenNamedList.add(k, value);
- }
- });
+ String k = attClass.getName() + '#' + key;
+
+ // map keys for "standard attributes":
+ if (ATTRIBUTE_MAPPING.containsKey(k)) {
+ k = ATTRIBUTE_MAPPING.get(k);
+ }
+
+ if (value instanceof BytesRef) {
+ final BytesRef p = (BytesRef) value;
+ value = p.toString();
+ }
+
+ tokenNamedList.add(k, value);
+ }
+ });
tokensNamedLists.add(tokenNamedList);
}
return tokensNamedLists;
}
-
- private String writeCharStream(NamedList<Object> out, Reader input ){
+
+ private String writeCharStream(NamedList<Object> out, Reader input) {
final int BUFFER_SIZE = 1024;
char[] buf = new char[BUFFER_SIZE];
int len = 0;
StringBuilder sb = new StringBuilder();
do {
try {
- len = input.read( buf, 0, BUFFER_SIZE );
+ len = input.read(buf, 0, BUFFER_SIZE);
} catch (IOException e) {
throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, e);
}
- if( len > 0 )
- sb.append(buf, 0, len);
- } while( len == BUFFER_SIZE );
- out.add( input.getClass().getName(), sb.toString());
+ if (len > 0) sb.append(buf, 0, len);
+ } while (len == BUFFER_SIZE);
+ out.add(input.getClass().getName(), sb.toString());
return sb.toString();
}
- // ================================================= Inner classes =================================================
+ // ================================================= Inner classes
+ // =================================================
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/core/backup/BackupFilePaths.java
##########
@@ -29,148 +30,160 @@
import java.util.Optional;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
-
-import static org.apache.solr.core.backup.BackupId.TRADITIONAL_BACKUP;
-import static org.apache.solr.core.backup.BackupManager.ZK_STATE_DIR;
+import org.apache.solr.core.backup.repository.BackupRepository;
/**
* Utility class for getting paths related to backups, or parsing information out of those paths.
*/
public class BackupFilePaths {
- private static final Pattern BACKUP_PROPS_ID_PTN = Pattern.compile("backup_([0-9]+).properties");
- private BackupRepository repository;
- private URI backupLoc;
-
- /**
- * Create a BackupFilePaths object.
- *
- * @param repository the repository; used primarily to resolve URIs.
- * @param backupLoc the root location for a named backup. For traditional backups this is expected to take the form
- * baseLocation/backupName. For incremental backups this is expected to be of the form
- * baseLocation/backupName/collectionName.
- */
- public BackupFilePaths(BackupRepository repository, URI backupLoc) {
- this.repository = repository;
- this.backupLoc = backupLoc;
- }
-
- /**
- * Return a URI for the 'index' location, responsible for holding index files for all backups at this location.
- *
- * Only valid for incremental backups.
- */
- public URI getIndexDir() {
- return repository.resolveDirectory(backupLoc, "index");
- }
-
- /**
- * Return a URI for the 'shard_backup_metadata' location, which contains metadata files about each shard backup.
- *
- * Only valid for incremental backups.
- */
- public URI getShardBackupMetadataDir() {
- return repository.resolveDirectory(backupLoc, "shard_backup_metadata");
+ private static final Pattern BACKUP_PROPS_ID_PTN = Pattern.compile("backup_([0-9]+).properties");
+ private BackupRepository repository;
+ private URI backupLoc;
+
+ /**
+ * Create a BackupFilePaths object.
+ *
+ * @param repository the repository; used primarily to resolve URIs.
+ * @param backupLoc the root location for a named backup. For traditional backups this is expected
+ * to take the form baseLocation/backupName. For incremental backups this is expected to be of
+ * the form baseLocation/backupName/collectionName.
+ */
+ public BackupFilePaths(BackupRepository repository, URI backupLoc) {
+ this.repository = repository;
+ this.backupLoc = backupLoc;
+ }
+
+ /**
+ * Return a URI for the 'index' location, responsible for holding index files for all backups at
+ * this location.
+ *
+ * <p>Only valid for incremental backups.
+ */
+ public URI getIndexDir() {
+ return repository.resolveDirectory(backupLoc, "index");
+ }
+
+ /**
+ * Return a URI for the 'shard_backup_metadata' location, which contains metadata files about each
+ * shard backup.
+ *
+ * <p>Only valid for incremental backups.
+ */
+ public URI getShardBackupMetadataDir() {
+ return repository.resolveDirectory(backupLoc, "shard_backup_metadata");
+ }
+
+ public URI getBackupLocation() {
+ return backupLoc;
+ }
+
+ /**
+ * Create all locations required to store an incremental backup.
+ *
+ * @throws IOException for issues encountered using repository to create directories
+ */
+ public void createIncrementalBackupFolders() throws IOException {
+ repository.createDirectory(backupLoc);
+ repository.createDirectory(getIndexDir());
+ repository.createDirectory(getShardBackupMetadataDir());
+ }
+
+ /**
+ * Get the directory name used to hold backed up ZK state
+ *
+ * <p>Valid for both incremental and traditional backups.
+ *
+ * @param id the ID of the backup in question
+ */
+ public static String getZkStateDir(BackupId id) {
+ if (id.id == TRADITIONAL_BACKUP) {
+ return ZK_STATE_DIR;
}
-
- public URI getBackupLocation() {
- return backupLoc;
- }
-
- /**
- * Create all locations required to store an incremental backup.
- *
- * @throws IOException for issues encountered using repository to create directories
- */
- public void createIncrementalBackupFolders() throws IOException {
- repository.createDirectory(backupLoc);
- repository.createDirectory(getIndexDir());
- repository.createDirectory(getShardBackupMetadataDir());
+ return String.format(Locale.ROOT, "%s_%d", ZK_STATE_DIR, id.id);
+ }
+
+ /**
+ * Get the filename of the top-level backup properties file
+ *
+ * <p>Valid for both incremental and traditional backups.
+ *
+ * @param id the ID of the backup in question
+ */
+ public static String getBackupPropsName(BackupId id) {
+ if (id.id == TRADITIONAL_BACKUP) {
+ return BackupManager.TRADITIONAL_BACKUP_PROPS_FILE;
}
-
- /**
- * Get the directory name used to hold backed up ZK state
- *
- * Valid for both incremental and traditional backups.
- *
- * @param id the ID of the backup in question
- */
- public static String getZkStateDir(BackupId id) {
- if (id.id == TRADITIONAL_BACKUP) {
- return ZK_STATE_DIR;
- }
- return String.format(Locale.ROOT, "%s_%d", ZK_STATE_DIR, id.id);
- }
-
- /**
- * Get the filename of the top-level backup properties file
- *
- * Valid for both incremental and traditional backups.
- *
- * @param id the ID of the backup in question
- */
- public static String getBackupPropsName(BackupId id) {
- if (id.id == TRADITIONAL_BACKUP) {
- return BackupManager.TRADITIONAL_BACKUP_PROPS_FILE;
- }
- return getBackupPropsName(id.id);
+ return getBackupPropsName(id.id);
+ }
+
+ /**
+ * Identify all strings which appear to be the filename of a top-level backup properties file.
+ *
+ * <p>Only valid for incremental backups.
+ *
+ * @param listFiles a list of strings, filenames which may or may not correspond to backup
+ * properties files
+ */
+ public static List<BackupId> findAllBackupIdsFromFileListing(String[] listFiles) {
+ List<BackupId> result = new ArrayList<>();
+ for (String file : listFiles) {
+ Matcher m = BACKUP_PROPS_ID_PTN.matcher(file);
+ if (m.find()) {
+ result.add(new BackupId(Integer.parseInt(m.group(1))));
+ }
}
- /**
- * Identify all strings which appear to be the filename of a top-level backup properties file.
- *
- * Only valid for incremental backups.
- *
- * @param listFiles a list of strings, filenames which may or may not correspond to backup properties files
- */
- public static List<BackupId> findAllBackupIdsFromFileListing(String[] listFiles) {
- List<BackupId> result = new ArrayList<>();
- for (String file: listFiles) {
- Matcher m = BACKUP_PROPS_ID_PTN.matcher(file);
- if (m.find()) {
- result.add(new BackupId(Integer.parseInt(m.group(1))));
- }
- }
-
- return result;
+ return result;
+ }
+
+ /**
+ * Identify the string from an array of filenames which represents the most recent top-level
+ * backup properties file.
+ *
+ * <p>Only valid for incremental backups.
+ *
+ * @param listFiles a list of strings, filenames which may or may not correspond to backup
+ * properties files.
+ */
+ public static Optional<BackupId> findMostRecentBackupIdFromFileListing(String[] listFiles) {
+ return findAllBackupIdsFromFileListing(listFiles).stream()
+ .max(Comparator.comparingInt(o -> o.id));
+ }
+
+ /**
+ * Builds the URI for the backup location given the user-provided 'location' and backup 'name'.
+ *
+ * @param repository the backup repository, used to list files and resolve URI's.
+ * @param location a URI representing the repository location holding each backup name
+ * @param backupName the specific backup name to create a URI for
+ */
+ public static URI buildExistingBackupLocationURI(
+ BackupRepository repository, URI location, String backupName) throws IOException {
+ final URI backupNameUri = repository.resolveDirectory(location, backupName);
+ final String[] entries = repository.listAll(backupNameUri);
+ final boolean incremental =
+ !Arrays.stream(entries)
+ .anyMatch(entry -> entry.equals(BackupManager.TRADITIONAL_BACKUP_PROPS_FILE));
+ if (incremental) {
+ // Incremental backups have an additional URI path component representing the collection that
+ // was backed up.
+ // This collection directory is the path assumed by other backup code.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/core/backup/BackupManager.java
##########
@@ -312,26 +345,31 @@ private void downloadConfigToRepo(ConfigSetService configSetService, String conf
}
}
- private void uploadConfigToSolrCloud(ConfigSetService configSetService, URI sourceDir, String configName, String filePrefix) throws IOException {
+ private void uploadConfigToSolrCloud(
+ ConfigSetService configSetService, URI sourceDir, String configName, String filePrefix)
+ throws IOException {
for (String file : repository.listAll(sourceDir)) {
String filePath = filePrefix + file;
URI path = repository.resolve(sourceDir, file);
BackupRepository.PathType t = repository.getPathType(path);
switch (t) {
- case FILE: {
- try (IndexInput is = repository.openInput(sourceDir, file, IOContext.DEFAULT)) {
- byte[] arr = new byte[(int) is.length()]; // probably ok since the config file should be small.
- is.readBytes(arr, 0, (int) is.length());
- configSetService.uploadFileToConfig(configName, filePath, arr, false);
+ case FILE:
+ {
+ try (IndexInput is = repository.openInput(sourceDir, file, IOContext.DEFAULT)) {
+ byte[] arr =
+ new byte[(int) is.length()]; // probably ok since the config file should be small.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/filestore/DistribPackageStore.java
##########
@@ -486,16 +501,20 @@ public void delete(String path) {
List<String> nodes = coreContainer.getPackageStoreAPI().shuffledNodes();
HttpClient client = coreContainer.getUpdateShardHandler().getDefaultHttpClient();
for (String node : nodes) {
- String baseUrl = coreContainer.getZkController().getZkStateReader().getBaseUrlForNodeName(node);
+ String baseUrl =
+ coreContainer.getZkController().getZkStateReader().getBaseUrlForNodeName(node);
String url = baseUrl.replace("/solr", "/api") + "/node/files" + path;
HttpDelete del = new HttpDelete(url);
- coreContainer.runAsync(() -> Utils.executeHttpMethod(client, url, null, del));//invoke delete command on all nodes asynchronously
+ coreContainer.runAsync(
+ () ->
+ Utils.executeHttpMethod(
+ client, url, null, del)); // invoke delete command on all nodes asynchronously
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/cluster/placement/plugins/AffinityPlacementFactory.java
##########
@@ -648,23 +821,34 @@ private void makePlacementDecisions(SolrCollection solrCollection, String shardN
}
}
- // Insert back a corrected entry for the winning AZ: one more replica living there and one less node that can accept new replicas
- // (the remaining candidate node list might be empty, in which case it will be cleaned up on the next iteration).
+ // Insert back a corrected entry for the winning AZ: one more replica living there and one
+ // less node that can accept new replicas
+ // (the remaining candidate node list might be empty, in which case it will be cleaned up on
+ // the next iteration).
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/core/snapshots/SolrSnapshotManager.java
##########
@@ -250,57 +266,66 @@ public void onCommit(List<? extends IndexCommit> commits)
* @param snapshots The snapshots to be preserved.
* @throws IOException in case of I/O errors.
*/
- public static void deleteNonSnapshotIndexFiles(SolrCore core, Directory dir, Collection<SnapshotMetaData> snapshots) throws IOException {
+ public static void deleteNonSnapshotIndexFiles(
+ SolrCore core, Directory dir, Collection<SnapshotMetaData> snapshots) throws IOException {
final Set<Long> genNumbers = new HashSet<>();
for (SnapshotMetaData m : snapshots) {
genNumbers.add(m.getGenerationNumber());
}
- deleteSnapshotIndexFiles(core, dir, new IndexDeletionPolicy() {
- @Override
- public void onInit(List<? extends IndexCommit> commits) throws IOException {
- for (IndexCommit ic : commits) {
- if (!genNumbers.contains(ic.getGeneration())) {
- if (log.isInfoEnabled()) {
- log.info("Deleting non-snapshotted index commit with generation {}", ic.getGeneration());
+ deleteSnapshotIndexFiles(
+ core,
+ dir,
+ new IndexDeletionPolicy() {
+ @Override
+ public void onInit(List<? extends IndexCommit> commits) throws IOException {
+ for (IndexCommit ic : commits) {
+ if (!genNumbers.contains(ic.getGeneration())) {
+ if (log.isInfoEnabled()) {
+ log.info(
+ "Deleting non-snapshotted index commit with generation {}",
+ ic.getGeneration());
+ }
+ ic.delete();
+ }
}
- ic.delete();
}
- }
- }
- @Override
- public void onCommit(List<? extends IndexCommit> commits)
- throws IOException {}
- });
+ @Override
+ public void onCommit(List<? extends IndexCommit> commits) throws IOException {}
+ });
}
/**
- * This method deletes index files of the {@linkplain IndexCommit} for the specified generation number.
+ * This method deletes index files of the {@linkplain IndexCommit} for the specified generation
+ * number.
*
* @param core The Solr core
* @param dir The index directory storing the snapshot.
* @throws IOException in case of I/O errors.
*/
-
@SuppressWarnings({"try", "unused"})
- private static void deleteSnapshotIndexFiles(SolrCore core, Directory dir, IndexDeletionPolicy delPolicy) throws IOException {
+ private static void deleteSnapshotIndexFiles(
+ SolrCore core, Directory dir, IndexDeletionPolicy delPolicy) throws IOException {
IndexWriterConfig conf = core.getSolrConfig().indexConfig.toIndexWriterConfig(core);
conf.setOpenMode(OpenMode.APPEND);
- conf.setMergePolicy(NoMergePolicy.INSTANCE);//Don't want to merge any commits here!
+ conf.setMergePolicy(NoMergePolicy.INSTANCE); // Don't want to merge any commits here!
conf.setIndexDeletionPolicy(delPolicy);
conf.setCodec(core.getCodec());
try (SolrIndexWriter iw = new SolrIndexWriter("SolrSnapshotCleaner", dir, conf)) {
- // Do nothing. The only purpose of opening index writer is to invoke the Lucene IndexDeletionPolicy#onInit
+ // Do nothing. The only purpose of opening index writer is to invoke the Lucene
+ // IndexDeletionPolicy#onInit
// method so that we can cleanup the files associated with specified index commit.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/core/ZkContainer.java
##########
@@ -47,38 +46,38 @@
import org.slf4j.LoggerFactory;
/**
- * Used by {@link CoreContainer} to hold ZooKeeper / SolrCloud info, especially {@link ZkController}.
- * Mainly it does some ZK initialization, and ensures a loading core registers in ZK.
- * Even when in standalone mode, perhaps surprisingly, an instance of this class exists.
- * If {@link #getZkController()} returns null then we're in standalone mode.
+ * Used by {@link CoreContainer} to hold ZooKeeper / SolrCloud info, especially {@link
+ * ZkController}. Mainly it does some ZK initialization, and ensures a loading core registers in ZK.
+ * Even when in standalone mode, perhaps surprisingly, an instance of this class exists. If {@link
+ * #getZkController()} returns null then we're in standalone mode.
*/
public class ZkContainer {
- // NOTE DWS: It's debatable if this in-between class is needed instead of folding it all into ZkController.
+ // NOTE DWS: It's debatable if this in-between class is needed instead of folding it all into
+ // ZkController.
// ZKC is huge though.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/CatStream.java
##########
@@ -229,12 +236,17 @@ private void findReadableFiles(CrawlFile seed, List<CrawlFile> foundFiles) {
foundFiles.add(seed);
} else if (Files.isDirectory(entry)) {
try (Stream<Path> directoryContents = Files.list(entry)) {
- directoryContents.sorted().forEach(iPath -> {
- // debatable: should the separator be OS/file-system specific, or perhaps always "/" ?
- final String displayPathSeparator = iPath.getFileSystem().getSeparator();
- final String itemDisplayPath = seed.displayPath + displayPathSeparator + iPath.getFileName();
- findReadableFiles(new CrawlFile(itemDisplayPath, iPath), foundFiles);
- });
+ directoryContents
+ .sorted()
+ .forEach(
+ iPath -> {
+ // debatable: should the separator be OS/file-system specific, or perhaps always
+ // "/" ?
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/IndexFetcher.java
##########
@@ -253,8 +280,9 @@ public IndexFetcher(final NamedList<?> initArgs, final ReplicationHandler handle
useInternalCompression = INTERNAL.equals(compress);
useExternalCompression = EXTERNAL.equals(compress);
connTimeout = getParameter(initArgs, HttpClientUtil.PROP_CONNECTION_TIMEOUT, 30000, null);
-
- // allow a leader override for tests - you specify this in /replication follower section of solrconfig and some
+
+ // allow a leader override for tests - you specify this in /replication follower section of
+ // solrconfig and some
// test don't want to define this
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/AnalysisRequestHandlerBase.java
##########
@@ -383,33 +387,42 @@ public boolean incrementToken() {
}
}
-
protected void addAttributes(AttributeSource attributeSource) {
- // note: ideally we wouldn't call addAttributeImpl which is marked internal. But nonetheless it's possible
- // this method is used by some custom attributes, especially since Solr doesn't provide a way to customize the
- // AttributeFactory which is the recommended way to choose which classes implement which attributes.
+ // note: ideally we wouldn't call addAttributeImpl which is marked internal. But nonetheless
+ // it's possible
+ // this method is used by some custom attributes, especially since Solr doesn't provide a way
+ // to customize the
+ // AttributeFactory which is the recommended way to choose which classes implement which
+ // attributes.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/DocumentAnalysisRequestHandler.java
##########
@@ -121,20 +121,19 @@ public String getDescription() {
return "Provides a breakdown of the analysis process of provided documents";
}
-
- //================================================ Helper Methods ==================================================
+ // ================================================ Helper Methods
+ // ==================================================
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/IndexFetcher.java
##########
@@ -1005,36 +1133,52 @@ private long downloadIndexFiles(boolean downloadCompleteIndex, Directory indexDi
}
if (log.isInfoEnabled()) {
- log.info("tmpIndexDir_type : {} , {}", tmpIndexDir.getClass(), FilterDirectory.unwrap(tmpIndexDir));
+ log.info(
+ "tmpIndexDir_type : {} , {}",
+ tmpIndexDir.getClass(),
+ FilterDirectory.unwrap(tmpIndexDir));
}
long usableSpace = usableDiskSpaceProvider.apply(tmpIndexDirPath);
if (getApproxTotalSpaceReqd(totalSpaceRequired) > usableSpace) {
deleteFilesInAdvance(indexDir, indexDirPath, totalSpaceRequired, usableSpace);
}
- for (Map<String,Object> file : filesToDownload) {
+ for (Map<String, Object> file : filesToDownload) {
String filename = (String) file.get(NAME);
long size = (Long) file.get(SIZE);
- CompareResult compareResult = compareFile(indexDir, filename, size, (Long) file.get(CHECKSUM));
+ CompareResult compareResult =
+ compareFile(indexDir, filename, size, (Long) file.get(CHECKSUM));
boolean alwaysDownload = filesToAlwaysDownloadIfNoChecksums(filename, size, compareResult);
if (log.isDebugEnabled()) {
- log.debug("Downloading file={} size={} checksum={} alwaysDownload={}", filename, size, file.get(CHECKSUM), alwaysDownload);
+ log.debug(
+ "Downloading file={} size={} checksum={} alwaysDownload={}",
+ filename,
+ size,
+ file.get(CHECKSUM),
+ alwaysDownload);
}
if (!compareResult.equal || downloadCompleteIndex || alwaysDownload) {
File localFile = new File(indexDirPath, filename);
- if (downloadCompleteIndex && doDifferentialCopy && compareResult.equal && compareResult.checkSummed
+ if (downloadCompleteIndex
+ && doDifferentialCopy
+ && compareResult.equal
+ && compareResult.checkSummed
&& localFile.exists()) {
if (log.isInfoEnabled()) {
- log.info("Don't need to download this file. Local file's path is: {}, checksum is: {}",
- localFile.getAbsolutePath(), file.get(CHECKSUM));
+ log.info(
+ "Don't need to download this file. Local file's path is: {}, checksum is: {}",
+ localFile.getAbsolutePath(),
+ file.get(CHECKSUM));
}
- // A hard link here should survive the eventual directory move, and should be more space efficient as
+ // A hard link here should survive the eventual directory move, and should be more space
+ // efficient as
// compared to a file copy. TODO: Maybe we could do a move safely here?
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/IndexFetcher.java
##########
@@ -900,7 +1015,8 @@ private StringBuilder readToStringBuilder(long replicationTime, String str) {
private void openNewSearcherAndUpdateCommitPoint() throws IOException {
RefCounted<SolrIndexSearcher> searcher = null;
IndexCommit commitPoint;
- // must get the latest solrCore object because the one we have might be closed because of a reload
+ // must get the latest solrCore object because the one we have might be closed because of a
+ // reload
// todo stop keeping solrCore around
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/IndexFetcher.java
##########
@@ -612,9 +707,9 @@ IndexFetchResult fetchLatestIndex(boolean forceReplication, boolean forceCoreRel
}
log.info("Configuration files are modified, core will be reloaded");
- logReplicationTimeAndConfFiles(modifiedConfFiles,
- successfulInstall);// write to a file time of replication and
- // conf files.
+ logReplicationTimeAndConfFiles(
+ modifiedConfFiles, successfulInstall); // write to a file time of replication and
+ // conf files.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/IndexFetcher.java
##########
@@ -507,29 +570,43 @@ IndexFetchResult fetchLatestIndex(boolean forceReplication, boolean forceCoreRel
}
// Create the sync service
- fsyncService = ExecutorUtil.newMDCAwareSingleThreadExecutor(new SolrNamedThreadFactory("fsyncService"));
+ fsyncService =
+ ExecutorUtil.newMDCAwareSingleThreadExecutor(new SolrNamedThreadFactory("fsyncService"));
// use a synchronized list because the list is read by other threads (to show details)
filesDownloaded = Collections.synchronizedList(new ArrayList<Map<String, Object>>());
- // if the generation of leader is older than that of the follower , it means they are not compatible to be copied
+ // if the generation of leader is older than that of the follower , it means they are not
+ // compatible to be copied
// then a new index directory to be created and all the files need to be copied
- boolean isFullCopyNeeded = IndexDeletionPolicyWrapper
- .getCommitTimestamp(commit) >= latestVersion
- || commit.getGeneration() >= latestGeneration || forceReplication;
+ boolean isFullCopyNeeded =
+ IndexDeletionPolicyWrapper.getCommitTimestamp(commit) >= latestVersion
+ || commit.getGeneration() >= latestGeneration
+ || forceReplication;
String timestamp = new SimpleDateFormat(SnapShooter.DATE_FMT, Locale.ROOT).format(new Date());
String tmpIdxDirName = "index." + timestamp;
tmpIndexDirPath = solrCore.getDataDir() + tmpIdxDirName;
- tmpIndexDir = solrCore.getDirectoryFactory().get(tmpIndexDirPath, DirContext.DEFAULT, solrCore.getSolrConfig().indexConfig.lockType);
+ tmpIndexDir =
+ solrCore
+ .getDirectoryFactory()
+ .get(
+ tmpIndexDirPath,
+ DirContext.DEFAULT,
+ solrCore.getSolrConfig().indexConfig.lockType);
// cindex dir...
indexDirPath = solrCore.getIndexDir();
- indexDir = solrCore.getDirectoryFactory().get(indexDirPath, DirContext.DEFAULT, solrCore.getSolrConfig().indexConfig.lockType);
+ indexDir =
+ solrCore
+ .getDirectoryFactory()
+ .get(indexDirPath, DirContext.DEFAULT, solrCore.getSolrConfig().indexConfig.lockType);
try {
- // We will compare all the index files from the leader vs the index files on disk to see if there is a mismatch
- // in the metadata. If there is a mismatch for the same index file then we download the entire index
+ // We will compare all the index files from the leader vs the index files on disk to see if
+ // there is a mismatch
+ // in the metadata. If there is a mismatch for the same index file then we download the
+ // entire index
// (except when differential copy is applicable) again.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/IndexFetcher.java
##########
@@ -507,29 +570,43 @@ IndexFetchResult fetchLatestIndex(boolean forceReplication, boolean forceCoreRel
}
// Create the sync service
- fsyncService = ExecutorUtil.newMDCAwareSingleThreadExecutor(new SolrNamedThreadFactory("fsyncService"));
+ fsyncService =
+ ExecutorUtil.newMDCAwareSingleThreadExecutor(new SolrNamedThreadFactory("fsyncService"));
// use a synchronized list because the list is read by other threads (to show details)
filesDownloaded = Collections.synchronizedList(new ArrayList<Map<String, Object>>());
- // if the generation of leader is older than that of the follower , it means they are not compatible to be copied
+ // if the generation of leader is older than that of the follower , it means they are not
+ // compatible to be copied
// then a new index directory to be created and all the files need to be copied
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/IndexFetcher.java
##########
@@ -1238,15 +1404,16 @@ private boolean isIndexStale(Directory dir) throws IOException {
/**
* Copy a file by the File#renameTo() method. If it fails, it is considered a failure
- * <p/>
+ *
+ * <p>
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/IndexFetcher.java
##########
@@ -1073,25 +1220,27 @@ private static Long getUsableSpace(String dir) {
}
}
-
-
private long getApproxTotalSpaceReqd(long totalSpaceRequired) {
- long approxTotalSpaceReqd = (long) (totalSpaceRequired * 1.05);// add 5% extra for safety
- approxTotalSpaceReqd += (100 * 1024 * 1024); //we should have an extra of 100MB free after everything is downloaded
+ long approxTotalSpaceReqd = (long) (totalSpaceRequired * 1.05); // add 5% extra for safety
+ approxTotalSpaceReqd +=
+ (100 * 1024 * 1024); // we should have an extra of 100MB free after everything is downloaded
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/MoreLikeThisHandler.java
##########
@@ -321,53 +312,66 @@ public MoreLikeThisHelper( SolrParams params, SolrIndexSearcher searcher )
}
}
String[] fields = list.toArray(new String[list.size()]);
- if( fields.length < 1 ) {
- throw new SolrException( SolrException.ErrorCode.BAD_REQUEST,
- "MoreLikeThis requires at least one similarity field: "+MoreLikeThisParams.SIMILARITY_FIELDS );
+ if (fields.length < 1) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "MoreLikeThis requires at least one similarity field: "
+ + MoreLikeThisParams.SIMILARITY_FIELDS);
}
-
- this.mlt = new MoreLikeThis( reader ); // TODO -- after LUCENE-896, we can use , searcher.getSimilarity() );
+
+ this.mlt =
+ new MoreLikeThis(
+ reader); // TODO -- after LUCENE-896, we can use , searcher.getSimilarity() );
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/ReplicationHandler.java
##########
@@ -267,12 +279,14 @@ public void handleRequestBody(SolrQueryRequest req, SolrQueryResponse rsp) throw
// in a catastrophic failure, but will result in the client getting an empty file list for
// the CMD_GET_FILE_LIST command.
//
- core.getDeletionPolicy().setReserveDuration(commitPoint.getGeneration(), reserveCommitDuration);
+ core.getDeletionPolicy()
+ .setReserveDuration(commitPoint.getGeneration(), reserveCommitDuration);
rsp.add(CMD_INDEX_VERSION, IndexDeletionPolicyWrapper.getCommitTimestamp(commitPoint));
rsp.add(GENERATION, commitPoint.getGeneration());
rsp.add(STATUS, OK_STATUS);
} else {
- // This happens when replication is not configured to happen after startup and no commit/optimize
+ // This happens when replication is not configured to happen after startup and no
+ // commit/optimize
// has happened yet.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/ReplicationHandler.java
##########
@@ -1373,45 +1468,45 @@ public void inform(SolrCore core) {
}
// check leader or follower is enabled
- private boolean isEnabled(NamedList<?> params ){
- if( params == null ) return false;
- Object enable = params.get( "enable" );
- if( enable == null ) return true;
- if( enable instanceof String )
- return StrUtils.parseBool( (String)enable );
- return Boolean.TRUE.equals( enable );
+ private boolean isEnabled(NamedList<?> params) {
+ if (params == null) return false;
+ Object enable = params.get("enable");
+ if (enable == null) return true;
+ if (enable instanceof String) return StrUtils.parseBool((String) enable);
+ return Boolean.TRUE.equals(enable);
}
- private final CloseHook startShutdownHook = new CloseHook() {
- @Override
- public void preClose(SolrCore core) {
- if (executorService != null)
- executorService.shutdown(); // we don't wait for shutdown - this can deadlock core reload
- }
+ private final CloseHook startShutdownHook =
+ new CloseHook() {
+ @Override
+ public void preClose(SolrCore core) {
+ if (executorService != null)
+ executorService
+ .shutdown(); // we don't wait for shutdown - this can deadlock core reload
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/ReplicationHandler.java
##########
@@ -508,8 +536,10 @@ private void restore(SolrParams params, SolrQueryResponse rsp, SolrQueryRequest
URI locationUri = repo.createDirectoryURI(location);
- //If name is not provided then look for the last unnamed( the ones with the snapshot.timestamp format)
- //snapshot folder since we allow snapshots to be taken without providing a name. Pick the latest timestamp.
+ // If name is not provided then look for the last unnamed( the ones with the snapshot.timestamp
+ // format)
+ // snapshot folder since we allow snapshots to be taken without providing a name. Pick the
+ // latest timestamp.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/IndexFetcher.java
##########
@@ -1046,23 +1190,26 @@ private long downloadIndexFiles(boolean downloadCompleteIndex, Directory indexDi
}
}
}
- log.info("Bytes downloaded: {}, Bytes skipped downloading: {}", bytesDownloaded, bytesSkippedCopying);
+ log.info(
+ "Bytes downloaded: {}, Bytes skipped downloading: {}",
+ bytesDownloaded,
+ bytesSkippedCopying);
return bytesDownloaded;
}
- //only for testing purposes. do not use this anywhere else
- //-----------START----------------------
+ // only for testing purposes. do not use this anywhere else
+ // -----------START----------------------
static BooleanSupplier testWait = () -> true;
static Function<String, Long> usableDiskSpaceProvider = dir -> getUsableSpace(dir);
- //------------ END---------------------
-
+ // ------------ END---------------------
private static Long getUsableSpace(String dir) {
try {
File file = new File(dir);
if (!file.exists()) {
file = file.getParentFile();
- if (!file.exists()) {//this is not a disk directory . so just pretend that there is enough space
+ if (!file.exists()) { // this is not a disk directory . so just pretend that there is enough
+ // space
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/ReplicationHandler.java
##########
@@ -1617,15 +1715,16 @@ public void write(OutputStream out) throws IOException {
fos.flush();
log.debug("Wrote {} bytes for file {}", offset + read, fileName); // nowarn
- //Pause if necessary
+ // Pause if necessary
maxBytesBeforePause += read;
if (maxBytesBeforePause >= rateLimiter.getMinPauseCheckBytes()) {
rateLimiter.pause(maxBytesBeforePause);
maxBytesBeforePause = 0;
}
if (read != buf.length) {
writeNothingAndFlush();
- fos.close(); // we close because DeflaterOutputStream requires a close call, but but the request outputstream is protected
+ fos.close(); // we close because DeflaterOutputStream requires a close call, but but
+ // the request outputstream is protected
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/IndexFetcher.java
##########
@@ -1635,156 +1818,176 @@ private int fetchPackets(FastInputStream fis) throws Exception {
}
long checkSumServer = -1;
fis.readFully(intbytes);
- //read the size of the packet
+ // read the size of the packet
int packetSize = readInt(intbytes);
if (packetSize <= 0) {
log.warn("No content received for file: {}", fileName);
return NO_CONTENT;
}
- //TODO consider recoding the remaining logic to not use/need buf[]; instead use the internal buffer of fis
+ // TODO consider recoding the remaining logic to not use/need buf[]; instead use the
+ // internal buffer of fis
if (buf.length < packetSize) {
- //This shouldn't happen since sender should use PACKET_SZ and we init the buf based on that too
+ // This shouldn't happen since sender should use PACKET_SZ and we init the buf based on
+ // that too
buf = new byte[packetSize];
}
if (checksum != null) {
- //read the checksum
+ // read the checksum
fis.readFully(longbytes);
checkSumServer = readLong(longbytes);
}
- //then read the packet of bytes
+ // then read the packet of bytes
fis.readFully(buf, 0, packetSize);
- //compare the checksum as sent from the leader
+ // compare the checksum as sent from the leader
if (includeChecksum) {
checksum.reset();
checksum.update(buf, 0, packetSize);
long checkSumClient = checksum.getValue();
if (checkSumClient != checkSumServer) {
log.error("Checksum not matched between client and server for file: {}", fileName);
- //if checksum is wrong it is a problem return (there doesn't seem to be a retry in this case.)
+ // if checksum is wrong it is a problem return (there doesn't seem to be a retry in
+ // this case.)
return 1;
}
}
- //if everything is fine, write down the packet to the file
+ // if everything is fine, write down the packet to the file
file.write(buf, packetSize);
bytesDownloaded += packetSize;
log.debug("Fetched and wrote {} bytes of file: {}", bytesDownloaded, fileName);
- //errorCount is always set to zero after a successful packet
+ // errorCount is always set to zero after a successful packet
errorCount = 0;
- if (bytesDownloaded >= size)
- return 0;
+ if (bytesDownloaded >= size) return 0;
}
} catch (ReplicationHandlerException e) {
throw e;
} catch (Exception e) {
- log.warn("Error in fetching file: {} (downloaded {} of {} bytes)",
- fileName, bytesDownloaded, size, e);
- //for any failure, increment the error count
+ log.warn(
+ "Error in fetching file: {} (downloaded {} of {} bytes)",
+ fileName,
+ bytesDownloaded,
+ size,
+ e);
+ // for any failure, increment the error count
errorCount++;
- //if it fails for the same packet for MAX_RETRIES fail and come out
+ // if it fails for the same packet for MAX_RETRIES fail and come out
if (errorCount > MAX_RETRIES) {
- throw new SolrException(SolrException.ErrorCode.SERVER_ERROR,
- "Failed to fetch file: " + fileName +
- " (downloaded " + bytesDownloaded + " of " + size + " bytes" +
- ", error count: " + errorCount + " > " + MAX_RETRIES + ")", e);
+ throw new SolrException(
+ SolrException.ErrorCode.SERVER_ERROR,
+ "Failed to fetch file: "
+ + fileName
+ + " (downloaded "
+ + bytesDownloaded
+ + " of "
+ + size
+ + " bytes"
+ + ", error count: "
+ + errorCount
+ + " > "
+ + MAX_RETRIES
+ + ")",
+ e);
}
return ERR;
}
}
/**
- * The webcontainer flushes the data only after it fills the buffer size. So, all data has to be read as readFully()
- * other wise it fails. So read everything as bytes and then extract an integer out of it
+ * The webcontainer flushes the data only after it fills the buffer size. So, all data has to be
+ * read as readFully() other wise it fails. So read everything as bytes and then extract an
+ * integer out of it
*/
private int readInt(byte[] b) {
- return (((b[0] & 0xff) << 24) | ((b[1] & 0xff) << 16)
- | ((b[2] & 0xff) << 8) | (b[3] & 0xff));
-
+ return (((b[0] & 0xff) << 24) | ((b[1] & 0xff) << 16) | ((b[2] & 0xff) << 8) | (b[3] & 0xff));
}
- /**
- * Same as above but to read longs from a byte array
- */
+ /** Same as above but to read longs from a byte array */
private long readLong(byte[] b) {
- return (((long) (b[0] & 0xff)) << 56) | (((long) (b[1] & 0xff)) << 48)
- | (((long) (b[2] & 0xff)) << 40) | (((long) (b[3] & 0xff)) << 32)
- | (((long) (b[4] & 0xff)) << 24) | ((b[5] & 0xff) << 16)
- | ((b[6] & 0xff) << 8) | ((b[7] & 0xff));
-
+ return (((long) (b[0] & 0xff)) << 56)
+ | (((long) (b[1] & 0xff)) << 48)
+ | (((long) (b[2] & 0xff)) << 40)
+ | (((long) (b[3] & 0xff)) << 32)
+ | (((long) (b[4] & 0xff)) << 24)
+ | ((b[5] & 0xff) << 16)
+ | ((b[6] & 0xff) << 8)
+ | ((b[7] & 0xff));
}
- /**
- * cleanup everything
- */
+ /** cleanup everything */
private void cleanup() {
try {
file.close();
- } catch (Exception e) {/* no-op */
+ } catch (Exception e) {
+ /* no-op */
log.error("Error closing file: {}", this.saveAs, e);
}
if (bytesDownloaded != size) {
- //if the download is not complete then
- //delete the file being downloaded
+ // if the download is not complete then
+ // delete the file being downloaded
try {
file.delete();
} catch (Exception e) {
log.error("Error deleting file: {}", this.saveAs, e);
}
- //if the failure is due to a user abort it is returned normally else an exception is thrown
+ // if the failure is due to a user abort it is returned normally else an exception is thrown
if (!aborted)
- throw new SolrException(SolrException.ErrorCode.SERVER_ERROR,
- "Unable to download " + fileName + " completely. Downloaded "
- + bytesDownloaded + "!=" + size);
+ throw new SolrException(
+ SolrException.ErrorCode.SERVER_ERROR,
+ "Unable to download "
+ + fileName
+ + " completely. Downloaded "
+ + bytesDownloaded
+ + "!="
+ + size);
}
}
- /**
- * Open a new stream using HttpClient
- */
+ /** Open a new stream using HttpClient */
private FastInputStream getStream() throws IOException {
ModifiableSolrParams params = new ModifiableSolrParams();
-// //the method is command=filecontent
+ // //the method is command=filecontent
params.set(COMMAND, CMD_GET_FILE);
params.set(GENERATION, Long.toString(indexGen));
params.set(CommonParams.QT, ReplicationHandler.PATH);
- //add the version to download. This is used to reserve the download
+ // add the version to download. This is used to reserve the download
params.set(solrParamOutput, fileName);
if (useInternalCompression) {
params.set(COMPRESSION, "true");
}
- //use checksum
+ // use checksum
if (this.includeChecksum) {
params.set(CHECKSUM, true);
}
- //wt=filestream this is a custom protocol
+ // wt=filestream this is a custom protocol
params.set(CommonParams.WT, FILE_STREAM);
- // This happen if there is a failure there is a retry. the offset=<sizedownloaded> ensures that
+ // This happen if there is a failure there is a retry. the offset=<sizedownloaded> ensures
+ // that
// the server starts from the offset
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/ReplicationHandler.java
##########
@@ -1560,14 +1654,17 @@ protected void initWrite() throws IOException {
}
buf = new byte[(len == -1 || len > PACKET_SZ) ? PACKET_SZ : len];
- //reserve commit point till write is complete
- if(indexGen != null) {
+ // reserve commit point till write is complete
+ if (indexGen != null) {
delPolicy.saveCommitPoint(indexGen);
}
}
protected void createOutputStream(OutputStream out) {
- out = new CloseShieldOutputStream(out); // DeflaterOutputStream requires a close call, but don't close the request outputstream
+ out =
+ new CloseShieldOutputStream(
+ out); // DeflaterOutputStream requires a close call, but don't close the request
+ // outputstream
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/ReplicationHandler.java
##########
@@ -1576,17 +1673,18 @@ protected void createOutputStream(OutputStream out) {
}
protected void extendReserveAndReleaseCommitPoint() {
- if(indexGen != null) {
- //Reserve the commit point for another 10s for the next file to be to fetched.
- //We need to keep extending the commit reservation between requests so that the replica can fetch
- //all the files correctly.
+ if (indexGen != null) {
+ // Reserve the commit point for another 10s for the next file to be to fetched.
+ // We need to keep extending the commit reservation between requests so that the replica can
+ // fetch
+ // all the files correctly.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/StreamHandler.java
##########
@@ -172,37 +181,41 @@ public void handleRequestBody(SolrQueryRequest req, SolrQueryResponse rsp) throw
TupleStream tupleStream;
try {
- StreamExpression streamExpression = StreamExpressionParser.parse(params.get(StreamParams.EXPR));
+ StreamExpression streamExpression =
+ StreamExpressionParser.parse(params.get(StreamParams.EXPR));
if (this.streamFactory.isEvaluator(streamExpression)) {
StreamExpression tupleExpression = new StreamExpression(StreamParams.TUPLE);
- tupleExpression.addParameter(new StreamExpressionNamedParameter(StreamParams.RETURN_VALUE, streamExpression));
+ tupleExpression.addParameter(
+ new StreamExpressionNamedParameter(StreamParams.RETURN_VALUE, streamExpression));
tupleStream = this.streamFactory.constructStream(tupleExpression);
} else {
tupleStream = this.streamFactory.constructStream(streamExpression);
}
} catch (Exception e) {
- // Catch exceptions that occur while the stream is being created. This will include streaming expression parse
+ // Catch exceptions that occur while the stream is being created. This will include streaming
+ // expression parse
// rules.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/admin/BackupCoreOp.java
##########
@@ -63,30 +64,43 @@ public void execute(CoreAdminHandler.CallInfo it) throws Exception {
if ("file".equals(locationUri.getScheme())) {
core.getCoreContainer().assertPathAllowed(Paths.get(locationUri));
}
- final ShardBackupId prevShardBackupId = prevShardBackupIdStr != null ? ShardBackupId.from(prevShardBackupIdStr) : null;
+ final ShardBackupId prevShardBackupId =
+ prevShardBackupIdStr != null ? ShardBackupId.from(prevShardBackupIdStr) : null;
BackupFilePaths incBackupFiles = new BackupFilePaths(repository, locationUri);
- IncrementalShardBackup incSnapShooter = new IncrementalShardBackup(repository, core, incBackupFiles,
- prevShardBackupId, shardBackupId, Optional.ofNullable(commitName));
+ IncrementalShardBackup incSnapShooter =
+ new IncrementalShardBackup(
+ repository,
+ core,
+ incBackupFiles,
+ prevShardBackupId,
+ shardBackupId,
+ Optional.ofNullable(commitName));
NamedList<Object> rsp = incSnapShooter.backup();
it.rsp.addResponse(rsp);
} else {
SnapShooter snapShooter = new SnapShooter(repository, core, locationUri, name, commitName);
// validateCreateSnapshot will create parent dirs instead of throw; that choice is dubious.
// But we want to throw. One reason is that
- // this dir really should, in fact must, already exist here if triggered via a collection backup on a shared
+ // this dir really should, in fact must, already exist here if triggered via a collection
+ // backup on a shared
// file system. Otherwise, perhaps the FS location isn't shared -- we want an error.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/ReplicationHandler.java
##########
@@ -1673,17 +1768,17 @@ public void write(OutputStream out) throws IOException {
if (Files.isReadable(file)) {
try (SeekableByteChannel channel = Files.newByteChannel(file)) {
- //if offset is mentioned move the pointer to that point
- if (offset != -1)
- channel.position(offset);
+ // if offset is mentioned move the pointer to that point
+ if (offset != -1) channel.position(offset);
ByteBuffer bb = ByteBuffer.wrap(buf);
while (true) {
bb.clear();
long bytesRead = channel.read(bb);
if (bytesRead <= 0) {
writeNothingAndFlush();
- fos.close(); // we close because DeflaterOutputStream requires a close call, but the request outputstream is protected
+ fos.close(); // we close because DeflaterOutputStream requires a close call, but
+ // the request outputstream is protected
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/admin/BaseHandlerApiSupport.java
##########
@@ -112,30 +110,36 @@ public void call(SolrQueryRequest req, SolrQueryResponse rsp) {
}
}
}
- wrapParams(req, new CommandOperation("", Collections.emptyMap()), commands.get(0), true);
+ wrapParams(
+ req, new CommandOperation("", Collections.emptyMap()), commands.get(0), true);
commands.get(0).invoke(req, rsp, apiHandler);
}
} catch (SolrException e) {
throw e;
} catch (Exception e) {
- throw new SolrException(BAD_REQUEST, e); //TODO BAD_REQUEST is a wild guess; should we flip the default? fail here to investigate how this happens in tests
+ throw new SolrException(
+ BAD_REQUEST,
+ e); // TODO BAD_REQUEST is a wild guess; should we flip the default? fail here to
+ // investigate how this happens in tests
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/admin/LukeRequestHandler.java
##########
@@ -558,18 +608,18 @@ private static void populateFieldInfo(IndexSchema schema,
field.add("copyDests", toListOfStringDests(schema.getCopyFieldsList(f.getName())));
field.add("copySources", schema.getCopySources(f.getName()));
+ fields.put(f.getName(), field);
- fields.put( f.getName(), field );
-
- List<String> v = typeusemap.get( ft.getTypeName() );
- if( v == null ) {
+ List<String> v = typeusemap.get(ft.getTypeName());
+ if (v == null) {
v = new ArrayList<>();
}
- v.add( f.getName() );
- typeusemap.put( ft.getTypeName(), v );
+ v.add(f.getName());
+ typeusemap.put(ft.getTypeName(), v);
}
- // This method just gets the top-most level of information. This was conflated with getting detailed info
+ // This method just gets the top-most level of information. This was conflated with getting
+ // detailed info
// for *all* the fields, called from CoreAdminHandler etc.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/admin/LukeRequestHandler.java
##########
@@ -344,44 +385,49 @@ private static String getFieldFlags( SchemaField f )
Set<String> fields = null;
String fl = params.get(CommonParams.FL);
if (fl != null) {
- fields = new TreeSet<>(Arrays.asList(fl.split( "[,\\s]+" )));
+ fields = new TreeSet<>(Arrays.asList(fl.split("[,\\s]+")));
}
LeafReader reader = searcher.getSlowAtomicReader();
IndexSchema schema = searcher.getSchema();
- // Don't be tempted to put this in the loop below, the whole point here is to alphabetize the fields!
+ // Don't be tempted to put this in the loop below, the whole point here is to alphabetize the
+ // fields!
Set<String> fieldNames = new TreeSet<>();
- for(FieldInfo fieldInfo : reader.getFieldInfos()) {
+ for (FieldInfo fieldInfo : reader.getFieldInfos()) {
fieldNames.add(fieldInfo.name);
}
// Walk the term enum and keep a priority queue for each map in our set
SimpleOrderedMap<Object> finfo = new SimpleOrderedMap<>();
for (String fieldName : fieldNames) {
- if (fields != null && ! fields.contains(fieldName) && ! fields.contains("*")) {
- continue; //we're not interested in this field Still an issue here
+ if (fields != null && !fields.contains(fieldName) && !fields.contains("*")) {
+ continue; // we're not interested in this field Still an issue here
}
SimpleOrderedMap<Object> fieldMap = new SimpleOrderedMap<>();
- SchemaField sfield = schema.getFieldOrNull( fieldName );
- FieldType ftype = (sfield==null)?null:sfield.getType();
+ SchemaField sfield = schema.getFieldOrNull(fieldName);
+ FieldType ftype = (sfield == null) ? null : sfield.getType();
- fieldMap.add( "type", (ftype==null)?null:ftype.getTypeName() );
+ fieldMap.add("type", (ftype == null) ? null : ftype.getTypeName());
fieldMap.add("schema", getFieldFlags(sfield));
- if (sfield != null && schema.isDynamicField(sfield.getName()) && schema.getDynamicPattern(sfield.getName()) != null) {
+ if (sfield != null
+ && schema.isDynamicField(sfield.getName())
+ && schema.getDynamicPattern(sfield.getName()) != null) {
fieldMap.add("dynamicBase", schema.getDynamicPattern(sfield.getName()));
}
Terms terms = reader.terms(fieldName);
- if (terms == null) { // Not indexed, so we need to report what we can (it made it through the fl param if specified)
- finfo.add( fieldName, fieldMap );
+ if (terms
+ == null) { // Not indexed, so we need to report what we can (it made it through the fl
+ // param if specified)
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/StreamHandler.java
##########
@@ -259,34 +280,47 @@ private void handleAdmin(SolrQueryRequest req, SolrQueryResponse rsp, SolrParams
String id = params.get(ID);
DaemonStream d = daemons.get(id);
if (d == null) {
- rsp.add(StreamParams.RESULT_SET, new DaemonResponseStream("Daemon:" + id + " not found on " + coreName));
+ rsp.add(
+ StreamParams.RESULT_SET,
+ new DaemonResponseStream("Daemon:" + id + " not found on " + coreName));
return;
}
switch (action) {
case "stop":
d.close();
- rsp.add(StreamParams.RESULT_SET, new DaemonResponseStream("Daemon:" + id + " stopped on " + coreName));
+ rsp.add(
+ StreamParams.RESULT_SET,
+ new DaemonResponseStream("Daemon:" + id + " stopped on " + coreName));
break;
case "start":
try {
d.open();
} catch (IOException e) {
- rsp.add(StreamParams.RESULT_SET, new DaemonResponseStream("Daemon: " + id + " error: " + e.getMessage()));
+ rsp.add(
+ StreamParams.RESULT_SET,
+ new DaemonResponseStream("Daemon: " + id + " error: " + e.getMessage()));
}
- rsp.add(StreamParams.RESULT_SET, new DaemonResponseStream("Daemon:" + id + " started on " + coreName));
+ rsp.add(
+ StreamParams.RESULT_SET,
+ new DaemonResponseStream("Daemon:" + id + " started on " + coreName));
break;
case "kill":
daemons.remove(id);
- d.close(); // we already found it in the daemons list, so we don't need to verify we removed it.
- rsp.add(StreamParams.RESULT_SET, new DaemonResponseStream("Daemon:" + id + " killed on " + coreName));
+ d.close(); // we already found it in the daemons list, so we don't need to verify we
+ // removed it.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/SnapShooter.java
##########
@@ -69,21 +67,34 @@
public SnapShooter(SolrCore core, String location, String snapshotName) {
String snapDirStr = null;
// Note - This logic is only applicable to the usecase where a shared file-system is exposed via
- // local file-system interface (primarily for backwards compatibility). For other use-cases, users
+ // local file-system interface (primarily for backwards compatibility). For other use-cases,
+ // users
// will be required to specify "location" where the backup should be stored.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/admin/LukeRequestHandler.java
##########
@@ -417,10 +463,13 @@ private static Document getFirstLiveDoc(Terms terms, LeafReader reader) throws I
PostingsEnum postingsEnum = null;
TermsEnum termsEnum = terms.iterator();
BytesRef text;
- // Deal with the chance that the first bunch of terms are in deleted documents. Is there a better way?
+ // Deal with the chance that the first bunch of terms are in deleted documents. Is there a
+ // better way?
for (int idx = 0; idx < 1000 && postingsEnum == null; ++idx) {
text = termsEnum.next();
- if (text == null) { // Ran off the end of the terms enum without finding any live docs with that field in them.
+ if (text
+ == null) { // Ran off the end of the terms enum without finding any live docs with that
+ // field in them.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/admin/PrepRecoveryOp.java
##########
@@ -63,105 +61,151 @@ public void execute(CallInfo it) throws Exception {
int conflictWaitMs = coreContainer.getZkController().getLeaderConflictResolveWait();
log.info(
"Going to wait for coreNodeName: {}, state: {}, checkLive: {}, onlyIfLeader: {}, onlyIfLeaderActive: {}",
- coreNodeName, waitForState, checkLive, onlyIfLeader, onlyIfLeaderActive);
+ coreNodeName,
+ waitForState,
+ checkLive,
+ onlyIfLeader,
+ onlyIfLeaderActive);
String collectionName;
CloudDescriptor cloudDescriptor;
try (SolrCore core = coreContainer.getCore(cname)) {
- if (core == null) throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "core not found:" + cname);
+ if (core == null)
+ throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "core not found:" + cname);
collectionName = core.getCoreDescriptor().getCloudDescriptor().getCollectionName();
- cloudDescriptor = core.getCoreDescriptor()
- .getCloudDescriptor();
+ cloudDescriptor = core.getCoreDescriptor().getCloudDescriptor();
}
AtomicReference<String> errorMessage = new AtomicReference<>();
try {
- coreContainer.getZkController().getZkStateReader().waitForState(collectionName, conflictWaitMs, TimeUnit.MILLISECONDS, (n, c) -> {
- if (c == null)
- return false;
-
- try (SolrCore core = coreContainer.getCore(cname)) {
- if (core == null) throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "core not found:" + cname);
- if (onlyIfLeader != null && onlyIfLeader) {
- if (!core.getCoreDescriptor().getCloudDescriptor().isLeader()) {
- throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "We are not the leader");
- }
- }
- }
-
- // wait until we are sure the recovering node is ready
- // to accept updates
- Replica.State state = null;
- boolean live = false;
- Slice slice = c.getSlice(cloudDescriptor.getShardId());
- if (slice != null) {
- final Replica replica = slice.getReplicasMap().get(coreNodeName);
- if (replica != null) {
- state = replica.getState();
- live = n.contains(nodeName);
-
- final Replica.State localState = cloudDescriptor.getLastPublished();
-
- // TODO: This is funky but I've seen this in testing where the replica asks the
- // leader to be in recovery? Need to track down how that happens ... in the meantime,
- // this is a safeguard
- boolean leaderDoesNotNeedRecovery = (onlyIfLeader != null &&
- onlyIfLeader &&
- cname.equals(replica.getStr("core")) &&
- waitForState == Replica.State.RECOVERING &&
- localState == Replica.State.ACTIVE &&
- state == Replica.State.ACTIVE);
-
- if (leaderDoesNotNeedRecovery) {
- log.warn("Leader {} ignoring request to be in the recovering state because it is live and active.", cname);
- }
-
- ZkShardTerms shardTerms = coreContainer.getZkController().getShardTerms(collectionName, slice.getName());
- // if the replica is waiting for leader to see recovery state, the leader should refresh its terms
- if (waitForState == Replica.State.RECOVERING && shardTerms.registered(coreNodeName)
- && shardTerms.skipSendingUpdatesTo(coreNodeName)) {
- // The replica changed it term, then published itself as RECOVERING.
- // This core already see replica as RECOVERING
- // so it is guarantees that a live-fetch will be enough for this core to see max term published
- shardTerms.refreshTerms();
- }
-
- boolean onlyIfActiveCheckResult = onlyIfLeaderActive != null && onlyIfLeaderActive
- && localState != Replica.State.ACTIVE;
- if (log.isInfoEnabled()) {
- log.info(
- "In WaitForState(" + waitForState + "): collection=" + collectionName + ", shard=" + slice.getName() +
- ", thisCore=" + cname + ", leaderDoesNotNeedRecovery=" + leaderDoesNotNeedRecovery +
- ", isLeader? " + cloudDescriptor.isLeader() +
- ", live=" + live + ", checkLive=" + checkLive + ", currentState=" + state
- + ", localState=" + localState + ", nodeName=" + nodeName +
- ", coreNodeName=" + coreNodeName + ", onlyIfActiveCheckResult=" + onlyIfActiveCheckResult
- + ", nodeProps: " + replica); //nowarn
- }
- if (!onlyIfActiveCheckResult && replica != null && (state == waitForState || leaderDoesNotNeedRecovery)) {
- if (checkLive == null) {
- return true;
- } else if (checkLive && live) {
- return true;
- } else if (!checkLive && !live) {
- return true;
- }
- }
- }
- }
-
- if (coreContainer.isShutDown()) {
- throw new SolrException(SolrException.ErrorCode.BAD_REQUEST,
- "Solr is shutting down");
- }
-
- return false;
- });
+ coreContainer
+ .getZkController()
+ .getZkStateReader()
+ .waitForState(
+ collectionName,
+ conflictWaitMs,
+ TimeUnit.MILLISECONDS,
+ (n, c) -> {
+ if (c == null) return false;
+
+ try (SolrCore core = coreContainer.getCore(cname)) {
+ if (core == null)
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST, "core not found:" + cname);
+ if (onlyIfLeader != null && onlyIfLeader) {
+ if (!core.getCoreDescriptor().getCloudDescriptor().isLeader()) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST, "We are not the leader");
+ }
+ }
+ }
+
+ // wait until we are sure the recovering node is ready
+ // to accept updates
+ Replica.State state = null;
+ boolean live = false;
+ Slice slice = c.getSlice(cloudDescriptor.getShardId());
+ if (slice != null) {
+ final Replica replica = slice.getReplicasMap().get(coreNodeName);
+ if (replica != null) {
+ state = replica.getState();
+ live = n.contains(nodeName);
+
+ final Replica.State localState = cloudDescriptor.getLastPublished();
+
+ // TODO: This is funky but I've seen this in testing where the replica asks the
+ // leader to be in recovery? Need to track down how that happens ... in the
+ // meantime,
+ // this is a safeguard
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/SnapShooter.java
##########
@@ -208,82 +227,103 @@ public static IndexCommit getAndSaveNamedIndexCommit(SolrCore solrCore, String c
if (namedCommit.isPresent()) {
final IndexCommit commit = namedCommit.get();
if (log.isDebugEnabled()) {
- log.debug("Using named commit: name={}, generation={}", commitName, commit.getGeneration());
+ log.debug(
+ "Using named commit: name={}, generation={}", commitName, commit.getGeneration());
}
delPolicy.saveCommitPoint(commit.getGeneration());
return commit;
}
} // else...
- throw new SolrException(ErrorCode.BAD_REQUEST, "Unable to find an index commit with name " +
- commitName + " for core " + solrCore.getName());
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST,
+ "Unable to find an index commit with name "
+ + commitName
+ + " for core "
+ + solrCore.getName());
}
- public void createSnapAsync(final int numberToKeep, Consumer<NamedList<?>> result) throws IOException {
- //TODO should use Solr's ExecutorUtil
- new Thread(() -> {
- NamedList<Object> snapShootDetails;
- try {
- snapShootDetails = createSnapshot();
- } catch (Exception e) {
- log.error("Exception while creating snapshot", e);
- snapShootDetails = new NamedList<>();
- snapShootDetails.add("exception", e.getMessage());
- }
- if (snapshotName == null) {
- try {
- deleteOldBackups(numberToKeep);
- } catch (IOException e) {
- log.warn("Unable to delete old snapshots ", e);
- }
- }
- if (null != snapShootDetails) result.accept(snapShootDetails);
- }, "CreateSnapshot").start();
-
+ public void createSnapAsync(final int numberToKeep, Consumer<NamedList<?>> result)
+ throws IOException {
+ // TODO should use Solr's ExecutorUtil
+ new Thread(
+ () -> {
+ NamedList<Object> snapShootDetails;
+ try {
+ snapShootDetails = createSnapshot();
+ } catch (Exception e) {
+ log.error("Exception while creating snapshot", e);
+ snapShootDetails = new NamedList<>();
+ snapShootDetails.add("exception", e.getMessage());
+ }
+ if (snapshotName == null) {
+ try {
+ deleteOldBackups(numberToKeep);
+ } catch (IOException e) {
+ log.warn("Unable to delete old snapshots ", e);
+ }
+ }
+ if (null != snapShootDetails) result.accept(snapShootDetails);
+ },
+ "CreateSnapshot")
+ .start();
}
/**
* Handles the logic of creating a snapshot
- * <p>
- * <b>NOTE:</b> The caller <em>MUST</em> ensure that the {@link IndexCommit} is saved prior to
- * calling this method, and released after calling this method, or there is no no garuntee that the
- * method will function correctly.
- * </p>
+ *
+ * <p><b>NOTE:</b> The caller <em>MUST</em> ensure that the {@link IndexCommit} is saved prior to
+ * calling this method, and released after calling this method, or there is no no garuntee that
+ * the method will function correctly.
*
* @see IndexDeletionPolicyWrapper#saveCommitPoint
* @see IndexDeletionPolicyWrapper#releaseCommitPoint
*/
protected NamedList<Object> createSnapshot(final IndexCommit indexCommit) throws Exception {
assert indexCommit != null;
if (log.isInfoEnabled()) {
- log.info("Creating backup snapshot {} at {}", (snapshotName == null ? "<not named>" : snapshotName), baseSnapDirPath);
+ log.info(
+ "Creating backup snapshot {} at {}",
+ (snapshotName == null ? "<not named>" : snapshotName),
+ baseSnapDirPath);
}
boolean success = false;
try {
NamedList<Object> details = new SimpleOrderedMap<>();
details.add("startTime", Instant.now().toString());
Collection<String> files = indexCommit.getFileNames();
- Directory dir = solrCore.getDirectoryFactory().get(solrCore.getIndexDir(), DirContext.DEFAULT, solrCore.getSolrConfig().indexConfig.lockType);
+ Directory dir =
+ solrCore
+ .getDirectoryFactory()
+ .get(
+ solrCore.getIndexDir(),
+ DirContext.DEFAULT,
+ solrCore.getSolrConfig().indexConfig.lockType);
try {
- for(String fileName : files) {
- log.debug("Copying fileName={} from dir={} to snapshot={}", fileName, dir, snapshotDirPath);
+ for (String fileName : files) {
+ log.debug(
+ "Copying fileName={} from dir={} to snapshot={}", fileName, dir, snapshotDirPath);
backupRepo.copyFileFrom(dir, fileName, snapshotDirPath);
}
} finally {
solrCore.getDirectoryFactory().release(dir);
}
String endTime = Instant.now().toString();
-
- details.add("fileCount", files.size()); // DEPRECATED: for removal, replaced with indexFileCount
+
+ details.add(
+ "fileCount", files.size()); // DEPRECATED: for removal, replaced with indexFileCount
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/admin/LukeRequestHandler.java
##########
@@ -601,62 +653,67 @@ private static void populateFieldInfo(IndexSchema schema,
interface IOSupplier {
boolean get() throws IOException;
}
-
+
private static Object closeSafe(IOSupplier isCurrent) {
try {
return isCurrent.get();
- }catch(AlreadyClosedException | IOException exception) {
+ } catch (AlreadyClosedException | IOException exception) {
}
return false;
}
-
/**
- * <p>A helper method that attempts to determine the file length of the the segments file for the
+ * A helper method that attempts to determine the file length of the the segments file for the
* specified IndexCommit from it's Directory.
- * </p>
- * <p>
- * If any sort of {@link IOException} occurs, this method will return "-1" and swallow the exception since
- * this may be normal if the IndexCommit is no longer "on disk". The specific type of the Exception will
- * affect how severely it is logged: {@link NoSuchFileException} is considered more "acceptible" then other
- * types of IOException which may indicate an actual problem with the Directory.
+ *
+ * <p>If any sort of {@link IOException} occurs, this method will return "-1" and swallow the
+ * exception since this may be normal if the IndexCommit is no longer "on disk". The specific type
+ * of the Exception will affect how severely it is logged: {@link NoSuchFileException} is
+ * considered more "acceptible" then other types of IOException which may indicate an actual
+ * problem with the Directory.
*/
private static long getSegmentsFileLength(IndexCommit commit) {
try {
return commit.getDirectory().fileLength(commit.getSegmentsFileName());
} catch (NoSuchFileException okException) {
- log.debug("Unable to determine the (optional) fileSize for the current IndexReader's segments file because it is "
- + "no longer in the Directory, this can happen if there are new commits since the Reader was opened"
- , okException);
+ log.debug(
+ "Unable to determine the (optional) fileSize for the current IndexReader's segments file because it is "
+ + "no longer in the Directory, this can happen if there are new commits since the Reader was opened",
+ okException);
} catch (IOException strangeException) {
- log.warn("Ignoring IOException wile attempting to determine the (optional) fileSize stat for the current IndexReader's segments file",
- strangeException);
+ log.warn(
+ "Ignoring IOException wile attempting to determine the (optional) fileSize stat for the current IndexReader's segments file",
+ strangeException);
}
return -1;
}
- // Get terribly detailed information about a particular field. This is a very expensive call, use it with caution
+ // Get terribly detailed information about a particular field. This is a very expensive call, use
+ // it with caution
// especially on large indexes!
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/admin/LukeRequestHandler.java
##########
@@ -601,62 +653,67 @@ private static void populateFieldInfo(IndexSchema schema,
interface IOSupplier {
boolean get() throws IOException;
}
-
+
private static Object closeSafe(IOSupplier isCurrent) {
try {
return isCurrent.get();
- }catch(AlreadyClosedException | IOException exception) {
+ } catch (AlreadyClosedException | IOException exception) {
}
return false;
}
-
/**
- * <p>A helper method that attempts to determine the file length of the the segments file for the
+ * A helper method that attempts to determine the file length of the the segments file for the
* specified IndexCommit from it's Directory.
- * </p>
- * <p>
- * If any sort of {@link IOException} occurs, this method will return "-1" and swallow the exception since
- * this may be normal if the IndexCommit is no longer "on disk". The specific type of the Exception will
- * affect how severely it is logged: {@link NoSuchFileException} is considered more "acceptible" then other
- * types of IOException which may indicate an actual problem with the Directory.
+ *
+ * <p>If any sort of {@link IOException} occurs, this method will return "-1" and swallow the
+ * exception since this may be normal if the IndexCommit is no longer "on disk". The specific type
+ * of the Exception will affect how severely it is logged: {@link NoSuchFileException} is
+ * considered more "acceptible" then other types of IOException which may indicate an actual
+ * problem with the Directory.
*/
private static long getSegmentsFileLength(IndexCommit commit) {
try {
return commit.getDirectory().fileLength(commit.getSegmentsFileName());
} catch (NoSuchFileException okException) {
- log.debug("Unable to determine the (optional) fileSize for the current IndexReader's segments file because it is "
- + "no longer in the Directory, this can happen if there are new commits since the Reader was opened"
- , okException);
+ log.debug(
+ "Unable to determine the (optional) fileSize for the current IndexReader's segments file because it is "
+ + "no longer in the Directory, this can happen if there are new commits since the Reader was opened",
+ okException);
} catch (IOException strangeException) {
- log.warn("Ignoring IOException wile attempting to determine the (optional) fileSize stat for the current IndexReader's segments file",
- strangeException);
+ log.warn(
+ "Ignoring IOException wile attempting to determine the (optional) fileSize stat for the current IndexReader's segments file",
+ strangeException);
}
return -1;
}
- // Get terribly detailed information about a particular field. This is a very expensive call, use it with caution
+ // Get terribly detailed information about a particular field. This is a very expensive call, use
+ // it with caution
// especially on large indexes!
- private static void getDetailedFieldInfo(SolrQueryRequest req, String field, SimpleOrderedMap<Object> fieldMap)
- throws IOException {
+ private static void getDetailedFieldInfo(
+ SolrQueryRequest req, String field, SimpleOrderedMap<Object> fieldMap) throws IOException {
SolrParams params = req.getParams();
- final int numTerms = params.getInt( NUMTERMS, DEFAULT_COUNT );
+ final int numTerms = params.getInt(NUMTERMS, DEFAULT_COUNT);
- TopTermQueue tiq = new TopTermQueue(numTerms + 1); // Something to collect the top N terms in.
+ TopTermQueue tiq = new TopTermQueue(numTerms + 1); // Something to collect the top N terms in.
final CharsRefBuilder spare = new CharsRefBuilder();
Terms terms = MultiTerms.getTerms(req.getSearcher().getIndexReader(), field);
- if (terms == null) { // field does not exist
+ if (terms == null) { // field does not exist
return;
}
TermsEnum termsEnum = terms.iterator();
BytesRef text;
int[] buckets = new int[HIST_ARRAY_SIZE];
while ((text = termsEnum.next()) != null) {
++tiq.distinctTerms;
- int freq = termsEnum.docFreq(); // This calculation seems odd, but it gives the same results as it used to.
+ int freq =
+ termsEnum
+ .docFreq(); // This calculation seems odd, but it gives the same results as it used
+ // to.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/admin/RebalanceLeaders.java
##########
@@ -126,14 +128,14 @@
void execute() throws KeeperException, InterruptedException {
DocCollection dc = checkParams();
-
int max = req.getParams().getInt(MAX_AT_ONCE_PROP, Integer.MAX_VALUE);
if (max <= 0) max = Integer.MAX_VALUE;
int maxWaitSecs = req.getParams().getInt(MAX_WAIT_SECONDS_PROP, 60);
-
- // If there are a maximum number of simultaneous requests specified, we have to pause when we have that many
- // outstanding requests and wait for at least one to finish before going on the the next rebalance.
+ // If there are a maximum number of simultaneous requests specified, we have to pause when we
+ // have that many
+ // outstanding requests and wait for at least one to finish before going on the the next
+ // rebalance.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/admin/LukeRequestHandler.java
##########
@@ -579,16 +629,18 @@ private static void populateFieldInfo(IndexSchema schema,
indexInfo.add("numDocs", reader.numDocs());
indexInfo.add("maxDoc", reader.maxDoc());
indexInfo.add("deletedDocs", reader.maxDoc() - reader.numDocs());
- indexInfo.add("version", reader.getVersion()); // TODO? Is this different then: IndexReader.getCurrentVersion( dir )?
+ indexInfo.add(
+ "version",
+ reader.getVersion()); // TODO? Is this different then: IndexReader.getCurrentVersion( dir )?
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/admin/MetricsHandler.java
##########
@@ -414,21 +452,28 @@ private MetricFilter parseMustMatchFilter(SolrParams params) {
private List<MetricType> parseMetricTypes(SolrParams params) {
String[] typeStr = params.getParams(TYPE_PARAM);
List<String> types = Collections.emptyList();
- if (typeStr != null && typeStr.length > 0) {
+ if (typeStr != null && typeStr.length > 0) {
types = new ArrayList<>();
for (String type : typeStr) {
types.addAll(StrUtils.splitSmart(type, ','));
}
}
- List<MetricType> metricTypes = Collections.singletonList(MetricType.all); // include all metrics by default
+ List<MetricType> metricTypes =
+ Collections.singletonList(MetricType.all); // include all metrics by default
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/admin/RebalanceLeaders.java
##########
@@ -215,19 +227,23 @@ private void checkLeaderStatus() throws InterruptedException, KeeperException {
// The process is:
// if the replica with preferredLeader is already the leader, do nothing
// Otherwise:
- // > if two nodes have the same sequence number and both point to the current leader, we presume that we've just
+ // > if two nodes have the same sequence number and both point to the current leader, we presume
+ // that we've just
// moved it, move the one that does _not_ have the preferredLeader to the end of the list.
- // > move the current leader to the end of the list. This _should_ mean that the current ephemeral node in the
- // leader election queue is removed and the only remaining node watching it is triggered to become leader.
+ // > move the current leader to the end of the list. This _should_ mean that the current ephemeral
+ // node in the
+ // leader election queue is removed and the only remaining node watching it is triggered to
+ // become leader.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/admin/RebalanceLeaders.java
##########
@@ -250,42 +268,53 @@ private void ensurePreferredIsLeader(Slice slice) throws KeeperException, Interr
// 1> if the preferred leader isn't first in line, tell it to re-queue itself.
// 2> tell the actual leader to re-queue itself.
- // Ok, the sorting for election nodes is a bit strange. If the sequence numbers are the same, then the whole
- // string is used, but that sorts nodes with the same sequence number by their session IDs from ZK.
- // While this is determinate, it's not quite what we need, so re-queue nodes that aren't us and are
+ // Ok, the sorting for election nodes is a bit strange. If the sequence numbers are the same,
+ // then the whole
+ // string is used, but that sorts nodes with the same sequence number by their session IDs
+ // from ZK.
+ // While this is determinate, it's not quite what we need, so re-queue nodes that aren't us
+ // and are
// watching the leader node..
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/admin/RebalanceLeaders.java
##########
@@ -172,20 +177,26 @@ private DocCollection checkParams() throws KeeperException, InterruptedException
collectionName = req.getParams().get(COLLECTION_PROP);
if (StringUtils.isBlank(collectionName)) {
- throw new SolrException(SolrException.ErrorCode.BAD_REQUEST,
- String.format(Locale.ROOT, "The " + COLLECTION_PROP + " is required for the Rebalance Leaders command."));
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ String.format(
+ Locale.ROOT,
+ "The " + COLLECTION_PROP + " is required for the Rebalance Leaders command."));
}
coreContainer.getZkController().getZkStateReader().forceUpdateCollection(collectionName);
ClusterState clusterState = coreContainer.getZkController().getClusterState();
DocCollection dc = clusterState.getCollection(collectionName);
if (dc == null) {
- throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "Collection '" + collectionName + "' does not exist, no action taken.");
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Collection '" + collectionName + "' does not exist, no action taken.");
}
return dc;
}
- // Once we've done all the fiddling with the queues, check on the way out to see if all the active preferred
+ // Once we've done all the fiddling with the queues, check on the way out to see if all the active
+ // preferred
// leaders that we intended to change are in fact the leaders.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/admin/PrepRecoveryOp.java
##########
@@ -63,105 +61,151 @@ public void execute(CallInfo it) throws Exception {
int conflictWaitMs = coreContainer.getZkController().getLeaderConflictResolveWait();
log.info(
"Going to wait for coreNodeName: {}, state: {}, checkLive: {}, onlyIfLeader: {}, onlyIfLeaderActive: {}",
- coreNodeName, waitForState, checkLive, onlyIfLeader, onlyIfLeaderActive);
+ coreNodeName,
+ waitForState,
+ checkLive,
+ onlyIfLeader,
+ onlyIfLeaderActive);
String collectionName;
CloudDescriptor cloudDescriptor;
try (SolrCore core = coreContainer.getCore(cname)) {
- if (core == null) throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "core not found:" + cname);
+ if (core == null)
+ throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "core not found:" + cname);
collectionName = core.getCoreDescriptor().getCloudDescriptor().getCollectionName();
- cloudDescriptor = core.getCoreDescriptor()
- .getCloudDescriptor();
+ cloudDescriptor = core.getCoreDescriptor().getCloudDescriptor();
}
AtomicReference<String> errorMessage = new AtomicReference<>();
try {
- coreContainer.getZkController().getZkStateReader().waitForState(collectionName, conflictWaitMs, TimeUnit.MILLISECONDS, (n, c) -> {
- if (c == null)
- return false;
-
- try (SolrCore core = coreContainer.getCore(cname)) {
- if (core == null) throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "core not found:" + cname);
- if (onlyIfLeader != null && onlyIfLeader) {
- if (!core.getCoreDescriptor().getCloudDescriptor().isLeader()) {
- throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "We are not the leader");
- }
- }
- }
-
- // wait until we are sure the recovering node is ready
- // to accept updates
- Replica.State state = null;
- boolean live = false;
- Slice slice = c.getSlice(cloudDescriptor.getShardId());
- if (slice != null) {
- final Replica replica = slice.getReplicasMap().get(coreNodeName);
- if (replica != null) {
- state = replica.getState();
- live = n.contains(nodeName);
-
- final Replica.State localState = cloudDescriptor.getLastPublished();
-
- // TODO: This is funky but I've seen this in testing where the replica asks the
- // leader to be in recovery? Need to track down how that happens ... in the meantime,
- // this is a safeguard
- boolean leaderDoesNotNeedRecovery = (onlyIfLeader != null &&
- onlyIfLeader &&
- cname.equals(replica.getStr("core")) &&
- waitForState == Replica.State.RECOVERING &&
- localState == Replica.State.ACTIVE &&
- state == Replica.State.ACTIVE);
-
- if (leaderDoesNotNeedRecovery) {
- log.warn("Leader {} ignoring request to be in the recovering state because it is live and active.", cname);
- }
-
- ZkShardTerms shardTerms = coreContainer.getZkController().getShardTerms(collectionName, slice.getName());
- // if the replica is waiting for leader to see recovery state, the leader should refresh its terms
- if (waitForState == Replica.State.RECOVERING && shardTerms.registered(coreNodeName)
- && shardTerms.skipSendingUpdatesTo(coreNodeName)) {
- // The replica changed it term, then published itself as RECOVERING.
- // This core already see replica as RECOVERING
- // so it is guarantees that a live-fetch will be enough for this core to see max term published
- shardTerms.refreshTerms();
- }
-
- boolean onlyIfActiveCheckResult = onlyIfLeaderActive != null && onlyIfLeaderActive
- && localState != Replica.State.ACTIVE;
- if (log.isInfoEnabled()) {
- log.info(
- "In WaitForState(" + waitForState + "): collection=" + collectionName + ", shard=" + slice.getName() +
- ", thisCore=" + cname + ", leaderDoesNotNeedRecovery=" + leaderDoesNotNeedRecovery +
- ", isLeader? " + cloudDescriptor.isLeader() +
- ", live=" + live + ", checkLive=" + checkLive + ", currentState=" + state
- + ", localState=" + localState + ", nodeName=" + nodeName +
- ", coreNodeName=" + coreNodeName + ", onlyIfActiveCheckResult=" + onlyIfActiveCheckResult
- + ", nodeProps: " + replica); //nowarn
- }
- if (!onlyIfActiveCheckResult && replica != null && (state == waitForState || leaderDoesNotNeedRecovery)) {
- if (checkLive == null) {
- return true;
- } else if (checkLive && live) {
- return true;
- } else if (!checkLive && !live) {
- return true;
- }
- }
- }
- }
-
- if (coreContainer.isShutDown()) {
- throw new SolrException(SolrException.ErrorCode.BAD_REQUEST,
- "Solr is shutting down");
- }
-
- return false;
- });
+ coreContainer
+ .getZkController()
+ .getZkStateReader()
+ .waitForState(
+ collectionName,
+ conflictWaitMs,
+ TimeUnit.MILLISECONDS,
+ (n, c) -> {
+ if (c == null) return false;
+
+ try (SolrCore core = coreContainer.getCore(cname)) {
+ if (core == null)
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST, "core not found:" + cname);
+ if (onlyIfLeader != null && onlyIfLeader) {
+ if (!core.getCoreDescriptor().getCloudDescriptor().isLeader()) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST, "We are not the leader");
+ }
+ }
+ }
+
+ // wait until we are sure the recovering node is ready
+ // to accept updates
+ Replica.State state = null;
+ boolean live = false;
+ Slice slice = c.getSlice(cloudDescriptor.getShardId());
+ if (slice != null) {
+ final Replica replica = slice.getReplicasMap().get(coreNodeName);
+ if (replica != null) {
+ state = replica.getState();
+ live = n.contains(nodeName);
+
+ final Replica.State localState = cloudDescriptor.getLastPublished();
+
+ // TODO: This is funky but I've seen this in testing where the replica asks the
+ // leader to be in recovery? Need to track down how that happens ... in the
+ // meantime,
+ // this is a safeguard
+ boolean leaderDoesNotNeedRecovery =
+ (onlyIfLeader != null
+ && onlyIfLeader
+ && cname.equals(replica.getStr("core"))
+ && waitForState == Replica.State.RECOVERING
+ && localState == Replica.State.ACTIVE
+ && state == Replica.State.ACTIVE);
+
+ if (leaderDoesNotNeedRecovery) {
+ log.warn(
+ "Leader {} ignoring request to be in the recovering state because it is live and active.",
+ cname);
+ }
+
+ ZkShardTerms shardTerms =
+ coreContainer
+ .getZkController()
+ .getShardTerms(collectionName, slice.getName());
+ // if the replica is waiting for leader to see recovery state, the leader should
+ // refresh its terms
+ if (waitForState == Replica.State.RECOVERING
+ && shardTerms.registered(coreNodeName)
+ && shardTerms.skipSendingUpdatesTo(coreNodeName)) {
+ // The replica changed it term, then published itself as RECOVERING.
+ // This core already see replica as RECOVERING
+ // so it is guarantees that a live-fetch will be enough for this core to see
+ // max term published
+ shardTerms.refreshTerms();
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/admin/RebalanceLeaders.java
##########
@@ -295,11 +324,18 @@ private void addInactiveToResults(Slice slice, Replica replica) {
}
SimpleOrderedMap<String> res = new SimpleOrderedMap<>();
res.add("status", "skipped");
- res.add("msg", "Replica " + replica.getName() + " is a referredLeader for shard " + slice.getName() + ", but is inactive. No change necessary");
+ res.add(
+ "msg",
+ "Replica "
+ + replica.getName()
+ + " is a referredLeader for shard "
+ + slice.getName()
+ + ", but is inactive. No change necessary");
inactives.add(replica.getName(), res);
}
- // Provide some feedback to the user about what actually happened, or in this case where no action was
+ // Provide some feedback to the user about what actually happened, or in this case where no action
+ // was
// necesary since this preferred replica was already the leader
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/admin/RebalanceLeaders.java
##########
@@ -250,42 +268,53 @@ private void ensurePreferredIsLeader(Slice slice) throws KeeperException, Interr
// 1> if the preferred leader isn't first in line, tell it to re-queue itself.
// 2> tell the actual leader to re-queue itself.
- // Ok, the sorting for election nodes is a bit strange. If the sequence numbers are the same, then the whole
- // string is used, but that sorts nodes with the same sequence number by their session IDs from ZK.
- // While this is determinate, it's not quite what we need, so re-queue nodes that aren't us and are
+ // Ok, the sorting for election nodes is a bit strange. If the sequence numbers are the same,
+ // then the whole
+ // string is used, but that sorts nodes with the same sequence number by their session IDs
+ // from ZK.
+ // While this is determinate, it's not quite what we need, so re-queue nodes that aren't us
+ // and are
// watching the leader node..
-
String firstWatcher = electionNodes.get(1);
if (LeaderElector.getNodeName(firstWatcher).equals(replica.getName()) == false) {
makeReplicaFirstWatcher(slice, replica);
}
- // This replica should be the leader at the end of the day, so let's record that information to check at the end
+ // This replica should be the leader at the end of the day, so let's record that information
+ // to check at the end
pendingOps.put(slice.getName(), replica.getName());
String leaderElectionNode = electionNodes.get(0);
- String coreName = slice.getReplica(LeaderElector.getNodeName(leaderElectionNode)).getStr(CORE_NAME_PROP);
+ String coreName =
+ slice.getReplica(LeaderElector.getNodeName(leaderElectionNode)).getStr(CORE_NAME_PROP);
rejoinElectionQueue(slice, leaderElectionNode, coreName, false);
waitForNodeChange(slice, leaderElectionNode);
return; // Done with this slice, skip the rest of the replicas.
}
}
- // Check that the election queue has some members! There really should be two or more for this to make any sense,
+ // Check that the election queue has some members! There really should be two or more for this to
+ // make any sense,
// if there's only one we can't change anything.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/admin/RebalanceLeaders.java
##########
@@ -250,42 +268,53 @@ private void ensurePreferredIsLeader(Slice slice) throws KeeperException, Interr
// 1> if the preferred leader isn't first in line, tell it to re-queue itself.
// 2> tell the actual leader to re-queue itself.
- // Ok, the sorting for election nodes is a bit strange. If the sequence numbers are the same, then the whole
- // string is used, but that sorts nodes with the same sequence number by their session IDs from ZK.
- // While this is determinate, it's not quite what we need, so re-queue nodes that aren't us and are
+ // Ok, the sorting for election nodes is a bit strange. If the sequence numbers are the same,
+ // then the whole
+ // string is used, but that sorts nodes with the same sequence number by their session IDs
+ // from ZK.
+ // While this is determinate, it's not quite what we need, so re-queue nodes that aren't us
+ // and are
// watching the leader node..
-
String firstWatcher = electionNodes.get(1);
if (LeaderElector.getNodeName(firstWatcher).equals(replica.getName()) == false) {
makeReplicaFirstWatcher(slice, replica);
}
- // This replica should be the leader at the end of the day, so let's record that information to check at the end
+ // This replica should be the leader at the end of the day, so let's record that information
+ // to check at the end
pendingOps.put(slice.getName(), replica.getName());
String leaderElectionNode = electionNodes.get(0);
- String coreName = slice.getReplica(LeaderElector.getNodeName(leaderElectionNode)).getStr(CORE_NAME_PROP);
+ String coreName =
+ slice.getReplica(LeaderElector.getNodeName(leaderElectionNode)).getStr(CORE_NAME_PROP);
rejoinElectionQueue(slice, leaderElectionNode, coreName, false);
waitForNodeChange(slice, leaderElectionNode);
return; // Done with this slice, skip the rest of the replicas.
}
}
- // Check that the election queue has some members! There really should be two or more for this to make any sense,
+ // Check that the election queue has some members! There really should be two or more for this to
+ // make any sense,
// if there's only one we can't change anything.
- private boolean electionQueueInBadState(List<String> electionNodes, Slice slice, Replica replica) {
- if (electionNodes.size() < 2) { // if there's only one node in the queue, should already be leader and we shouldn't be here anyway.
- log.warn("Rebalancing leaders and slice {} has less than two elements in the leader election queue, but replica {} doesn't think it's the leader."
- , slice.getName(), replica.getName());
+ private boolean electionQueueInBadState(
+ List<String> electionNodes, Slice slice, Replica replica) {
+ if (electionNodes.size()
+ < 2) { // if there's only one node in the queue, should already be leader and we shouldn't
+ // be here anyway.
+ log.warn(
+ "Rebalancing leaders and slice {} has less than two elements in the leader election queue, but replica {} doesn't think it's the leader.",
+ slice.getName(),
+ replica.getName());
return true;
}
return false;
}
- // Provide some feedback to the user about what actually happened, or in this case where no action was
+ // Provide some feedback to the user about what actually happened, or in this case where no action
+ // was
// possible
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/admin/RebalanceLeaders.java
##########
@@ -309,20 +345,31 @@ private void addAlreadyLeaderToResults(Slice slice, Replica replica) {
}
SimpleOrderedMap<String> res = new SimpleOrderedMap<>();
res.add("status", "skipped");
- res.add("msg", "Replica " + replica.getName() + " is already the leader for shard " + slice.getName() + ". No change necessary");
+ res.add(
+ "msg",
+ "Replica "
+ + replica.getName()
+ + " is already the leader for shard "
+ + slice.getName()
+ + ". No change necessary");
alreadyLeaders.add(replica.getName(), res);
}
- // Put the replica in at the head of the queue and send all nodes with the same sequence number to the back of the list
- // There can be "ties", i.e. replicas in the queue with the same sequence number. Sorting doesn't necessarily sort
- // the one we most care about first. So put the node we _don't care about at the end of the election queue_
+ // Put the replica in at the head of the queue and send all nodes with the same sequence number to
+ // the back of the list
+ // There can be "ties", i.e. replicas in the queue with the same sequence number. Sorting doesn't
+ // necessarily sort
+ // the one we most care about first. So put the node we _don't care about at the end of the
+ // election queue_
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/admin/RebalanceLeaders.java
##########
@@ -367,25 +418,31 @@ void makeReplicaFirstWatcher(Slice slice, Replica replica)
}
// We won't get here for the preferredLeader node
if (LeaderElector.getSeq(thisNode) == newSeq) {
- String coreName = slice.getReplica(LeaderElector.getNodeName(thisNode)).getStr(CORE_NAME_PROP);
+ String coreName =
+ slice.getReplica(LeaderElector.getNodeName(thisNode)).getStr(CORE_NAME_PROP);
rejoinElectionQueue(slice, thisNode, coreName, false);
waitForNodeChange(slice, thisNode);
}
}
}
- // We're just waiting for the electionNode to rejoin the queue with a _different_ node, indicating that any
+ // We're just waiting for the electionNode to rejoin the queue with a _different_ node, indicating
+ // that any
// requeueing we've done has happened.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/admin/RebalanceLeaders.java
##########
@@ -468,16 +538,23 @@ private void addToSuccesses(Slice slice, Replica replica) {
results.add("successes", successes);
}
if (log.isInfoEnabled()) {
- log.info("Successfully changed leader of shard {} to replica {}", slice.getName(), replica.getName());
+ log.info(
+ "Successfully changed leader of shard {} to replica {}",
+ slice.getName(),
+ replica.getName());
}
SimpleOrderedMap<String> res = new SimpleOrderedMap<>();
res.add("status", "success");
- res.add("msg", "Successfully changed leader of slice " + slice.getName() + " to " + replica.getName());
+ res.add(
+ "msg",
+ "Successfully changed leader of slice " + slice.getName() + " to " + replica.getName());
successes.add(slice.getName(), res);
}
- // If for any reason we were supposed to change leadership, that should be recorded in changingLeaders. Any
- // time we verified that the change actually occurred, that entry should have been removed. So report anything
+ // If for any reason we were supposed to change leadership, that should be recorded in
+ // changingLeaders. Any
+ // time we verified that the change actually occurred, that entry should have been removed. So
+ // report anything
// left over as a failure.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/admin/SecurityConfHandler.java
##########
@@ -269,44 +277,54 @@ public String toString() {
synchronized (this) {
if (apis == null) {
Collection<Api> apis = new ArrayList<>();
- final SpecProvider authcCommands = Utils.getSpec("cluster.security.authentication.Commands");
- final SpecProvider authzCommands = Utils.getSpec("cluster.security.authorization.Commands");
+ final SpecProvider authcCommands =
+ Utils.getSpec("cluster.security.authentication.Commands");
+ final SpecProvider authzCommands =
+ Utils.getSpec("cluster.security.authorization.Commands");
apis.add(new ReqHandlerToApi(this, Utils.getSpec("cluster.security.authentication")));
apis.add(new ReqHandlerToApi(this, Utils.getSpec("cluster.security.authorization")));
- SpecProvider authcSpecProvider = () -> {
- AuthenticationPlugin authcPlugin = cores.getAuthenticationPlugin();
- return authcPlugin != null && authcPlugin instanceof SpecProvider ?
- ((SpecProvider) authcPlugin).getSpec() :
- authcCommands.getSpec();
- };
-
- apis.add(new ReqHandlerToApi(this, authcSpecProvider) {
- @Override
- public synchronized Map<String, JsonSchemaValidator> getCommandSchema() {
- //it is possible that the Authentication plugin is modified since the last call. invalidate the
- // the cached commandSchema
- if(SecurityConfHandler.this.authcPlugin != cores.getAuthenticationPlugin()) commandSchema = null;
- SecurityConfHandler.this.authcPlugin = cores.getAuthenticationPlugin();
- return super.getCommandSchema();
- }
- });
-
- SpecProvider authzSpecProvider = () -> {
- AuthorizationPlugin authzPlugin = cores.getAuthorizationPlugin();
- return authzPlugin != null && authzPlugin instanceof SpecProvider ?
- ((SpecProvider) authzPlugin).getSpec() :
- authzCommands.getSpec();
- };
- apis.add(new ApiBag.ReqHandlerToApi(this, authzSpecProvider) {
- @Override
- public synchronized Map<String, JsonSchemaValidator> getCommandSchema() {
- //it is possible that the Authorization plugin is modified since the last call. invalidate the
- // the cached commandSchema
- if(SecurityConfHandler.this.authzPlugin != cores.getAuthorizationPlugin()) commandSchema = null;
- SecurityConfHandler.this.authzPlugin = cores.getAuthorizationPlugin();
- return super.getCommandSchema();
- }
- });
+ SpecProvider authcSpecProvider =
+ () -> {
+ AuthenticationPlugin authcPlugin = cores.getAuthenticationPlugin();
+ return authcPlugin != null && authcPlugin instanceof SpecProvider
+ ? ((SpecProvider) authcPlugin).getSpec()
+ : authcCommands.getSpec();
+ };
+
+ apis.add(
+ new ReqHandlerToApi(this, authcSpecProvider) {
+ @Override
+ public synchronized Map<String, JsonSchemaValidator> getCommandSchema() {
+ // it is possible that the Authentication plugin is modified since the last call.
+ // invalidate the
+ // the cached commandSchema
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/admin/RebalanceLeaders.java
##########
@@ -395,9 +452,11 @@ int waitForNodeChange(Slice slice, String electionNode) throws InterruptedExcept
return -1;
}
- // Move an election node to some other place in the queue. If rejoinAtHead==false, then at the end, otherwise
+ // Move an election node to some other place in the queue. If rejoinAtHead==false, then at the
+ // end, otherwise
// the new node should point at the leader.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/admin/RebalanceLeaders.java
##########
@@ -447,9 +514,12 @@ private boolean waitAsyncRequests(final int maxWaitSecs, Boolean waitForAll)
}
}
// We're done if we're processing a few at a time or all requests are processed.
- // We don't want to change, say, 100s of leaders simultaneously. So if the request specifies some limit,
- // and we're at that limit, we want to return to the caller so it can immediately add another request.
- // That's the purpose of the first clause here. Otherwise, of course, just return if all requests are
+ // We don't want to change, say, 100s of leaders simultaneously. So if the request specifies
+ // some limit,
+ // and we're at that limit, we want to return to the caller so it can immediately add another
+ // request.
+ // That's the purpose of the first clause here. Otherwise, of course, just return if all
+ // requests are
// processed.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/admin/RebalanceLeaders.java
##########
@@ -250,42 +268,53 @@ private void ensurePreferredIsLeader(Slice slice) throws KeeperException, Interr
// 1> if the preferred leader isn't first in line, tell it to re-queue itself.
// 2> tell the actual leader to re-queue itself.
- // Ok, the sorting for election nodes is a bit strange. If the sequence numbers are the same, then the whole
- // string is used, but that sorts nodes with the same sequence number by their session IDs from ZK.
- // While this is determinate, it's not quite what we need, so re-queue nodes that aren't us and are
+ // Ok, the sorting for election nodes is a bit strange. If the sequence numbers are the same,
+ // then the whole
+ // string is used, but that sorts nodes with the same sequence number by their session IDs
+ // from ZK.
+ // While this is determinate, it's not quite what we need, so re-queue nodes that aren't us
+ // and are
// watching the leader node..
-
String firstWatcher = electionNodes.get(1);
if (LeaderElector.getNodeName(firstWatcher).equals(replica.getName()) == false) {
makeReplicaFirstWatcher(slice, replica);
}
- // This replica should be the leader at the end of the day, so let's record that information to check at the end
+ // This replica should be the leader at the end of the day, so let's record that information
+ // to check at the end
pendingOps.put(slice.getName(), replica.getName());
String leaderElectionNode = electionNodes.get(0);
- String coreName = slice.getReplica(LeaderElector.getNodeName(leaderElectionNode)).getStr(CORE_NAME_PROP);
+ String coreName =
+ slice.getReplica(LeaderElector.getNodeName(leaderElectionNode)).getStr(CORE_NAME_PROP);
rejoinElectionQueue(slice, leaderElectionNode, coreName, false);
waitForNodeChange(slice, leaderElectionNode);
return; // Done with this slice, skip the rest of the replicas.
}
}
- // Check that the election queue has some members! There really should be two or more for this to make any sense,
+ // Check that the election queue has some members! There really should be two or more for this to
+ // make any sense,
// if there's only one we can't change anything.
- private boolean electionQueueInBadState(List<String> electionNodes, Slice slice, Replica replica) {
- if (electionNodes.size() < 2) { // if there's only one node in the queue, should already be leader and we shouldn't be here anyway.
- log.warn("Rebalancing leaders and slice {} has less than two elements in the leader election queue, but replica {} doesn't think it's the leader."
- , slice.getName(), replica.getName());
+ private boolean electionQueueInBadState(
+ List<String> electionNodes, Slice slice, Replica replica) {
+ if (electionNodes.size()
+ < 2) { // if there's only one node in the queue, should already be leader and we shouldn't
+ // be here anyway.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/admin/SecurityConfHandler.java
##########
@@ -269,44 +277,54 @@ public String toString() {
synchronized (this) {
if (apis == null) {
Collection<Api> apis = new ArrayList<>();
- final SpecProvider authcCommands = Utils.getSpec("cluster.security.authentication.Commands");
- final SpecProvider authzCommands = Utils.getSpec("cluster.security.authorization.Commands");
+ final SpecProvider authcCommands =
+ Utils.getSpec("cluster.security.authentication.Commands");
+ final SpecProvider authzCommands =
+ Utils.getSpec("cluster.security.authorization.Commands");
apis.add(new ReqHandlerToApi(this, Utils.getSpec("cluster.security.authentication")));
apis.add(new ReqHandlerToApi(this, Utils.getSpec("cluster.security.authorization")));
- SpecProvider authcSpecProvider = () -> {
- AuthenticationPlugin authcPlugin = cores.getAuthenticationPlugin();
- return authcPlugin != null && authcPlugin instanceof SpecProvider ?
- ((SpecProvider) authcPlugin).getSpec() :
- authcCommands.getSpec();
- };
-
- apis.add(new ReqHandlerToApi(this, authcSpecProvider) {
- @Override
- public synchronized Map<String, JsonSchemaValidator> getCommandSchema() {
- //it is possible that the Authentication plugin is modified since the last call. invalidate the
- // the cached commandSchema
- if(SecurityConfHandler.this.authcPlugin != cores.getAuthenticationPlugin()) commandSchema = null;
- SecurityConfHandler.this.authcPlugin = cores.getAuthenticationPlugin();
- return super.getCommandSchema();
- }
- });
-
- SpecProvider authzSpecProvider = () -> {
- AuthorizationPlugin authzPlugin = cores.getAuthorizationPlugin();
- return authzPlugin != null && authzPlugin instanceof SpecProvider ?
- ((SpecProvider) authzPlugin).getSpec() :
- authzCommands.getSpec();
- };
- apis.add(new ApiBag.ReqHandlerToApi(this, authzSpecProvider) {
- @Override
- public synchronized Map<String, JsonSchemaValidator> getCommandSchema() {
- //it is possible that the Authorization plugin is modified since the last call. invalidate the
- // the cached commandSchema
- if(SecurityConfHandler.this.authzPlugin != cores.getAuthorizationPlugin()) commandSchema = null;
- SecurityConfHandler.this.authzPlugin = cores.getAuthorizationPlugin();
- return super.getCommandSchema();
- }
- });
+ SpecProvider authcSpecProvider =
+ () -> {
+ AuthenticationPlugin authcPlugin = cores.getAuthenticationPlugin();
+ return authcPlugin != null && authcPlugin instanceof SpecProvider
+ ? ((SpecProvider) authcPlugin).getSpec()
+ : authcCommands.getSpec();
+ };
+
+ apis.add(
+ new ReqHandlerToApi(this, authcSpecProvider) {
+ @Override
+ public synchronized Map<String, JsonSchemaValidator> getCommandSchema() {
+ // it is possible that the Authentication plugin is modified since the last call.
+ // invalidate the
+ // the cached commandSchema
+ if (SecurityConfHandler.this.authcPlugin != cores.getAuthenticationPlugin())
+ commandSchema = null;
+ SecurityConfHandler.this.authcPlugin = cores.getAuthenticationPlugin();
+ return super.getCommandSchema();
+ }
+ });
+
+ SpecProvider authzSpecProvider =
+ () -> {
+ AuthorizationPlugin authzPlugin = cores.getAuthorizationPlugin();
+ return authzPlugin != null && authzPlugin instanceof SpecProvider
+ ? ((SpecProvider) authzPlugin).getSpec()
+ : authzCommands.getSpec();
+ };
+ apis.add(
+ new ApiBag.ReqHandlerToApi(this, authzSpecProvider) {
+ @Override
+ public synchronized Map<String, JsonSchemaValidator> getCommandSchema() {
+ // it is possible that the Authorization plugin is modified since the last call.
+ // invalidate the
+ // the cached commandSchema
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/admin/SplitOp.java
##########
@@ -329,11 +358,12 @@ static String toSplitString(Collection<DocRouter.Range> splits) throws Exception
String termStr = term.utf8ToString();
int firstSep = termStr.indexOf(CompositeIdRouter.SEPARATOR);
// truncate to first separator since we don't support multiple levels currently
- // NOTE: this does not currently work for tri-level composite ids since the number of bits allocated to the first ID is 16 for a 2 part id
+ // NOTE: this does not currently work for tri-level composite ids since the number of bits
+ // allocated to the first ID is 16 for a 2 part id
// and 8 for a 3 part id!
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/admin/ShowFileRequestHandler.java
##########
@@ -341,19 +345,20 @@ public static String getAdminFileFromZooKeeper(SolrQueryRequest req, SolrQueryRe
// Make sure the file exists, is readable and is not a hidden file
if (!zkClient.exists(adminFile, true)) {
log.error("Can not find: {}", adminFile);
- rsp.setException(new SolrException(SolrException.ErrorCode.NOT_FOUND, "Can not find: "
- + adminFile));
+ rsp.setException(
+ new SolrException(SolrException.ErrorCode.NOT_FOUND, "Can not find: " + adminFile));
return null;
}
return adminFile;
}
-
- // Find the file indicated by the "file=XXX" parameter or the root of the conf directory on the local
- // file system. Respects all the "interesting" stuff around what the resource loader does to find files.
- public static Path getAdminFileFromFileSystem(SolrQueryRequest req, SolrQueryResponse rsp,
- Set<String> hiddenFiles) {
+ // Find the file indicated by the "file=XXX" parameter or the root of the conf directory on the
+ // local
+ // file system. Respects all the "interesting" stuff around what the resource loader does to find
+ // files.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/admin/ShowFileRequestHandler.java
##########
@@ -286,23 +284,30 @@ public static String getSafeContentType(String contentType) {
//////////////////////// Static methods //////////////////////////////
- public static boolean isHiddenFile(SolrQueryRequest req, SolrQueryResponse rsp, String fnameIn, boolean reportError,
- Set<String> hiddenFiles) {
+ public static boolean isHiddenFile(
+ SolrQueryRequest req,
+ SolrQueryResponse rsp,
+ String fnameIn,
+ boolean reportError,
+ Set<String> hiddenFiles) {
String fname = fnameIn.toUpperCase(Locale.ROOT);
if (hiddenFiles.contains(fname) || hiddenFiles.contains("*")) {
if (reportError) {
log.error("Cannot access {}", fname);
- rsp.setException(new SolrException(SolrException.ErrorCode.FORBIDDEN, "Can not access: " + fnameIn));
+ rsp.setException(
+ new SolrException(SolrException.ErrorCode.FORBIDDEN, "Can not access: " + fnameIn));
}
return true;
}
- // This is slightly off, a valid path is something like ./schema.xml. I don't think it's worth the effort though
+ // This is slightly off, a valid path is something like ./schema.xml. I don't think it's worth
+ // the effort though
// to fix it to handle all possibilities though.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/admin/SplitOp.java
##########
@@ -420,13 +460,15 @@ static String toSplitString(Collection<DocRouter.Range> splits) throws Exception
// resize if needed
if (currPrefix.length < term.length) {
- currPrefix.bytes = new byte[term.length+10];
+ currPrefix.bytes = new byte[term.length + 10];
}
- // Copy the bytes up to and including the separator, and set the length if the separator is found.
- // If there was no separator, then length remains 0 and it's the indicator that we have no prefix bucket
+ // Copy the bytes up to and including the separator, and set the length if the separator is
+ // found.
+ // If there was no separator, then length remains 0 and it's the indicator that we have no
+ // prefix bucket
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/admin/SplitOp.java
##########
@@ -448,12 +496,15 @@ static String toSplitString(Collection<DocRouter.Range> splits) throws Exception
/*
* Returns the list of recommended splits, or null if there is not enough information
*/
- static Collection<DocRouter.Range> getSplits(Collection<RangeCount> rawCounts, DocRouter.Range currentRange) throws Exception {
+ static Collection<DocRouter.Range> getSplits(
+ Collection<RangeCount> rawCounts, DocRouter.Range currentRange) throws Exception {
int totalCount = 0;
- RangeCount biggest = null; // keep track of the largest in case we need to split it out into it's own shard
- RangeCount last = null; // keep track of what the last range is
+ RangeCount biggest =
+ null; // keep track of the largest in case we need to split it out into it's own shard
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/admin/ZookeeperInfoHandler.java
##########
@@ -404,44 +391,45 @@ public void handleRequestBody(SolrQueryRequest req, SolrQueryResponse rsp) throw
String filterType = params.get("filterType");
if (filterType != null) {
filterType = filterType.trim().toLowerCase(Locale.ROOT);
- if (filterType.length() == 0)
- filterType = null;
+ if (filterType.length() == 0) filterType = null;
}
FilterType type = (filterType != null) ? FilterType.valueOf(filterType) : FilterType.none;
String filter = (type != FilterType.none) ? params.get("filter") : null;
if (filter != null) {
filter = filter.trim();
- if (filter.length() == 0)
- filter = null;
+ if (filter.length() == 0) filter = null;
}
ZKPrinter printer = new ZKPrinter(cores.getZkController());
printer.detail = detail;
printer.dump = dump;
boolean isGraphView = "graph".equals(params.get("view"));
- // There is no znode /clusterstate.json (removed in Solr 9), but we do as if there's one and return collection listing
- // Need to change services.js if cleaning up here, collection list is used from Admin UI Cloud - Graph
+ // There is no znode /clusterstate.json (removed in Solr 9), but we do as if there's one and
+ // return collection listing
+ // Need to change services.js if cleaning up here, collection list is used from Admin UI Cloud -
+ // Graph
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/component/CloudReplicaSource.java
##########
@@ -170,7 +197,8 @@ private void addSlices(Map<String, Slice> target, ClusterState state, SolrParams
@Override
public List<String> getSliceNames() {
// This is maybe a bug?
- return Collections.unmodifiableList(Arrays.asList(slices)); // Do not use List.of because slices could have null
+ return Collections.unmodifiableList(
+ Arrays.asList(slices)); // Do not use List.of because slices could have null
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/admin/SplitOp.java
##########
@@ -472,27 +523,30 @@ static String toSplitString(Collection<DocRouter.Range> splits) throws Exception
return null;
}
-
List<DocRouter.Range> targetRanges = new ArrayList<>();
if (counts.size() == 1) {
// We have a single range, so we should split it.
- // Currently, we only split a prefix/bucket when we have just one, but this could be changed/controlled
- // in the future via a allowedSizeDifference parameter (i.e. if just separating prefix buckets results in
+ // Currently, we only split a prefix/bucket when we have just one, but this could be
+ // changed/controlled
+ // in the future via a allowedSizeDifference parameter (i.e. if just separating prefix buckets
+ // results in
// too large of an imbalanced, allow splitting within a prefix)
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/component/ActiveTasksListComponent.java
##########
@@ -16,102 +16,114 @@
*/
package org.apache.solr.handler.component;
-import org.apache.solr.common.MapWriter;
-import org.apache.solr.common.util.NamedList;
-
import java.io.IOException;
import java.util.Iterator;
import java.util.LinkedHashMap;
import java.util.Map;
+import org.apache.solr.common.MapWriter;
+import org.apache.solr.common.util.NamedList;
/** List the active tasks that can be cancelled */
public class ActiveTasksListComponent extends SearchComponent {
- public static final String COMPONENT_NAME = "activetaskslist";
+ public static final String COMPONENT_NAME = "activetaskslist";
- private boolean shouldProcess;
+ private boolean shouldProcess;
- @Override
- public void prepare(ResponseBuilder rb) throws IOException {
- if (rb.isTaskListRequest()) {
- shouldProcess = true;
- }
+ @Override
+ public void prepare(ResponseBuilder rb) throws IOException {
+ if (rb.isTaskListRequest()) {
+ shouldProcess = true;
}
+ }
- @Override
- public void process(ResponseBuilder rb) {
- if (!shouldProcess) {
- return;
- }
+ @Override
+ public void process(ResponseBuilder rb) {
+ if (!shouldProcess) {
+ return;
+ }
- if (rb.getTaskStatusCheckUUID() != null) {
- boolean isActiveOnThisShard = rb.req.getCore().getCancellableQueryTracker().isQueryIdActive(rb.getTaskStatusCheckUUID());
+ if (rb.getTaskStatusCheckUUID() != null) {
+ boolean isActiveOnThisShard =
+ rb.req
+ .getCore()
+ .getCancellableQueryTracker()
+ .isQueryIdActive(rb.getTaskStatusCheckUUID());
- rb.rsp.add("taskStatus", isActiveOnThisShard);
- return;
- }
+ rb.rsp.add("taskStatus", isActiveOnThisShard);
+ return;
+ }
- rb.rsp.add("taskList", (MapWriter) ew -> {
- Iterator<Map.Entry<String, String>> iterator = rb.req.getCore().getCancellableQueryTracker().getActiveQueriesGenerated();
+ rb.rsp.add(
+ "taskList",
+ (MapWriter)
+ ew -> {
+ Iterator<Map.Entry<String, String>> iterator =
+ rb.req.getCore().getCancellableQueryTracker().getActiveQueriesGenerated();
- while (iterator.hasNext()) {
+ while (iterator.hasNext()) {
Map.Entry<String, String> entry = iterator.next();
ew.put(entry.getKey(), entry.getValue());
- }
- });
+ }
+ });
+ }
+
+ @Override
+ @SuppressWarnings("unchecked")
+ public void handleResponses(ResponseBuilder rb, ShardRequest sreq) {
+ if (!shouldProcess) {
+ return;
}
- @Override
- @SuppressWarnings("unchecked")
- public void handleResponses(ResponseBuilder rb, ShardRequest sreq) {
- if (!shouldProcess) {
- return;
- }
+ NamedList<String> resultList = new NamedList<>();
- NamedList<String> resultList = new NamedList<>();
+ for (ShardResponse r : sreq.responses) {
- for (ShardResponse r : sreq.responses) {
+ if (rb.getTaskStatusCheckUUID() != null) {
+ boolean isTaskActiveOnShard = r.getSolrResponse().getResponse().getBooleanArg("taskStatus");
- if (rb.getTaskStatusCheckUUID() != null) {
- boolean isTaskActiveOnShard = r.getSolrResponse().getResponse().getBooleanArg("taskStatus");
-
- if (isTaskActiveOnShard) {
- rb.rsp.getValues().add("taskStatus", "id:" + rb.getTaskStatusCheckUUID() + ", status: active");
- return;
- } else {
- continue;
- }
- }
-
- LinkedHashMap<String, String> result = (LinkedHashMap<String, String>) r.getSolrResponse()
- .getResponse().get("taskList");
-
- Iterator<Map.Entry<String, String>> iterator = result.entrySet().iterator();
+ if (isTaskActiveOnShard) {
+ rb.rsp
+ .getValues()
+ .add("taskStatus", "id:" + rb.getTaskStatusCheckUUID() + ", status: active");
+ return;
+ } else {
+ continue;
+ }
+ }
- while (iterator.hasNext()) {
- Map.Entry<String, String> entry = iterator.next();
+ LinkedHashMap<String, String> result =
+ (LinkedHashMap<String, String>) r.getSolrResponse().getResponse().get("taskList");
- resultList.add(entry.getKey(), entry.getValue());
- }
- }
+ Iterator<Map.Entry<String, String>> iterator = result.entrySet().iterator();
- if (rb.getTaskStatusCheckUUID() != null) {
- // We got here with the specific taskID check being specified -- this means that the taskID was not
- // found in active tasks on any shard
- rb.rsp.getValues().add("taskStatus", "id:" + rb.getTaskStatusCheckUUID() + ", status: inactive");
- return;
- }
+ while (iterator.hasNext()) {
+ Map.Entry<String, String> entry = iterator.next();
- rb.rsp.getValues().add("taskList", resultList);
+ resultList.add(entry.getKey(), entry.getValue());
+ }
}
- @Override
- public String getDescription() {
- return "Responsible for listing all active cancellable tasks and also supports checking the status of " +
- "a particular task";
+ if (rb.getTaskStatusCheckUUID() != null) {
+ // We got here with the specific taskID check being specified -- this means that the taskID
+ // was not
+ // found in active tasks on any shard
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/component/ExpandComponent.java
##########
@@ -353,15 +367,17 @@ public void process(ResponseBuilder rb) throws IOException {
}
final boolean expandNullGroup =
- params.getBool(ExpandParams.EXPAND_NULL, false) &&
- // Our GroupCollector can typically ignore nulls (and the user's nullGroup param) unless the
- // current page had any - but if expand.q was specified, current page doesn't mater: We
- // need look for nulls if the user asked us to because we don't know what the expand.q will match
- (nullGroupOnCurrentPage || (null != query));
+ params.getBool(ExpandParams.EXPAND_NULL, false)
+ &&
+ // Our GroupCollector can typically ignore nulls (and the user's nullGroup param) unless
+ // the
+ // current page had any - but if expand.q was specified, current page doesn't mater: We
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/component/HttpShardHandler.java
##########
@@ -342,20 +366,34 @@ private static String createSliceShardsStr(final List<String> shardUrls) {
return sliceShardsStr.toString();
}
- private boolean canShortCircuit(String[] slices, boolean onlyNrtReplicas, SolrParams params, CloudDescriptor cloudDescriptor) {
- // Are we hosting the shard that this request is for, and are we active? If so, then handle it ourselves
+ private boolean canShortCircuit(
+ String[] slices,
+ boolean onlyNrtReplicas,
+ SolrParams params,
+ CloudDescriptor cloudDescriptor) {
+ // Are we hosting the shard that this request is for, and are we active? If so, then handle it
+ // ourselves
// and make it a non-distributed request.
String ourSlice = cloudDescriptor.getShardId();
String ourCollection = cloudDescriptor.getCollectionName();
// Some requests may only be fulfilled by replicas of type Replica.Type.NRT
- if (slices.length == 1 && slices[0] != null
- && (slices[0].equals(ourSlice) || slices[0].equals(ourCollection + "_" + ourSlice)) // handle the <collection>_<slice> format
+ if (slices.length == 1
+ && slices[0] != null
+ && (slices[0].equals(ourSlice)
+ || slices[0].equals(
+ ourCollection + "_" + ourSlice)) // handle the <collection>_<slice> format
&& cloudDescriptor.getLastPublished() == Replica.State.ACTIVE
&& (!onlyNrtReplicas || cloudDescriptor.getReplicaType() == Replica.Type.NRT)) {
- boolean shortCircuit = params.getBool("shortCircuit", true); // currently just a debugging parameter to check distrib search on a single node
+ boolean shortCircuit =
+ params.getBool(
+ "shortCircuit",
+ true); // currently just a debugging parameter to check distrib search on a single
+ // node
String targetHandler = params.get(ShardParams.SHARDS_QT);
- shortCircuit = shortCircuit && targetHandler == null; // if a different handler is specified, don't short-circuit
+ shortCircuit =
+ shortCircuit
+ && targetHandler == null; // if a different handler is specified, don't short-circuit
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/component/HttpShardHandler.java
##########
@@ -342,20 +366,34 @@ private static String createSliceShardsStr(final List<String> shardUrls) {
return sliceShardsStr.toString();
}
- private boolean canShortCircuit(String[] slices, boolean onlyNrtReplicas, SolrParams params, CloudDescriptor cloudDescriptor) {
- // Are we hosting the shard that this request is for, and are we active? If so, then handle it ourselves
+ private boolean canShortCircuit(
+ String[] slices,
+ boolean onlyNrtReplicas,
+ SolrParams params,
+ CloudDescriptor cloudDescriptor) {
+ // Are we hosting the shard that this request is for, and are we active? If so, then handle it
+ // ourselves
// and make it a non-distributed request.
String ourSlice = cloudDescriptor.getShardId();
String ourCollection = cloudDescriptor.getCollectionName();
// Some requests may only be fulfilled by replicas of type Replica.Type.NRT
- if (slices.length == 1 && slices[0] != null
- && (slices[0].equals(ourSlice) || slices[0].equals(ourCollection + "_" + ourSlice)) // handle the <collection>_<slice> format
+ if (slices.length == 1
+ && slices[0] != null
+ && (slices[0].equals(ourSlice)
+ || slices[0].equals(
+ ourCollection + "_" + ourSlice)) // handle the <collection>_<slice> format
&& cloudDescriptor.getLastPublished() == Replica.State.ACTIVE
&& (!onlyNrtReplicas || cloudDescriptor.getReplicaType() == Replica.Type.NRT)) {
- boolean shortCircuit = params.getBool("shortCircuit", true); // currently just a debugging parameter to check distrib search on a single node
+ boolean shortCircuit =
+ params.getBool(
+ "shortCircuit",
+ true); // currently just a debugging parameter to check distrib search on a single
+ // node
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/component/ExpandComponent.java
##########
@@ -718,30 +784,34 @@ private DocSlice collectorToDocSlice(Collector groupCollector, SolrIndexSearcher
}
return null;
}
-
- private void addGroupSliceToOutputMap(NamedList<DocSlice> outMap, CharsRefBuilder charsRef,
- long groupValue, DocSlice slice) {
- if(fieldType instanceof StrField) {
- final BytesRef bytesRef = ordBytes.get((int)groupValue);
+
+ private void addGroupSliceToOutputMap(
+ NamedList<DocSlice> outMap, CharsRefBuilder charsRef, long groupValue, DocSlice slice) {
+ if (fieldType instanceof StrField) {
+ final BytesRef bytesRef = ordBytes.get((int) groupValue);
fieldType.indexedToReadable(bytesRef, charsRef);
String group = charsRef.toString();
outMap.add(group, slice);
} else {
outMap.add(numericToString(fieldType, groupValue), slice);
}
}
-
+
@Override
public ScoreMode scoreMode() {
final LongObjectMap<Collector> groups = getGroups();
if (groups.isEmpty()) {
return ScoreMode.COMPLETE; // doesn't matter?
} else {
- return groups.iterator().next().value.scoreMode(); // we assume all the collectors should have the same nature
+ return groups
+ .iterator()
+ .next()
+ .value
+ .scoreMode(); // we assume all the collectors should have the same nature
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/component/HttpShardHandler.java
##########
@@ -342,20 +366,34 @@ private static String createSliceShardsStr(final List<String> shardUrls) {
return sliceShardsStr.toString();
}
- private boolean canShortCircuit(String[] slices, boolean onlyNrtReplicas, SolrParams params, CloudDescriptor cloudDescriptor) {
- // Are we hosting the shard that this request is for, and are we active? If so, then handle it ourselves
+ private boolean canShortCircuit(
+ String[] slices,
+ boolean onlyNrtReplicas,
+ SolrParams params,
+ CloudDescriptor cloudDescriptor) {
+ // Are we hosting the shard that this request is for, and are we active? If so, then handle it
+ // ourselves
// and make it a non-distributed request.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/component/HttpShardHandler.java
##########
@@ -342,20 +366,34 @@ private static String createSliceShardsStr(final List<String> shardUrls) {
return sliceShardsStr.toString();
}
- private boolean canShortCircuit(String[] slices, boolean onlyNrtReplicas, SolrParams params, CloudDescriptor cloudDescriptor) {
- // Are we hosting the shard that this request is for, and are we active? If so, then handle it ourselves
+ private boolean canShortCircuit(
+ String[] slices,
+ boolean onlyNrtReplicas,
+ SolrParams params,
+ CloudDescriptor cloudDescriptor) {
+ // Are we hosting the shard that this request is for, and are we active? If so, then handle it
+ // ourselves
// and make it a non-distributed request.
String ourSlice = cloudDescriptor.getShardId();
String ourCollection = cloudDescriptor.getCollectionName();
// Some requests may only be fulfilled by replicas of type Replica.Type.NRT
- if (slices.length == 1 && slices[0] != null
- && (slices[0].equals(ourSlice) || slices[0].equals(ourCollection + "_" + ourSlice)) // handle the <collection>_<slice> format
+ if (slices.length == 1
+ && slices[0] != null
+ && (slices[0].equals(ourSlice)
+ || slices[0].equals(
+ ourCollection + "_" + ourSlice)) // handle the <collection>_<slice> format
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/component/PhrasesIdentificationComponent.java
##########
@@ -232,23 +234,25 @@ public void finishStage(ResponseBuilder rb) {
scoreAndAddResultsToResponse(rb, contextData);
}
-
@Override
public void process(ResponseBuilder rb) throws IOException {
- final PhrasesContextData contextData = (PhrasesContextData) rb.req.getContext().get(this.getClass());
+ final PhrasesContextData contextData =
+ (PhrasesContextData) rb.req.getContext().get(this.getClass());
if (null == contextData) {
// if prepare didn't give us anything to work with, then we should do nothing
return;
}
// regardless of single node / shard, we need local stats...
- Phrase.populateStats(contextData.allPhrases, contextData.fieldWeights.keySet(), rb.req.getSearcher());
+ Phrase.populateStats(
+ contextData.allPhrases, contextData.fieldWeights.keySet(), rb.req.getSearcher());
- if ( rb.req.getParams().getBool(ShardParams.IS_SHARD, false) ) {
+ if (rb.req.getParams().getBool(ShardParams.IS_SHARD, false)) {
// shard request, return stats for all phrases (in original order)
SimpleOrderedMap<Object> output = new SimpleOrderedMap<>();
output.add("_all", Phrase.formatShardResponse(contextData.allPhrases));
- // TODO: might want to add numDocs() & getSumTotalTermFreq(f)/getDocCount(f) stats from each field...
+ // TODO: might want to add numDocs() & getSumTotalTermFreq(f)/getDocCount(f) stats from each
+ // field...
// so that we can sum/merge them for use in scoring?
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/component/QueryCancellationComponent.java
##########
@@ -16,89 +16,95 @@
*/
package org.apache.solr.handler.component;
-import org.apache.solr.client.solrj.util.Cancellable;
-
import java.io.IOException;
+import org.apache.solr.client.solrj.util.Cancellable;
/** Responsible for handling query cancellation requests */
public class QueryCancellationComponent extends SearchComponent {
- public static final String COMPONENT_NAME = "querycancellation";
+ public static final String COMPONENT_NAME = "querycancellation";
- private boolean shouldProcess;
+ private boolean shouldProcess;
- @Override
- public void prepare(ResponseBuilder rb) throws IOException
- {
- if (rb.isCancellation()) {
- shouldProcess = true;
- }
+ @Override
+ public void prepare(ResponseBuilder rb) throws IOException {
+ if (rb.isCancellation()) {
+ shouldProcess = true;
}
+ }
- @Override
- public void process(ResponseBuilder rb) {
- if (!shouldProcess) {
- return;
- }
-
- String cancellationUUID = rb.getCancellationUUID();
-
- if (cancellationUUID == null) {
- throw new RuntimeException("Null query UUID seen");
- }
+ @Override
+ public void process(ResponseBuilder rb) {
+ if (!shouldProcess) {
+ return;
+ }
- Cancellable cancellableTask = rb.req.getCore().getCancellableQueryTracker().getCancellableTask(cancellationUUID);
+ String cancellationUUID = rb.getCancellationUUID();
- if (cancellableTask != null) {
- cancellableTask.cancel();
- rb.rsp.add("cancellationResult", "success");
- } else {
- rb.rsp.add("cancellationResult", "not found");
- }
+ if (cancellationUUID == null) {
+ throw new RuntimeException("Null query UUID seen");
}
- @Override
- @SuppressWarnings("unchecked")
- public void handleResponses(ResponseBuilder rb, ShardRequest sreq) {
- if (!shouldProcess) {
- return;
- }
+ Cancellable cancellableTask =
+ rb.req.getCore().getCancellableQueryTracker().getCancellableTask(cancellationUUID);
- boolean queryFound = false;
+ if (cancellableTask != null) {
+ cancellableTask.cancel();
+ rb.rsp.add("cancellationResult", "success");
+ } else {
+ rb.rsp.add("cancellationResult", "not found");
+ }
+ }
- for (ShardResponse r : sreq.responses) {
+ @Override
+ @SuppressWarnings("unchecked")
+ public void handleResponses(ResponseBuilder rb, ShardRequest sreq) {
+ if (!shouldProcess) {
+ return;
+ }
- String cancellationResult = (String) r.getSolrResponse()
- .getResponse().get("cancellationResult");
+ boolean queryFound = false;
- if (cancellationResult.equalsIgnoreCase("success")) {
- queryFound = true;
+ for (ShardResponse r : sreq.responses) {
- break;
- }
- }
+ String cancellationResult =
+ (String) r.getSolrResponse().getResponse().get("cancellationResult");
- // If any shard sees the query as present, then we mark the query as successfully cancelled. If no shard found
- // the query, then that can denote that the query was not found. This is important since the query cancellation
- // request is broadcast to all shards, and the query might have completed on some shards but not on others
+ if (cancellationResult.equalsIgnoreCase("success")) {
+ queryFound = true;
- if(queryFound) {
- rb.rsp.getValues().add("status", "Query with queryID " + rb.getCancellationUUID() +
- " cancelled successfully");
- rb.rsp.getValues().add("responseCode", 200 /* HTTP OK */);
- } else {
- rb.rsp.getValues().add("status", "Query with queryID " + rb.getCancellationUUID() +
- " not found");
- rb.rsp.getValues().add("responseCode", 404 /* HTTP NOT FOUND */);
- }
+ break;
+ }
}
- @Override
- public String getDescription() {
- return "Supports cancellation of queries which are cancellable";
+ // If any shard sees the query as present, then we mark the query as successfully cancelled. If
+ // no shard found
+ // the query, then that can denote that the query was not found. This is important since the
+ // query cancellation
+ // request is broadcast to all shards, and the query might have completed on some shards but not
+ // on others
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/component/HttpShardHandlerFactory.java
##########
@@ -233,59 +246,73 @@ public void init(PluginInfo info) {
r.setSeed(Long.parseLong(v));
}
- BlockingQueue<Runnable> blockingQueue = (this.queueSize == -1) ?
- new SynchronousQueue<Runnable>(this.accessPolicy) :
- new ArrayBlockingQueue<Runnable>(this.queueSize, this.accessPolicy);
-
- this.commExecutor = new ExecutorUtil.MDCAwareThreadPoolExecutor(
- this.corePoolSize,
- this.maximumPoolSize,
- this.keepAliveTime, TimeUnit.SECONDS,
- blockingQueue,
- new SolrNamedThreadFactory("httpShardExecutor"),
- // the Runnable added to this executor handles all exceptions so we disable stack trace collection as an optimization
- // see SOLR-11880 for more details
- false
- );
+ BlockingQueue<Runnable> blockingQueue =
+ (this.queueSize == -1)
+ ? new SynchronousQueue<Runnable>(this.accessPolicy)
+ : new ArrayBlockingQueue<Runnable>(this.queueSize, this.accessPolicy);
+
+ this.commExecutor =
+ new ExecutorUtil.MDCAwareThreadPoolExecutor(
+ this.corePoolSize,
+ this.maximumPoolSize,
+ this.keepAliveTime,
+ TimeUnit.SECONDS,
+ blockingQueue,
+ new SolrNamedThreadFactory("httpShardExecutor"),
+ // the Runnable added to this executor handles all exceptions so we disable stack trace
+ // collection as an optimization
+ // see SOLR-11880 for more details
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/component/PhrasesIdentificationComponent.java
##########
@@ -175,41 +173,44 @@ public int distributedProcess(ResponseBuilder rb) {
sreq.params.remove(ShardParams.SHARDS);
rb.addRequest(this, sreq);
return ResponseBuilder.STAGE_GET_FIELDS;
-
+
} else if (rb.stage == ResponseBuilder.STAGE_GET_FIELDS) {
// NOTE: we don't do any actual work in this stage, but we need to ensure that even if
- // we are being used in isolation w/o QueryComponent that SearchHandler "tracks" a STAGE_GET_FIELDS
+ // we are being used in isolation w/o QueryComponent that SearchHandler "tracks" a
+ // STAGE_GET_FIELDS
// so that finishStage(STAGE_GET_FIELDS) is called on us and we can add our merged results
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/component/PhrasesIdentificationComponent.java
##########
@@ -470,48 +508,57 @@ public String summarize(final List<Phrase> results) {
return out.toString();
}
}
-
-
- /**
- * Model the data known about a single (candidate) Phrase -- which may or may not be indexed
+
+ /**
+ * Model the data known about a single (candidate) Phrase -- which may or may not be indexed
+ *
* @lucene.internal
*/
public static final class Phrase {
/**
- * Factory method for constructing a list of Phrases given the specified input and using the analyzer
- * for the specified field. The <code>maxIndexedPositionLength</code> and
- * <code>maxQueryPositionLength</code> provided *must* match the effective values used by
- * respective analyzers.
+ * Factory method for constructing a list of Phrases given the specified input and using the
+ * analyzer for the specified field. The <code>maxIndexedPositionLength</code> and <code>
+ * maxQueryPositionLength</code> provided *must* match the effective values used by respective
+ * analyzers.
*/
- public static List<Phrase> extractPhrases(final String input, final SchemaField analysisField,
- final int maxIndexedPositionLength,
- final int maxQueryPositionLength) {
+ public static List<Phrase> extractPhrases(
+ final String input,
+ final SchemaField analysisField,
+ final int maxIndexedPositionLength,
+ final int maxQueryPositionLength) {
- // TODO: rather then requiring the query analyzer to produce the Phrases for us (assuming Shingles)
- // we could potentially just require that it produces unigrams compatible with the unigrams in the
- // indexed fields, and then build our own Phrases at query time -- making the maxQueryPositionLength
+ // TODO: rather then requiring the query analyzer to produce the Phrases for us (assuming
+ // Shingles)
+ // we could potentially just require that it produces unigrams compatible with the unigrams in
+ // the
+ // indexed fields, and then build our own Phrases at query time -- making the
+ // maxQueryPositionLength
// a 100% run time configuration option.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/component/QueryComponent.java
##########
@@ -399,7 +403,8 @@ public void process(ResponseBuilder rb) throws IOException
//
GroupingSpecification groupingSpec = rb.getGroupingSpec();
if (groupingSpec != null) {
- cmd.setSegmentTerminateEarly(false); // not supported, silently ignore any segmentTerminateEarly flag
+ cmd.setSegmentTerminateEarly(
+ false); // not supported, silently ignore any segmentTerminateEarly flag
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/component/PhrasesIdentificationComponent.java
##########
@@ -1028,78 +1112,93 @@ private static double computeFieldScore(final Phrase input,
// sub-phrase of every other phrase (or at least: all siblings of diff sizes that add up to
// an existing phrase). As well as require us to give up on a predictible "range" of
// legal values for scores (IIUC from the LLR docs)
-
+
final long phrase_ttf = words.getTTF(field);
final long phrase_df = words.getDocFreq(field);
final long words_conj_count = words.getConjunctionDocCount(field);
max_sub_conj_count = Math.max(words_conj_count, max_sub_conj_count);
-
- final double max_wrapper_phrase_probability =
- words.getIndexedSuperPhrases().stream()
- .mapToDouble(p -> p.getConjunctionDocCount(field) <= 0 ?
- // special case check -- we already know *our* conj count > 0,
- // but we need a similar check for wrapper phrases: if <= 0, their probability is 0
- 0.0D : ((double)p.getDocFreq(field) / p.getConjunctionDocCount(field))).max().orElse(0.0D);
-
- final LongSummaryStatistics words_ttfs =
- words.getIndividualIndexedTerms().stream()
- .collect(Collectors.summarizingLong(t -> t.getTTF(field)));
-
- final double words_phrase_prob = (phrase_ttf / (double)words_ttfs.getMin());
- final double words_not_phrase_prob = (phrase_ttf / (double)words_ttfs.getMax());
-
- final double phrase_prob = (phrase_conj_count / (double)words_conj_count);
-
-
- final double phrase_score = words_phrase_prob * (phrase_prob - max_wrapper_phrase_probability);
- final double not_phrase_score = words_not_phrase_prob * (1 - (phrase_prob - max_wrapper_phrase_probability));
+
+ final double max_wrapper_phrase_probability =
+ words.getIndexedSuperPhrases().stream()
+ .mapToDouble(
+ p ->
+ p.getConjunctionDocCount(field) <= 0
+ ?
+ // special case check -- we already know *our* conj count > 0,
+ // but we need a similar check for wrapper phrases: if <= 0, their
+ // probability is 0
+ 0.0D
+ : ((double) p.getDocFreq(field) / p.getConjunctionDocCount(field)))
+ .max()
+ .orElse(0.0D);
+
+ final LongSummaryStatistics words_ttfs =
+ words.getIndividualIndexedTerms().stream()
+ .collect(Collectors.summarizingLong(t -> t.getTTF(field)));
+
+ final double words_phrase_prob = (phrase_ttf / (double) words_ttfs.getMin());
+ final double words_not_phrase_prob = (phrase_ttf / (double) words_ttfs.getMax());
+
+ final double phrase_prob = (phrase_conj_count / (double) words_conj_count);
+
+ final double phrase_score =
+ words_phrase_prob * (phrase_prob - max_wrapper_phrase_probability);
+ final double not_phrase_score =
+ words_not_phrase_prob * (1 - (phrase_prob - max_wrapper_phrase_probability));
final double words_score = phrase_score - not_phrase_score;
-
+
field_score += words_score;
}
- // NOTE: the "scaling" factors below can "increase" negative scores (by reducing the unsigned value)
- // when they should ideally be penalizing the scores further, but since we currently don't care
+ // NOTE: the "scaling" factors below can "increase" negative scores (by reducing the unsigned
+ // value)
+ // when they should ideally be penalizing the scores further, but since we currently don't
+ // care
// about any score lower then 0, it's not worth worrying about.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/component/PhrasesIdentificationComponent.java
##########
@@ -1028,78 +1112,93 @@ private static double computeFieldScore(final Phrase input,
// sub-phrase of every other phrase (or at least: all siblings of diff sizes that add up to
// an existing phrase). As well as require us to give up on a predictible "range" of
// legal values for scores (IIUC from the LLR docs)
-
+
final long phrase_ttf = words.getTTF(field);
final long phrase_df = words.getDocFreq(field);
final long words_conj_count = words.getConjunctionDocCount(field);
max_sub_conj_count = Math.max(words_conj_count, max_sub_conj_count);
-
- final double max_wrapper_phrase_probability =
- words.getIndexedSuperPhrases().stream()
- .mapToDouble(p -> p.getConjunctionDocCount(field) <= 0 ?
- // special case check -- we already know *our* conj count > 0,
- // but we need a similar check for wrapper phrases: if <= 0, their probability is 0
- 0.0D : ((double)p.getDocFreq(field) / p.getConjunctionDocCount(field))).max().orElse(0.0D);
-
- final LongSummaryStatistics words_ttfs =
- words.getIndividualIndexedTerms().stream()
- .collect(Collectors.summarizingLong(t -> t.getTTF(field)));
-
- final double words_phrase_prob = (phrase_ttf / (double)words_ttfs.getMin());
- final double words_not_phrase_prob = (phrase_ttf / (double)words_ttfs.getMax());
-
- final double phrase_prob = (phrase_conj_count / (double)words_conj_count);
-
-
- final double phrase_score = words_phrase_prob * (phrase_prob - max_wrapper_phrase_probability);
- final double not_phrase_score = words_not_phrase_prob * (1 - (phrase_prob - max_wrapper_phrase_probability));
+
+ final double max_wrapper_phrase_probability =
+ words.getIndexedSuperPhrases().stream()
+ .mapToDouble(
+ p ->
+ p.getConjunctionDocCount(field) <= 0
+ ?
+ // special case check -- we already know *our* conj count > 0,
+ // but we need a similar check for wrapper phrases: if <= 0, their
+ // probability is 0
+ 0.0D
+ : ((double) p.getDocFreq(field) / p.getConjunctionDocCount(field)))
+ .max()
+ .orElse(0.0D);
+
+ final LongSummaryStatistics words_ttfs =
+ words.getIndividualIndexedTerms().stream()
+ .collect(Collectors.summarizingLong(t -> t.getTTF(field)));
+
+ final double words_phrase_prob = (phrase_ttf / (double) words_ttfs.getMin());
+ final double words_not_phrase_prob = (phrase_ttf / (double) words_ttfs.getMax());
+
+ final double phrase_prob = (phrase_conj_count / (double) words_conj_count);
+
+ final double phrase_score =
+ words_phrase_prob * (phrase_prob - max_wrapper_phrase_probability);
+ final double not_phrase_score =
+ words_not_phrase_prob * (1 - (phrase_prob - max_wrapper_phrase_probability));
final double words_score = phrase_score - not_phrase_score;
-
+
field_score += words_score;
}
- // NOTE: the "scaling" factors below can "increase" negative scores (by reducing the unsigned value)
- // when they should ideally be penalizing the scores further, but since we currently don't care
+ // NOTE: the "scaling" factors below can "increase" negative scores (by reducing the unsigned
+ // value)
+ // when they should ideally be penalizing the scores further, but since we currently don't
+ // care
// about any score lower then 0, it's not worth worrying about.
-
- // Average the accumulated score over the number of actual indexed sub-phrases that contributed
+
+ // Average the accumulated score over the number of actual indexed sub-phrases that
+ // contributed
//
- // NOTE: since we subsequently want to multiply the score by a fraction with num_indexed_sub_phrases
+ // NOTE: since we subsequently want to multiply the score by a fraction with
+ // num_indexed_sub_phrases
// in the numerator, we can skip this...
// SEE BELOW // field_score /= (double) num_indexed_sub_phrases;
-
+
// If we leave field_score as is, then a phrase longer then the maxIndexedPositionLength
- // will never score higher then the highest scoring sub-phrase it has (because we've averaged them)
+ // will never score higher then the highest scoring sub-phrase it has (because we've averaged
+ // them)
// so we scale the scores against the longest possible phrase length we're considering
//
// NOTE: We don't use num_indexed_sub_phrases in the numerator since we skipped it when
// averating above...
- field_score *= ( 1.0D // SEE ABOVE // * ( (double)num_indexed_sub_phrases )
- / (1 + maxQueryPositionLength - maxIndexedPositionLength) );
-
+ field_score *=
+ (1.0D // SEE ABOVE // * ( (double)num_indexed_sub_phrases )
+ / (1 + maxQueryPositionLength - maxIndexedPositionLength));
+
// scale the field_score based on the ratio of the conjunction docCount for the whole phrase
- // realtive to the largest conjunction docCount of it's (largest indexed) sub phrases, to penalize
+ // realtive to the largest conjunction docCount of it's (largest indexed) sub phrases, to
+ // penalize
// the scores of very long phrases that exist very rarely relative to the how often their
// sub phrases exist in the index
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/component/PivotFacetFieldValueCollection.java
##########
@@ -121,16 +115,18 @@ public int size() {
if (numRefinableValues < facetFieldOffset) {
return Collections.<PivotFacetValue>emptyList();
}
-
- final int offsetPlusCount = (facetFieldLimit >= 0)
- ? Math.min(facetFieldLimit + facetFieldOffset, numRefinableValues)
- : numRefinableValues;
-
+
+ final int offsetPlusCount =
+ (facetFieldLimit >= 0)
+ ? Math.min(facetFieldLimit + facetFieldOffset, numRefinableValues)
+ : numRefinableValues;
+
if (1 < facetFieldMinimumCount && facetFieldSort.equals(FacetParams.FACET_SORT_INDEX)) {
// we have to skip any values that (still) don't meet the mincount
//
// TODO: in theory we could avoid this extra check by trimming sooner (SOLR-6331)
- // but since that's a destructive op that blows away the `valuesMap` which we (might?) still need
+ // but since that's a destructive op that blows away the `valuesMap` which we (might?) still
+ // need
// (and pre-emptively skips the offsets) we're avoiding re-working that optimization
// for now until/unless someone gives it more careful thought...
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/component/QueryComponent.java
##########
@@ -1229,42 +1271,50 @@ protected void returnFields(ResponseBuilder rb, ShardRequest sreq) {
for (ShardResponse srsp : sreq.responses) {
if (srsp.getException() != null) {
// Don't try to get the documents if there was an exception in the shard
- if(rb.req.getParams().getBool(ShardParams.SHARDS_INFO, false)) {
+ if (rb.req.getParams().getBool(ShardParams.SHARDS_INFO, false)) {
@SuppressWarnings("unchecked")
- NamedList<Object> shardInfo = (NamedList<Object>) rb.rsp.getValues().get(ShardParams.SHARDS_INFO);
+ NamedList<Object> shardInfo =
+ (NamedList<Object>) rb.rsp.getValues().get(ShardParams.SHARDS_INFO);
@SuppressWarnings("unchecked")
SimpleOrderedMap<Object> nl = (SimpleOrderedMap<Object>) shardInfo.get(srsp.getShard());
if (nl.get("error") == null) {
// Add the error to the shards info section if it wasn't added before
Throwable t = srsp.getException();
- if(t instanceof SolrServerException) {
- t = ((SolrServerException)t).getCause();
+ if (t instanceof SolrServerException) {
+ t = ((SolrServerException) t).getCause();
}
- nl.add("error", t.toString() );
+ nl.add("error", t.toString());
StringWriter trace = new StringWriter();
t.printStackTrace(new PrintWriter(trace));
- nl.add("trace", trace.toString() );
+ nl.add("trace", trace.toString());
}
}
-
+
continue;
}
{
- NamedList<?> responseHeader = (NamedList<?>)srsp.getSolrResponse().getResponse().get("responseHeader");
- if (Boolean.TRUE.equals(responseHeader.getBooleanArg(SolrQueryResponse.RESPONSE_HEADER_PARTIAL_RESULTS_KEY))) {
- rb.rsp.getResponseHeader().asShallowMap()
- .put(SolrQueryResponse.RESPONSE_HEADER_PARTIAL_RESULTS_KEY, Boolean.TRUE);
+ NamedList<?> responseHeader =
+ (NamedList<?>) srsp.getSolrResponse().getResponse().get("responseHeader");
+ if (Boolean.TRUE.equals(
+ responseHeader.getBooleanArg(
+ SolrQueryResponse.RESPONSE_HEADER_PARTIAL_RESULTS_KEY))) {
+ rb.rsp
+ .getResponseHeader()
+ .asShallowMap()
+ .put(SolrQueryResponse.RESPONSE_HEADER_PARTIAL_RESULTS_KEY, Boolean.TRUE);
}
}
- SolrDocumentList docs = (SolrDocumentList) srsp.getSolrResponse().getResponse().get("response");
+ SolrDocumentList docs =
+ (SolrDocumentList) srsp.getSolrResponse().getResponse().get("response");
for (SolrDocument doc : docs) {
Object id = doc.getFieldValue(keyFieldName);
ShardDoc sdoc = rb.resultIds.get(id.toString());
if (sdoc != null) {
if (returnScores) {
doc.setField("score", sdoc.score);
} else {
- // Score might have been added (in createMainQuery) to shard-requests (and therefore in shard-response-docs)
+ // Score might have been added (in createMainQuery) to shard-requests (and therefore
+ // in shard-response-docs)
// Remove score if the outer request did not ask for it returned
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/component/QueryElevationComponent.java
##########
@@ -622,44 +669,57 @@ private void addDebugInfo(ResponseBuilder rb, Elevation elevation) {
rb.addDebugInfo("queryBoosting", dbg);
}
- //---------------------------------------------------------------------------------
+ // ---------------------------------------------------------------------------------
// Boosted docs helper
- //---------------------------------------------------------------------------------
+ // ---------------------------------------------------------------------------------
/**
- * Resolves a set of boosted docs by uniqueKey to a map of docIds mapped to a priority value > 0.
+ * Resolves a set of boosted docs by uniqueKey to a map of docIds mapped to a priority value >
+ * 0.
+ *
* @param indexSearcher the SolrIndexSearcher; required
- * @param boosted are the set of uniqueKey values to be boosted in priority order. If null; returns null.
- * @param context the {@link SolrQueryRequest#getContext()} or null if none. We'll cache our results here.
+ * @param boosted are the set of uniqueKey values to be boosted in priority order. If null;
+ * returns null.
+ * @param context the {@link SolrQueryRequest#getContext()} or null if none. We'll cache our
+ * results here.
*/
- //TODO consider simplifying to remove "boosted" arg which can be looked up in context via BOOSTED key?
- public static IntIntHashMap getBoostDocs(SolrIndexSearcher indexSearcher, Set<BytesRef> boosted,
- Map<Object,Object> context) throws IOException {
+ // TODO consider simplifying to remove "boosted" arg which can be looked up in context via BOOSTED
+ // key?
+ public static IntIntHashMap getBoostDocs(
+ SolrIndexSearcher indexSearcher, Set<BytesRef> boosted, Map<Object, Object> context)
+ throws IOException {
IntIntHashMap boostDocs = null;
if (boosted != null) {
- //First see if it's already in the request context. Could have been put there by another caller.
+ // First see if it's already in the request context. Could have been put there by another
+ // caller.
if (context != null) {
boostDocs = (IntIntHashMap) context.get(BOOSTED_DOCIDS);
if (boostDocs != null) {
return boostDocs;
}
}
- //Not in the context yet so load it.
+ // Not in the context yet so load it.
boostDocs = new IntIntHashMap(boosted.size()); // docId to boost
- int priority = boosted.size() + 1; // the corresponding priority for each boosted key (starts at this; decrements down)
+ int priority =
+ boosted.size()
+ + 1; // the corresponding priority for each boosted key (starts at this; decrements
+ // down)
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/component/QueryElevationComponent.java
##########
@@ -1183,7 +1277,8 @@ protected void doSetNextReader(LeafReaderContext context) {
@Override
public int compare(int slot1, int slot2) {
- return values[slot1] - values[slot2]; // values will be small enough that there is no overflow concern
+ return values[slot1]
+ - values[slot2]; // values will be small enough that there is no overflow concern
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/component/RangeFacetProcessor.java
##########
@@ -208,23 +229,27 @@ public void getFacetRangeCounts(RangeFacetRequest rangeFacetRequest, NamedList<O
intervals.add(after);
}
- IntervalFacets.FacetInterval[] intervalsArray = intervals.toArray(new IntervalFacets.FacetInterval[intervals.size()]);
+ IntervalFacets.FacetInterval[] intervalsArray =
+ intervals.toArray(new IntervalFacets.FacetInterval[intervals.size()]);
// don't use the ArrayList anymore
intervals = null;
new IntervalFacets(sf, searcher, computeDocSet(docsOrig, rfr.getExcludeTags()), intervalsArray);
int intervalIndex = 0;
int lastIntervalIndex = intervalsArray.length - 1;
- // if the user requested "BEFORE", it will be the first of the intervals. Needs to be added to the
+ // if the user requested "BEFORE", it will be the first of the intervals. Needs to be added to
+ // the
// response named list instead of with the counts
if (includeBefore) {
res.add(intervalsArray[intervalIndex].getKey(), intervalsArray[intervalIndex].getCount());
intervalIndex++;
}
- // if the user requested "BETWEEN", it will be the first or second of the intervals (depending on if
- // "BEFORE" was also requested). Needs to be added to the response named list instead of with the counts
+ // if the user requested "BETWEEN", it will be the first or second of the intervals (depending
+ // on if
+ // "BEFORE" was also requested). Needs to be added to the response named list instead of with
+ // the counts
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/component/QueryElevationComponent.java
##########
@@ -622,44 +669,57 @@ private void addDebugInfo(ResponseBuilder rb, Elevation elevation) {
rb.addDebugInfo("queryBoosting", dbg);
}
- //---------------------------------------------------------------------------------
+ // ---------------------------------------------------------------------------------
// Boosted docs helper
- //---------------------------------------------------------------------------------
+ // ---------------------------------------------------------------------------------
/**
- * Resolves a set of boosted docs by uniqueKey to a map of docIds mapped to a priority value > 0.
+ * Resolves a set of boosted docs by uniqueKey to a map of docIds mapped to a priority value >
+ * 0.
+ *
* @param indexSearcher the SolrIndexSearcher; required
- * @param boosted are the set of uniqueKey values to be boosted in priority order. If null; returns null.
- * @param context the {@link SolrQueryRequest#getContext()} or null if none. We'll cache our results here.
+ * @param boosted are the set of uniqueKey values to be boosted in priority order. If null;
+ * returns null.
+ * @param context the {@link SolrQueryRequest#getContext()} or null if none. We'll cache our
+ * results here.
*/
- //TODO consider simplifying to remove "boosted" arg which can be looked up in context via BOOSTED key?
- public static IntIntHashMap getBoostDocs(SolrIndexSearcher indexSearcher, Set<BytesRef> boosted,
- Map<Object,Object> context) throws IOException {
+ // TODO consider simplifying to remove "boosted" arg which can be looked up in context via BOOSTED
+ // key?
+ public static IntIntHashMap getBoostDocs(
+ SolrIndexSearcher indexSearcher, Set<BytesRef> boosted, Map<Object, Object> context)
+ throws IOException {
IntIntHashMap boostDocs = null;
if (boosted != null) {
- //First see if it's already in the request context. Could have been put there by another caller.
+ // First see if it's already in the request context. Could have been put there by another
+ // caller.
if (context != null) {
boostDocs = (IntIntHashMap) context.get(BOOSTED_DOCIDS);
if (boostDocs != null) {
return boostDocs;
}
}
- //Not in the context yet so load it.
+ // Not in the context yet so load it.
boostDocs = new IntIntHashMap(boosted.size()); // docId to boost
- int priority = boosted.size() + 1; // the corresponding priority for each boosted key (starts at this; decrements down)
+ int priority =
+ boosted.size()
+ + 1; // the corresponding priority for each boosted key (starts at this; decrements
+ // down)
for (BytesRef uniqueKey : boosted) {
priority--; // therefore first == bosted.size(); last will be 1
- long segAndId = indexSearcher.lookupId(uniqueKey); // higher 32 bits == segment ID, low 32 bits == doc ID
+ long segAndId =
+ indexSearcher.lookupId(
+ uniqueKey); // higher 32 bits == segment ID, low 32 bits == doc ID
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/component/RealTimeGetComponent.java
##########
@@ -126,35 +122,40 @@ public void process(ResponseBuilder rb) throws IOException
Replica.Type replicaType = cloudDesc.getReplicaType();
if (replicaType != null) {
if (replicaType == Replica.Type.PULL) {
- throw new SolrException(ErrorCode.BAD_REQUEST,
- String.format(Locale.ROOT, "%s can't handle realtime get requests. Replicas of type %s do not support these type of requests",
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST,
+ String.format(
+ Locale.ROOT,
+ "%s can't handle realtime get requests. Replicas of type %s do not support these type of requests",
cloudDesc.getCoreNodeName(),
Replica.Type.PULL));
- }
- // non-leader TLOG replicas should not respond to distrib /get requests, but internal requests are OK
+ }
+ // non-leader TLOG replicas should not respond to distrib /get requests, but internal
+ // requests are OK
}
}
-
+
if (!params.getBool(COMPONENT_NAME, true)) {
return;
}
- //TODO remove this at Solr 10
- //After SOLR-14641 other nodes won't call RTG with this param.
- //Just keeping here for backward-compatibility, if we remove this, nodes with older versions will
- //assume that this node can't handle version ranges.
+ // TODO remove this at Solr 10
+ // After SOLR-14641 other nodes won't call RTG with this param.
+ // Just keeping here for backward-compatibility, if we remove this, nodes with older versions
+ // will
+ // assume that this node can't handle version ranges.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/component/QueryElevationComponent.java
##########
@@ -425,7 +449,8 @@ protected ElevationProvider loadElevationProvider(XmlConfigFile config) {
elevationBuilder.addElevatedIds(Collections.singletonList(id));
}
- // It is allowed to define multiple times different elevations for the same query. In this case the elevations
+ // It is allowed to define multiple times different elevations for the same query. In this
+ // case the elevations
// are merged in the ElevationBuilder (they will be triggered at the same time).
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/component/QueryComponent.java
##########
@@ -823,255 +841,282 @@ protected void createMainQuery(ResponseBuilder rb) {
rb.addRequest(this, sreq);
}
-
+
protected boolean addFL(StringBuilder fl, String field, boolean additionalAdded) {
if (additionalAdded) fl.append(",");
fl.append(field);
return true;
}
protected void mergeIds(ResponseBuilder rb, ShardRequest sreq) {
- List<MergeStrategy> mergeStrategies = rb.getMergeStrategies();
- if(mergeStrategies != null) {
- mergeStrategies.sort(MergeStrategy.MERGE_COMP);
- boolean idsMerged = false;
- for(MergeStrategy mergeStrategy : mergeStrategies) {
- mergeStrategy.merge(rb, sreq);
- if(mergeStrategy.mergesIds()) {
- idsMerged = true;
- }
+ List<MergeStrategy> mergeStrategies = rb.getMergeStrategies();
+ if (mergeStrategies != null) {
+ mergeStrategies.sort(MergeStrategy.MERGE_COMP);
+ boolean idsMerged = false;
+ for (MergeStrategy mergeStrategy : mergeStrategies) {
+ mergeStrategy.merge(rb, sreq);
+ if (mergeStrategy.mergesIds()) {
+ idsMerged = true;
}
+ }
- if(idsMerged) {
- return; //ids were merged above so return.
- }
+ if (idsMerged) {
+ return; // ids were merged above so return.
}
+ }
- SortSpec ss = rb.getSortSpec();
- Sort sort = ss.getSort();
+ SortSpec ss = rb.getSortSpec();
+ Sort sort = ss.getSort();
- SortField[] sortFields = null;
- if(sort != null) sortFields = sort.getSort();
- else {
- sortFields = new SortField[]{SortField.FIELD_SCORE};
- }
-
- IndexSchema schema = rb.req.getSchema();
- SchemaField uniqueKeyField = schema.getUniqueKeyField();
+ SortField[] sortFields = null;
+ if (sort != null) sortFields = sort.getSort();
+ else {
+ sortFields = new SortField[] {SortField.FIELD_SCORE};
+ }
+ IndexSchema schema = rb.req.getSchema();
+ SchemaField uniqueKeyField = schema.getUniqueKeyField();
- // id to shard mapping, to eliminate any accidental dups
- HashMap<Object,String> uniqueDoc = new HashMap<>();
+ // id to shard mapping, to eliminate any accidental dups
+ HashMap<Object, String> uniqueDoc = new HashMap<>();
- // Merge the docs via a priority queue so we don't have to sort *all* of the
- // documents... we only need to order the top (rows+start)
- final ShardFieldSortedHitQueue queue = new ShardFieldSortedHitQueue(sortFields, ss.getOffset() + ss.getCount(), rb.req.getSearcher());
+ // Merge the docs via a priority queue so we don't have to sort *all* of the
+ // documents... we only need to order the top (rows+start)
+ final ShardFieldSortedHitQueue queue =
+ new ShardFieldSortedHitQueue(
+ sortFields, ss.getOffset() + ss.getCount(), rb.req.getSearcher());
- NamedList<Object> shardInfo = null;
- if(rb.req.getParams().getBool(ShardParams.SHARDS_INFO, false)) {
- shardInfo = new SimpleOrderedMap<>();
- rb.rsp.getValues().add(ShardParams.SHARDS_INFO,shardInfo);
- }
-
- long numFound = 0;
- boolean hitCountIsExact = true;
- Float maxScore=null;
- boolean thereArePartialResults = false;
- Boolean segmentTerminatedEarly = null;
- for (ShardResponse srsp : sreq.responses) {
- SolrDocumentList docs = null;
- NamedList<?> responseHeader = null;
-
- if(shardInfo!=null) {
- SimpleOrderedMap<Object> nl = new SimpleOrderedMap<>();
-
- if (srsp.getException() != null) {
- Throwable t = srsp.getException();
- if(t instanceof SolrServerException) {
- t = ((SolrServerException)t).getCause();
- }
- nl.add("error", t.toString() );
- StringWriter trace = new StringWriter();
- t.printStackTrace(new PrintWriter(trace));
- nl.add("trace", trace.toString() );
- if (srsp.getShardAddress() != null) {
- nl.add("shardAddress", srsp.getShardAddress());
- }
+ NamedList<Object> shardInfo = null;
+ if (rb.req.getParams().getBool(ShardParams.SHARDS_INFO, false)) {
+ shardInfo = new SimpleOrderedMap<>();
+ rb.rsp.getValues().add(ShardParams.SHARDS_INFO, shardInfo);
+ }
+
+ long numFound = 0;
+ boolean hitCountIsExact = true;
+ Float maxScore = null;
+ boolean thereArePartialResults = false;
+ Boolean segmentTerminatedEarly = null;
+ for (ShardResponse srsp : sreq.responses) {
+ SolrDocumentList docs = null;
+ NamedList<?> responseHeader = null;
+
+ if (shardInfo != null) {
+ SimpleOrderedMap<Object> nl = new SimpleOrderedMap<>();
+
+ if (srsp.getException() != null) {
+ Throwable t = srsp.getException();
+ if (t instanceof SolrServerException) {
+ t = ((SolrServerException) t).getCause();
}
- else {
- responseHeader = (NamedList<?>)srsp.getSolrResponse().getResponse().get("responseHeader");
- final Object rhste = responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY);
- if (rhste != null) {
- nl.add(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY, rhste);
- }
- docs = (SolrDocumentList)srsp.getSolrResponse().getResponse().get("response");
- nl.add("numFound", docs.getNumFound());
- nl.add("numFoundExact", docs.getNumFoundExact());
- nl.add("maxScore", docs.getMaxScore());
+ nl.add("error", t.toString());
+ StringWriter trace = new StringWriter();
+ t.printStackTrace(new PrintWriter(trace));
+ nl.add("trace", trace.toString());
+ if (srsp.getShardAddress() != null) {
nl.add("shardAddress", srsp.getShardAddress());
}
- if(srsp.getSolrResponse()!=null) {
- nl.add("time", srsp.getSolrResponse().getElapsedTime());
+ } else {
+ responseHeader =
+ (NamedList<?>) srsp.getSolrResponse().getResponse().get("responseHeader");
+ final Object rhste =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY);
+ if (rhste != null) {
+ nl.add(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY, rhste);
}
-
- shardInfo.add(srsp.getShard(), nl);
- }
- // now that we've added the shard info, let's only proceed if we have no error.
- if (srsp.getException() != null) {
- thereArePartialResults = true;
- continue;
- }
-
- if (docs == null) { // could have been initialized in the shards info block above
- docs = (SolrDocumentList)srsp.getSolrResponse().getResponse().get("response");
+ docs = (SolrDocumentList) srsp.getSolrResponse().getResponse().get("response");
+ nl.add("numFound", docs.getNumFound());
+ nl.add("numFoundExact", docs.getNumFoundExact());
+ nl.add("maxScore", docs.getMaxScore());
+ nl.add("shardAddress", srsp.getShardAddress());
}
-
- if (responseHeader == null) { // could have been initialized in the shards info block above
- responseHeader = (NamedList<?>)srsp.getSolrResponse().getResponse().get("responseHeader");
+ if (srsp.getSolrResponse() != null) {
+ nl.add("time", srsp.getSolrResponse().getElapsedTime());
}
- final boolean thisResponseIsPartial;
- thisResponseIsPartial = Boolean.TRUE.equals(responseHeader.getBooleanArg(SolrQueryResponse.RESPONSE_HEADER_PARTIAL_RESULTS_KEY));
- thereArePartialResults |= thisResponseIsPartial;
-
- if (!Boolean.TRUE.equals(segmentTerminatedEarly)) {
- final Object ste = responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY);
- if (Boolean.TRUE.equals(ste)) {
- segmentTerminatedEarly = Boolean.TRUE;
- } else if (Boolean.FALSE.equals(ste)) {
- segmentTerminatedEarly = Boolean.FALSE;
- }
- }
-
- // calculate global maxScore and numDocsFound
- if (docs.getMaxScore() != null) {
- maxScore = maxScore==null ? docs.getMaxScore() : Math.max(maxScore, docs.getMaxScore());
- }
- numFound += docs.getNumFound();
-
- if (hitCountIsExact && Boolean.FALSE.equals(docs.getNumFoundExact())) {
- hitCountIsExact = false;
- }
+ shardInfo.add(srsp.getShard(), nl);
+ }
+ // now that we've added the shard info, let's only proceed if we have no error.
+ if (srsp.getException() != null) {
+ thereArePartialResults = true;
+ continue;
+ }
- @SuppressWarnings("unchecked")
- NamedList<List<Object>> sortFieldValues = (NamedList<List<Object>>)(srsp.getSolrResponse().getResponse().get("sort_values"));
- if (null == sortFieldValues) {
- sortFieldValues = new NamedList<>();
- }
+ if (docs == null) { // could have been initialized in the shards info block above
+ docs = (SolrDocumentList) srsp.getSolrResponse().getResponse().get("response");
+ }
- // if the SortSpec contains a field besides score or the Lucene docid, then the values will need to be unmarshalled from
- // sortFieldValues.
- boolean needsUnmarshalling = ss.includesNonScoreOrDocField();
+ if (responseHeader == null) { // could have been initialized in the shards info block above
+ responseHeader = (NamedList<?>) srsp.getSolrResponse().getResponse().get("responseHeader");
+ }
- // if we need to unmarshal the sortFieldValues for sorting but we have none, which can happen if partial results are
- // being returned from the shard, then skip merging the results for the shard. This avoids an exception below.
- // if the shard returned partial results but we don't need to unmarshal (a normal scoring query), then merge what we got.
- if (thisResponseIsPartial && sortFieldValues.size() == 0 && needsUnmarshalling) {
- continue;
+ final boolean thisResponseIsPartial;
+ thisResponseIsPartial =
+ Boolean.TRUE.equals(
+ responseHeader.getBooleanArg(SolrQueryResponse.RESPONSE_HEADER_PARTIAL_RESULTS_KEY));
+ thereArePartialResults |= thisResponseIsPartial;
+
+ if (!Boolean.TRUE.equals(segmentTerminatedEarly)) {
+ final Object ste =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY);
+ if (Boolean.TRUE.equals(ste)) {
+ segmentTerminatedEarly = Boolean.TRUE;
+ } else if (Boolean.FALSE.equals(ste)) {
+ segmentTerminatedEarly = Boolean.FALSE;
}
+ }
- // Checking needsUnmarshalling saves on iterating the SortFields in the SortSpec again.
- NamedList<List<Object>> unmarshalledSortFieldValues = needsUnmarshalling ? unmarshalSortValues(ss, sortFieldValues, schema) : new NamedList<>();
-
- // go through every doc in this response, construct a ShardDoc, and
- // put it in the priority queue so it can be ordered.
- for (int i=0; i<docs.size(); i++) {
- SolrDocument doc = docs.get(i);
- Object id = doc.getFieldValue(uniqueKeyField.getName());
-
- String prevShard = uniqueDoc.put(id, srsp.getShard());
- if (prevShard != null) {
- // duplicate detected
- numFound--;
-
- // For now, just always use the first encountered since we can't currently
- // remove the previous one added to the priority queue. If we switched
- // to the Java5 PriorityQueue, this would be easier.
- continue;
- // make which duplicate is used deterministic based on shard
- // if (prevShard.compareTo(srsp.shard) >= 0) {
- // TODO: remove previous from priority queue
- // continue;
- // }
- }
+ // calculate global maxScore and numDocsFound
+ if (docs.getMaxScore() != null) {
+ maxScore = maxScore == null ? docs.getMaxScore() : Math.max(maxScore, docs.getMaxScore());
+ }
+ numFound += docs.getNumFound();
- ShardDoc shardDoc = new ShardDoc();
- shardDoc.id = id;
- shardDoc.shard = srsp.getShard();
- shardDoc.orderInShard = i;
- Object scoreObj = doc.getFieldValue("score");
- if (scoreObj != null) {
- if (scoreObj instanceof String) {
- shardDoc.score = Float.parseFloat((String)scoreObj);
- } else {
- shardDoc.score = (Float)scoreObj;
- }
- }
+ if (hitCountIsExact && Boolean.FALSE.equals(docs.getNumFoundExact())) {
+ hitCountIsExact = false;
+ }
- shardDoc.sortFieldValues = unmarshalledSortFieldValues;
-
- queue.insertWithOverflow(shardDoc);
- } // end for-each-doc-in-response
- } // end for-each-response
-
- // The queue now has 0 -> queuesize docs, where queuesize <= start + rows
- // So we want to pop the last documents off the queue to get
- // the docs offset -> queuesize
- int resultSize = queue.size() - ss.getOffset();
- resultSize = Math.max(0, resultSize); // there may not be any docs in range
-
- Map<Object,ShardDoc> resultIds = new HashMap<>();
- for (int i=resultSize-1; i>=0; i--) {
- ShardDoc shardDoc = queue.pop();
- shardDoc.positionInResponse = i;
- // Need the toString() for correlation with other lists that must
- // be strings (like keys in highlighting, explain, etc)
- resultIds.put(shardDoc.id.toString(), shardDoc);
+ @SuppressWarnings("unchecked")
+ NamedList<List<Object>> sortFieldValues =
+ (NamedList<List<Object>>) (srsp.getSolrResponse().getResponse().get("sort_values"));
+ if (null == sortFieldValues) {
+ sortFieldValues = new NamedList<>();
}
- // Add hits for distributed requests
- // https://issues.apache.org/jira/browse/SOLR-3518
- rb.rsp.addToLog("hits", numFound);
-
- SolrDocumentList responseDocs = new SolrDocumentList();
- if (maxScore!=null) responseDocs.setMaxScore(maxScore);
- responseDocs.setNumFound(numFound);
- responseDocs.setNumFoundExact(hitCountIsExact);
- responseDocs.setStart(ss.getOffset());
- // size appropriately
- for (int i=0; i<resultSize; i++) responseDocs.add(null);
-
- // save these results in a private area so we can access them
- // again when retrieving stored fields.
- // TODO: use ResponseBuilder (w/ comments) or the request context?
- rb.resultIds = resultIds;
- rb.setResponseDocs(responseDocs);
-
- populateNextCursorMarkFromMergedShards(rb);
-
- if (thereArePartialResults) {
- rb.rsp.getResponseHeader().asShallowMap()
- .put(SolrQueryResponse.RESPONSE_HEADER_PARTIAL_RESULTS_KEY, Boolean.TRUE);
+ // if the SortSpec contains a field besides score or the Lucene docid, then the values will
+ // need to be unmarshalled from
+ // sortFieldValues.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/component/RangeFacetProcessor.java
##########
@@ -208,23 +229,27 @@ public void getFacetRangeCounts(RangeFacetRequest rangeFacetRequest, NamedList<O
intervals.add(after);
}
- IntervalFacets.FacetInterval[] intervalsArray = intervals.toArray(new IntervalFacets.FacetInterval[intervals.size()]);
+ IntervalFacets.FacetInterval[] intervalsArray =
+ intervals.toArray(new IntervalFacets.FacetInterval[intervals.size()]);
// don't use the ArrayList anymore
intervals = null;
new IntervalFacets(sf, searcher, computeDocSet(docsOrig, rfr.getExcludeTags()), intervalsArray);
int intervalIndex = 0;
int lastIntervalIndex = intervalsArray.length - 1;
- // if the user requested "BEFORE", it will be the first of the intervals. Needs to be added to the
+ // if the user requested "BEFORE", it will be the first of the intervals. Needs to be added to
+ // the
// response named list instead of with the counts
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/component/RealTimeGetComponent.java
##########
@@ -227,138 +228,158 @@ public void process(ResponseBuilder rb) throws IOException
SolrDocumentList docList = new SolrDocumentList();
UpdateLog ulog = core.getUpdateHandler().getUpdateLog();
- SearcherInfo searcherInfo = new SearcherInfo(core);
-
+ SearcherInfo searcherInfo = new SearcherInfo(core);
+
// this is initialized & set on the context *after* any searcher (re-)opening
ResultContext resultContext = null;
final DocTransformer transformer = rsp.getReturnFields().getTransformer();
// true in any situation where we have to use a realtime searcher rather then returning docs
// directly from the UpdateLog
final boolean mustUseRealtimeSearcher =
- // if we have filters, we need to check those against the indexed form of the doc
- (rb.getFilters() != null)
- || ((null != transformer) && transformer.needsSolrIndexSearcher());
-
- try {
-
- boolean opennedRealtimeSearcher = false;
- BytesRefBuilder idBytes = new BytesRefBuilder();
- for (String idStr : reqIds.allIds) {
- fieldType.readableToIndexed(idStr, idBytes);
- // if _route_ is passed, id is a child doc. TODO remove in SOLR-15064
- if (!opennedRealtimeSearcher && !params.get(ShardParams._ROUTE_, idStr).equals(idStr)) {
- searcherInfo.clear();
- resultContext = null;
- ulog.openRealtimeSearcher(); // force open a new realtime searcher
- opennedRealtimeSearcher = true;
- } else if (ulog != null) {
- Object o = ulog.lookup(idBytes.get());
- if (o != null) {
- // should currently be a List<Oper,Ver,Doc/Id>
- List<?> entry = (List<?>)o;
- assert entry.size() >= 3;
- int oper = (Integer)entry.get(UpdateLog.FLAGS_IDX) & UpdateLog.OPERATION_MASK;
- switch (oper) {
- case UpdateLog.UPDATE_INPLACE: // fall through to ADD
- case UpdateLog.ADD:
-
- if (mustUseRealtimeSearcher) {
- // close handles to current searchers & result context
- if (!opennedRealtimeSearcher) {
- searcherInfo.clear();
- resultContext = null;
- ulog.openRealtimeSearcher(); // force open a new realtime searcher
- opennedRealtimeSearcher = true;
- }
- o = null; // pretend we never found this record and fall through to use the searcher
- break;
- }
-
- SolrDocument doc;
- if (oper == UpdateLog.ADD) {
- doc = toSolrDoc((SolrInputDocument)entry.get(entry.size()-1), core.getLatestSchema());
- // toSolrDoc filtered copy-field targets already
- if (transformer!=null) {
- transformer.transform(doc, -1); // unknown docID
- }
- } else if (oper == UpdateLog.UPDATE_INPLACE) {
- assert entry.size() == 5;
- // For in-place update case, we have obtained the partial document till now. We need to
- // resolve it to a full document to be returned to the user.
- // resolveFullDocument applies the transformer, if present.
- doc = resolveFullDocument(core, idBytes.get(), rsp.getReturnFields(), (SolrInputDocument)entry.get(entry.size()-1), entry);
- if (doc == null) {
- break; // document has been deleted as the resolve was going on
- }
- doc.visitSelfAndNestedDocs((label, d) -> removeCopyFieldTargets(d, req.getSchema()));
- } else {
- throw new SolrException(ErrorCode.INVALID_STATE, "Expected ADD or UPDATE_INPLACE. Got: " + oper);
- }
-
- docList.add(doc);
- break;
- case UpdateLog.DELETE:
- break;
- default:
- throw new SolrException(SolrException.ErrorCode.SERVER_ERROR, "Unknown Operation! " + oper);
- }
- if (o != null) continue;
- }
- }
-
- // didn't find it in the update log, so it should be in the newest searcher opened
- searcherInfo.init();
- // don't bother with ResultContext yet, we won't need it if doc doesn't match filters
-
- int docid = -1;
- long segAndId = searcherInfo.getSearcher().lookupId(idBytes.get());
- if (segAndId >= 0) {
- int segid = (int) segAndId;
- LeafReaderContext ctx = searcherInfo.getSearcher().getTopReaderContext().leaves().get((int) (segAndId >> 32));
- docid = segid + ctx.docBase;
-
- if (rb.getFilters() != null) {
- for (Query raw : rb.getFilters()) {
- Query q = raw.rewrite(searcherInfo.getSearcher().getIndexReader());
- Scorer scorer = searcherInfo.getSearcher().createWeight(q, ScoreMode.COMPLETE_NO_SCORES, 1f).scorer(ctx);
- if (scorer == null || segid != scorer.iterator().advance(segid)) {
- // filter doesn't match.
- docid = -1;
- break;
- }
- }
- }
- }
-
- if (docid < 0) continue;
-
- Document luceneDocument = searcherInfo.getSearcher().doc(docid, rsp.getReturnFields().getLuceneFieldNames());
- SolrDocument doc = toSolrDoc(luceneDocument, core.getLatestSchema());
- SolrDocumentFetcher docFetcher = searcherInfo.getSearcher().getDocFetcher();
- docFetcher.decorateDocValueFields(doc, docid, docFetcher.getNonStoredDVs(true));
- if ( null != transformer) {
- if (null == resultContext) {
- // either first pass, or we've re-opened searcher - either way now we setContext
- resultContext = new RTGResultContext(rsp.getReturnFields(), searcherInfo.getSearcher(), req);
- transformer.setContext(resultContext); // we avoid calling setContext unless searcher is new/changed
- }
- transformer.transform(doc, docid);
- }
- docList.add(doc);
- } // loop on ids
-
- } finally {
- searcherInfo.clear();
- }
-
- addDocListToResponse(rb, docList);
+ // if we have filters, we need to check those against the indexed form of the doc
+ (rb.getFilters() != null)
+ || ((null != transformer) && transformer.needsSolrIndexSearcher());
+
+ try {
+
+ boolean opennedRealtimeSearcher = false;
+ BytesRefBuilder idBytes = new BytesRefBuilder();
+ for (String idStr : reqIds.allIds) {
+ fieldType.readableToIndexed(idStr, idBytes);
+ // if _route_ is passed, id is a child doc. TODO remove in SOLR-15064
+ if (!opennedRealtimeSearcher && !params.get(ShardParams._ROUTE_, idStr).equals(idStr)) {
+ searcherInfo.clear();
+ resultContext = null;
+ ulog.openRealtimeSearcher(); // force open a new realtime searcher
+ opennedRealtimeSearcher = true;
+ } else if (ulog != null) {
+ Object o = ulog.lookup(idBytes.get());
+ if (o != null) {
+ // should currently be a List<Oper,Ver,Doc/Id>
+ List<?> entry = (List<?>) o;
+ assert entry.size() >= 3;
+ int oper = (Integer) entry.get(UpdateLog.FLAGS_IDX) & UpdateLog.OPERATION_MASK;
+ switch (oper) {
+ case UpdateLog.UPDATE_INPLACE: // fall through to ADD
+ case UpdateLog.ADD:
+ if (mustUseRealtimeSearcher) {
+ // close handles to current searchers & result context
+ if (!opennedRealtimeSearcher) {
+ searcherInfo.clear();
+ resultContext = null;
+ ulog.openRealtimeSearcher(); // force open a new realtime searcher
+ opennedRealtimeSearcher = true;
+ }
+ o = null; // pretend we never found this record and fall through to use the
+ // searcher
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/component/QueryComponent.java
##########
@@ -823,255 +841,282 @@ protected void createMainQuery(ResponseBuilder rb) {
rb.addRequest(this, sreq);
}
-
+
protected boolean addFL(StringBuilder fl, String field, boolean additionalAdded) {
if (additionalAdded) fl.append(",");
fl.append(field);
return true;
}
protected void mergeIds(ResponseBuilder rb, ShardRequest sreq) {
- List<MergeStrategy> mergeStrategies = rb.getMergeStrategies();
- if(mergeStrategies != null) {
- mergeStrategies.sort(MergeStrategy.MERGE_COMP);
- boolean idsMerged = false;
- for(MergeStrategy mergeStrategy : mergeStrategies) {
- mergeStrategy.merge(rb, sreq);
- if(mergeStrategy.mergesIds()) {
- idsMerged = true;
- }
+ List<MergeStrategy> mergeStrategies = rb.getMergeStrategies();
+ if (mergeStrategies != null) {
+ mergeStrategies.sort(MergeStrategy.MERGE_COMP);
+ boolean idsMerged = false;
+ for (MergeStrategy mergeStrategy : mergeStrategies) {
+ mergeStrategy.merge(rb, sreq);
+ if (mergeStrategy.mergesIds()) {
+ idsMerged = true;
}
+ }
- if(idsMerged) {
- return; //ids were merged above so return.
- }
+ if (idsMerged) {
+ return; // ids were merged above so return.
}
+ }
- SortSpec ss = rb.getSortSpec();
- Sort sort = ss.getSort();
+ SortSpec ss = rb.getSortSpec();
+ Sort sort = ss.getSort();
- SortField[] sortFields = null;
- if(sort != null) sortFields = sort.getSort();
- else {
- sortFields = new SortField[]{SortField.FIELD_SCORE};
- }
-
- IndexSchema schema = rb.req.getSchema();
- SchemaField uniqueKeyField = schema.getUniqueKeyField();
+ SortField[] sortFields = null;
+ if (sort != null) sortFields = sort.getSort();
+ else {
+ sortFields = new SortField[] {SortField.FIELD_SCORE};
+ }
+ IndexSchema schema = rb.req.getSchema();
+ SchemaField uniqueKeyField = schema.getUniqueKeyField();
- // id to shard mapping, to eliminate any accidental dups
- HashMap<Object,String> uniqueDoc = new HashMap<>();
+ // id to shard mapping, to eliminate any accidental dups
+ HashMap<Object, String> uniqueDoc = new HashMap<>();
- // Merge the docs via a priority queue so we don't have to sort *all* of the
- // documents... we only need to order the top (rows+start)
- final ShardFieldSortedHitQueue queue = new ShardFieldSortedHitQueue(sortFields, ss.getOffset() + ss.getCount(), rb.req.getSearcher());
+ // Merge the docs via a priority queue so we don't have to sort *all* of the
+ // documents... we only need to order the top (rows+start)
+ final ShardFieldSortedHitQueue queue =
+ new ShardFieldSortedHitQueue(
+ sortFields, ss.getOffset() + ss.getCount(), rb.req.getSearcher());
- NamedList<Object> shardInfo = null;
- if(rb.req.getParams().getBool(ShardParams.SHARDS_INFO, false)) {
- shardInfo = new SimpleOrderedMap<>();
- rb.rsp.getValues().add(ShardParams.SHARDS_INFO,shardInfo);
- }
-
- long numFound = 0;
- boolean hitCountIsExact = true;
- Float maxScore=null;
- boolean thereArePartialResults = false;
- Boolean segmentTerminatedEarly = null;
- for (ShardResponse srsp : sreq.responses) {
- SolrDocumentList docs = null;
- NamedList<?> responseHeader = null;
-
- if(shardInfo!=null) {
- SimpleOrderedMap<Object> nl = new SimpleOrderedMap<>();
-
- if (srsp.getException() != null) {
- Throwable t = srsp.getException();
- if(t instanceof SolrServerException) {
- t = ((SolrServerException)t).getCause();
- }
- nl.add("error", t.toString() );
- StringWriter trace = new StringWriter();
- t.printStackTrace(new PrintWriter(trace));
- nl.add("trace", trace.toString() );
- if (srsp.getShardAddress() != null) {
- nl.add("shardAddress", srsp.getShardAddress());
- }
+ NamedList<Object> shardInfo = null;
+ if (rb.req.getParams().getBool(ShardParams.SHARDS_INFO, false)) {
+ shardInfo = new SimpleOrderedMap<>();
+ rb.rsp.getValues().add(ShardParams.SHARDS_INFO, shardInfo);
+ }
+
+ long numFound = 0;
+ boolean hitCountIsExact = true;
+ Float maxScore = null;
+ boolean thereArePartialResults = false;
+ Boolean segmentTerminatedEarly = null;
+ for (ShardResponse srsp : sreq.responses) {
+ SolrDocumentList docs = null;
+ NamedList<?> responseHeader = null;
+
+ if (shardInfo != null) {
+ SimpleOrderedMap<Object> nl = new SimpleOrderedMap<>();
+
+ if (srsp.getException() != null) {
+ Throwable t = srsp.getException();
+ if (t instanceof SolrServerException) {
+ t = ((SolrServerException) t).getCause();
}
- else {
- responseHeader = (NamedList<?>)srsp.getSolrResponse().getResponse().get("responseHeader");
- final Object rhste = responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY);
- if (rhste != null) {
- nl.add(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY, rhste);
- }
- docs = (SolrDocumentList)srsp.getSolrResponse().getResponse().get("response");
- nl.add("numFound", docs.getNumFound());
- nl.add("numFoundExact", docs.getNumFoundExact());
- nl.add("maxScore", docs.getMaxScore());
+ nl.add("error", t.toString());
+ StringWriter trace = new StringWriter();
+ t.printStackTrace(new PrintWriter(trace));
+ nl.add("trace", trace.toString());
+ if (srsp.getShardAddress() != null) {
nl.add("shardAddress", srsp.getShardAddress());
}
- if(srsp.getSolrResponse()!=null) {
- nl.add("time", srsp.getSolrResponse().getElapsedTime());
+ } else {
+ responseHeader =
+ (NamedList<?>) srsp.getSolrResponse().getResponse().get("responseHeader");
+ final Object rhste =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY);
+ if (rhste != null) {
+ nl.add(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY, rhste);
}
-
- shardInfo.add(srsp.getShard(), nl);
- }
- // now that we've added the shard info, let's only proceed if we have no error.
- if (srsp.getException() != null) {
- thereArePartialResults = true;
- continue;
- }
-
- if (docs == null) { // could have been initialized in the shards info block above
- docs = (SolrDocumentList)srsp.getSolrResponse().getResponse().get("response");
+ docs = (SolrDocumentList) srsp.getSolrResponse().getResponse().get("response");
+ nl.add("numFound", docs.getNumFound());
+ nl.add("numFoundExact", docs.getNumFoundExact());
+ nl.add("maxScore", docs.getMaxScore());
+ nl.add("shardAddress", srsp.getShardAddress());
}
-
- if (responseHeader == null) { // could have been initialized in the shards info block above
- responseHeader = (NamedList<?>)srsp.getSolrResponse().getResponse().get("responseHeader");
+ if (srsp.getSolrResponse() != null) {
+ nl.add("time", srsp.getSolrResponse().getElapsedTime());
}
- final boolean thisResponseIsPartial;
- thisResponseIsPartial = Boolean.TRUE.equals(responseHeader.getBooleanArg(SolrQueryResponse.RESPONSE_HEADER_PARTIAL_RESULTS_KEY));
- thereArePartialResults |= thisResponseIsPartial;
-
- if (!Boolean.TRUE.equals(segmentTerminatedEarly)) {
- final Object ste = responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY);
- if (Boolean.TRUE.equals(ste)) {
- segmentTerminatedEarly = Boolean.TRUE;
- } else if (Boolean.FALSE.equals(ste)) {
- segmentTerminatedEarly = Boolean.FALSE;
- }
- }
-
- // calculate global maxScore and numDocsFound
- if (docs.getMaxScore() != null) {
- maxScore = maxScore==null ? docs.getMaxScore() : Math.max(maxScore, docs.getMaxScore());
- }
- numFound += docs.getNumFound();
-
- if (hitCountIsExact && Boolean.FALSE.equals(docs.getNumFoundExact())) {
- hitCountIsExact = false;
- }
+ shardInfo.add(srsp.getShard(), nl);
+ }
+ // now that we've added the shard info, let's only proceed if we have no error.
+ if (srsp.getException() != null) {
+ thereArePartialResults = true;
+ continue;
+ }
- @SuppressWarnings("unchecked")
- NamedList<List<Object>> sortFieldValues = (NamedList<List<Object>>)(srsp.getSolrResponse().getResponse().get("sort_values"));
- if (null == sortFieldValues) {
- sortFieldValues = new NamedList<>();
- }
+ if (docs == null) { // could have been initialized in the shards info block above
+ docs = (SolrDocumentList) srsp.getSolrResponse().getResponse().get("response");
+ }
- // if the SortSpec contains a field besides score or the Lucene docid, then the values will need to be unmarshalled from
- // sortFieldValues.
- boolean needsUnmarshalling = ss.includesNonScoreOrDocField();
+ if (responseHeader == null) { // could have been initialized in the shards info block above
+ responseHeader = (NamedList<?>) srsp.getSolrResponse().getResponse().get("responseHeader");
+ }
- // if we need to unmarshal the sortFieldValues for sorting but we have none, which can happen if partial results are
- // being returned from the shard, then skip merging the results for the shard. This avoids an exception below.
- // if the shard returned partial results but we don't need to unmarshal (a normal scoring query), then merge what we got.
- if (thisResponseIsPartial && sortFieldValues.size() == 0 && needsUnmarshalling) {
- continue;
+ final boolean thisResponseIsPartial;
+ thisResponseIsPartial =
+ Boolean.TRUE.equals(
+ responseHeader.getBooleanArg(SolrQueryResponse.RESPONSE_HEADER_PARTIAL_RESULTS_KEY));
+ thereArePartialResults |= thisResponseIsPartial;
+
+ if (!Boolean.TRUE.equals(segmentTerminatedEarly)) {
+ final Object ste =
+ responseHeader.get(SolrQueryResponse.RESPONSE_HEADER_SEGMENT_TERMINATED_EARLY_KEY);
+ if (Boolean.TRUE.equals(ste)) {
+ segmentTerminatedEarly = Boolean.TRUE;
+ } else if (Boolean.FALSE.equals(ste)) {
+ segmentTerminatedEarly = Boolean.FALSE;
}
+ }
- // Checking needsUnmarshalling saves on iterating the SortFields in the SortSpec again.
- NamedList<List<Object>> unmarshalledSortFieldValues = needsUnmarshalling ? unmarshalSortValues(ss, sortFieldValues, schema) : new NamedList<>();
-
- // go through every doc in this response, construct a ShardDoc, and
- // put it in the priority queue so it can be ordered.
- for (int i=0; i<docs.size(); i++) {
- SolrDocument doc = docs.get(i);
- Object id = doc.getFieldValue(uniqueKeyField.getName());
-
- String prevShard = uniqueDoc.put(id, srsp.getShard());
- if (prevShard != null) {
- // duplicate detected
- numFound--;
-
- // For now, just always use the first encountered since we can't currently
- // remove the previous one added to the priority queue. If we switched
- // to the Java5 PriorityQueue, this would be easier.
- continue;
- // make which duplicate is used deterministic based on shard
- // if (prevShard.compareTo(srsp.shard) >= 0) {
- // TODO: remove previous from priority queue
- // continue;
- // }
- }
+ // calculate global maxScore and numDocsFound
+ if (docs.getMaxScore() != null) {
+ maxScore = maxScore == null ? docs.getMaxScore() : Math.max(maxScore, docs.getMaxScore());
+ }
+ numFound += docs.getNumFound();
- ShardDoc shardDoc = new ShardDoc();
- shardDoc.id = id;
- shardDoc.shard = srsp.getShard();
- shardDoc.orderInShard = i;
- Object scoreObj = doc.getFieldValue("score");
- if (scoreObj != null) {
- if (scoreObj instanceof String) {
- shardDoc.score = Float.parseFloat((String)scoreObj);
- } else {
- shardDoc.score = (Float)scoreObj;
- }
- }
+ if (hitCountIsExact && Boolean.FALSE.equals(docs.getNumFoundExact())) {
+ hitCountIsExact = false;
+ }
- shardDoc.sortFieldValues = unmarshalledSortFieldValues;
-
- queue.insertWithOverflow(shardDoc);
- } // end for-each-doc-in-response
- } // end for-each-response
-
- // The queue now has 0 -> queuesize docs, where queuesize <= start + rows
- // So we want to pop the last documents off the queue to get
- // the docs offset -> queuesize
- int resultSize = queue.size() - ss.getOffset();
- resultSize = Math.max(0, resultSize); // there may not be any docs in range
-
- Map<Object,ShardDoc> resultIds = new HashMap<>();
- for (int i=resultSize-1; i>=0; i--) {
- ShardDoc shardDoc = queue.pop();
- shardDoc.positionInResponse = i;
- // Need the toString() for correlation with other lists that must
- // be strings (like keys in highlighting, explain, etc)
- resultIds.put(shardDoc.id.toString(), shardDoc);
+ @SuppressWarnings("unchecked")
+ NamedList<List<Object>> sortFieldValues =
+ (NamedList<List<Object>>) (srsp.getSolrResponse().getResponse().get("sort_values"));
+ if (null == sortFieldValues) {
+ sortFieldValues = new NamedList<>();
}
- // Add hits for distributed requests
- // https://issues.apache.org/jira/browse/SOLR-3518
- rb.rsp.addToLog("hits", numFound);
-
- SolrDocumentList responseDocs = new SolrDocumentList();
- if (maxScore!=null) responseDocs.setMaxScore(maxScore);
- responseDocs.setNumFound(numFound);
- responseDocs.setNumFoundExact(hitCountIsExact);
- responseDocs.setStart(ss.getOffset());
- // size appropriately
- for (int i=0; i<resultSize; i++) responseDocs.add(null);
-
- // save these results in a private area so we can access them
- // again when retrieving stored fields.
- // TODO: use ResponseBuilder (w/ comments) or the request context?
- rb.resultIds = resultIds;
- rb.setResponseDocs(responseDocs);
-
- populateNextCursorMarkFromMergedShards(rb);
-
- if (thereArePartialResults) {
- rb.rsp.getResponseHeader().asShallowMap()
- .put(SolrQueryResponse.RESPONSE_HEADER_PARTIAL_RESULTS_KEY, Boolean.TRUE);
+ // if the SortSpec contains a field besides score or the Lucene docid, then the values will
+ // need to be unmarshalled from
+ // sortFieldValues.
+ boolean needsUnmarshalling = ss.includesNonScoreOrDocField();
+
+ // if we need to unmarshal the sortFieldValues for sorting but we have none, which can happen
+ // if partial results are
+ // being returned from the shard, then skip merging the results for the shard. This avoids an
+ // exception below.
+ // if the shard returned partial results but we don't need to unmarshal (a normal scoring
+ // query), then merge what we got.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/component/RealTimeGetComponent.java
##########
@@ -227,138 +228,158 @@ public void process(ResponseBuilder rb) throws IOException
SolrDocumentList docList = new SolrDocumentList();
UpdateLog ulog = core.getUpdateHandler().getUpdateLog();
- SearcherInfo searcherInfo = new SearcherInfo(core);
-
+ SearcherInfo searcherInfo = new SearcherInfo(core);
+
// this is initialized & set on the context *after* any searcher (re-)opening
ResultContext resultContext = null;
final DocTransformer transformer = rsp.getReturnFields().getTransformer();
// true in any situation where we have to use a realtime searcher rather then returning docs
// directly from the UpdateLog
final boolean mustUseRealtimeSearcher =
- // if we have filters, we need to check those against the indexed form of the doc
- (rb.getFilters() != null)
- || ((null != transformer) && transformer.needsSolrIndexSearcher());
-
- try {
-
- boolean opennedRealtimeSearcher = false;
- BytesRefBuilder idBytes = new BytesRefBuilder();
- for (String idStr : reqIds.allIds) {
- fieldType.readableToIndexed(idStr, idBytes);
- // if _route_ is passed, id is a child doc. TODO remove in SOLR-15064
- if (!opennedRealtimeSearcher && !params.get(ShardParams._ROUTE_, idStr).equals(idStr)) {
- searcherInfo.clear();
- resultContext = null;
- ulog.openRealtimeSearcher(); // force open a new realtime searcher
- opennedRealtimeSearcher = true;
- } else if (ulog != null) {
- Object o = ulog.lookup(idBytes.get());
- if (o != null) {
- // should currently be a List<Oper,Ver,Doc/Id>
- List<?> entry = (List<?>)o;
- assert entry.size() >= 3;
- int oper = (Integer)entry.get(UpdateLog.FLAGS_IDX) & UpdateLog.OPERATION_MASK;
- switch (oper) {
- case UpdateLog.UPDATE_INPLACE: // fall through to ADD
- case UpdateLog.ADD:
-
- if (mustUseRealtimeSearcher) {
- // close handles to current searchers & result context
- if (!opennedRealtimeSearcher) {
- searcherInfo.clear();
- resultContext = null;
- ulog.openRealtimeSearcher(); // force open a new realtime searcher
- opennedRealtimeSearcher = true;
- }
- o = null; // pretend we never found this record and fall through to use the searcher
- break;
- }
-
- SolrDocument doc;
- if (oper == UpdateLog.ADD) {
- doc = toSolrDoc((SolrInputDocument)entry.get(entry.size()-1), core.getLatestSchema());
- // toSolrDoc filtered copy-field targets already
- if (transformer!=null) {
- transformer.transform(doc, -1); // unknown docID
- }
- } else if (oper == UpdateLog.UPDATE_INPLACE) {
- assert entry.size() == 5;
- // For in-place update case, we have obtained the partial document till now. We need to
- // resolve it to a full document to be returned to the user.
- // resolveFullDocument applies the transformer, if present.
- doc = resolveFullDocument(core, idBytes.get(), rsp.getReturnFields(), (SolrInputDocument)entry.get(entry.size()-1), entry);
- if (doc == null) {
- break; // document has been deleted as the resolve was going on
- }
- doc.visitSelfAndNestedDocs((label, d) -> removeCopyFieldTargets(d, req.getSchema()));
- } else {
- throw new SolrException(ErrorCode.INVALID_STATE, "Expected ADD or UPDATE_INPLACE. Got: " + oper);
- }
-
- docList.add(doc);
- break;
- case UpdateLog.DELETE:
- break;
- default:
- throw new SolrException(SolrException.ErrorCode.SERVER_ERROR, "Unknown Operation! " + oper);
- }
- if (o != null) continue;
- }
- }
-
- // didn't find it in the update log, so it should be in the newest searcher opened
- searcherInfo.init();
- // don't bother with ResultContext yet, we won't need it if doc doesn't match filters
-
- int docid = -1;
- long segAndId = searcherInfo.getSearcher().lookupId(idBytes.get());
- if (segAndId >= 0) {
- int segid = (int) segAndId;
- LeafReaderContext ctx = searcherInfo.getSearcher().getTopReaderContext().leaves().get((int) (segAndId >> 32));
- docid = segid + ctx.docBase;
-
- if (rb.getFilters() != null) {
- for (Query raw : rb.getFilters()) {
- Query q = raw.rewrite(searcherInfo.getSearcher().getIndexReader());
- Scorer scorer = searcherInfo.getSearcher().createWeight(q, ScoreMode.COMPLETE_NO_SCORES, 1f).scorer(ctx);
- if (scorer == null || segid != scorer.iterator().advance(segid)) {
- // filter doesn't match.
- docid = -1;
- break;
- }
- }
- }
- }
-
- if (docid < 0) continue;
-
- Document luceneDocument = searcherInfo.getSearcher().doc(docid, rsp.getReturnFields().getLuceneFieldNames());
- SolrDocument doc = toSolrDoc(luceneDocument, core.getLatestSchema());
- SolrDocumentFetcher docFetcher = searcherInfo.getSearcher().getDocFetcher();
- docFetcher.decorateDocValueFields(doc, docid, docFetcher.getNonStoredDVs(true));
- if ( null != transformer) {
- if (null == resultContext) {
- // either first pass, or we've re-opened searcher - either way now we setContext
- resultContext = new RTGResultContext(rsp.getReturnFields(), searcherInfo.getSearcher(), req);
- transformer.setContext(resultContext); // we avoid calling setContext unless searcher is new/changed
- }
- transformer.transform(doc, docid);
- }
- docList.add(doc);
- } // loop on ids
-
- } finally {
- searcherInfo.clear();
- }
-
- addDocListToResponse(rb, docList);
+ // if we have filters, we need to check those against the indexed form of the doc
+ (rb.getFilters() != null)
+ || ((null != transformer) && transformer.needsSolrIndexSearcher());
+
+ try {
+
+ boolean opennedRealtimeSearcher = false;
+ BytesRefBuilder idBytes = new BytesRefBuilder();
+ for (String idStr : reqIds.allIds) {
+ fieldType.readableToIndexed(idStr, idBytes);
+ // if _route_ is passed, id is a child doc. TODO remove in SOLR-15064
+ if (!opennedRealtimeSearcher && !params.get(ShardParams._ROUTE_, idStr).equals(idStr)) {
+ searcherInfo.clear();
+ resultContext = null;
+ ulog.openRealtimeSearcher(); // force open a new realtime searcher
+ opennedRealtimeSearcher = true;
+ } else if (ulog != null) {
+ Object o = ulog.lookup(idBytes.get());
+ if (o != null) {
+ // should currently be a List<Oper,Ver,Doc/Id>
+ List<?> entry = (List<?>) o;
+ assert entry.size() >= 3;
+ int oper = (Integer) entry.get(UpdateLog.FLAGS_IDX) & UpdateLog.OPERATION_MASK;
+ switch (oper) {
+ case UpdateLog.UPDATE_INPLACE: // fall through to ADD
+ case UpdateLog.ADD:
+ if (mustUseRealtimeSearcher) {
+ // close handles to current searchers & result context
+ if (!opennedRealtimeSearcher) {
+ searcherInfo.clear();
+ resultContext = null;
+ ulog.openRealtimeSearcher(); // force open a new realtime searcher
+ opennedRealtimeSearcher = true;
+ }
+ o = null; // pretend we never found this record and fall through to use the
+ // searcher
+ break;
+ }
+
+ SolrDocument doc;
+ if (oper == UpdateLog.ADD) {
+ doc =
+ toSolrDoc(
+ (SolrInputDocument) entry.get(entry.size() - 1), core.getLatestSchema());
+ // toSolrDoc filtered copy-field targets already
+ if (transformer != null) {
+ transformer.transform(doc, -1); // unknown docID
+ }
+ } else if (oper == UpdateLog.UPDATE_INPLACE) {
+ assert entry.size() == 5;
+ // For in-place update case, we have obtained the partial document till now. We
+ // need to
+ // resolve it to a full document to be returned to the user.
+ // resolveFullDocument applies the transformer, if present.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/component/RealTimeGetComponent.java
##########
@@ -632,35 +694,43 @@ private static SolrInputDocument getInputDocumentFromTlog(
return iDoc;
}
- /** returns the SolrInputDocument from the current tlog, or DELETED if it has been deleted, or
- * null if there is no record of it in the current update log. If null is returned, it could
- * still be in the latest index. Copy-field target fields are excluded.
+ /**
+ * returns the SolrInputDocument from the current tlog, or DELETED if it has been deleted, or null
+ * if there is no record of it in the current update log. If null is returned, it could still be
+ * in the latest index. Copy-field target fields are excluded.
+ *
* @param idBytes doc ID to find; never a child doc.
- * @param versionReturned If a non-null AtomicLong is passed in, it is set to the version of the update returned from the TLog.
+ * @param versionReturned If a non-null AtomicLong is passed in, it is set to the version of the
+ * update returned from the TLog.
*/
@SuppressWarnings({"fallthrough"})
- public static SolrInputDocument getInputDocumentFromTlog(SolrCore core, BytesRef idBytes, AtomicLong versionReturned,
- Set<String> onlyTheseFields, Resolution resolution) {
+ public static SolrInputDocument getInputDocumentFromTlog(
+ SolrCore core,
+ BytesRef idBytes,
+ AtomicLong versionReturned,
+ Set<String> onlyTheseFields,
+ Resolution resolution) {
UpdateLog ulog = core.getUpdateHandler().getUpdateLog();
if (ulog != null) {
Object o = ulog.lookup(idBytes);
if (o != null) {
// should currently be a List<Oper,Ver,Doc/Id>
- List<?> entry = (List<?>)o;
+ List<?> entry = (List<?>) o;
assert entry.size() >= 3;
- int oper = (Integer)entry.get(0) & UpdateLog.OPERATION_MASK;
+ int oper = (Integer) entry.get(0) & UpdateLog.OPERATION_MASK;
if (versionReturned != null) {
- versionReturned.set((long)entry.get(UpdateLog.VERSION_IDX));
+ versionReturned.set((long) entry.get(UpdateLog.VERSION_IDX));
}
switch (oper) {
case UpdateLog.UPDATE_INPLACE:
assert entry.size() == 5;
-
+
if (resolution != Resolution.PARTIAL) {
- SolrInputDocument doc = (SolrInputDocument)entry.get(entry.size()-1);
+ SolrInputDocument doc = (SolrInputDocument) entry.get(entry.size() - 1);
try {
- // For in-place update case, we have obtained the partial document till now. We need to
+ // For in-place update case, we have obtained the partial document till now. We need
+ // to
// resolve it to a full document to be returned to the user.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/component/RealTimeGetComponent.java
##########
@@ -450,7 +483,8 @@ private static SolrDocument resolveFullDocument(SolrCore core, BytesRef idBytes,
if (lastPrevPointer == -1) { // full document was not found in tlog, but exists in index
return mergePartialDocWithFullDocFromIndex(core, idBytes, returnFields, partialDoc);
} else if (lastPrevPointer > 0) {
- // We were supposed to have found the last full doc also in the tlogs, but the prevPointer links led to nowhere
+ // We were supposed to have found the last full doc also in the tlogs, but the prevPointer
+ // links led to nowhere
// We should reopen a new RT searcher and get the doc. This should be a rare occurrence
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/component/RealTimeGetComponent.java
##########
@@ -775,59 +857,65 @@ private static boolean hasRootTerm(SolrIndexSearcher searcher, BytesRef rootIdBy
}
/** Traverse the doc looking for a doc with the specified ID. */
- private static SolrInputDocument findNestedDocById(SolrInputDocument iDoc, BytesRef idBytes, IndexSchema schema) {
+ private static SolrInputDocument findNestedDocById(
+ SolrInputDocument iDoc, BytesRef idBytes, IndexSchema schema) {
assert schema.printableUniqueKey(iDoc) != null : "need IDs";
// traverse nested doc, looking for the node with the ID we are looking for
SolrInputDocument[] found = new SolrInputDocument[1];
String idStr = schema.printableUniqueKey(idBytes);
- BiConsumer<String, SolrInputDocument> finder = (label, childDoc) -> {
- if (found[0] == null && idStr.equals(schema.printableUniqueKey(childDoc))) {
- found[0] = childDoc;
- }
- };
+ BiConsumer<String, SolrInputDocument> finder =
+ (label, childDoc) -> {
+ if (found[0] == null && idStr.equals(schema.printableUniqueKey(childDoc))) {
+ found[0] = childDoc;
+ }
+ };
iDoc.visitSelfAndNestedDocs(finder);
return found[0];
}
- private static SolrReturnFields makeReturnFields(SolrCore core, Set<String> requestedFields, Resolution resolution) {
+ private static SolrReturnFields makeReturnFields(
+ SolrCore core, Set<String> requestedFields, Resolution resolution) {
DocTransformer docTransformer;
- if (resolution == Resolution.ROOT_WITH_CHILDREN && core.getLatestSchema().isUsableForChildDocs()) {
+ if (resolution == Resolution.ROOT_WITH_CHILDREN
+ && core.getLatestSchema().isUsableForChildDocs()) {
SolrParams params = new ModifiableSolrParams().set("limit", "-1");
try (LocalSolrQueryRequest req = new LocalSolrQueryRequest(core, params)) {
docTransformer = core.getTransformerFactory("child").create(null, params, req);
}
} else {
docTransformer = null;
}
- // TODO optimization: add feature to SolrReturnFields to exclude copyFieldTargets from wildcard matching.
+ // TODO optimization: add feature to SolrReturnFields to exclude copyFieldTargets from wildcard
+ // matching.
// Today, we filter this data out later before returning, but it's already been fetched.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/component/RealTimeGetComponent.java
##########
@@ -881,7 +970,8 @@ private static IndexableField materialize(IndexableField in) {
return new ClonedField(in);
}
- private static class ClonedField extends Field { // TODO Lucene Field has no copy constructor; maybe it should?
+ private static class ClonedField
+ extends Field { // TODO Lucene Field has no copy constructor; maybe it should?
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/component/RealTimeGetComponent.java
##########
@@ -537,18 +580,28 @@ private static SolrDocument mergePartialDocWithFullDocFromIndex(SolrCore core, B
SolrDocument doc = fetchSolrDoc(searcher, docid, returnFields);
if (!doc.containsKey(VERSION_FIELD)) {
- searcher.getDocFetcher().decorateDocValueFields(doc, docid, Collections.singleton(VERSION_FIELD));
+ searcher
+ .getDocFetcher()
+ .decorateDocValueFields(doc, docid, Collections.singleton(VERSION_FIELD));
}
long docVersion = (long) doc.getFirstValue(VERSION_FIELD);
Object partialVersionObj = partialDoc.getFieldValue(VERSION_FIELD);
- long partialDocVersion = partialVersionObj instanceof Field? ((Field) partialVersionObj).numericValue().longValue():
- partialVersionObj instanceof Number? ((Number) partialVersionObj).longValue(): Long.parseLong(partialVersionObj.toString());
+ long partialDocVersion =
+ partialVersionObj instanceof Field
+ ? ((Field) partialVersionObj).numericValue().longValue()
+ : partialVersionObj instanceof Number
+ ? ((Number) partialVersionObj).longValue()
+ : Long.parseLong(partialVersionObj.toString());
if (docVersion > partialDocVersion) {
return doc;
}
- for (String fieldName: partialDoc.getFieldNames()) {
- doc.setField(fieldName.toString(), partialDoc.getFieldValue(fieldName)); // since partial doc will only contain single valued fields, this is fine
+ for (String fieldName : partialDoc.getFieldNames()) {
+ doc.setField(
+ fieldName.toString(),
+ partialDoc.getFieldValue(
+ fieldName)); // since partial doc will only contain single valued fields, this is
+ // fine
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/component/SearchHandler.java
##########
@@ -246,20 +248,24 @@ public ShardHandler getAndPrepShardHandler(SolrQueryRequest req, ResponseBuilder
final String shards = req.getParams().get(ShardParams.SHARDS);
rb.isDistrib = ((shards != null) && (shards.indexOf('/') > 0));
}
-
+
if (rb.isDistrib) {
shardHandler = shardHandlerFactory.getShardHandler();
shardHandler.prepDistributed(rb);
if (!rb.isDistrib) {
- shardHandler = null; // request is not distributed after all and so the shard handler is not needed
+ shardHandler =
+ null; // request is not distributed after all and so the shard handler is not needed
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/component/SearchHandler.java
##########
@@ -317,32 +324,38 @@ public void handleRequestBody(SolrQueryRequest req, SolrQueryResponse rsp) throw
if (trippedCircuitBreakers != null) {
String errorMessage = CircuitBreakerManager.toErrorMessage(trippedCircuitBreakers);
rsp.add(STATUS, FAILURE);
- rsp.setException(new SolrException(SolrException.ErrorCode.SERVICE_UNAVAILABLE, "Circuit Breakers tripped " + errorMessage));
+ rsp.setException(
+ new SolrException(
+ SolrException.ErrorCode.SERVICE_UNAVAILABLE,
+ "Circuit Breakers tripped " + errorMessage));
return;
}
}
- final ShardHandler shardHandler1 = getAndPrepShardHandler(req, rb); // creates a ShardHandler object only if it's needed
+ final ShardHandler shardHandler1 =
+ getAndPrepShardHandler(req, rb); // creates a ShardHandler object only if it's needed
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/component/SearchHandler.java
##########
@@ -416,32 +430,34 @@ public void handleRequestBody(SolrQueryRequest req, SolrQueryResponse rsp) throw
nextStage = ResponseBuilder.STAGE_DONE;
// call all components
- for( SearchComponent c : components ) {
+ for (SearchComponent c : components) {
// the next stage is the minimum of what all components report
nextStage = Math.min(nextStage, c.distributedProcess(rb));
}
-
// check the outgoing queue and send requests
while (rb.outgoing.size() > 0) {
// submit all current request tasks at once
while (rb.outgoing.size() > 0) {
ShardRequest sreq = rb.outgoing.remove(0);
sreq.actualShards = sreq.shards;
- if (sreq.actualShards==ShardRequest.ALL_SHARDS) {
+ if (sreq.actualShards == ShardRequest.ALL_SHARDS) {
sreq.actualShards = rb.shards;
}
- sreq.responses = new ArrayList<>(sreq.actualShards.length); // presume we'll get a response from each shard we send to
+ sreq.responses =
+ new ArrayList<>(
+ sreq.actualShards
+ .length); // presume we'll get a response from each shard we send to
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/component/SpatialHeatmapFacets.java
##########
@@ -83,32 +99,43 @@ private static FacetRequest createHeatmapRequest(String fieldKey, String fieldNa
return heatmapFacets;
}
- /** Called by FacetComponent's impl of
- * {@link org.apache.solr.handler.component.SearchComponent#modifyRequest(ResponseBuilder, SearchComponent, ShardRequest)}. */
- public static void distribModifyRequest(ShardRequest sreq, LinkedHashMap<String, HeatmapFacet> heatmapFacets) {
- // Set the format to PNG because it's compressed and it's the only format we have code to read at the moment.
- // We re-write the facet.heatmap list with PNG format in local-params where it has highest precedence.
-
- //Remove existing heatmap field param vals; we will rewrite
+ /**
+ * Called by FacetComponent's impl of {@link
+ * org.apache.solr.handler.component.SearchComponent#modifyRequest(ResponseBuilder,
+ * SearchComponent, ShardRequest)}.
+ */
+ public static void distribModifyRequest(
+ ShardRequest sreq, LinkedHashMap<String, HeatmapFacet> heatmapFacets) {
+ // Set the format to PNG because it's compressed and it's the only format we have code to read
+ // at the moment.
+ // We re-write the facet.heatmap list with PNG format in local-params where it has highest
+ // precedence.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/component/SearchHandler.java
##########
@@ -468,117 +484,128 @@ public void handleRequestBody(SolrQueryRequest req, SolrQueryResponse rsp) throw
}
}
-
// now wait for replies, but if anyone puts more requests on
// the outgoing queue, send them out immediately (by exiting
// this loop)
boolean tolerant = ShardParams.getShardsTolerantAsBool(rb.req.getParams());
while (rb.outgoing.size() == 0) {
- ShardResponse srsp = tolerant ?
- shardHandler1.takeCompletedIncludingErrors():
- shardHandler1.takeCompletedOrError();
- if (srsp == null) break; // no more requests to wait for
+ ShardResponse srsp =
+ tolerant
+ ? shardHandler1.takeCompletedIncludingErrors()
+ : shardHandler1.takeCompletedOrError();
+ if (srsp == null) break; // no more requests to wait for
- // Was there an exception?
+ // Was there an exception?
if (srsp.getException() != null) {
// If things are not tolerant, abort everything and rethrow
- if(!tolerant) {
+ if (!tolerant) {
shardHandler1.cancelAll();
if (srsp.getException() instanceof SolrException) {
- throw (SolrException)srsp.getException();
+ throw (SolrException) srsp.getException();
} else {
- throw new SolrException(SolrException.ErrorCode.SERVER_ERROR, srsp.getException());
+ throw new SolrException(
+ SolrException.ErrorCode.SERVER_ERROR, srsp.getException());
}
} else {
- rsp.getResponseHeader().asShallowMap().put(SolrQueryResponse.RESPONSE_HEADER_PARTIAL_RESULTS_KEY, Boolean.TRUE);
+ rsp.getResponseHeader()
+ .asShallowMap()
+ .put(SolrQueryResponse.RESPONSE_HEADER_PARTIAL_RESULTS_KEY, Boolean.TRUE);
}
}
rb.finished.add(srsp.getShardRequest());
// let the components see the responses to the request
- for(SearchComponent c : components) {
+ for (SearchComponent c : components) {
c.handleResponses(rb, srsp.getShardRequest());
}
}
}
- for(SearchComponent c : components) {
+ for (SearchComponent c : components) {
c.finishStage(rb);
}
// we are done when the next stage is MAX_VALUE
} while (nextStage != Integer.MAX_VALUE);
}
-
+
// SOLR-5550: still provide shards.info if requested even for a short circuited distrib request
- if(!rb.isDistrib && req.getParams().getBool(ShardParams.SHARDS_INFO, false) && rb.shortCircuitedURL != null) {
+ if (!rb.isDistrib
+ && req.getParams().getBool(ShardParams.SHARDS_INFO, false)
+ && rb.shortCircuitedURL != null) {
NamedList<Object> shardInfo = new SimpleOrderedMap<Object>();
- SimpleOrderedMap<Object> nl = new SimpleOrderedMap<Object>();
+ SimpleOrderedMap<Object> nl = new SimpleOrderedMap<Object>();
if (rsp.getException() != null) {
Throwable cause = rsp.getException();
if (cause instanceof SolrServerException) {
- cause = ((SolrServerException)cause).getRootCause();
+ cause = ((SolrServerException) cause).getRootCause();
} else {
if (cause.getCause() != null) {
cause = cause.getCause();
- }
+ }
}
- nl.add("error", cause.toString() );
+ nl.add("error", cause.toString());
StringWriter trace = new StringWriter();
cause.printStackTrace(new PrintWriter(trace));
- nl.add("trace", trace.toString() );
+ nl.add("trace", trace.toString());
} else if (rb.getResults() != null) {
nl.add("numFound", rb.getResults().docList.matches());
- nl.add("numFoundExact", rb.getResults().docList.hitCountRelation() == TotalHits.Relation.EQUAL_TO);
+ nl.add(
+ "numFoundExact",
+ rb.getResults().docList.hitCountRelation() == TotalHits.Relation.EQUAL_TO);
nl.add("maxScore", rb.getResults().docList.maxScore());
}
nl.add("shardAddress", rb.shortCircuitedURL);
nl.add("time", req.getRequestTimer().getTime()); // elapsed time of this request so far
-
- int pos = rb.shortCircuitedURL.indexOf("://");
- String shardInfoName = pos != -1 ? rb.shortCircuitedURL.substring(pos+3) : rb.shortCircuitedURL;
- shardInfo.add(shardInfoName, nl);
- rsp.getValues().add(ShardParams.SHARDS_INFO,shardInfo);
+
+ int pos = rb.shortCircuitedURL.indexOf("://");
+ String shardInfoName =
+ pos != -1 ? rb.shortCircuitedURL.substring(pos + 3) : rb.shortCircuitedURL;
+ shardInfo.add(shardInfoName, nl);
+ rsp.getValues().add(ShardParams.SHARDS_INFO, shardInfo);
}
}
private void tagRequestWithRequestId(ResponseBuilder rb) {
- final boolean ridTaggingDisabled = rb.req.getParams().getBool(CommonParams.DISABLE_REQUEST_ID, false);
- if (! ridTaggingDisabled) {
+ final boolean ridTaggingDisabled =
+ rb.req.getParams().getBool(CommonParams.DISABLE_REQUEST_ID, false);
+ if (!ridTaggingDisabled) {
String rid = getOrGenerateRequestId(rb.req);
// NOTE: SearchHandler explicitly never clears/removes this MDC value...
- // We want it to live for the entire request, beyond the scope of SearchHandler's processing, and trust
+ // We want it to live for the entire request, beyond the scope of SearchHandler's processing,
+ // and trust
// SolrDispatchFilter to clean it up at the end of the request.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/component/SearchHandler.java
##########
@@ -317,32 +324,38 @@ public void handleRequestBody(SolrQueryRequest req, SolrQueryResponse rsp) throw
if (trippedCircuitBreakers != null) {
String errorMessage = CircuitBreakerManager.toErrorMessage(trippedCircuitBreakers);
rsp.add(STATUS, FAILURE);
- rsp.setException(new SolrException(SolrException.ErrorCode.SERVICE_UNAVAILABLE, "Circuit Breakers tripped " + errorMessage));
+ rsp.setException(
+ new SolrException(
+ SolrException.ErrorCode.SERVICE_UNAVAILABLE,
+ "Circuit Breakers tripped " + errorMessage));
return;
}
}
- final ShardHandler shardHandler1 = getAndPrepShardHandler(req, rb); // creates a ShardHandler object only if it's needed
+ final ShardHandler shardHandler1 =
+ getAndPrepShardHandler(req, rb); // creates a ShardHandler object only if it's needed
if (timer == null) {
// non-debugging prepare phase
- for( SearchComponent c : components ) {
+ for (SearchComponent c : components) {
c.prepare(rb);
}
} else {
// debugging prepare phase
- RTimerTree subt = timer.sub( "prepare" );
- for( SearchComponent c : components ) {
- rb.setTimer(subt.sub( c.getName() ) );
+ RTimerTree subt = timer.sub("prepare");
+ for (SearchComponent c : components) {
+ rb.setTimer(subt.sub(c.getName()));
c.prepare(rb);
rb.getTimer().stop();
}
subt.stop();
}
{ // Once all of our components have been prepared, check if this request involves a SortSpec.
- // If it does, and if our request includes a cursorMark param, then parse & init the CursorMark state
- // (This must happen after the prepare() of all components, because any component may have modified the SortSpec)
+ // If it does, and if our request includes a cursorMark param, then parse & init the
+ // CursorMark state
+ // (This must happen after the prepare() of all components, because any component may have
+ // modified the SortSpec)
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/component/SuggestComponent.java
##########
@@ -473,24 +498,32 @@ private SuggesterResult toSuggesterResult(SimpleOrderedMap<SimpleOrderedMap<Name
}
return result;
}
-
+
/** Listener to build or reload the maintained {@link SolrSuggester} by this component */
private static class SuggesterListener implements SolrEventListener {
private final SolrCore core;
private final SolrSuggester suggester;
private final boolean buildOnCommit;
private final boolean buildOnOptimize;
private final boolean buildOnStartup;
-
- // On core reload, immediately after the core is created a new searcher is opened, causing the suggester
- // to trigger a "buildOnCommit". The only event that we want to trigger in that situation is "buildOnStartup"
- // so if buildOnCommit is true and this is a core being reloaded, we will skip the first time this listener
- // is called.
+
+ // On core reload, immediately after the core is created a new searcher is opened, causing the
+ // suggester
+ // to trigger a "buildOnCommit". The only event that we want to trigger in that situation is
+ // "buildOnStartup"
+ // so if buildOnCommit is true and this is a core being reloaded, we will skip the first time
+ // this listener
+ // is called.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/component/SpellCheckComponent.java
##########
@@ -211,25 +217,30 @@ public void process(ResponseBuilder rb) throws IOException {
rb.rsp.add("spellcheck", response);
} else {
- throw new SolrException(SolrException.ErrorCode.NOT_FOUND,
- "Specified dictionaries do not exist: " + getDictionaryNameAsSingleString(getDictionaryNames(params)));
+ throw new SolrException(
+ SolrException.ErrorCode.NOT_FOUND,
+ "Specified dictionaries do not exist: "
+ + getDictionaryNameAsSingleString(getDictionaryNames(params)));
}
}
}
private Integer maxResultsForSuggest(ResponseBuilder rb) {
SolrParams params = rb.req.getParams();
- float maxResultsForSuggestParamValue = params.getFloat(SpellingParams.SPELLCHECK_MAX_RESULTS_FOR_SUGGEST, 0.0f);
+ float maxResultsForSuggestParamValue =
+ params.getFloat(SpellingParams.SPELLCHECK_MAX_RESULTS_FOR_SUGGEST, 0.0f);
Integer maxResultsForSuggest = null;
if (maxResultsForSuggestParamValue > 0.0f) {
if (maxResultsForSuggestParamValue == (int) maxResultsForSuggestParamValue) {
// If a whole number was passed in, this is a discrete number of documents
maxResultsForSuggest = (int) maxResultsForSuggestParamValue;
} else {
- // If a fractional value was passed in, this is the % of documents returned by the specified filter
+ // If a fractional value was passed in, this is the % of documents returned by the specified
+ // filter
// If no specified filter, we use the most restrictive filter of the fq parameters
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/designer/DefaultSchemaSuggester.java
##########
@@ -309,8 +365,10 @@ protected String isIntOrLong(List<Object> values, Locale locale) {
}
}
- // if all values are less than some smallish threshold, then it's likely this field holds small numbers
- // but be very conservative here as it's simply an optimization and we can always fall back to long
+ // if all values are less than some smallish threshold, then it's likely this field holds small
+ // numbers
+ // but be very conservative here as it's simply an optimization and we can always fall back to
+ // long
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/admin/CollectionsHandler.java
##########
@@ -254,8 +265,10 @@ void invokeAction(SolrQueryRequest req, SolrQueryResponse rsp, CoreContainer cor
rsp.setException(exp);
}
- // Even if Overseer does wait for the collection to be created, it sees a different cluster state than this node,
- // so this wait is required to make sure the local node Zookeeper watches fired and now see the collection.
+ // Even if Overseer does wait for the collection to be created, it sees a different cluster
+ // state than this node,
+ // so this wait is required to make sure the local node Zookeeper watches fired and now see the
+ // collection.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/export/ExportWriter.java
##########
@@ -202,37 +208,52 @@ private void _write(OutputStream os) throws IOException {
SortSpec sortSpec = info.getResponseBuilder().getSortSpec();
if (sortSpec == null) {
- writeException((new IOException(new SyntaxError("No sort criteria was provided."))), writer, true);
+ writeException(
+ (new IOException(new SyntaxError("No sort criteria was provided."))), writer, true);
return;
}
SolrIndexSearcher searcher = req.getSearcher();
Sort sort = searcher.weightSort(sortSpec.getSort());
if (sort == null) {
- writeException((new IOException(new SyntaxError("No sort criteria was provided."))), writer, true);
+ writeException(
+ (new IOException(new SyntaxError("No sort criteria was provided."))), writer, true);
return;
}
if (sort != null && sort.needsScores()) {
- writeException((new IOException(new SyntaxError("Scoring is not currently supported with xsort."))), writer, true);
+ writeException(
+ (new IOException(new SyntaxError("Scoring is not currently supported with xsort."))),
+ writer,
+ true);
return;
}
- // There is a bailout in SolrIndexSearcher.getDocListNC when there are _no_ docs in the index at all.
+ // There is a bailout in SolrIndexSearcher.getDocListNC when there are _no_ docs in the index at
+ // all.
// if (lastDocRequested <= 0) {
// That causes the totalHits and export entries in the context to _not_ get set.
- // The only time that really matters is when we search against an _empty_ set. That's too obscure
- // a condition to handle as part of this patch, if someone wants to pursue it it can be reproduced with:
- // ant test -Dtestcase=StreamingTest -Dtests.method=testAllValidExportTypes -Dtests.seed=10F13879D0D1D6AD -Dtests.slow=true -Dtests.locale=es-PA -Dtests.timezone=America/Bahia_Banderas -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
+ // The only time that really matters is when we search against an _empty_ set. That's too
+ // obscure
+ // a condition to handle as part of this patch, if someone wants to pursue it it can be
+ // reproduced with:
+ // ant test -Dtestcase=StreamingTest -Dtests.method=testAllValidExportTypes
+ // -Dtests.seed=10F13879D0D1D6AD -Dtests.slow=true -Dtests.locale=es-PA
+ // -Dtests.timezone=America/Bahia_Banderas -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/export/ExportWriter.java
##########
@@ -597,12 +630,15 @@ SortDoc getSortDoc(SolrIndexSearcher searcher, SortField[] sortFields) throws IO
sortValues[i] = new StringValue(vals, field, new IntComp.IntAsc());
}
} else {
- throw new IOException("Sort fields must be one of the following types: int,float,long,double,string,date,boolean,SortableText");
+ throw new IOException(
+ "Sort fields must be one of the following types: int,float,long,double,string,date,boolean,SortableText");
}
}
- //SingleValueSortDoc etc are specialized classes which don't have array lookups. On benchmarking large datasets
- //This is faster than the using an array in SortDoc . So upto 4 sort fields we still want to keep specialized classes.
- //SOLR-12616 has more details
+ // SingleValueSortDoc etc are specialized classes which don't have array lookups. On
+ // benchmarking large datasets
+ // This is faster than the using an array in SortDoc . So upto 4 sort fields we still want to
+ // keep specialized classes.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/designer/SchemaDesignerAPI.java
##########
@@ -831,7 +955,8 @@ protected SchemaDesignerSettings getMutableSchemaForConfigSet(final String confi
}
if (publishedVersion != -1) {
- // keep track of the version of the configSet the mutable is derived from in case another user
+ // keep track of the version of the configSet the mutable is derived from in case another
+ // user
// changes the derived from schema before we publish the mutable on top of it
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/designer/DefaultSchemaSuggester.java
##########
@@ -139,57 +172,74 @@ protected void validateNumericTypeChange(SchemaField field, FieldType toType, Li
throw new IllegalStateException("FieldType '" + fieldTypeName + "' not found in the schema!");
}
- Map<String, String> fieldProps = guessFieldProps(fieldName, fieldType, sampleValues, isMV, schema);
+ Map<String, String> fieldProps =
+ guessFieldProps(fieldName, fieldType, sampleValues, isMV, schema);
SchemaField schemaField = schema.newField(fieldName, fieldTypeName, fieldProps);
return Optional.of(schemaField);
}
@Override
- public ManagedIndexSchema adaptExistingFieldToData(SchemaField schemaField, List<Object> sampleValues, ManagedIndexSchema schema) {
+ public ManagedIndexSchema adaptExistingFieldToData(
+ SchemaField schemaField, List<Object> sampleValues, ManagedIndexSchema schema) {
// Promote a single-valued to multi-valued if needed
if (!schemaField.multiValued() && isMultiValued(sampleValues)) {
// this existing field needs to be promoted to multi-valued
SimpleOrderedMap<Object> fieldProps = schemaField.getNamedPropertyValues(false);
fieldProps.add("multiValued", true);
fieldProps.remove("name");
fieldProps.remove("type");
- schema = schema.replaceField(schemaField.getName(), schemaField.getType(), fieldProps.asShallowMap());
+ schema =
+ schema.replaceField(
+ schemaField.getName(), schemaField.getType(), fieldProps.asShallowMap());
}
- // TODO: other "healing" type operations here ... but we have to be careful about overriding explicit user changes
- // such as a user making a text field a string field, we wouldn't want to revert that field back to text
+ // TODO: other "healing" type operations here ... but we have to be careful about overriding
+ // explicit user changes
+ // such as a user making a text field a string field, we wouldn't want to revert that field back
+ // to text
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/admin/CollectionsHandler.java
##########
@@ -267,21 +280,33 @@ void invokeAction(SolrQueryRequest req, SolrQueryResponse rsp, CoreContainer cor
public static long DEFAULT_COLLECTION_OP_TIMEOUT = 180 * 1000;
- public SolrResponse submitCollectionApiCommand(ZkNodeProps m, CollectionAction action) throws KeeperException, InterruptedException {
+ public SolrResponse submitCollectionApiCommand(ZkNodeProps m, CollectionAction action)
+ throws KeeperException, InterruptedException {
return submitCollectionApiCommand(m, action, DEFAULT_COLLECTION_OP_TIMEOUT);
}
- public SolrResponse submitCollectionApiCommand(ZkNodeProps m, CollectionAction action, long timeout) throws KeeperException, InterruptedException {
- // Collection API messages are either sent to Overseer and processed there, or processed locally.
- // Distributing Collection API implies we're also distributing Cluster State Updates. Indeed collection creation
- // with non distributed cluster state updates requires for "Per Replica States" that the Collection API be running
- // on Overseer, which means that it is not possible to distributed Collection API while keeping cluster state updates
- // on Overseer. See the call to CollectionCommandContext.submitIntraProcessMessage() in CreateCollectionCmd.call() which
- // can only be done if the Collection API command runs on the same JVM as the Overseer based cluster state update...
- // The configuration handling includes these checks to not allow distributing collection API without distributing
+ public SolrResponse submitCollectionApiCommand(
+ ZkNodeProps m, CollectionAction action, long timeout)
+ throws KeeperException, InterruptedException {
+ // Collection API messages are either sent to Overseer and processed there, or processed
+ // locally.
+ // Distributing Collection API implies we're also distributing Cluster State Updates. Indeed
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/core/SolrCore.java
##########
@@ -442,7 +450,9 @@ private String getIndexPropertyFromPropFile(Directory dir) throws IOException {
// All other exceptions are will propagate to caller.
return dataDir + "index/";
}
- final InputStream is = new PropertiesInputStream(input); // c'tor just assigns a variable here, no exception thrown.
+ final InputStream is =
+ new PropertiesInputStream(
+ input); // c'tor just assigns a variable here, no exception thrown.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/core/SolrCore.java
##########
@@ -846,44 +892,54 @@ void initIndex(boolean passOnPreviousState, boolean reload) throws IOException {
log.debug("Solr index directory '{}' doesn't exist. Creating new index...", indexDir);
SolrIndexWriter writer = null;
try {
- writer = SolrIndexWriter.create(this, "SolrCore.initIndex", indexDir, getDirectoryFactory(), true,
- getLatestSchema(), solrConfig.indexConfig, solrDelPolicy, codec);
+ writer =
+ SolrIndexWriter.create(
+ this,
+ "SolrCore.initIndex",
+ indexDir,
+ getDirectoryFactory(),
+ true,
+ getLatestSchema(),
+ solrConfig.indexConfig,
+ solrDelPolicy,
+ codec);
} finally {
IOUtils.closeQuietly(writer);
}
-
}
cleanupOldIndexDirectories(reload);
}
-
/**
- * Creates an instance by trying a constructor that accepts a SolrCore before
- * trying the default (no arg) constructor.
+ * Creates an instance by trying a constructor that accepts a SolrCore before trying the default
+ * (no arg) constructor.
*
* @param className the instance class to create
- * @param cast the class or interface that the instance should extend or implement
- * @param msg a message helping compose the exception error if any occurs.
- * @param core The SolrCore instance for which this object needs to be loaded
+ * @param cast the class or interface that the instance should extend or implement
+ * @param msg a message helping compose the exception error if any occurs.
+ * @param core The SolrCore instance for which this object needs to be loaded
* @return the desired instance
* @throws SolrException if the object could not be instantiated
*/
- public static <T> T createInstance(String className, Class<T> cast, String msg, SolrCore core, ResourceLoader resourceLoader) {
+ public static <T> T createInstance(
+ String className, Class<T> cast, String msg, SolrCore core, ResourceLoader resourceLoader) {
Class<? extends T> clazz = null;
if (msg == null) msg = "SolrCore Object";
try {
clazz = resourceLoader.findClass(className, cast);
- //most of the classes do not have constructors which takes SolrCore argument. It is recommended to obtain SolrCore by implementing SolrCoreAware.
- // So invariably always it will cause a NoSuchMethodException. So iterate though the list of available constructors
+ // most of the classes do not have constructors which takes SolrCore argument. It is
+ // recommended to obtain SolrCore by implementing SolrCoreAware.
+ // So invariably always it will cause a NoSuchMethodException. So iterate though the list of
+ // available constructors
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/core/SolrCore.java
##########
@@ -1957,56 +2110,84 @@ boolean areAllSearcherReferencesEmpty() {
}
/**
- * Computes fingerprint of a segment and caches it only if all the version in segment are included in the fingerprint.
- * We can't use computeIfAbsent as caching is conditional (as described above)
- * There is chance that two threads may compute fingerprint on the same segment. It might be OK to do so rather than locking entire map.
+ * Computes fingerprint of a segment and caches it only if all the version in segment are included
+ * in the fingerprint. We can't use computeIfAbsent as caching is conditional (as described above)
+ * There is chance that two threads may compute fingerprint on the same segment. It might be OK to
+ * do so rather than locking entire map.
*
- * @param searcher searcher that includes specified LeaderReaderContext
- * @param ctx LeafReaderContext of a segment to compute fingerprint of
+ * @param searcher searcher that includes specified LeaderReaderContext
+ * @param ctx LeafReaderContext of a segment to compute fingerprint of
* @param maxVersion maximum version number to consider for fingerprint computation
* @return IndexFingerprint of the segment
* @throws IOException Can throw IOException
*/
- public IndexFingerprint getIndexFingerprint(SolrIndexSearcher searcher, LeafReaderContext ctx, long maxVersion)
- throws IOException {
+ public IndexFingerprint getIndexFingerprint(
+ SolrIndexSearcher searcher, LeafReaderContext ctx, long maxVersion) throws IOException {
IndexReader.CacheHelper cacheHelper = ctx.reader().getReaderCacheHelper();
if (cacheHelper == null) {
if (log.isDebugEnabled()) {
- log.debug("Cannot cache IndexFingerprint as reader does not support caching. searcher:{} reader:{} readerHash:{} maxVersion:{}", searcher, ctx.reader(), ctx.reader().hashCode(), maxVersion);
+ log.debug(
+ "Cannot cache IndexFingerprint as reader does not support caching. searcher:{} reader:{} readerHash:{} maxVersion:{}",
+ searcher,
+ ctx.reader(),
+ ctx.reader().hashCode(),
+ maxVersion);
}
return IndexFingerprint.getFingerprint(searcher, ctx, maxVersion);
}
IndexFingerprint f = null;
f = perSegmentFingerprintCache.get(cacheHelper.getKey());
// fingerprint is either not cached or
- // if we want fingerprint only up to a version less than maxVersionEncountered in the segment, or
+ // if we want fingerprint only up to a version less than maxVersionEncountered in the segment,
+ // or
// documents were deleted from segment for which fingerprint was cached
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/admin/CollectionsHandler.java
##########
@@ -1494,78 +1902,87 @@ public static void waitForActiveCollection(String collectionName, CoreContainer
Integer seconds = ccfg.getCreateCollectionWaitTimeTillActive();
Boolean checkLeaderOnly = ccfg.isCreateCollectionCheckLeaderActive();
if (log.isInfoEnabled()) {
- log.info("Wait for new collection to be active for at most {} seconds. Check all shard {}"
- , seconds, (checkLeaderOnly ? "leaders" : "replicas"));
+ log.info(
+ "Wait for new collection to be active for at most {} seconds. Check all shard {}",
+ seconds,
+ (checkLeaderOnly ? "leaders" : "replicas"));
}
try {
- cc.getZkController().getZkStateReader().waitForState(collectionName, seconds, TimeUnit.SECONDS, (n, c) -> {
-
- if (c == null) {
- // the collection was not created, don't wait
- return true;
- }
-
- if (c.getSlices() != null) {
- Collection<Slice> shards = c.getSlices();
- int replicaNotAliveCnt = 0;
- for (Slice shard : shards) {
- Collection<Replica> replicas;
- if (!checkLeaderOnly) replicas = shard.getReplicas();
- else {
- replicas = new ArrayList<Replica>();
- replicas.add(shard.getLeader());
- }
- for (Replica replica : replicas) {
- String state = replica.getStr(ZkStateReader.STATE_PROP);
- if (log.isDebugEnabled()) {
- log.debug("Checking replica status, collection={} replica={} state={}", collectionName,
- replica.getCoreUrl(), state);
- }
- if (!n.contains(replica.getNodeName())
- || !state.equals(Replica.State.ACTIVE.toString())) {
- replicaNotAliveCnt++;
+ cc.getZkController()
+ .getZkStateReader()
+ .waitForState(
+ collectionName,
+ seconds,
+ TimeUnit.SECONDS,
+ (n, c) -> {
+ if (c == null) {
+ // the collection was not created, don't wait
+ return true;
+ }
+
+ if (c.getSlices() != null) {
+ Collection<Slice> shards = c.getSlices();
+ int replicaNotAliveCnt = 0;
+ for (Slice shard : shards) {
+ Collection<Replica> replicas;
+ if (!checkLeaderOnly) replicas = shard.getReplicas();
+ else {
+ replicas = new ArrayList<Replica>();
+ replicas.add(shard.getLeader());
+ }
+ for (Replica replica : replicas) {
+ String state = replica.getStr(ZkStateReader.STATE_PROP);
+ if (log.isDebugEnabled()) {
+ log.debug(
+ "Checking replica status, collection={} replica={} state={}",
+ collectionName,
+ replica.getCoreUrl(),
+ state);
+ }
+ if (!n.contains(replica.getNodeName())
+ || !state.equals(Replica.State.ACTIVE.toString())) {
+ replicaNotAliveCnt++;
+ return false;
+ }
+ }
+ }
+
+ return (replicaNotAliveCnt == 0) || (replicaNotAliveCnt <= replicaFailCount);
+ }
return false;
- }
- }
- }
-
- return (replicaNotAliveCnt == 0) || (replicaNotAliveCnt <= replicaFailCount);
- }
- return false;
- });
+ });
} catch (TimeoutException | InterruptedException e) {
- String error = "Timeout waiting for active collection " + collectionName + " with timeout=" + seconds;
+ String error =
+ "Timeout waiting for active collection " + collectionName + " with timeout=" + seconds;
throw new NotInClusterStateException(ErrorCode.SERVER_ERROR, error);
}
-
}
-
private static void verifyShardsParam(String shardsParam) {
for (String shard : shardsParam.split(",")) {
SolrIdentifierValidator.validateShardName(shard);
}
}
interface CollectionOp {
- Map<String, Object> execute(SolrQueryRequest req, SolrQueryResponse rsp, CollectionsHandler h) throws Exception;
-
+ Map<String, Object> execute(SolrQueryRequest req, SolrQueryResponse rsp, CollectionsHandler h)
+ throws Exception;
}
@Override
public Boolean registerV2() {
return Boolean.TRUE;
}
- // These "copy" methods were once SolrParams.getAll but were moved here as there is no universal way that
+ // These "copy" methods were once SolrParams.getAll but were moved here as there is no universal
+ // way that
// a SolrParams can be represented in a Map; there are various choices.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/core/SolrCore.java
##########
@@ -2491,45 +2718,47 @@ public IndexFingerprint getIndexFingerprint(SolrIndexSearcher searcher, LeafRead
}
if (searchHolder != null) {
- searchHolder.decref(); // decrement 1 for _searcher (searchHolder will never become _searcher now)
+ searchHolder
+ .decref(); // decrement 1 for _searcher (searchHolder will never become _searcher now)
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/core/SolrCore.java
##########
@@ -2330,26 +2554,25 @@ public IndexFingerprint getIndexFingerprint(SolrIndexSearcher searcher, LeafRead
}
// a signal to decrement onDeckSearchers if something goes wrong.
- final boolean[] decrementOnDeckCount = new boolean[]{true};
- RefCounted<SolrIndexSearcher> currSearcherHolder = null; // searcher we are autowarming from
+ final boolean[] decrementOnDeckCount = new boolean[] {true};
+ RefCounted<SolrIndexSearcher> currSearcherHolder = null; // searcher we are autowarming from
RefCounted<SolrIndexSearcher> searchHolder = null;
boolean success = false;
openSearcherLock.lock();
Timer.Context timerContext = newSearcherTimer.time();
try {
searchHolder = openNewSearcher(updateHandlerReopens, false);
- // the searchHolder will be incremented once already (and it will eventually be assigned to _searcher when registered)
+ // the searchHolder will be incremented once already (and it will eventually be assigned to
+ // _searcher when registered)
// increment it again if we are going to return it to the caller.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/core/SolrCore.java
##########
@@ -2182,21 +2386,32 @@ public IndexFingerprint getIndexFingerprint(SolrIndexSearcher searcher, LeafRead
} finally {
writer.decref();
}
- tmp = new SolrIndexSearcher(this, newIndexDir, getLatestSchema(),
- (realtime ? "realtime" : "main"), newReader, true, !realtime, true, directoryFactory);
+ tmp =
+ new SolrIndexSearcher(
+ this,
+ newIndexDir,
+ getLatestSchema(),
+ (realtime ? "realtime" : "main"),
+ newReader,
+ true,
+ !realtime,
+ true,
+ directoryFactory);
}
}
List<RefCounted<SolrIndexSearcher>> searcherList = realtime ? _realtimeSearchers : _searchers;
- RefCounted<SolrIndexSearcher> newSearcher = newHolder(tmp, searcherList); // refcount now at 1
+ RefCounted<SolrIndexSearcher> newSearcher = newHolder(tmp, searcherList); // refcount now at 1
// Increment reference again for "realtimeSearcher" variable. It should be at 2 after.
- // When it's decremented by both the caller of this method, and by realtimeSearcher being replaced,
+ // When it's decremented by both the caller of this method, and by realtimeSearcher being
+ // replaced,
// it will be closed.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/admin/CollectionsHandler.java
##########
@@ -388,982 +426,1324 @@ public Category getCategory() {
return Category.ADMIN;
}
- private static void createSysConfigSet(CoreContainer coreContainer) throws KeeperException, InterruptedException {
+ private static void createSysConfigSet(CoreContainer coreContainer)
+ throws KeeperException, InterruptedException {
SolrZkClient zk = coreContainer.getZkController().getZkStateReader().getZkClient();
ZkCmdExecutor cmdExecutor = new ZkCmdExecutor(zk.getZkClientTimeout());
cmdExecutor.ensureExists(ZkStateReader.CONFIGS_ZKNODE, zk);
- cmdExecutor.ensureExists(ZkStateReader.CONFIGS_ZKNODE + "/" + CollectionAdminParams.SYSTEM_COLL, zk);
+ cmdExecutor.ensureExists(
+ ZkStateReader.CONFIGS_ZKNODE + "/" + CollectionAdminParams.SYSTEM_COLL, zk);
try {
- String path = ZkStateReader.CONFIGS_ZKNODE + "/" + CollectionAdminParams.SYSTEM_COLL + "/schema.xml";
- byte[] data = IOUtils.toByteArray(CollectionsHandler.class.getResourceAsStream("/SystemCollectionSchema.xml"));
+ String path =
+ ZkStateReader.CONFIGS_ZKNODE + "/" + CollectionAdminParams.SYSTEM_COLL + "/schema.xml";
+ byte[] data =
+ IOUtils.toByteArray(
+ CollectionsHandler.class.getResourceAsStream("/SystemCollectionSchema.xml"));
assert data != null && data.length > 0;
cmdExecutor.ensureExists(path, data, CreateMode.PERSISTENT, zk);
- path = ZkStateReader.CONFIGS_ZKNODE + "/" + CollectionAdminParams.SYSTEM_COLL + "/solrconfig.xml";
- data = IOUtils.toByteArray(CollectionsHandler.class.getResourceAsStream("/SystemCollectionSolrConfig.xml"));
+ path =
+ ZkStateReader.CONFIGS_ZKNODE
+ + "/"
+ + CollectionAdminParams.SYSTEM_COLL
+ + "/solrconfig.xml";
+ data =
+ IOUtils.toByteArray(
+ CollectionsHandler.class.getResourceAsStream("/SystemCollectionSolrConfig.xml"));
assert data != null && data.length > 0;
cmdExecutor.ensureExists(path, data, CreateMode.PERSISTENT, zk);
} catch (IOException e) {
throw new SolrException(ErrorCode.SERVER_ERROR, e);
}
-
-
}
- private static void addStatusToResponse(NamedList<Object> results, RequestStatusState state, String msg) {
+ private static void addStatusToResponse(
+ NamedList<Object> results, RequestStatusState state, String msg) {
SimpleOrderedMap<String> status = new SimpleOrderedMap<>();
status.add("state", state.getKey());
status.add("msg", msg);
results.add("status", status);
}
public enum CollectionOperation implements CollectionOp {
- CREATE_OP(CREATE, (req, rsp, h) -> {
- Map<String, Object> props = copy(req.getParams().required(), null, NAME);
- props.put("fromApi", "true");
- copy(req.getParams(), props,
- REPLICATION_FACTOR,
- COLL_CONF,
- NUM_SLICES,
- CREATE_NODE_SET,
- CREATE_NODE_SET_SHUFFLE,
- SHARDS_PROP,
- PULL_REPLICAS,
- TLOG_REPLICAS,
- NRT_REPLICAS,
- WAIT_FOR_FINAL_STATE,
- PER_REPLICA_STATE,
- ALIAS);
-
- if (props.get(REPLICATION_FACTOR) != null && props.get(NRT_REPLICAS) != null) {
- //TODO: Remove this in 8.0 . Keep this for SolrJ client back-compat. See SOLR-11676 for more details
- int replicationFactor = Integer.parseInt((String) props.get(REPLICATION_FACTOR));
- int nrtReplicas = Integer.parseInt((String) props.get(NRT_REPLICAS));
- if (replicationFactor != nrtReplicas) {
- throw new SolrException(ErrorCode.BAD_REQUEST,
- "Cannot specify both replicationFactor and nrtReplicas as they mean the same thing");
- }
- }
- if (props.get(REPLICATION_FACTOR) != null) {
- props.put(NRT_REPLICAS, props.get(REPLICATION_FACTOR));
- } else if (props.get(NRT_REPLICAS) != null) {
- props.put(REPLICATION_FACTOR, props.get(NRT_REPLICAS));
- }
-
- final String collectionName = SolrIdentifierValidator.validateCollectionName((String) props.get(NAME));
- final String shardsParam = (String) props.get(SHARDS_PROP);
- if (StringUtils.isNotEmpty(shardsParam)) {
- verifyShardsParam(shardsParam);
- }
- if (CollectionAdminParams.SYSTEM_COLL.equals(collectionName)) {
- //We must always create a .system collection with only a single shard
- props.put(NUM_SLICES, 1);
- props.remove(SHARDS_PROP);
- createSysConfigSet(h.coreContainer);
-
- }
- if (shardsParam == null) h.copyFromClusterProp(props, NUM_SLICES);
- for (String prop : ImmutableSet.of(NRT_REPLICAS, PULL_REPLICAS, TLOG_REPLICAS))
- h.copyFromClusterProp(props, prop);
- copyPropertiesWithPrefix(req.getParams(), props, PROPERTY_PREFIX);
- return copyPropertiesWithPrefix(req.getParams(), props, "router.");
+ CREATE_OP(
+ CREATE,
+ (req, rsp, h) -> {
+ Map<String, Object> props = copy(req.getParams().required(), null, NAME);
+ props.put("fromApi", "true");
+ copy(
+ req.getParams(),
+ props,
+ REPLICATION_FACTOR,
+ COLL_CONF,
+ NUM_SLICES,
+ CREATE_NODE_SET,
+ CREATE_NODE_SET_SHUFFLE,
+ SHARDS_PROP,
+ PULL_REPLICAS,
+ TLOG_REPLICAS,
+ NRT_REPLICAS,
+ WAIT_FOR_FINAL_STATE,
+ PER_REPLICA_STATE,
+ ALIAS);
+
+ if (props.get(REPLICATION_FACTOR) != null && props.get(NRT_REPLICAS) != null) {
+ // TODO: Remove this in 8.0 . Keep this for SolrJ client back-compat. See SOLR-11676 for
+ // more details
+ int replicationFactor = Integer.parseInt((String) props.get(REPLICATION_FACTOR));
+ int nrtReplicas = Integer.parseInt((String) props.get(NRT_REPLICAS));
+ if (replicationFactor != nrtReplicas) {
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST,
+ "Cannot specify both replicationFactor and nrtReplicas as they mean the same thing");
+ }
+ }
+ if (props.get(REPLICATION_FACTOR) != null) {
+ props.put(NRT_REPLICAS, props.get(REPLICATION_FACTOR));
+ } else if (props.get(NRT_REPLICAS) != null) {
+ props.put(REPLICATION_FACTOR, props.get(NRT_REPLICAS));
+ }
- }),
+ final String collectionName =
+ SolrIdentifierValidator.validateCollectionName((String) props.get(NAME));
+ final String shardsParam = (String) props.get(SHARDS_PROP);
+ if (StringUtils.isNotEmpty(shardsParam)) {
+ verifyShardsParam(shardsParam);
+ }
+ if (CollectionAdminParams.SYSTEM_COLL.equals(collectionName)) {
+ // We must always create a .system collection with only a single shard
+ props.put(NUM_SLICES, 1);
+ props.remove(SHARDS_PROP);
+ createSysConfigSet(h.coreContainer);
+ }
+ if (shardsParam == null) h.copyFromClusterProp(props, NUM_SLICES);
+ for (String prop : ImmutableSet.of(NRT_REPLICAS, PULL_REPLICAS, TLOG_REPLICAS))
+ h.copyFromClusterProp(props, prop);
+ copyPropertiesWithPrefix(req.getParams(), props, PROPERTY_PREFIX);
+ return copyPropertiesWithPrefix(req.getParams(), props, "router.");
+ }),
@SuppressWarnings({"unchecked"})
- COLSTATUS_OP(COLSTATUS, (req, rsp, h) -> {
- Map<String, Object> props = copy(req.getParams(), null,
- COLLECTION_PROP,
- ColStatus.CORE_INFO_PROP,
- ColStatus.SEGMENTS_PROP,
- ColStatus.FIELD_INFO_PROP,
- ColStatus.RAW_SIZE_PROP,
- ColStatus.RAW_SIZE_SUMMARY_PROP,
- ColStatus.RAW_SIZE_DETAILS_PROP,
- ColStatus.RAW_SIZE_SAMPLING_PERCENT_PROP,
- ColStatus.SIZE_INFO_PROP);
-
- new ColStatus(h.coreContainer.getSolrClientCache(),
- h.coreContainer.getZkController().getZkStateReader().getClusterState(), new ZkNodeProps(props))
- .getColStatus(rsp.getValues());
- return null;
- }),
- DELETE_OP(DELETE, (req, rsp, h) -> {
- Map<String, Object> map = copy(req.getParams().required(), null, NAME);
- return copy(req.getParams(), map, FOLLOW_ALIASES);
- }),
+ COLSTATUS_OP(
+ COLSTATUS,
+ (req, rsp, h) -> {
+ Map<String, Object> props =
+ copy(
+ req.getParams(),
+ null,
+ COLLECTION_PROP,
+ ColStatus.CORE_INFO_PROP,
+ ColStatus.SEGMENTS_PROP,
+ ColStatus.FIELD_INFO_PROP,
+ ColStatus.RAW_SIZE_PROP,
+ ColStatus.RAW_SIZE_SUMMARY_PROP,
+ ColStatus.RAW_SIZE_DETAILS_PROP,
+ ColStatus.RAW_SIZE_SAMPLING_PERCENT_PROP,
+ ColStatus.SIZE_INFO_PROP);
+
+ new ColStatus(
+ h.coreContainer.getSolrClientCache(),
+ h.coreContainer.getZkController().getZkStateReader().getClusterState(),
+ new ZkNodeProps(props))
+ .getColStatus(rsp.getValues());
+ return null;
+ }),
+ DELETE_OP(
+ DELETE,
+ (req, rsp, h) -> {
+ Map<String, Object> map = copy(req.getParams().required(), null, NAME);
+ return copy(req.getParams(), map, FOLLOW_ALIASES);
+ }),
// XXX should this command support followAliases?
- RELOAD_OP(RELOAD, (req, rsp, h) -> {
- Map<String, Object> map = copy(req.getParams().required(), null, NAME);
- return copy(req.getParams(), map);
- }),
-
- RENAME_OP(RENAME, (req, rsp, h) -> {
- Map<String, Object> map = copy(req.getParams().required(), null, NAME, CollectionAdminParams.TARGET);
- return copy(req.getParams(), map, FOLLOW_ALIASES);
- }),
-
- REINDEXCOLLECTION_OP(REINDEXCOLLECTION, (req, rsp, h) -> {
- Map<String, Object> m = copy(req.getParams().required(), null, NAME);
- copy(req.getParams(), m,
- ReindexCollectionCmd.COMMAND,
- ReindexCollectionCmd.REMOVE_SOURCE,
- ReindexCollectionCmd.TARGET,
- ZkStateReader.CONFIGNAME_PROP,
- NUM_SLICES,
- NRT_REPLICAS,
- PULL_REPLICAS,
- TLOG_REPLICAS,
- REPLICATION_FACTOR,
- CREATE_NODE_SET,
- CREATE_NODE_SET_SHUFFLE,
- "shards",
- CommonParams.ROWS,
- CommonParams.Q,
- CommonParams.FL,
- FOLLOW_ALIASES);
- if (req.getParams().get("collection." + ZkStateReader.CONFIGNAME_PROP) != null) {
- m.put(ZkStateReader.CONFIGNAME_PROP, req.getParams().get("collection." + ZkStateReader.CONFIGNAME_PROP));
- }
- copyPropertiesWithPrefix(req.getParams(), m, "router.");
- return m;
- }),
-
- SYNCSHARD_OP(SYNCSHARD, (req, rsp, h) -> {
- String extCollection = req.getParams().required().get("collection");
- String collection = h.coreContainer.getZkController().getZkStateReader().getAliases().resolveSimpleAlias(extCollection);
- String shard = req.getParams().required().get("shard");
-
- ClusterState clusterState = h.coreContainer.getZkController().getClusterState();
-
- DocCollection docCollection = clusterState.getCollection(collection);
- ZkNodeProps leaderProps = docCollection.getLeader(shard);
- ZkCoreNodeProps nodeProps = new ZkCoreNodeProps(leaderProps);
-
- try (HttpSolrClient client = new Builder(nodeProps.getBaseUrl())
- .withConnectionTimeout(15000)
- .withSocketTimeout(60000)
- .build()) {
- RequestSyncShard reqSyncShard = new RequestSyncShard();
- reqSyncShard.setCollection(collection);
- reqSyncShard.setShard(shard);
- reqSyncShard.setCoreName(nodeProps.getCoreName());
- client.request(reqSyncShard);
- }
- return null;
- }),
-
- CREATEALIAS_OP(CREATEALIAS, (req, rsp, h) -> {
- String alias = req.getParams().get(NAME);
- SolrIdentifierValidator.validateAliasName(alias);
- String collections = req.getParams().get("collections");
- RoutedAlias routedAlias = null;
- Exception ex = null;
- HashMap<String,Object> possiblyModifiedParams = new HashMap<>();
- try {
- // note that RA specific validation occurs here.
- req.getParams().toMap(possiblyModifiedParams);
- @SuppressWarnings({"unchecked", "rawtypes"})
- // This is awful because RoutedAlias lies about what types it wants
- Map<String, String> temp = (Map<String, String>) (Map) possiblyModifiedParams;
- routedAlias = RoutedAlias.fromProps(alias, temp);
- } catch (SolrException e) {
- // we'll throw this later if we are in fact creating a routed alias.
- ex = e;
- }
- ModifiableSolrParams finalParams = new ModifiableSolrParams();
- for (Map.Entry<String, Object> entry : possiblyModifiedParams.entrySet()) {
- if (entry.getValue().getClass().isArray() ) {
- // v2 api hits this case
- for (Object o : (Object[]) entry.getValue()) {
- finalParams.add(entry.getKey(),o.toString());
+ RELOAD_OP(
+ RELOAD,
+ (req, rsp, h) -> {
+ Map<String, Object> map = copy(req.getParams().required(), null, NAME);
+ return copy(req.getParams(), map);
+ }),
+
+ RENAME_OP(
+ RENAME,
+ (req, rsp, h) -> {
+ Map<String, Object> map =
+ copy(req.getParams().required(), null, NAME, CollectionAdminParams.TARGET);
+ return copy(req.getParams(), map, FOLLOW_ALIASES);
+ }),
+
+ REINDEXCOLLECTION_OP(
+ REINDEXCOLLECTION,
+ (req, rsp, h) -> {
+ Map<String, Object> m = copy(req.getParams().required(), null, NAME);
+ copy(
+ req.getParams(),
+ m,
+ ReindexCollectionCmd.COMMAND,
+ ReindexCollectionCmd.REMOVE_SOURCE,
+ ReindexCollectionCmd.TARGET,
+ ZkStateReader.CONFIGNAME_PROP,
+ NUM_SLICES,
+ NRT_REPLICAS,
+ PULL_REPLICAS,
+ TLOG_REPLICAS,
+ REPLICATION_FACTOR,
+ CREATE_NODE_SET,
+ CREATE_NODE_SET_SHUFFLE,
+ "shards",
+ CommonParams.ROWS,
+ CommonParams.Q,
+ CommonParams.FL,
+ FOLLOW_ALIASES);
+ if (req.getParams().get("collection." + ZkStateReader.CONFIGNAME_PROP) != null) {
+ m.put(
+ ZkStateReader.CONFIGNAME_PROP,
+ req.getParams().get("collection." + ZkStateReader.CONFIGNAME_PROP));
+ }
+ copyPropertiesWithPrefix(req.getParams(), m, "router.");
+ return m;
+ }),
+
+ SYNCSHARD_OP(
+ SYNCSHARD,
+ (req, rsp, h) -> {
+ String extCollection = req.getParams().required().get("collection");
+ String collection =
+ h.coreContainer
+ .getZkController()
+ .getZkStateReader()
+ .getAliases()
+ .resolveSimpleAlias(extCollection);
+ String shard = req.getParams().required().get("shard");
+
+ ClusterState clusterState = h.coreContainer.getZkController().getClusterState();
+
+ DocCollection docCollection = clusterState.getCollection(collection);
+ ZkNodeProps leaderProps = docCollection.getLeader(shard);
+ ZkCoreNodeProps nodeProps = new ZkCoreNodeProps(leaderProps);
+
+ try (HttpSolrClient client =
+ new Builder(nodeProps.getBaseUrl())
+ .withConnectionTimeout(15000)
+ .withSocketTimeout(60000)
+ .build()) {
+ RequestSyncShard reqSyncShard = new RequestSyncShard();
+ reqSyncShard.setCollection(collection);
+ reqSyncShard.setShard(shard);
+ reqSyncShard.setCoreName(nodeProps.getCoreName());
+ client.request(reqSyncShard);
+ }
+ return null;
+ }),
+
+ CREATEALIAS_OP(
+ CREATEALIAS,
+ (req, rsp, h) -> {
+ String alias = req.getParams().get(NAME);
+ SolrIdentifierValidator.validateAliasName(alias);
+ String collections = req.getParams().get("collections");
+ RoutedAlias routedAlias = null;
+ Exception ex = null;
+ HashMap<String, Object> possiblyModifiedParams = new HashMap<>();
+ try {
+ // note that RA specific validation occurs here.
+ req.getParams().toMap(possiblyModifiedParams);
+ @SuppressWarnings({"unchecked", "rawtypes"})
+ // This is awful because RoutedAlias lies about what types it wants
+ Map<String, String> temp = (Map<String, String>) (Map) possiblyModifiedParams;
+ routedAlias = RoutedAlias.fromProps(alias, temp);
+ } catch (SolrException e) {
+ // we'll throw this later if we are in fact creating a routed alias.
+ ex = e;
+ }
+ ModifiableSolrParams finalParams = new ModifiableSolrParams();
+ for (Map.Entry<String, Object> entry : possiblyModifiedParams.entrySet()) {
+ if (entry.getValue().getClass().isArray()) {
+ // v2 api hits this case
+ for (Object o : (Object[]) entry.getValue()) {
+ finalParams.add(entry.getKey(), o.toString());
+ }
+ } else {
+ finalParams.add(entry.getKey(), entry.getValue().toString());
+ }
}
- } else {
- finalParams.add(entry.getKey(),entry.getValue().toString());
- }
- }
- if (collections != null) {
- if (routedAlias != null) {
- throw new SolrException(BAD_REQUEST, "Collections cannot be specified when creating a routed alias.");
- } else {
- //////////////////////////////////////
- // Regular alias creation indicated //
- //////////////////////////////////////
- return copy(finalParams.required(), null, NAME, "collections");
- }
- }
+ if (collections != null) {
+ if (routedAlias != null) {
+ throw new SolrException(
+ BAD_REQUEST, "Collections cannot be specified when creating a routed alias.");
+ } else {
+ //////////////////////////////////////
+ // Regular alias creation indicated //
+ //////////////////////////////////////
+ return copy(finalParams.required(), null, NAME, "collections");
+ }
+ }
- /////////////////////////////////////////////////
- // We are creating a routed alias from here on //
- /////////////////////////////////////////////////
+ /////////////////////////////////////////////////
+ // We are creating a routed alias from here on //
+ /////////////////////////////////////////////////
- // If our prior creation attempt had issues expose them now.
- if (ex != null) {
- throw ex;
- }
+ // If our prior creation attempt had issues expose them now.
+ if (ex != null) {
+ throw ex;
+ }
- // Now filter out just the parameters we care about from the request
- assert routedAlias != null;
- Map<String, Object> result = copy(finalParams, null, routedAlias.getRequiredParams());
- copy(finalParams, result, routedAlias.getOptionalParams());
-
- ModifiableSolrParams createCollParams = new ModifiableSolrParams(); // without prefix
-
- // add to result params that start with "create-collection.".
- // Additionally, save these without the prefix to createCollParams
- for (Map.Entry<String, String[]> entry : finalParams) {
- final String p = entry.getKey();
- if (p.startsWith(CREATE_COLLECTION_PREFIX)) {
- // This is what SolrParams#getAll(Map, Collection)} does
- final String[] v = entry.getValue();
- if (v.length == 1) {
- result.put(p, v[0]);
- } else {
- result.put(p, v);
+ // Now filter out just the parameters we care about from the request
+ assert routedAlias != null;
+ Map<String, Object> result = copy(finalParams, null, routedAlias.getRequiredParams());
+ copy(finalParams, result, routedAlias.getOptionalParams());
+
+ ModifiableSolrParams createCollParams = new ModifiableSolrParams(); // without prefix
+
+ // add to result params that start with "create-collection.".
+ // Additionally, save these without the prefix to createCollParams
+ for (Map.Entry<String, String[]> entry : finalParams) {
+ final String p = entry.getKey();
+ if (p.startsWith(CREATE_COLLECTION_PREFIX)) {
+ // This is what SolrParams#getAll(Map, Collection)} does
+ final String[] v = entry.getValue();
+ if (v.length == 1) {
+ result.put(p, v[0]);
+ } else {
+ result.put(p, v);
+ }
+ createCollParams.set(p.substring(CREATE_COLLECTION_PREFIX.length()), v);
+ }
}
- createCollParams.set(p.substring(CREATE_COLLECTION_PREFIX.length()), v);
- }
- }
- // Verify that the create-collection prefix'ed params appear to be valid.
- if (createCollParams.get(NAME) != null) {
- throw new SolrException(BAD_REQUEST, "routed aliases calculate names for their " +
- "dependent collections, you cannot specify the name.");
- }
- if (createCollParams.get(COLL_CONF) == null) {
- throw new SolrException(SolrException.ErrorCode.BAD_REQUEST,
- "We require an explicit " + COLL_CONF);
- }
- // note: could insist on a config name here as well.... or wait to throw at overseer
- createCollParams.add(NAME, "TMP_name_TMP_name_TMP"); // just to pass validation
- CREATE_OP.execute(new LocalSolrQueryRequest(null, createCollParams), rsp, h); // ignore results
+ // Verify that the create-collection prefix'ed params appear to be valid.
+ if (createCollParams.get(NAME) != null) {
+ throw new SolrException(
+ BAD_REQUEST,
+ "routed aliases calculate names for their "
+ + "dependent collections, you cannot specify the name.");
+ }
+ if (createCollParams.get(COLL_CONF) == null) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST, "We require an explicit " + COLL_CONF);
+ }
+ // note: could insist on a config name here as well.... or wait to throw at overseer
+ createCollParams.add(NAME, "TMP_name_TMP_name_TMP"); // just to pass validation
+ CREATE_OP.execute(
+ new LocalSolrQueryRequest(null, createCollParams), rsp, h); // ignore results
- return result;
- }),
+ return result;
+ }),
DELETEALIAS_OP(DELETEALIAS, (req, rsp, h) -> copy(req.getParams().required(), null, NAME)),
/**
* Change properties for an alias (use CREATEALIAS_OP to change the actual value of the alias)
*/
- ALIASPROP_OP(ALIASPROP, (req, rsp, h) -> {
- Map<String, Object> params = copy(req.getParams().required(), null, NAME);
-
- // Note: success/no-op in the event of no properties supplied is intentional. Keeps code simple and one less case
- // for api-callers to check for.
- return convertPrefixToMap(req.getParams(), params, "property");
- }),
-
- /**
- * List the aliases and associated properties.
- */
+ ALIASPROP_OP(
+ ALIASPROP,
+ (req, rsp, h) -> {
+ Map<String, Object> params = copy(req.getParams().required(), null, NAME);
+
+ // Note: success/no-op in the event of no properties supplied is intentional. Keeps code
+ // simple and one less case
+ // for api-callers to check for.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/core/SolrCore.java
##########
@@ -958,15 +1048,21 @@ public CoreContainer getCoreContainer() {
}
SolrCore(CoreContainer coreContainer, CoreDescriptor cd, ConfigSet configSet) {
- this(coreContainer, cd, configSet, null,
- null, null, null, false);
+ this(coreContainer, cd, configSet, null, null, null, null, false);
}
- private SolrCore(CoreContainer coreContainer, CoreDescriptor coreDescriptor, ConfigSet configSet,
- String dataDir, UpdateHandler updateHandler,
- IndexDeletionPolicyWrapper delPolicy, SolrCore prev, boolean reload) {
+ private SolrCore(
+ CoreContainer coreContainer,
+ CoreDescriptor coreDescriptor,
+ ConfigSet configSet,
+ String dataDir,
+ UpdateHandler updateHandler,
+ IndexDeletionPolicyWrapper delPolicy,
+ SolrCore prev,
+ boolean reload) {
- assert ObjectReleaseTracker.track(searcherExecutor); // ensure that in unclean shutdown tests we still close this
+ assert ObjectReleaseTracker.track(
+ searcherExecutor); // ensure that in unclean shutdown tests we still close this
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/core/SolrCore.java
##########
@@ -2182,21 +2386,32 @@ public IndexFingerprint getIndexFingerprint(SolrIndexSearcher searcher, LeafRead
} finally {
writer.decref();
}
- tmp = new SolrIndexSearcher(this, newIndexDir, getLatestSchema(),
- (realtime ? "realtime" : "main"), newReader, true, !realtime, true, directoryFactory);
+ tmp =
+ new SolrIndexSearcher(
+ this,
+ newIndexDir,
+ getLatestSchema(),
+ (realtime ? "realtime" : "main"),
+ newReader,
+ true,
+ !realtime,
+ true,
+ directoryFactory);
}
}
List<RefCounted<SolrIndexSearcher>> searcherList = realtime ? _realtimeSearchers : _searchers;
- RefCounted<SolrIndexSearcher> newSearcher = newHolder(tmp, searcherList); // refcount now at 1
+ RefCounted<SolrIndexSearcher> newSearcher = newHolder(tmp, searcherList); // refcount now at 1
// Increment reference again for "realtimeSearcher" variable. It should be at 2 after.
- // When it's decremented by both the caller of this method, and by realtimeSearcher being replaced,
+ // When it's decremented by both the caller of this method, and by realtimeSearcher being
+ // replaced,
// it will be closed.
newSearcher.incref();
synchronized (searcherLock) {
- // Check if the core is closed again inside the lock in case this method is racing with a close. If the core is
+ // Check if the core is closed again inside the lock in case this method is racing with a
+ // close. If the core is
// closed, clean up the new searcher and bail.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/core/SolrCore.java
##########
@@ -894,25 +950,43 @@ void initIndex(boolean passOnPreviousState, boolean reload) throws IOException {
throw inner;
}
- throw new SolrException(ErrorCode.SERVER_ERROR, "Error Instantiating " + msg + ", " + className + " failed to instantiate " + cast.getName(), e);
+ throw new SolrException(
+ ErrorCode.SERVER_ERROR,
+ "Error Instantiating "
+ + msg
+ + ", "
+ + className
+ + " failed to instantiate "
+ + cast.getName(),
+ e);
}
}
- private UpdateHandler createReloadedUpdateHandler(String className, String msg, UpdateHandler updateHandler) {
+ private UpdateHandler createReloadedUpdateHandler(
+ String className, String msg, UpdateHandler updateHandler) {
Class<? extends UpdateHandler> clazz = null;
if (msg == null) msg = "SolrCore Object";
try {
clazz = getResourceLoader().findClass(className, UpdateHandler.class);
- //most of the classes do not have constructors which takes SolrCore argument. It is recommended to obtain SolrCore by implementing SolrCoreAware.
- // So invariably always it will cause a NoSuchMethodException. So iterate though the list of available constructors
+ // most of the classes do not have constructors which takes SolrCore argument. It is
+ // recommended to obtain SolrCore by implementing SolrCoreAware.
+ // So invariably always it will cause a NoSuchMethodException. So iterate though the list of
+ // available constructors
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/admin/CollectionsHandler.java
##########
@@ -388,982 +426,1324 @@ public Category getCategory() {
return Category.ADMIN;
}
- private static void createSysConfigSet(CoreContainer coreContainer) throws KeeperException, InterruptedException {
+ private static void createSysConfigSet(CoreContainer coreContainer)
+ throws KeeperException, InterruptedException {
SolrZkClient zk = coreContainer.getZkController().getZkStateReader().getZkClient();
ZkCmdExecutor cmdExecutor = new ZkCmdExecutor(zk.getZkClientTimeout());
cmdExecutor.ensureExists(ZkStateReader.CONFIGS_ZKNODE, zk);
- cmdExecutor.ensureExists(ZkStateReader.CONFIGS_ZKNODE + "/" + CollectionAdminParams.SYSTEM_COLL, zk);
+ cmdExecutor.ensureExists(
+ ZkStateReader.CONFIGS_ZKNODE + "/" + CollectionAdminParams.SYSTEM_COLL, zk);
try {
- String path = ZkStateReader.CONFIGS_ZKNODE + "/" + CollectionAdminParams.SYSTEM_COLL + "/schema.xml";
- byte[] data = IOUtils.toByteArray(CollectionsHandler.class.getResourceAsStream("/SystemCollectionSchema.xml"));
+ String path =
+ ZkStateReader.CONFIGS_ZKNODE + "/" + CollectionAdminParams.SYSTEM_COLL + "/schema.xml";
+ byte[] data =
+ IOUtils.toByteArray(
+ CollectionsHandler.class.getResourceAsStream("/SystemCollectionSchema.xml"));
assert data != null && data.length > 0;
cmdExecutor.ensureExists(path, data, CreateMode.PERSISTENT, zk);
- path = ZkStateReader.CONFIGS_ZKNODE + "/" + CollectionAdminParams.SYSTEM_COLL + "/solrconfig.xml";
- data = IOUtils.toByteArray(CollectionsHandler.class.getResourceAsStream("/SystemCollectionSolrConfig.xml"));
+ path =
+ ZkStateReader.CONFIGS_ZKNODE
+ + "/"
+ + CollectionAdminParams.SYSTEM_COLL
+ + "/solrconfig.xml";
+ data =
+ IOUtils.toByteArray(
+ CollectionsHandler.class.getResourceAsStream("/SystemCollectionSolrConfig.xml"));
assert data != null && data.length > 0;
cmdExecutor.ensureExists(path, data, CreateMode.PERSISTENT, zk);
} catch (IOException e) {
throw new SolrException(ErrorCode.SERVER_ERROR, e);
}
-
-
}
- private static void addStatusToResponse(NamedList<Object> results, RequestStatusState state, String msg) {
+ private static void addStatusToResponse(
+ NamedList<Object> results, RequestStatusState state, String msg) {
SimpleOrderedMap<String> status = new SimpleOrderedMap<>();
status.add("state", state.getKey());
status.add("msg", msg);
results.add("status", status);
}
public enum CollectionOperation implements CollectionOp {
- CREATE_OP(CREATE, (req, rsp, h) -> {
- Map<String, Object> props = copy(req.getParams().required(), null, NAME);
- props.put("fromApi", "true");
- copy(req.getParams(), props,
- REPLICATION_FACTOR,
- COLL_CONF,
- NUM_SLICES,
- CREATE_NODE_SET,
- CREATE_NODE_SET_SHUFFLE,
- SHARDS_PROP,
- PULL_REPLICAS,
- TLOG_REPLICAS,
- NRT_REPLICAS,
- WAIT_FOR_FINAL_STATE,
- PER_REPLICA_STATE,
- ALIAS);
-
- if (props.get(REPLICATION_FACTOR) != null && props.get(NRT_REPLICAS) != null) {
- //TODO: Remove this in 8.0 . Keep this for SolrJ client back-compat. See SOLR-11676 for more details
- int replicationFactor = Integer.parseInt((String) props.get(REPLICATION_FACTOR));
- int nrtReplicas = Integer.parseInt((String) props.get(NRT_REPLICAS));
- if (replicationFactor != nrtReplicas) {
- throw new SolrException(ErrorCode.BAD_REQUEST,
- "Cannot specify both replicationFactor and nrtReplicas as they mean the same thing");
- }
- }
- if (props.get(REPLICATION_FACTOR) != null) {
- props.put(NRT_REPLICAS, props.get(REPLICATION_FACTOR));
- } else if (props.get(NRT_REPLICAS) != null) {
- props.put(REPLICATION_FACTOR, props.get(NRT_REPLICAS));
- }
-
- final String collectionName = SolrIdentifierValidator.validateCollectionName((String) props.get(NAME));
- final String shardsParam = (String) props.get(SHARDS_PROP);
- if (StringUtils.isNotEmpty(shardsParam)) {
- verifyShardsParam(shardsParam);
- }
- if (CollectionAdminParams.SYSTEM_COLL.equals(collectionName)) {
- //We must always create a .system collection with only a single shard
- props.put(NUM_SLICES, 1);
- props.remove(SHARDS_PROP);
- createSysConfigSet(h.coreContainer);
-
- }
- if (shardsParam == null) h.copyFromClusterProp(props, NUM_SLICES);
- for (String prop : ImmutableSet.of(NRT_REPLICAS, PULL_REPLICAS, TLOG_REPLICAS))
- h.copyFromClusterProp(props, prop);
- copyPropertiesWithPrefix(req.getParams(), props, PROPERTY_PREFIX);
- return copyPropertiesWithPrefix(req.getParams(), props, "router.");
+ CREATE_OP(
+ CREATE,
+ (req, rsp, h) -> {
+ Map<String, Object> props = copy(req.getParams().required(), null, NAME);
+ props.put("fromApi", "true");
+ copy(
+ req.getParams(),
+ props,
+ REPLICATION_FACTOR,
+ COLL_CONF,
+ NUM_SLICES,
+ CREATE_NODE_SET,
+ CREATE_NODE_SET_SHUFFLE,
+ SHARDS_PROP,
+ PULL_REPLICAS,
+ TLOG_REPLICAS,
+ NRT_REPLICAS,
+ WAIT_FOR_FINAL_STATE,
+ PER_REPLICA_STATE,
+ ALIAS);
+
+ if (props.get(REPLICATION_FACTOR) != null && props.get(NRT_REPLICAS) != null) {
+ // TODO: Remove this in 8.0 . Keep this for SolrJ client back-compat. See SOLR-11676 for
+ // more details
+ int replicationFactor = Integer.parseInt((String) props.get(REPLICATION_FACTOR));
+ int nrtReplicas = Integer.parseInt((String) props.get(NRT_REPLICAS));
+ if (replicationFactor != nrtReplicas) {
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST,
+ "Cannot specify both replicationFactor and nrtReplicas as they mean the same thing");
+ }
+ }
+ if (props.get(REPLICATION_FACTOR) != null) {
+ props.put(NRT_REPLICAS, props.get(REPLICATION_FACTOR));
+ } else if (props.get(NRT_REPLICAS) != null) {
+ props.put(REPLICATION_FACTOR, props.get(NRT_REPLICAS));
+ }
- }),
+ final String collectionName =
+ SolrIdentifierValidator.validateCollectionName((String) props.get(NAME));
+ final String shardsParam = (String) props.get(SHARDS_PROP);
+ if (StringUtils.isNotEmpty(shardsParam)) {
+ verifyShardsParam(shardsParam);
+ }
+ if (CollectionAdminParams.SYSTEM_COLL.equals(collectionName)) {
+ // We must always create a .system collection with only a single shard
+ props.put(NUM_SLICES, 1);
+ props.remove(SHARDS_PROP);
+ createSysConfigSet(h.coreContainer);
+ }
+ if (shardsParam == null) h.copyFromClusterProp(props, NUM_SLICES);
+ for (String prop : ImmutableSet.of(NRT_REPLICAS, PULL_REPLICAS, TLOG_REPLICAS))
+ h.copyFromClusterProp(props, prop);
+ copyPropertiesWithPrefix(req.getParams(), props, PROPERTY_PREFIX);
+ return copyPropertiesWithPrefix(req.getParams(), props, "router.");
+ }),
@SuppressWarnings({"unchecked"})
- COLSTATUS_OP(COLSTATUS, (req, rsp, h) -> {
- Map<String, Object> props = copy(req.getParams(), null,
- COLLECTION_PROP,
- ColStatus.CORE_INFO_PROP,
- ColStatus.SEGMENTS_PROP,
- ColStatus.FIELD_INFO_PROP,
- ColStatus.RAW_SIZE_PROP,
- ColStatus.RAW_SIZE_SUMMARY_PROP,
- ColStatus.RAW_SIZE_DETAILS_PROP,
- ColStatus.RAW_SIZE_SAMPLING_PERCENT_PROP,
- ColStatus.SIZE_INFO_PROP);
-
- new ColStatus(h.coreContainer.getSolrClientCache(),
- h.coreContainer.getZkController().getZkStateReader().getClusterState(), new ZkNodeProps(props))
- .getColStatus(rsp.getValues());
- return null;
- }),
- DELETE_OP(DELETE, (req, rsp, h) -> {
- Map<String, Object> map = copy(req.getParams().required(), null, NAME);
- return copy(req.getParams(), map, FOLLOW_ALIASES);
- }),
+ COLSTATUS_OP(
+ COLSTATUS,
+ (req, rsp, h) -> {
+ Map<String, Object> props =
+ copy(
+ req.getParams(),
+ null,
+ COLLECTION_PROP,
+ ColStatus.CORE_INFO_PROP,
+ ColStatus.SEGMENTS_PROP,
+ ColStatus.FIELD_INFO_PROP,
+ ColStatus.RAW_SIZE_PROP,
+ ColStatus.RAW_SIZE_SUMMARY_PROP,
+ ColStatus.RAW_SIZE_DETAILS_PROP,
+ ColStatus.RAW_SIZE_SAMPLING_PERCENT_PROP,
+ ColStatus.SIZE_INFO_PROP);
+
+ new ColStatus(
+ h.coreContainer.getSolrClientCache(),
+ h.coreContainer.getZkController().getZkStateReader().getClusterState(),
+ new ZkNodeProps(props))
+ .getColStatus(rsp.getValues());
+ return null;
+ }),
+ DELETE_OP(
+ DELETE,
+ (req, rsp, h) -> {
+ Map<String, Object> map = copy(req.getParams().required(), null, NAME);
+ return copy(req.getParams(), map, FOLLOW_ALIASES);
+ }),
// XXX should this command support followAliases?
- RELOAD_OP(RELOAD, (req, rsp, h) -> {
- Map<String, Object> map = copy(req.getParams().required(), null, NAME);
- return copy(req.getParams(), map);
- }),
-
- RENAME_OP(RENAME, (req, rsp, h) -> {
- Map<String, Object> map = copy(req.getParams().required(), null, NAME, CollectionAdminParams.TARGET);
- return copy(req.getParams(), map, FOLLOW_ALIASES);
- }),
-
- REINDEXCOLLECTION_OP(REINDEXCOLLECTION, (req, rsp, h) -> {
- Map<String, Object> m = copy(req.getParams().required(), null, NAME);
- copy(req.getParams(), m,
- ReindexCollectionCmd.COMMAND,
- ReindexCollectionCmd.REMOVE_SOURCE,
- ReindexCollectionCmd.TARGET,
- ZkStateReader.CONFIGNAME_PROP,
- NUM_SLICES,
- NRT_REPLICAS,
- PULL_REPLICAS,
- TLOG_REPLICAS,
- REPLICATION_FACTOR,
- CREATE_NODE_SET,
- CREATE_NODE_SET_SHUFFLE,
- "shards",
- CommonParams.ROWS,
- CommonParams.Q,
- CommonParams.FL,
- FOLLOW_ALIASES);
- if (req.getParams().get("collection." + ZkStateReader.CONFIGNAME_PROP) != null) {
- m.put(ZkStateReader.CONFIGNAME_PROP, req.getParams().get("collection." + ZkStateReader.CONFIGNAME_PROP));
- }
- copyPropertiesWithPrefix(req.getParams(), m, "router.");
- return m;
- }),
-
- SYNCSHARD_OP(SYNCSHARD, (req, rsp, h) -> {
- String extCollection = req.getParams().required().get("collection");
- String collection = h.coreContainer.getZkController().getZkStateReader().getAliases().resolveSimpleAlias(extCollection);
- String shard = req.getParams().required().get("shard");
-
- ClusterState clusterState = h.coreContainer.getZkController().getClusterState();
-
- DocCollection docCollection = clusterState.getCollection(collection);
- ZkNodeProps leaderProps = docCollection.getLeader(shard);
- ZkCoreNodeProps nodeProps = new ZkCoreNodeProps(leaderProps);
-
- try (HttpSolrClient client = new Builder(nodeProps.getBaseUrl())
- .withConnectionTimeout(15000)
- .withSocketTimeout(60000)
- .build()) {
- RequestSyncShard reqSyncShard = new RequestSyncShard();
- reqSyncShard.setCollection(collection);
- reqSyncShard.setShard(shard);
- reqSyncShard.setCoreName(nodeProps.getCoreName());
- client.request(reqSyncShard);
- }
- return null;
- }),
-
- CREATEALIAS_OP(CREATEALIAS, (req, rsp, h) -> {
- String alias = req.getParams().get(NAME);
- SolrIdentifierValidator.validateAliasName(alias);
- String collections = req.getParams().get("collections");
- RoutedAlias routedAlias = null;
- Exception ex = null;
- HashMap<String,Object> possiblyModifiedParams = new HashMap<>();
- try {
- // note that RA specific validation occurs here.
- req.getParams().toMap(possiblyModifiedParams);
- @SuppressWarnings({"unchecked", "rawtypes"})
- // This is awful because RoutedAlias lies about what types it wants
- Map<String, String> temp = (Map<String, String>) (Map) possiblyModifiedParams;
- routedAlias = RoutedAlias.fromProps(alias, temp);
- } catch (SolrException e) {
- // we'll throw this later if we are in fact creating a routed alias.
- ex = e;
- }
- ModifiableSolrParams finalParams = new ModifiableSolrParams();
- for (Map.Entry<String, Object> entry : possiblyModifiedParams.entrySet()) {
- if (entry.getValue().getClass().isArray() ) {
- // v2 api hits this case
- for (Object o : (Object[]) entry.getValue()) {
- finalParams.add(entry.getKey(),o.toString());
+ RELOAD_OP(
+ RELOAD,
+ (req, rsp, h) -> {
+ Map<String, Object> map = copy(req.getParams().required(), null, NAME);
+ return copy(req.getParams(), map);
+ }),
+
+ RENAME_OP(
+ RENAME,
+ (req, rsp, h) -> {
+ Map<String, Object> map =
+ copy(req.getParams().required(), null, NAME, CollectionAdminParams.TARGET);
+ return copy(req.getParams(), map, FOLLOW_ALIASES);
+ }),
+
+ REINDEXCOLLECTION_OP(
+ REINDEXCOLLECTION,
+ (req, rsp, h) -> {
+ Map<String, Object> m = copy(req.getParams().required(), null, NAME);
+ copy(
+ req.getParams(),
+ m,
+ ReindexCollectionCmd.COMMAND,
+ ReindexCollectionCmd.REMOVE_SOURCE,
+ ReindexCollectionCmd.TARGET,
+ ZkStateReader.CONFIGNAME_PROP,
+ NUM_SLICES,
+ NRT_REPLICAS,
+ PULL_REPLICAS,
+ TLOG_REPLICAS,
+ REPLICATION_FACTOR,
+ CREATE_NODE_SET,
+ CREATE_NODE_SET_SHUFFLE,
+ "shards",
+ CommonParams.ROWS,
+ CommonParams.Q,
+ CommonParams.FL,
+ FOLLOW_ALIASES);
+ if (req.getParams().get("collection." + ZkStateReader.CONFIGNAME_PROP) != null) {
+ m.put(
+ ZkStateReader.CONFIGNAME_PROP,
+ req.getParams().get("collection." + ZkStateReader.CONFIGNAME_PROP));
+ }
+ copyPropertiesWithPrefix(req.getParams(), m, "router.");
+ return m;
+ }),
+
+ SYNCSHARD_OP(
+ SYNCSHARD,
+ (req, rsp, h) -> {
+ String extCollection = req.getParams().required().get("collection");
+ String collection =
+ h.coreContainer
+ .getZkController()
+ .getZkStateReader()
+ .getAliases()
+ .resolveSimpleAlias(extCollection);
+ String shard = req.getParams().required().get("shard");
+
+ ClusterState clusterState = h.coreContainer.getZkController().getClusterState();
+
+ DocCollection docCollection = clusterState.getCollection(collection);
+ ZkNodeProps leaderProps = docCollection.getLeader(shard);
+ ZkCoreNodeProps nodeProps = new ZkCoreNodeProps(leaderProps);
+
+ try (HttpSolrClient client =
+ new Builder(nodeProps.getBaseUrl())
+ .withConnectionTimeout(15000)
+ .withSocketTimeout(60000)
+ .build()) {
+ RequestSyncShard reqSyncShard = new RequestSyncShard();
+ reqSyncShard.setCollection(collection);
+ reqSyncShard.setShard(shard);
+ reqSyncShard.setCoreName(nodeProps.getCoreName());
+ client.request(reqSyncShard);
+ }
+ return null;
+ }),
+
+ CREATEALIAS_OP(
+ CREATEALIAS,
+ (req, rsp, h) -> {
+ String alias = req.getParams().get(NAME);
+ SolrIdentifierValidator.validateAliasName(alias);
+ String collections = req.getParams().get("collections");
+ RoutedAlias routedAlias = null;
+ Exception ex = null;
+ HashMap<String, Object> possiblyModifiedParams = new HashMap<>();
+ try {
+ // note that RA specific validation occurs here.
+ req.getParams().toMap(possiblyModifiedParams);
+ @SuppressWarnings({"unchecked", "rawtypes"})
+ // This is awful because RoutedAlias lies about what types it wants
+ Map<String, String> temp = (Map<String, String>) (Map) possiblyModifiedParams;
+ routedAlias = RoutedAlias.fromProps(alias, temp);
+ } catch (SolrException e) {
+ // we'll throw this later if we are in fact creating a routed alias.
+ ex = e;
+ }
+ ModifiableSolrParams finalParams = new ModifiableSolrParams();
+ for (Map.Entry<String, Object> entry : possiblyModifiedParams.entrySet()) {
+ if (entry.getValue().getClass().isArray()) {
+ // v2 api hits this case
+ for (Object o : (Object[]) entry.getValue()) {
+ finalParams.add(entry.getKey(), o.toString());
+ }
+ } else {
+ finalParams.add(entry.getKey(), entry.getValue().toString());
+ }
}
- } else {
- finalParams.add(entry.getKey(),entry.getValue().toString());
- }
- }
- if (collections != null) {
- if (routedAlias != null) {
- throw new SolrException(BAD_REQUEST, "Collections cannot be specified when creating a routed alias.");
- } else {
- //////////////////////////////////////
- // Regular alias creation indicated //
- //////////////////////////////////////
- return copy(finalParams.required(), null, NAME, "collections");
- }
- }
+ if (collections != null) {
+ if (routedAlias != null) {
+ throw new SolrException(
+ BAD_REQUEST, "Collections cannot be specified when creating a routed alias.");
+ } else {
+ //////////////////////////////////////
+ // Regular alias creation indicated //
+ //////////////////////////////////////
+ return copy(finalParams.required(), null, NAME, "collections");
+ }
+ }
- /////////////////////////////////////////////////
- // We are creating a routed alias from here on //
- /////////////////////////////////////////////////
+ /////////////////////////////////////////////////
+ // We are creating a routed alias from here on //
+ /////////////////////////////////////////////////
- // If our prior creation attempt had issues expose them now.
- if (ex != null) {
- throw ex;
- }
+ // If our prior creation attempt had issues expose them now.
+ if (ex != null) {
+ throw ex;
+ }
- // Now filter out just the parameters we care about from the request
- assert routedAlias != null;
- Map<String, Object> result = copy(finalParams, null, routedAlias.getRequiredParams());
- copy(finalParams, result, routedAlias.getOptionalParams());
-
- ModifiableSolrParams createCollParams = new ModifiableSolrParams(); // without prefix
-
- // add to result params that start with "create-collection.".
- // Additionally, save these without the prefix to createCollParams
- for (Map.Entry<String, String[]> entry : finalParams) {
- final String p = entry.getKey();
- if (p.startsWith(CREATE_COLLECTION_PREFIX)) {
- // This is what SolrParams#getAll(Map, Collection)} does
- final String[] v = entry.getValue();
- if (v.length == 1) {
- result.put(p, v[0]);
- } else {
- result.put(p, v);
+ // Now filter out just the parameters we care about from the request
+ assert routedAlias != null;
+ Map<String, Object> result = copy(finalParams, null, routedAlias.getRequiredParams());
+ copy(finalParams, result, routedAlias.getOptionalParams());
+
+ ModifiableSolrParams createCollParams = new ModifiableSolrParams(); // without prefix
+
+ // add to result params that start with "create-collection.".
+ // Additionally, save these without the prefix to createCollParams
+ for (Map.Entry<String, String[]> entry : finalParams) {
+ final String p = entry.getKey();
+ if (p.startsWith(CREATE_COLLECTION_PREFIX)) {
+ // This is what SolrParams#getAll(Map, Collection)} does
+ final String[] v = entry.getValue();
+ if (v.length == 1) {
+ result.put(p, v[0]);
+ } else {
+ result.put(p, v);
+ }
+ createCollParams.set(p.substring(CREATE_COLLECTION_PREFIX.length()), v);
+ }
}
- createCollParams.set(p.substring(CREATE_COLLECTION_PREFIX.length()), v);
- }
- }
- // Verify that the create-collection prefix'ed params appear to be valid.
- if (createCollParams.get(NAME) != null) {
- throw new SolrException(BAD_REQUEST, "routed aliases calculate names for their " +
- "dependent collections, you cannot specify the name.");
- }
- if (createCollParams.get(COLL_CONF) == null) {
- throw new SolrException(SolrException.ErrorCode.BAD_REQUEST,
- "We require an explicit " + COLL_CONF);
- }
- // note: could insist on a config name here as well.... or wait to throw at overseer
- createCollParams.add(NAME, "TMP_name_TMP_name_TMP"); // just to pass validation
- CREATE_OP.execute(new LocalSolrQueryRequest(null, createCollParams), rsp, h); // ignore results
+ // Verify that the create-collection prefix'ed params appear to be valid.
+ if (createCollParams.get(NAME) != null) {
+ throw new SolrException(
+ BAD_REQUEST,
+ "routed aliases calculate names for their "
+ + "dependent collections, you cannot specify the name.");
+ }
+ if (createCollParams.get(COLL_CONF) == null) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST, "We require an explicit " + COLL_CONF);
+ }
+ // note: could insist on a config name here as well.... or wait to throw at overseer
+ createCollParams.add(NAME, "TMP_name_TMP_name_TMP"); // just to pass validation
+ CREATE_OP.execute(
+ new LocalSolrQueryRequest(null, createCollParams), rsp, h); // ignore results
- return result;
- }),
+ return result;
+ }),
DELETEALIAS_OP(DELETEALIAS, (req, rsp, h) -> copy(req.getParams().required(), null, NAME)),
/**
* Change properties for an alias (use CREATEALIAS_OP to change the actual value of the alias)
*/
- ALIASPROP_OP(ALIASPROP, (req, rsp, h) -> {
- Map<String, Object> params = copy(req.getParams().required(), null, NAME);
-
- // Note: success/no-op in the event of no properties supplied is intentional. Keeps code simple and one less case
- // for api-callers to check for.
- return convertPrefixToMap(req.getParams(), params, "property");
- }),
-
- /**
- * List the aliases and associated properties.
- */
+ ALIASPROP_OP(
+ ALIASPROP,
+ (req, rsp, h) -> {
+ Map<String, Object> params = copy(req.getParams().required(), null, NAME);
+
+ // Note: success/no-op in the event of no properties supplied is intentional. Keeps code
+ // simple and one less case
+ // for api-callers to check for.
+ return convertPrefixToMap(req.getParams(), params, "property");
+ }),
+
+ /** List the aliases and associated properties. */
@SuppressWarnings({"unchecked"})
- LISTALIASES_OP(LISTALIASES, (req, rsp, h) -> {
- ZkStateReader zkStateReader = h.coreContainer.getZkController().getZkStateReader();
- // if someone calls listAliases, lets ensure we return an up to date response
- zkStateReader.aliasesManager.update();
- Aliases aliases = zkStateReader.getAliases();
- if (aliases != null) {
- // the aliases themselves...
- rsp.getValues().add("aliases", aliases.getCollectionAliasMap());
- // Any properties for the above aliases.
- Map<String, Map<String, String>> meta = new LinkedHashMap<>();
- for (String alias : aliases.getCollectionAliasListMap().keySet()) {
- Map<String, String> collectionAliasProperties = aliases.getCollectionAliasProperties(alias);
- if (!collectionAliasProperties.isEmpty()) {
- meta.put(alias, collectionAliasProperties);
+ LISTALIASES_OP(
+ LISTALIASES,
+ (req, rsp, h) -> {
+ ZkStateReader zkStateReader = h.coreContainer.getZkController().getZkStateReader();
+ // if someone calls listAliases, lets ensure we return an up to date response
+ zkStateReader.aliasesManager.update();
+ Aliases aliases = zkStateReader.getAliases();
+ if (aliases != null) {
+ // the aliases themselves...
+ rsp.getValues().add("aliases", aliases.getCollectionAliasMap());
+ // Any properties for the above aliases.
+ Map<String, Map<String, String>> meta = new LinkedHashMap<>();
+ for (String alias : aliases.getCollectionAliasListMap().keySet()) {
+ Map<String, String> collectionAliasProperties =
+ aliases.getCollectionAliasProperties(alias);
+ if (!collectionAliasProperties.isEmpty()) {
+ meta.put(alias, collectionAliasProperties);
+ }
+ }
+ rsp.getValues().add("properties", meta);
+ }
+ return null;
+ }),
+ SPLITSHARD_OP(
+ SPLITSHARD,
+ DEFAULT_COLLECTION_OP_TIMEOUT * 5,
+ (req, rsp, h) -> {
+ String name = req.getParams().required().get(COLLECTION_PROP);
+ // TODO : add support for multiple shards
+ String shard = req.getParams().get(SHARD_ID_PROP);
+ String rangesStr = req.getParams().get(CoreAdminParams.RANGES);
+ String splitKey = req.getParams().get("split.key");
+ String numSubShards = req.getParams().get(NUM_SUB_SHARDS);
+ String fuzz = req.getParams().get(SPLIT_FUZZ);
+
+ if (splitKey == null && shard == null) {
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST, "At least one of shard, or split.key should be specified.");
+ }
+ if (splitKey != null && shard != null) {
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST, "Only one of 'shard' or 'split.key' should be specified");
+ }
+ if (splitKey != null && rangesStr != null) {
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST, "Only one of 'ranges' or 'split.key' should be specified");
+ }
+ if (numSubShards != null && (splitKey != null || rangesStr != null)) {
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST,
+ "numSubShards can not be specified with split.key or ranges parameters");
+ }
+ if (fuzz != null && (splitKey != null || rangesStr != null)) {
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST,
+ "fuzz can not be specified with split.key or ranges parameters");
}
- }
- rsp.getValues().add("properties", meta);
- }
- return null;
- }),
- SPLITSHARD_OP(SPLITSHARD, DEFAULT_COLLECTION_OP_TIMEOUT * 5, (req, rsp, h) -> {
- String name = req.getParams().required().get(COLLECTION_PROP);
- // TODO : add support for multiple shards
- String shard = req.getParams().get(SHARD_ID_PROP);
- String rangesStr = req.getParams().get(CoreAdminParams.RANGES);
- String splitKey = req.getParams().get("split.key");
- String numSubShards = req.getParams().get(NUM_SUB_SHARDS);
- String fuzz = req.getParams().get(SPLIT_FUZZ);
-
- if (splitKey == null && shard == null) {
- throw new SolrException(ErrorCode.BAD_REQUEST, "At least one of shard, or split.key should be specified.");
- }
- if (splitKey != null && shard != null) {
- throw new SolrException(ErrorCode.BAD_REQUEST,
- "Only one of 'shard' or 'split.key' should be specified");
- }
- if (splitKey != null && rangesStr != null) {
- throw new SolrException(ErrorCode.BAD_REQUEST,
- "Only one of 'ranges' or 'split.key' should be specified");
- }
- if (numSubShards != null && (splitKey != null || rangesStr != null)) {
- throw new SolrException(ErrorCode.BAD_REQUEST,
- "numSubShards can not be specified with split.key or ranges parameters");
- }
- if (fuzz != null && (splitKey != null || rangesStr != null)) {
- throw new SolrException(ErrorCode.BAD_REQUEST,
- "fuzz can not be specified with split.key or ranges parameters");
- }
- Map<String, Object> map = copy(req.getParams(), null,
- COLLECTION_PROP,
- SHARD_ID_PROP,
- "split.key",
- CoreAdminParams.RANGES,
- WAIT_FOR_FINAL_STATE,
- TIMING,
- SPLIT_METHOD,
- NUM_SUB_SHARDS,
- SPLIT_FUZZ,
- SPLIT_BY_PREFIX,
- FOLLOW_ALIASES);
- return copyPropertiesWithPrefix(req.getParams(), map, PROPERTY_PREFIX);
- }),
- DELETESHARD_OP(DELETESHARD, (req, rsp, h) -> {
- Map<String, Object> map = copy(req.getParams().required(), null,
- COLLECTION_PROP,
- SHARD_ID_PROP);
- copy(req.getParams(), map,
- DELETE_INDEX,
- DELETE_DATA_DIR,
- DELETE_INSTANCE_DIR,
- FOLLOW_ALIASES);
- return map;
- }),
- FORCELEADER_OP(FORCELEADER, (req, rsp, h) -> {
- forceLeaderElection(req, h);
- return null;
- }),
- CREATESHARD_OP(CREATESHARD, (req, rsp, h) -> {
- Map<String, Object> map = copy(req.getParams().required(), null,
- COLLECTION_PROP,
- SHARD_ID_PROP);
- ClusterState clusterState = h.coreContainer.getZkController().getClusterState();
- final String newShardName = SolrIdentifierValidator.validateShardName(req.getParams().get(SHARD_ID_PROP));
- boolean followAliases = req.getParams().getBool(FOLLOW_ALIASES, false);
- String extCollectionName = req.getParams().get(COLLECTION_PROP);
- String collectionName = followAliases ? h.coreContainer.getZkController().getZkStateReader()
- .getAliases().resolveSimpleAlias(extCollectionName) : extCollectionName;
- if (!ImplicitDocRouter.NAME.equals(((Map<?,?>) clusterState.getCollection(collectionName).get(DOC_ROUTER)).get(NAME)))
- throw new SolrException(ErrorCode.BAD_REQUEST, "shards can be added only to 'implicit' collections");
- copy(req.getParams(), map,
- REPLICATION_FACTOR,
- NRT_REPLICAS,
- TLOG_REPLICAS,
- PULL_REPLICAS,
- CREATE_NODE_SET,
- WAIT_FOR_FINAL_STATE,
- FOLLOW_ALIASES);
- return copyPropertiesWithPrefix(req.getParams(), map, PROPERTY_PREFIX);
- }),
- DELETEREPLICA_OP(DELETEREPLICA, (req, rsp, h) -> {
- Map<String, Object> map = copy(req.getParams().required(), null,
- COLLECTION_PROP);
-
- return copy(req.getParams(), map,
- DELETE_INDEX,
- DELETE_DATA_DIR,
- DELETE_INSTANCE_DIR,
- COUNT_PROP, REPLICA_PROP,
- SHARD_ID_PROP,
- ONLY_IF_DOWN,
- FOLLOW_ALIASES);
- }),
- MIGRATE_OP(MIGRATE, (req, rsp, h) -> {
- Map<String, Object> map = copy(req.getParams().required(), null, COLLECTION_PROP, "split.key", "target.collection");
- return copy(req.getParams(), map, "forward.timeout", FOLLOW_ALIASES);
- }),
- ADDROLE_OP(ADDROLE, (req, rsp, h) -> {
- Map<String, Object> map = copy(req.getParams().required(), null, "role", "node");
- if (!KNOWN_ROLES.contains(map.get("role")))
- throw new SolrException(ErrorCode.BAD_REQUEST, "Unknown role. Supported roles are ," + KNOWN_ROLES);
- return map;
- }),
- REMOVEROLE_OP(REMOVEROLE, (req, rsp, h) -> {
- Map<String, Object> map = copy(req.getParams().required(), null, "role", "node");
- if (!KNOWN_ROLES.contains(map.get("role")))
- throw new SolrException(ErrorCode.BAD_REQUEST, "Unknown role. Supported roles are ," + KNOWN_ROLES);
- return map;
- }),
- CLUSTERPROP_OP(CLUSTERPROP, (req, rsp, h) -> {
- String name = req.getParams().required().get(NAME);
- String val = req.getParams().get(VALUE_LONG);
- ClusterProperties cp = new ClusterProperties(h.coreContainer.getZkController().getZkClient());
- cp.setClusterProperty(name, val);
- return null;
- }),
- COLLECTIONPROP_OP(COLLECTIONPROP, (req, rsp, h) -> {
- String extCollection = req.getParams().required().get(NAME);
- String collection = h.coreContainer.getZkController().getZkStateReader().getAliases().resolveSimpleAlias(extCollection);
- String name = req.getParams().required().get(PROPERTY_NAME);
- String val = req.getParams().get(PROPERTY_VALUE);
- CollectionProperties cp = new CollectionProperties(h.coreContainer.getZkController().getZkClient());
- cp.setCollectionProperty(collection, name, val);
- return null;
- }),
+ Map<String, Object> map =
+ copy(
+ req.getParams(),
+ null,
+ COLLECTION_PROP,
+ SHARD_ID_PROP,
+ "split.key",
+ CoreAdminParams.RANGES,
+ WAIT_FOR_FINAL_STATE,
+ TIMING,
+ SPLIT_METHOD,
+ NUM_SUB_SHARDS,
+ SPLIT_FUZZ,
+ SPLIT_BY_PREFIX,
+ FOLLOW_ALIASES);
+ return copyPropertiesWithPrefix(req.getParams(), map, PROPERTY_PREFIX);
+ }),
+ DELETESHARD_OP(
+ DELETESHARD,
+ (req, rsp, h) -> {
+ Map<String, Object> map =
+ copy(req.getParams().required(), null, COLLECTION_PROP, SHARD_ID_PROP);
+ copy(
+ req.getParams(),
+ map,
+ DELETE_INDEX,
+ DELETE_DATA_DIR,
+ DELETE_INSTANCE_DIR,
+ FOLLOW_ALIASES);
+ return map;
+ }),
+ FORCELEADER_OP(
+ FORCELEADER,
+ (req, rsp, h) -> {
+ forceLeaderElection(req, h);
+ return null;
+ }),
+ CREATESHARD_OP(
+ CREATESHARD,
+ (req, rsp, h) -> {
+ Map<String, Object> map =
+ copy(req.getParams().required(), null, COLLECTION_PROP, SHARD_ID_PROP);
+ ClusterState clusterState = h.coreContainer.getZkController().getClusterState();
+ final String newShardName =
+ SolrIdentifierValidator.validateShardName(req.getParams().get(SHARD_ID_PROP));
+ boolean followAliases = req.getParams().getBool(FOLLOW_ALIASES, false);
+ String extCollectionName = req.getParams().get(COLLECTION_PROP);
+ String collectionName =
+ followAliases
+ ? h.coreContainer
+ .getZkController()
+ .getZkStateReader()
+ .getAliases()
+ .resolveSimpleAlias(extCollectionName)
+ : extCollectionName;
+ if (!ImplicitDocRouter.NAME.equals(
+ ((Map<?, ?>) clusterState.getCollection(collectionName).get(DOC_ROUTER)).get(NAME)))
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST, "shards can be added only to 'implicit' collections");
+ copy(
+ req.getParams(),
+ map,
+ REPLICATION_FACTOR,
+ NRT_REPLICAS,
+ TLOG_REPLICAS,
+ PULL_REPLICAS,
+ CREATE_NODE_SET,
+ WAIT_FOR_FINAL_STATE,
+ FOLLOW_ALIASES);
+ return copyPropertiesWithPrefix(req.getParams(), map, PROPERTY_PREFIX);
+ }),
+ DELETEREPLICA_OP(
+ DELETEREPLICA,
+ (req, rsp, h) -> {
+ Map<String, Object> map = copy(req.getParams().required(), null, COLLECTION_PROP);
+
+ return copy(
+ req.getParams(),
+ map,
+ DELETE_INDEX,
+ DELETE_DATA_DIR,
+ DELETE_INSTANCE_DIR,
+ COUNT_PROP,
+ REPLICA_PROP,
+ SHARD_ID_PROP,
+ ONLY_IF_DOWN,
+ FOLLOW_ALIASES);
+ }),
+ MIGRATE_OP(
+ MIGRATE,
+ (req, rsp, h) -> {
+ Map<String, Object> map =
+ copy(
+ req.getParams().required(),
+ null,
+ COLLECTION_PROP,
+ "split.key",
+ "target.collection");
+ return copy(req.getParams(), map, "forward.timeout", FOLLOW_ALIASES);
+ }),
+ ADDROLE_OP(
+ ADDROLE,
+ (req, rsp, h) -> {
+ Map<String, Object> map = copy(req.getParams().required(), null, "role", "node");
+ if (!KNOWN_ROLES.contains(map.get("role")))
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST, "Unknown role. Supported roles are ," + KNOWN_ROLES);
+ return map;
+ }),
+ REMOVEROLE_OP(
+ REMOVEROLE,
+ (req, rsp, h) -> {
+ Map<String, Object> map = copy(req.getParams().required(), null, "role", "node");
+ if (!KNOWN_ROLES.contains(map.get("role")))
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST, "Unknown role. Supported roles are ," + KNOWN_ROLES);
+ return map;
+ }),
+ CLUSTERPROP_OP(
+ CLUSTERPROP,
+ (req, rsp, h) -> {
+ String name = req.getParams().required().get(NAME);
+ String val = req.getParams().get(VALUE_LONG);
+ ClusterProperties cp =
+ new ClusterProperties(h.coreContainer.getZkController().getZkClient());
+ cp.setClusterProperty(name, val);
+ return null;
+ }),
+ COLLECTIONPROP_OP(
+ COLLECTIONPROP,
+ (req, rsp, h) -> {
+ String extCollection = req.getParams().required().get(NAME);
+ String collection =
+ h.coreContainer
+ .getZkController()
+ .getZkStateReader()
+ .getAliases()
+ .resolveSimpleAlias(extCollection);
+ String name = req.getParams().required().get(PROPERTY_NAME);
+ String val = req.getParams().get(PROPERTY_VALUE);
+ CollectionProperties cp =
+ new CollectionProperties(h.coreContainer.getZkController().getZkClient());
+ cp.setCollectionProperty(collection, name, val);
+ return null;
+ }),
@SuppressWarnings({"unchecked"})
- REQUESTSTATUS_OP(REQUESTSTATUS, (req, rsp, h) -> {
- req.getParams().required().check(REQUESTID);
-
- final CoreContainer coreContainer = h.coreContainer;
- final String requestId = req.getParams().get(REQUESTID);
- final ZkController zkController = coreContainer.getZkController();
-
- final NamedList<Object> status = new NamedList<>();
- if (coreContainer.getDistributedCollectionCommandRunner().isEmpty()) {
- if (zkController.getOverseerCompletedMap().contains(requestId)) {
- final byte[] mapEntry = zkController.getOverseerCompletedMap().get(requestId);
- rsp.getValues().addAll(OverseerSolrResponseSerializer.deserialize(mapEntry).getResponse());
- addStatusToResponse(status, COMPLETED, "found [" + requestId + "] in completed tasks");
- } else if (zkController.getOverseerFailureMap().contains(requestId)) {
- final byte[] mapEntry = zkController.getOverseerFailureMap().get(requestId);
- rsp.getValues().addAll(OverseerSolrResponseSerializer.deserialize(mapEntry).getResponse());
- addStatusToResponse(status, FAILED, "found [" + requestId + "] in failed tasks");
- } else if (zkController.getOverseerRunningMap().contains(requestId)) {
- addStatusToResponse(status, RUNNING, "found [" + requestId + "] in running tasks");
- } else if (h.overseerCollectionQueueContains(requestId)) {
- addStatusToResponse(status, SUBMITTED, "found [" + requestId + "] in submitted tasks");
- } else {
- addStatusToResponse(status, NOT_FOUND, "Did not find [" + requestId + "] in any tasks queue");
- }
- } else {
- Pair<RequestStatusState, OverseerSolrResponse> sr =
- coreContainer.getDistributedCollectionCommandRunner().get().getAsyncTaskRequestStatus(requestId);
- final String message;
- switch (sr.first()) {
- case COMPLETED:
- message = "found [" + requestId + "] in completed tasks";
- rsp.getValues().addAll(sr.second().getResponse());
- break;
- case FAILED:
- message = "found [" + requestId + "] in failed tasks";
- rsp.getValues().addAll(sr.second().getResponse());
- break;
- case RUNNING:
- message = "found [" + requestId + "] in running tasks";
- break;
- case SUBMITTED:
- message = "found [" + requestId + "] in submitted tasks";
- break;
- default:
- message = "Did not find [" + requestId + "] in any tasks queue";
- }
- addStatusToResponse(status, sr.first(), message);
- }
-
- rsp.getValues().addAll(status);
- return null;
- }),
- DELETESTATUS_OP(DELETESTATUS, new CollectionOp() {
- @SuppressWarnings("unchecked")
- @Override
- public Map<String, Object> execute(SolrQueryRequest req, SolrQueryResponse rsp, CollectionsHandler h) throws Exception {
- final CoreContainer coreContainer = h.coreContainer;
- final String requestId = req.getParams().get(REQUESTID);
- final ZkController zkController = coreContainer.getZkController();
- Boolean flush = req.getParams().getBool(CollectionAdminParams.FLUSH, false);
-
- if (requestId == null && !flush) {
- throw new SolrException(ErrorCode.BAD_REQUEST, "Either requestid or flush parameter must be specified.");
- }
-
- if (requestId != null && flush) {
- throw new SolrException(ErrorCode.BAD_REQUEST,
- "Both requestid and flush parameters can not be specified together.");
- }
-
- if (coreContainer.getDistributedCollectionCommandRunner().isEmpty()) {
- if (flush) {
- Collection<String> completed = zkController.getOverseerCompletedMap().keys();
- Collection<String> failed = zkController.getOverseerFailureMap().keys();
- for (String asyncId : completed) {
- zkController.getOverseerCompletedMap().remove(asyncId);
- zkController.clearAsyncId(asyncId);
- }
- for (String asyncId : failed) {
- zkController.getOverseerFailureMap().remove(asyncId);
- zkController.clearAsyncId(asyncId);
+ REQUESTSTATUS_OP(
+ REQUESTSTATUS,
+ (req, rsp, h) -> {
+ req.getParams().required().check(REQUESTID);
+
+ final CoreContainer coreContainer = h.coreContainer;
+ final String requestId = req.getParams().get(REQUESTID);
+ final ZkController zkController = coreContainer.getZkController();
+
+ final NamedList<Object> status = new NamedList<>();
+ if (coreContainer.getDistributedCollectionCommandRunner().isEmpty()) {
+ if (zkController.getOverseerCompletedMap().contains(requestId)) {
+ final byte[] mapEntry = zkController.getOverseerCompletedMap().get(requestId);
+ rsp.getValues()
+ .addAll(OverseerSolrResponseSerializer.deserialize(mapEntry).getResponse());
+ addStatusToResponse(
+ status, COMPLETED, "found [" + requestId + "] in completed tasks");
+ } else if (zkController.getOverseerFailureMap().contains(requestId)) {
+ final byte[] mapEntry = zkController.getOverseerFailureMap().get(requestId);
+ rsp.getValues()
+ .addAll(OverseerSolrResponseSerializer.deserialize(mapEntry).getResponse());
+ addStatusToResponse(status, FAILED, "found [" + requestId + "] in failed tasks");
+ } else if (zkController.getOverseerRunningMap().contains(requestId)) {
+ addStatusToResponse(status, RUNNING, "found [" + requestId + "] in running tasks");
+ } else if (h.overseerCollectionQueueContains(requestId)) {
+ addStatusToResponse(
+ status, SUBMITTED, "found [" + requestId + "] in submitted tasks");
+ } else {
+ addStatusToResponse(
+ status, NOT_FOUND, "Did not find [" + requestId + "] in any tasks queue");
}
- rsp.getValues().add("status", "successfully cleared stored collection api responses");
} else {
- // Request to cleanup
- if (zkController.getOverseerCompletedMap().remove(requestId)) {
- zkController.clearAsyncId(requestId);
- rsp.getValues().add("status", "successfully removed stored response for [" + requestId + "]");
- } else if (zkController.getOverseerFailureMap().remove(requestId)) {
- zkController.clearAsyncId(requestId);
- rsp.getValues().add("status", "successfully removed stored response for [" + requestId + "]");
- } else {
- rsp.getValues().add("status", "[" + requestId + "] not found in stored responses");
- // Don't call zkController.clearAsyncId for this, since it could be a running/pending task
+ Pair<RequestStatusState, OverseerSolrResponse> sr =
+ coreContainer
+ .getDistributedCollectionCommandRunner()
+ .get()
+ .getAsyncTaskRequestStatus(requestId);
+ final String message;
+ switch (sr.first()) {
+ case COMPLETED:
+ message = "found [" + requestId + "] in completed tasks";
+ rsp.getValues().addAll(sr.second().getResponse());
+ break;
+ case FAILED:
+ message = "found [" + requestId + "] in failed tasks";
+ rsp.getValues().addAll(sr.second().getResponse());
+ break;
+ case RUNNING:
+ message = "found [" + requestId + "] in running tasks";
+ break;
+ case SUBMITTED:
+ message = "found [" + requestId + "] in submitted tasks";
+ break;
+ default:
+ message = "Did not find [" + requestId + "] in any tasks queue";
}
+ addStatusToResponse(status, sr.first(), message);
}
- } else {
- if (flush) {
- coreContainer.getDistributedCollectionCommandRunner().get().deleteAllAsyncIds();
- rsp.getValues().add("status", "successfully cleared stored collection api responses");
- } else {
- if (coreContainer.getDistributedCollectionCommandRunner().get().deleteSingleAsyncId(requestId)) {
- rsp.getValues().add("status", "successfully removed stored response for [" + requestId + "]");
+
+ rsp.getValues().addAll(status);
+ return null;
+ }),
+ DELETESTATUS_OP(
+ DELETESTATUS,
+ new CollectionOp() {
+ @SuppressWarnings("unchecked")
+ @Override
+ public Map<String, Object> execute(
+ SolrQueryRequest req, SolrQueryResponse rsp, CollectionsHandler h) throws Exception {
+ final CoreContainer coreContainer = h.coreContainer;
+ final String requestId = req.getParams().get(REQUESTID);
+ final ZkController zkController = coreContainer.getZkController();
+ Boolean flush = req.getParams().getBool(CollectionAdminParams.FLUSH, false);
+
+ if (requestId == null && !flush) {
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST, "Either requestid or flush parameter must be specified.");
+ }
+
+ if (requestId != null && flush) {
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST,
+ "Both requestid and flush parameters can not be specified together.");
+ }
+
+ if (coreContainer.getDistributedCollectionCommandRunner().isEmpty()) {
+ if (flush) {
+ Collection<String> completed = zkController.getOverseerCompletedMap().keys();
+ Collection<String> failed = zkController.getOverseerFailureMap().keys();
+ for (String asyncId : completed) {
+ zkController.getOverseerCompletedMap().remove(asyncId);
+ zkController.clearAsyncId(asyncId);
+ }
+ for (String asyncId : failed) {
+ zkController.getOverseerFailureMap().remove(asyncId);
+ zkController.clearAsyncId(asyncId);
+ }
+ rsp.getValues()
+ .add("status", "successfully cleared stored collection api responses");
+ } else {
+ // Request to cleanup
+ if (zkController.getOverseerCompletedMap().remove(requestId)) {
+ zkController.clearAsyncId(requestId);
+ rsp.getValues()
+ .add(
+ "status", "successfully removed stored response for [" + requestId + "]");
+ } else if (zkController.getOverseerFailureMap().remove(requestId)) {
+ zkController.clearAsyncId(requestId);
+ rsp.getValues()
+ .add(
+ "status", "successfully removed stored response for [" + requestId + "]");
+ } else {
+ rsp.getValues()
+ .add("status", "[" + requestId + "] not found in stored responses");
+ // Don't call zkController.clearAsyncId for this, since it could be a
+ // running/pending task
+ }
+ }
} else {
- rsp.getValues().add("status", "[" + requestId + "] not found in stored responses");
+ if (flush) {
+ coreContainer.getDistributedCollectionCommandRunner().get().deleteAllAsyncIds();
+ rsp.getValues()
+ .add("status", "successfully cleared stored collection api responses");
+ } else {
+ if (coreContainer
+ .getDistributedCollectionCommandRunner()
+ .get()
+ .deleteSingleAsyncId(requestId)) {
+ rsp.getValues()
+ .add(
+ "status", "successfully removed stored response for [" + requestId + "]");
+ } else {
+ rsp.getValues()
+ .add("status", "[" + requestId + "] not found in stored responses");
+ }
+ }
}
+ return null;
}
- }
- return null;
- }
- }),
- ADDREPLICA_OP(ADDREPLICA, (req, rsp, h) -> {
- Map<String, Object> props = copy(req.getParams(), null,
- COLLECTION_PROP,
- "node",
- SHARD_ID_PROP,
- _ROUTE_,
- CoreAdminParams.NAME,
- INSTANCE_DIR,
- DATA_DIR,
- ULOG_DIR,
- REPLICA_TYPE,
- WAIT_FOR_FINAL_STATE,
- NRT_REPLICAS,
- TLOG_REPLICAS,
- PULL_REPLICAS,
- CREATE_NODE_SET,
- FOLLOW_ALIASES,
- SKIP_NODE_ASSIGNMENT);
- return copyPropertiesWithPrefix(req.getParams(), props, PROPERTY_PREFIX);
- }),
+ }),
+ ADDREPLICA_OP(
+ ADDREPLICA,
+ (req, rsp, h) -> {
+ Map<String, Object> props =
+ copy(
+ req.getParams(),
+ null,
+ COLLECTION_PROP,
+ "node",
+ SHARD_ID_PROP,
+ _ROUTE_,
+ CoreAdminParams.NAME,
+ INSTANCE_DIR,
+ DATA_DIR,
+ ULOG_DIR,
+ REPLICA_TYPE,
+ WAIT_FOR_FINAL_STATE,
+ NRT_REPLICAS,
+ TLOG_REPLICAS,
+ PULL_REPLICAS,
+ CREATE_NODE_SET,
+ FOLLOW_ALIASES,
+ SKIP_NODE_ASSIGNMENT);
+ return copyPropertiesWithPrefix(req.getParams(), props, PROPERTY_PREFIX);
+ }),
OVERSEERSTATUS_OP(OVERSEERSTATUS, (req, rsp, h) -> new LinkedHashMap<>()),
@SuppressWarnings({"unchecked"})
- DISTRIBUTEDAPIPROCESSING_OP(DISTRIBUTEDAPIPROCESSING, (req, rsp, h) -> {
- NamedList<Object> results = new NamedList<>();
- boolean isDistributedApi = h.coreContainer.getDistributedCollectionCommandRunner().isPresent();
- results.add("isDistributedApi", isDistributedApi);
- rsp.getValues().addAll(results);
- return null;
- }),
- /**
- * Handle list collection request.
- * Do list collection request to zk host
- */
+ DISTRIBUTEDAPIPROCESSING_OP(
+ DISTRIBUTEDAPIPROCESSING,
+ (req, rsp, h) -> {
+ NamedList<Object> results = new NamedList<>();
+ boolean isDistributedApi =
+ h.coreContainer.getDistributedCollectionCommandRunner().isPresent();
+ results.add("isDistributedApi", isDistributedApi);
+ rsp.getValues().addAll(results);
+ return null;
+ }),
+ /** Handle list collection request. Do list collection request to zk host */
@SuppressWarnings({"unchecked"})
- LIST_OP(LIST, (req, rsp, h) -> {
- NamedList<Object> results = new NamedList<>();
- Map<String, DocCollection> collections = h.coreContainer.getZkController().getZkStateReader().getClusterState().getCollectionsMap();
- List<String> collectionList = new ArrayList<>(collections.keySet());
- Collections.sort(collectionList);
- // XXX should we add aliases here?
- results.add("collections", collectionList);
- SolrResponse response = new OverseerSolrResponse(results);
- rsp.getValues().addAll(response.getResponse());
- return null;
- }),
+ LIST_OP(
+ LIST,
+ (req, rsp, h) -> {
+ NamedList<Object> results = new NamedList<>();
+ Map<String, DocCollection> collections =
+ h.coreContainer
+ .getZkController()
+ .getZkStateReader()
+ .getClusterState()
+ .getCollectionsMap();
+ List<String> collectionList = new ArrayList<>(collections.keySet());
+ Collections.sort(collectionList);
+ // XXX should we add aliases here?
+ results.add("collections", collectionList);
+ SolrResponse response = new OverseerSolrResponse(results);
+ rsp.getValues().addAll(response.getResponse());
+ return null;
+ }),
/**
- * Handle cluster status request.
- * Can return status per specific collection/shard or per all collections.
+ * Handle cluster status request. Can return status per specific collection/shard or per all
+ * collections.
*/
- CLUSTERSTATUS_OP(CLUSTERSTATUS, (req, rsp, h) -> {
- Map<String, Object> all = copy(req.getParams(), null,
- COLLECTION_PROP,
- SHARD_ID_PROP,
- _ROUTE_);
- new ClusterStatus(h.coreContainer.getZkController().getZkStateReader(),
- new ZkNodeProps(all)).getClusterStatus(rsp.getValues());
- return null;
- }),
- ADDREPLICAPROP_OP(ADDREPLICAPROP, (req, rsp, h) -> {
- Map<String, Object> map = copy(req.getParams().required(), null,
- COLLECTION_PROP,
- PROPERTY_PROP,
- SHARD_ID_PROP,
- REPLICA_PROP,
- PROPERTY_VALUE_PROP);
- copy(req.getParams(), map, SHARD_UNIQUE);
- String property = (String) map.get(PROPERTY_PROP);
- if (!property.startsWith(PROPERTY_PREFIX)) {
- property = PROPERTY_PREFIX + property;
- }
+ CLUSTERSTATUS_OP(
+ CLUSTERSTATUS,
+ (req, rsp, h) -> {
+ Map<String, Object> all =
+ copy(req.getParams(), null, COLLECTION_PROP, SHARD_ID_PROP, _ROUTE_);
+ new ClusterStatus(
+ h.coreContainer.getZkController().getZkStateReader(), new ZkNodeProps(all))
+ .getClusterStatus(rsp.getValues());
+ return null;
+ }),
+ ADDREPLICAPROP_OP(
+ ADDREPLICAPROP,
+ (req, rsp, h) -> {
+ Map<String, Object> map =
+ copy(
+ req.getParams().required(),
+ null,
+ COLLECTION_PROP,
+ PROPERTY_PROP,
+ SHARD_ID_PROP,
+ REPLICA_PROP,
+ PROPERTY_VALUE_PROP);
+ copy(req.getParams(), map, SHARD_UNIQUE);
+ String property = (String) map.get(PROPERTY_PROP);
+ if (!property.startsWith(PROPERTY_PREFIX)) {
+ property = PROPERTY_PREFIX + property;
+ }
- boolean uniquePerSlice = Boolean.parseBoolean((String) map.get(SHARD_UNIQUE));
-
- // Check if we're trying to set a property with parameters that allow us to set the property on multiple replicas
- // in a slice on properties that are known to only be one-per-slice and error out if so.
- if (StringUtils.isNotBlank((String) map.get(SHARD_UNIQUE)) &&
- SliceMutator.SLICE_UNIQUE_BOOLEAN_PROPERTIES.contains(property.toLowerCase(Locale.ROOT)) &&
- uniquePerSlice == false) {
- throw new SolrException(ErrorCode.BAD_REQUEST,
- "Overseer replica property command received for property " + property +
- " with the " + SHARD_UNIQUE +
- " parameter set to something other than 'true'. No action taken.");
- }
- return map;
- }),
+ boolean uniquePerSlice = Boolean.parseBoolean((String) map.get(SHARD_UNIQUE));
+
+ // Check if we're trying to set a property with parameters that allow us to set the
+ // property on multiple replicas
+ // in a slice on properties that are known to only be one-per-slice and error out if so.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/core/SolrCore.java
##########
@@ -2631,9 +2864,12 @@ public void execute(SolrRequestHandler handler, SolrQueryRequest req, SolrQueryR
requestLog.debug(rsp.getToLogAsString());
}
- // TODO: this doesn't seem to be working correctly and causes problems with the example server and distrib (for example /spell)
- // if (req.getParams().getBool(ShardParams.IS_SHARD,false) && !(handler instanceof SearchHandler))
- // throw new SolrException(SolrException.ErrorCode.BAD_REQUEST,"isShard is only acceptable with search handlers");
+ // TODO: this doesn't seem to be working correctly and causes problems with the example server
+ // and distrib (for example /spell)
+ // if (req.getParams().getBool(ShardParams.IS_SHARD,false) && !(handler instanceof
+ // SearchHandler))
+ // throw new SolrException(SolrException.ErrorCode.BAD_REQUEST,"isShard is only acceptable
+ // with search handlers");
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/admin/CollectionsHandler.java
##########
@@ -388,982 +426,1324 @@ public Category getCategory() {
return Category.ADMIN;
}
- private static void createSysConfigSet(CoreContainer coreContainer) throws KeeperException, InterruptedException {
+ private static void createSysConfigSet(CoreContainer coreContainer)
+ throws KeeperException, InterruptedException {
SolrZkClient zk = coreContainer.getZkController().getZkStateReader().getZkClient();
ZkCmdExecutor cmdExecutor = new ZkCmdExecutor(zk.getZkClientTimeout());
cmdExecutor.ensureExists(ZkStateReader.CONFIGS_ZKNODE, zk);
- cmdExecutor.ensureExists(ZkStateReader.CONFIGS_ZKNODE + "/" + CollectionAdminParams.SYSTEM_COLL, zk);
+ cmdExecutor.ensureExists(
+ ZkStateReader.CONFIGS_ZKNODE + "/" + CollectionAdminParams.SYSTEM_COLL, zk);
try {
- String path = ZkStateReader.CONFIGS_ZKNODE + "/" + CollectionAdminParams.SYSTEM_COLL + "/schema.xml";
- byte[] data = IOUtils.toByteArray(CollectionsHandler.class.getResourceAsStream("/SystemCollectionSchema.xml"));
+ String path =
+ ZkStateReader.CONFIGS_ZKNODE + "/" + CollectionAdminParams.SYSTEM_COLL + "/schema.xml";
+ byte[] data =
+ IOUtils.toByteArray(
+ CollectionsHandler.class.getResourceAsStream("/SystemCollectionSchema.xml"));
assert data != null && data.length > 0;
cmdExecutor.ensureExists(path, data, CreateMode.PERSISTENT, zk);
- path = ZkStateReader.CONFIGS_ZKNODE + "/" + CollectionAdminParams.SYSTEM_COLL + "/solrconfig.xml";
- data = IOUtils.toByteArray(CollectionsHandler.class.getResourceAsStream("/SystemCollectionSolrConfig.xml"));
+ path =
+ ZkStateReader.CONFIGS_ZKNODE
+ + "/"
+ + CollectionAdminParams.SYSTEM_COLL
+ + "/solrconfig.xml";
+ data =
+ IOUtils.toByteArray(
+ CollectionsHandler.class.getResourceAsStream("/SystemCollectionSolrConfig.xml"));
assert data != null && data.length > 0;
cmdExecutor.ensureExists(path, data, CreateMode.PERSISTENT, zk);
} catch (IOException e) {
throw new SolrException(ErrorCode.SERVER_ERROR, e);
}
-
-
}
- private static void addStatusToResponse(NamedList<Object> results, RequestStatusState state, String msg) {
+ private static void addStatusToResponse(
+ NamedList<Object> results, RequestStatusState state, String msg) {
SimpleOrderedMap<String> status = new SimpleOrderedMap<>();
status.add("state", state.getKey());
status.add("msg", msg);
results.add("status", status);
}
public enum CollectionOperation implements CollectionOp {
- CREATE_OP(CREATE, (req, rsp, h) -> {
- Map<String, Object> props = copy(req.getParams().required(), null, NAME);
- props.put("fromApi", "true");
- copy(req.getParams(), props,
- REPLICATION_FACTOR,
- COLL_CONF,
- NUM_SLICES,
- CREATE_NODE_SET,
- CREATE_NODE_SET_SHUFFLE,
- SHARDS_PROP,
- PULL_REPLICAS,
- TLOG_REPLICAS,
- NRT_REPLICAS,
- WAIT_FOR_FINAL_STATE,
- PER_REPLICA_STATE,
- ALIAS);
-
- if (props.get(REPLICATION_FACTOR) != null && props.get(NRT_REPLICAS) != null) {
- //TODO: Remove this in 8.0 . Keep this for SolrJ client back-compat. See SOLR-11676 for more details
- int replicationFactor = Integer.parseInt((String) props.get(REPLICATION_FACTOR));
- int nrtReplicas = Integer.parseInt((String) props.get(NRT_REPLICAS));
- if (replicationFactor != nrtReplicas) {
- throw new SolrException(ErrorCode.BAD_REQUEST,
- "Cannot specify both replicationFactor and nrtReplicas as they mean the same thing");
- }
- }
- if (props.get(REPLICATION_FACTOR) != null) {
- props.put(NRT_REPLICAS, props.get(REPLICATION_FACTOR));
- } else if (props.get(NRT_REPLICAS) != null) {
- props.put(REPLICATION_FACTOR, props.get(NRT_REPLICAS));
- }
-
- final String collectionName = SolrIdentifierValidator.validateCollectionName((String) props.get(NAME));
- final String shardsParam = (String) props.get(SHARDS_PROP);
- if (StringUtils.isNotEmpty(shardsParam)) {
- verifyShardsParam(shardsParam);
- }
- if (CollectionAdminParams.SYSTEM_COLL.equals(collectionName)) {
- //We must always create a .system collection with only a single shard
- props.put(NUM_SLICES, 1);
- props.remove(SHARDS_PROP);
- createSysConfigSet(h.coreContainer);
-
- }
- if (shardsParam == null) h.copyFromClusterProp(props, NUM_SLICES);
- for (String prop : ImmutableSet.of(NRT_REPLICAS, PULL_REPLICAS, TLOG_REPLICAS))
- h.copyFromClusterProp(props, prop);
- copyPropertiesWithPrefix(req.getParams(), props, PROPERTY_PREFIX);
- return copyPropertiesWithPrefix(req.getParams(), props, "router.");
+ CREATE_OP(
+ CREATE,
+ (req, rsp, h) -> {
+ Map<String, Object> props = copy(req.getParams().required(), null, NAME);
+ props.put("fromApi", "true");
+ copy(
+ req.getParams(),
+ props,
+ REPLICATION_FACTOR,
+ COLL_CONF,
+ NUM_SLICES,
+ CREATE_NODE_SET,
+ CREATE_NODE_SET_SHUFFLE,
+ SHARDS_PROP,
+ PULL_REPLICAS,
+ TLOG_REPLICAS,
+ NRT_REPLICAS,
+ WAIT_FOR_FINAL_STATE,
+ PER_REPLICA_STATE,
+ ALIAS);
+
+ if (props.get(REPLICATION_FACTOR) != null && props.get(NRT_REPLICAS) != null) {
+ // TODO: Remove this in 8.0 . Keep this for SolrJ client back-compat. See SOLR-11676 for
+ // more details
+ int replicationFactor = Integer.parseInt((String) props.get(REPLICATION_FACTOR));
+ int nrtReplicas = Integer.parseInt((String) props.get(NRT_REPLICAS));
+ if (replicationFactor != nrtReplicas) {
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST,
+ "Cannot specify both replicationFactor and nrtReplicas as they mean the same thing");
+ }
+ }
+ if (props.get(REPLICATION_FACTOR) != null) {
+ props.put(NRT_REPLICAS, props.get(REPLICATION_FACTOR));
+ } else if (props.get(NRT_REPLICAS) != null) {
+ props.put(REPLICATION_FACTOR, props.get(NRT_REPLICAS));
+ }
- }),
+ final String collectionName =
+ SolrIdentifierValidator.validateCollectionName((String) props.get(NAME));
+ final String shardsParam = (String) props.get(SHARDS_PROP);
+ if (StringUtils.isNotEmpty(shardsParam)) {
+ verifyShardsParam(shardsParam);
+ }
+ if (CollectionAdminParams.SYSTEM_COLL.equals(collectionName)) {
+ // We must always create a .system collection with only a single shard
+ props.put(NUM_SLICES, 1);
+ props.remove(SHARDS_PROP);
+ createSysConfigSet(h.coreContainer);
+ }
+ if (shardsParam == null) h.copyFromClusterProp(props, NUM_SLICES);
+ for (String prop : ImmutableSet.of(NRT_REPLICAS, PULL_REPLICAS, TLOG_REPLICAS))
+ h.copyFromClusterProp(props, prop);
+ copyPropertiesWithPrefix(req.getParams(), props, PROPERTY_PREFIX);
+ return copyPropertiesWithPrefix(req.getParams(), props, "router.");
+ }),
@SuppressWarnings({"unchecked"})
- COLSTATUS_OP(COLSTATUS, (req, rsp, h) -> {
- Map<String, Object> props = copy(req.getParams(), null,
- COLLECTION_PROP,
- ColStatus.CORE_INFO_PROP,
- ColStatus.SEGMENTS_PROP,
- ColStatus.FIELD_INFO_PROP,
- ColStatus.RAW_SIZE_PROP,
- ColStatus.RAW_SIZE_SUMMARY_PROP,
- ColStatus.RAW_SIZE_DETAILS_PROP,
- ColStatus.RAW_SIZE_SAMPLING_PERCENT_PROP,
- ColStatus.SIZE_INFO_PROP);
-
- new ColStatus(h.coreContainer.getSolrClientCache(),
- h.coreContainer.getZkController().getZkStateReader().getClusterState(), new ZkNodeProps(props))
- .getColStatus(rsp.getValues());
- return null;
- }),
- DELETE_OP(DELETE, (req, rsp, h) -> {
- Map<String, Object> map = copy(req.getParams().required(), null, NAME);
- return copy(req.getParams(), map, FOLLOW_ALIASES);
- }),
+ COLSTATUS_OP(
+ COLSTATUS,
+ (req, rsp, h) -> {
+ Map<String, Object> props =
+ copy(
+ req.getParams(),
+ null,
+ COLLECTION_PROP,
+ ColStatus.CORE_INFO_PROP,
+ ColStatus.SEGMENTS_PROP,
+ ColStatus.FIELD_INFO_PROP,
+ ColStatus.RAW_SIZE_PROP,
+ ColStatus.RAW_SIZE_SUMMARY_PROP,
+ ColStatus.RAW_SIZE_DETAILS_PROP,
+ ColStatus.RAW_SIZE_SAMPLING_PERCENT_PROP,
+ ColStatus.SIZE_INFO_PROP);
+
+ new ColStatus(
+ h.coreContainer.getSolrClientCache(),
+ h.coreContainer.getZkController().getZkStateReader().getClusterState(),
+ new ZkNodeProps(props))
+ .getColStatus(rsp.getValues());
+ return null;
+ }),
+ DELETE_OP(
+ DELETE,
+ (req, rsp, h) -> {
+ Map<String, Object> map = copy(req.getParams().required(), null, NAME);
+ return copy(req.getParams(), map, FOLLOW_ALIASES);
+ }),
// XXX should this command support followAliases?
- RELOAD_OP(RELOAD, (req, rsp, h) -> {
- Map<String, Object> map = copy(req.getParams().required(), null, NAME);
- return copy(req.getParams(), map);
- }),
-
- RENAME_OP(RENAME, (req, rsp, h) -> {
- Map<String, Object> map = copy(req.getParams().required(), null, NAME, CollectionAdminParams.TARGET);
- return copy(req.getParams(), map, FOLLOW_ALIASES);
- }),
-
- REINDEXCOLLECTION_OP(REINDEXCOLLECTION, (req, rsp, h) -> {
- Map<String, Object> m = copy(req.getParams().required(), null, NAME);
- copy(req.getParams(), m,
- ReindexCollectionCmd.COMMAND,
- ReindexCollectionCmd.REMOVE_SOURCE,
- ReindexCollectionCmd.TARGET,
- ZkStateReader.CONFIGNAME_PROP,
- NUM_SLICES,
- NRT_REPLICAS,
- PULL_REPLICAS,
- TLOG_REPLICAS,
- REPLICATION_FACTOR,
- CREATE_NODE_SET,
- CREATE_NODE_SET_SHUFFLE,
- "shards",
- CommonParams.ROWS,
- CommonParams.Q,
- CommonParams.FL,
- FOLLOW_ALIASES);
- if (req.getParams().get("collection." + ZkStateReader.CONFIGNAME_PROP) != null) {
- m.put(ZkStateReader.CONFIGNAME_PROP, req.getParams().get("collection." + ZkStateReader.CONFIGNAME_PROP));
- }
- copyPropertiesWithPrefix(req.getParams(), m, "router.");
- return m;
- }),
-
- SYNCSHARD_OP(SYNCSHARD, (req, rsp, h) -> {
- String extCollection = req.getParams().required().get("collection");
- String collection = h.coreContainer.getZkController().getZkStateReader().getAliases().resolveSimpleAlias(extCollection);
- String shard = req.getParams().required().get("shard");
-
- ClusterState clusterState = h.coreContainer.getZkController().getClusterState();
-
- DocCollection docCollection = clusterState.getCollection(collection);
- ZkNodeProps leaderProps = docCollection.getLeader(shard);
- ZkCoreNodeProps nodeProps = new ZkCoreNodeProps(leaderProps);
-
- try (HttpSolrClient client = new Builder(nodeProps.getBaseUrl())
- .withConnectionTimeout(15000)
- .withSocketTimeout(60000)
- .build()) {
- RequestSyncShard reqSyncShard = new RequestSyncShard();
- reqSyncShard.setCollection(collection);
- reqSyncShard.setShard(shard);
- reqSyncShard.setCoreName(nodeProps.getCoreName());
- client.request(reqSyncShard);
- }
- return null;
- }),
-
- CREATEALIAS_OP(CREATEALIAS, (req, rsp, h) -> {
- String alias = req.getParams().get(NAME);
- SolrIdentifierValidator.validateAliasName(alias);
- String collections = req.getParams().get("collections");
- RoutedAlias routedAlias = null;
- Exception ex = null;
- HashMap<String,Object> possiblyModifiedParams = new HashMap<>();
- try {
- // note that RA specific validation occurs here.
- req.getParams().toMap(possiblyModifiedParams);
- @SuppressWarnings({"unchecked", "rawtypes"})
- // This is awful because RoutedAlias lies about what types it wants
- Map<String, String> temp = (Map<String, String>) (Map) possiblyModifiedParams;
- routedAlias = RoutedAlias.fromProps(alias, temp);
- } catch (SolrException e) {
- // we'll throw this later if we are in fact creating a routed alias.
- ex = e;
- }
- ModifiableSolrParams finalParams = new ModifiableSolrParams();
- for (Map.Entry<String, Object> entry : possiblyModifiedParams.entrySet()) {
- if (entry.getValue().getClass().isArray() ) {
- // v2 api hits this case
- for (Object o : (Object[]) entry.getValue()) {
- finalParams.add(entry.getKey(),o.toString());
+ RELOAD_OP(
+ RELOAD,
+ (req, rsp, h) -> {
+ Map<String, Object> map = copy(req.getParams().required(), null, NAME);
+ return copy(req.getParams(), map);
+ }),
+
+ RENAME_OP(
+ RENAME,
+ (req, rsp, h) -> {
+ Map<String, Object> map =
+ copy(req.getParams().required(), null, NAME, CollectionAdminParams.TARGET);
+ return copy(req.getParams(), map, FOLLOW_ALIASES);
+ }),
+
+ REINDEXCOLLECTION_OP(
+ REINDEXCOLLECTION,
+ (req, rsp, h) -> {
+ Map<String, Object> m = copy(req.getParams().required(), null, NAME);
+ copy(
+ req.getParams(),
+ m,
+ ReindexCollectionCmd.COMMAND,
+ ReindexCollectionCmd.REMOVE_SOURCE,
+ ReindexCollectionCmd.TARGET,
+ ZkStateReader.CONFIGNAME_PROP,
+ NUM_SLICES,
+ NRT_REPLICAS,
+ PULL_REPLICAS,
+ TLOG_REPLICAS,
+ REPLICATION_FACTOR,
+ CREATE_NODE_SET,
+ CREATE_NODE_SET_SHUFFLE,
+ "shards",
+ CommonParams.ROWS,
+ CommonParams.Q,
+ CommonParams.FL,
+ FOLLOW_ALIASES);
+ if (req.getParams().get("collection." + ZkStateReader.CONFIGNAME_PROP) != null) {
+ m.put(
+ ZkStateReader.CONFIGNAME_PROP,
+ req.getParams().get("collection." + ZkStateReader.CONFIGNAME_PROP));
+ }
+ copyPropertiesWithPrefix(req.getParams(), m, "router.");
+ return m;
+ }),
+
+ SYNCSHARD_OP(
+ SYNCSHARD,
+ (req, rsp, h) -> {
+ String extCollection = req.getParams().required().get("collection");
+ String collection =
+ h.coreContainer
+ .getZkController()
+ .getZkStateReader()
+ .getAliases()
+ .resolveSimpleAlias(extCollection);
+ String shard = req.getParams().required().get("shard");
+
+ ClusterState clusterState = h.coreContainer.getZkController().getClusterState();
+
+ DocCollection docCollection = clusterState.getCollection(collection);
+ ZkNodeProps leaderProps = docCollection.getLeader(shard);
+ ZkCoreNodeProps nodeProps = new ZkCoreNodeProps(leaderProps);
+
+ try (HttpSolrClient client =
+ new Builder(nodeProps.getBaseUrl())
+ .withConnectionTimeout(15000)
+ .withSocketTimeout(60000)
+ .build()) {
+ RequestSyncShard reqSyncShard = new RequestSyncShard();
+ reqSyncShard.setCollection(collection);
+ reqSyncShard.setShard(shard);
+ reqSyncShard.setCoreName(nodeProps.getCoreName());
+ client.request(reqSyncShard);
+ }
+ return null;
+ }),
+
+ CREATEALIAS_OP(
+ CREATEALIAS,
+ (req, rsp, h) -> {
+ String alias = req.getParams().get(NAME);
+ SolrIdentifierValidator.validateAliasName(alias);
+ String collections = req.getParams().get("collections");
+ RoutedAlias routedAlias = null;
+ Exception ex = null;
+ HashMap<String, Object> possiblyModifiedParams = new HashMap<>();
+ try {
+ // note that RA specific validation occurs here.
+ req.getParams().toMap(possiblyModifiedParams);
+ @SuppressWarnings({"unchecked", "rawtypes"})
+ // This is awful because RoutedAlias lies about what types it wants
+ Map<String, String> temp = (Map<String, String>) (Map) possiblyModifiedParams;
+ routedAlias = RoutedAlias.fromProps(alias, temp);
+ } catch (SolrException e) {
+ // we'll throw this later if we are in fact creating a routed alias.
+ ex = e;
+ }
+ ModifiableSolrParams finalParams = new ModifiableSolrParams();
+ for (Map.Entry<String, Object> entry : possiblyModifiedParams.entrySet()) {
+ if (entry.getValue().getClass().isArray()) {
+ // v2 api hits this case
+ for (Object o : (Object[]) entry.getValue()) {
+ finalParams.add(entry.getKey(), o.toString());
+ }
+ } else {
+ finalParams.add(entry.getKey(), entry.getValue().toString());
+ }
}
- } else {
- finalParams.add(entry.getKey(),entry.getValue().toString());
- }
- }
- if (collections != null) {
- if (routedAlias != null) {
- throw new SolrException(BAD_REQUEST, "Collections cannot be specified when creating a routed alias.");
- } else {
- //////////////////////////////////////
- // Regular alias creation indicated //
- //////////////////////////////////////
- return copy(finalParams.required(), null, NAME, "collections");
- }
- }
+ if (collections != null) {
+ if (routedAlias != null) {
+ throw new SolrException(
+ BAD_REQUEST, "Collections cannot be specified when creating a routed alias.");
+ } else {
+ //////////////////////////////////////
+ // Regular alias creation indicated //
+ //////////////////////////////////////
+ return copy(finalParams.required(), null, NAME, "collections");
+ }
+ }
- /////////////////////////////////////////////////
- // We are creating a routed alias from here on //
- /////////////////////////////////////////////////
+ /////////////////////////////////////////////////
+ // We are creating a routed alias from here on //
+ /////////////////////////////////////////////////
- // If our prior creation attempt had issues expose them now.
- if (ex != null) {
- throw ex;
- }
+ // If our prior creation attempt had issues expose them now.
+ if (ex != null) {
+ throw ex;
+ }
- // Now filter out just the parameters we care about from the request
- assert routedAlias != null;
- Map<String, Object> result = copy(finalParams, null, routedAlias.getRequiredParams());
- copy(finalParams, result, routedAlias.getOptionalParams());
-
- ModifiableSolrParams createCollParams = new ModifiableSolrParams(); // without prefix
-
- // add to result params that start with "create-collection.".
- // Additionally, save these without the prefix to createCollParams
- for (Map.Entry<String, String[]> entry : finalParams) {
- final String p = entry.getKey();
- if (p.startsWith(CREATE_COLLECTION_PREFIX)) {
- // This is what SolrParams#getAll(Map, Collection)} does
- final String[] v = entry.getValue();
- if (v.length == 1) {
- result.put(p, v[0]);
- } else {
- result.put(p, v);
+ // Now filter out just the parameters we care about from the request
+ assert routedAlias != null;
+ Map<String, Object> result = copy(finalParams, null, routedAlias.getRequiredParams());
+ copy(finalParams, result, routedAlias.getOptionalParams());
+
+ ModifiableSolrParams createCollParams = new ModifiableSolrParams(); // without prefix
+
+ // add to result params that start with "create-collection.".
+ // Additionally, save these without the prefix to createCollParams
+ for (Map.Entry<String, String[]> entry : finalParams) {
+ final String p = entry.getKey();
+ if (p.startsWith(CREATE_COLLECTION_PREFIX)) {
+ // This is what SolrParams#getAll(Map, Collection)} does
+ final String[] v = entry.getValue();
+ if (v.length == 1) {
+ result.put(p, v[0]);
+ } else {
+ result.put(p, v);
+ }
+ createCollParams.set(p.substring(CREATE_COLLECTION_PREFIX.length()), v);
+ }
}
- createCollParams.set(p.substring(CREATE_COLLECTION_PREFIX.length()), v);
- }
- }
- // Verify that the create-collection prefix'ed params appear to be valid.
- if (createCollParams.get(NAME) != null) {
- throw new SolrException(BAD_REQUEST, "routed aliases calculate names for their " +
- "dependent collections, you cannot specify the name.");
- }
- if (createCollParams.get(COLL_CONF) == null) {
- throw new SolrException(SolrException.ErrorCode.BAD_REQUEST,
- "We require an explicit " + COLL_CONF);
- }
- // note: could insist on a config name here as well.... or wait to throw at overseer
- createCollParams.add(NAME, "TMP_name_TMP_name_TMP"); // just to pass validation
- CREATE_OP.execute(new LocalSolrQueryRequest(null, createCollParams), rsp, h); // ignore results
+ // Verify that the create-collection prefix'ed params appear to be valid.
+ if (createCollParams.get(NAME) != null) {
+ throw new SolrException(
+ BAD_REQUEST,
+ "routed aliases calculate names for their "
+ + "dependent collections, you cannot specify the name.");
+ }
+ if (createCollParams.get(COLL_CONF) == null) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST, "We require an explicit " + COLL_CONF);
+ }
+ // note: could insist on a config name here as well.... or wait to throw at overseer
+ createCollParams.add(NAME, "TMP_name_TMP_name_TMP"); // just to pass validation
+ CREATE_OP.execute(
+ new LocalSolrQueryRequest(null, createCollParams), rsp, h); // ignore results
- return result;
- }),
+ return result;
+ }),
DELETEALIAS_OP(DELETEALIAS, (req, rsp, h) -> copy(req.getParams().required(), null, NAME)),
/**
* Change properties for an alias (use CREATEALIAS_OP to change the actual value of the alias)
*/
- ALIASPROP_OP(ALIASPROP, (req, rsp, h) -> {
- Map<String, Object> params = copy(req.getParams().required(), null, NAME);
-
- // Note: success/no-op in the event of no properties supplied is intentional. Keeps code simple and one less case
- // for api-callers to check for.
- return convertPrefixToMap(req.getParams(), params, "property");
- }),
-
- /**
- * List the aliases and associated properties.
- */
+ ALIASPROP_OP(
+ ALIASPROP,
+ (req, rsp, h) -> {
+ Map<String, Object> params = copy(req.getParams().required(), null, NAME);
+
+ // Note: success/no-op in the event of no properties supplied is intentional. Keeps code
+ // simple and one less case
+ // for api-callers to check for.
+ return convertPrefixToMap(req.getParams(), params, "property");
+ }),
+
+ /** List the aliases and associated properties. */
@SuppressWarnings({"unchecked"})
- LISTALIASES_OP(LISTALIASES, (req, rsp, h) -> {
- ZkStateReader zkStateReader = h.coreContainer.getZkController().getZkStateReader();
- // if someone calls listAliases, lets ensure we return an up to date response
- zkStateReader.aliasesManager.update();
- Aliases aliases = zkStateReader.getAliases();
- if (aliases != null) {
- // the aliases themselves...
- rsp.getValues().add("aliases", aliases.getCollectionAliasMap());
- // Any properties for the above aliases.
- Map<String, Map<String, String>> meta = new LinkedHashMap<>();
- for (String alias : aliases.getCollectionAliasListMap().keySet()) {
- Map<String, String> collectionAliasProperties = aliases.getCollectionAliasProperties(alias);
- if (!collectionAliasProperties.isEmpty()) {
- meta.put(alias, collectionAliasProperties);
+ LISTALIASES_OP(
+ LISTALIASES,
+ (req, rsp, h) -> {
+ ZkStateReader zkStateReader = h.coreContainer.getZkController().getZkStateReader();
+ // if someone calls listAliases, lets ensure we return an up to date response
+ zkStateReader.aliasesManager.update();
+ Aliases aliases = zkStateReader.getAliases();
+ if (aliases != null) {
+ // the aliases themselves...
+ rsp.getValues().add("aliases", aliases.getCollectionAliasMap());
+ // Any properties for the above aliases.
+ Map<String, Map<String, String>> meta = new LinkedHashMap<>();
+ for (String alias : aliases.getCollectionAliasListMap().keySet()) {
+ Map<String, String> collectionAliasProperties =
+ aliases.getCollectionAliasProperties(alias);
+ if (!collectionAliasProperties.isEmpty()) {
+ meta.put(alias, collectionAliasProperties);
+ }
+ }
+ rsp.getValues().add("properties", meta);
+ }
+ return null;
+ }),
+ SPLITSHARD_OP(
+ SPLITSHARD,
+ DEFAULT_COLLECTION_OP_TIMEOUT * 5,
+ (req, rsp, h) -> {
+ String name = req.getParams().required().get(COLLECTION_PROP);
+ // TODO : add support for multiple shards
+ String shard = req.getParams().get(SHARD_ID_PROP);
+ String rangesStr = req.getParams().get(CoreAdminParams.RANGES);
+ String splitKey = req.getParams().get("split.key");
+ String numSubShards = req.getParams().get(NUM_SUB_SHARDS);
+ String fuzz = req.getParams().get(SPLIT_FUZZ);
+
+ if (splitKey == null && shard == null) {
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST, "At least one of shard, or split.key should be specified.");
+ }
+ if (splitKey != null && shard != null) {
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST, "Only one of 'shard' or 'split.key' should be specified");
+ }
+ if (splitKey != null && rangesStr != null) {
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST, "Only one of 'ranges' or 'split.key' should be specified");
+ }
+ if (numSubShards != null && (splitKey != null || rangesStr != null)) {
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST,
+ "numSubShards can not be specified with split.key or ranges parameters");
+ }
+ if (fuzz != null && (splitKey != null || rangesStr != null)) {
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST,
+ "fuzz can not be specified with split.key or ranges parameters");
}
- }
- rsp.getValues().add("properties", meta);
- }
- return null;
- }),
- SPLITSHARD_OP(SPLITSHARD, DEFAULT_COLLECTION_OP_TIMEOUT * 5, (req, rsp, h) -> {
- String name = req.getParams().required().get(COLLECTION_PROP);
- // TODO : add support for multiple shards
- String shard = req.getParams().get(SHARD_ID_PROP);
- String rangesStr = req.getParams().get(CoreAdminParams.RANGES);
- String splitKey = req.getParams().get("split.key");
- String numSubShards = req.getParams().get(NUM_SUB_SHARDS);
- String fuzz = req.getParams().get(SPLIT_FUZZ);
-
- if (splitKey == null && shard == null) {
- throw new SolrException(ErrorCode.BAD_REQUEST, "At least one of shard, or split.key should be specified.");
- }
- if (splitKey != null && shard != null) {
- throw new SolrException(ErrorCode.BAD_REQUEST,
- "Only one of 'shard' or 'split.key' should be specified");
- }
- if (splitKey != null && rangesStr != null) {
- throw new SolrException(ErrorCode.BAD_REQUEST,
- "Only one of 'ranges' or 'split.key' should be specified");
- }
- if (numSubShards != null && (splitKey != null || rangesStr != null)) {
- throw new SolrException(ErrorCode.BAD_REQUEST,
- "numSubShards can not be specified with split.key or ranges parameters");
- }
- if (fuzz != null && (splitKey != null || rangesStr != null)) {
- throw new SolrException(ErrorCode.BAD_REQUEST,
- "fuzz can not be specified with split.key or ranges parameters");
- }
- Map<String, Object> map = copy(req.getParams(), null,
- COLLECTION_PROP,
- SHARD_ID_PROP,
- "split.key",
- CoreAdminParams.RANGES,
- WAIT_FOR_FINAL_STATE,
- TIMING,
- SPLIT_METHOD,
- NUM_SUB_SHARDS,
- SPLIT_FUZZ,
- SPLIT_BY_PREFIX,
- FOLLOW_ALIASES);
- return copyPropertiesWithPrefix(req.getParams(), map, PROPERTY_PREFIX);
- }),
- DELETESHARD_OP(DELETESHARD, (req, rsp, h) -> {
- Map<String, Object> map = copy(req.getParams().required(), null,
- COLLECTION_PROP,
- SHARD_ID_PROP);
- copy(req.getParams(), map,
- DELETE_INDEX,
- DELETE_DATA_DIR,
- DELETE_INSTANCE_DIR,
- FOLLOW_ALIASES);
- return map;
- }),
- FORCELEADER_OP(FORCELEADER, (req, rsp, h) -> {
- forceLeaderElection(req, h);
- return null;
- }),
- CREATESHARD_OP(CREATESHARD, (req, rsp, h) -> {
- Map<String, Object> map = copy(req.getParams().required(), null,
- COLLECTION_PROP,
- SHARD_ID_PROP);
- ClusterState clusterState = h.coreContainer.getZkController().getClusterState();
- final String newShardName = SolrIdentifierValidator.validateShardName(req.getParams().get(SHARD_ID_PROP));
- boolean followAliases = req.getParams().getBool(FOLLOW_ALIASES, false);
- String extCollectionName = req.getParams().get(COLLECTION_PROP);
- String collectionName = followAliases ? h.coreContainer.getZkController().getZkStateReader()
- .getAliases().resolveSimpleAlias(extCollectionName) : extCollectionName;
- if (!ImplicitDocRouter.NAME.equals(((Map<?,?>) clusterState.getCollection(collectionName).get(DOC_ROUTER)).get(NAME)))
- throw new SolrException(ErrorCode.BAD_REQUEST, "shards can be added only to 'implicit' collections");
- copy(req.getParams(), map,
- REPLICATION_FACTOR,
- NRT_REPLICAS,
- TLOG_REPLICAS,
- PULL_REPLICAS,
- CREATE_NODE_SET,
- WAIT_FOR_FINAL_STATE,
- FOLLOW_ALIASES);
- return copyPropertiesWithPrefix(req.getParams(), map, PROPERTY_PREFIX);
- }),
- DELETEREPLICA_OP(DELETEREPLICA, (req, rsp, h) -> {
- Map<String, Object> map = copy(req.getParams().required(), null,
- COLLECTION_PROP);
-
- return copy(req.getParams(), map,
- DELETE_INDEX,
- DELETE_DATA_DIR,
- DELETE_INSTANCE_DIR,
- COUNT_PROP, REPLICA_PROP,
- SHARD_ID_PROP,
- ONLY_IF_DOWN,
- FOLLOW_ALIASES);
- }),
- MIGRATE_OP(MIGRATE, (req, rsp, h) -> {
- Map<String, Object> map = copy(req.getParams().required(), null, COLLECTION_PROP, "split.key", "target.collection");
- return copy(req.getParams(), map, "forward.timeout", FOLLOW_ALIASES);
- }),
- ADDROLE_OP(ADDROLE, (req, rsp, h) -> {
- Map<String, Object> map = copy(req.getParams().required(), null, "role", "node");
- if (!KNOWN_ROLES.contains(map.get("role")))
- throw new SolrException(ErrorCode.BAD_REQUEST, "Unknown role. Supported roles are ," + KNOWN_ROLES);
- return map;
- }),
- REMOVEROLE_OP(REMOVEROLE, (req, rsp, h) -> {
- Map<String, Object> map = copy(req.getParams().required(), null, "role", "node");
- if (!KNOWN_ROLES.contains(map.get("role")))
- throw new SolrException(ErrorCode.BAD_REQUEST, "Unknown role. Supported roles are ," + KNOWN_ROLES);
- return map;
- }),
- CLUSTERPROP_OP(CLUSTERPROP, (req, rsp, h) -> {
- String name = req.getParams().required().get(NAME);
- String val = req.getParams().get(VALUE_LONG);
- ClusterProperties cp = new ClusterProperties(h.coreContainer.getZkController().getZkClient());
- cp.setClusterProperty(name, val);
- return null;
- }),
- COLLECTIONPROP_OP(COLLECTIONPROP, (req, rsp, h) -> {
- String extCollection = req.getParams().required().get(NAME);
- String collection = h.coreContainer.getZkController().getZkStateReader().getAliases().resolveSimpleAlias(extCollection);
- String name = req.getParams().required().get(PROPERTY_NAME);
- String val = req.getParams().get(PROPERTY_VALUE);
- CollectionProperties cp = new CollectionProperties(h.coreContainer.getZkController().getZkClient());
- cp.setCollectionProperty(collection, name, val);
- return null;
- }),
+ Map<String, Object> map =
+ copy(
+ req.getParams(),
+ null,
+ COLLECTION_PROP,
+ SHARD_ID_PROP,
+ "split.key",
+ CoreAdminParams.RANGES,
+ WAIT_FOR_FINAL_STATE,
+ TIMING,
+ SPLIT_METHOD,
+ NUM_SUB_SHARDS,
+ SPLIT_FUZZ,
+ SPLIT_BY_PREFIX,
+ FOLLOW_ALIASES);
+ return copyPropertiesWithPrefix(req.getParams(), map, PROPERTY_PREFIX);
+ }),
+ DELETESHARD_OP(
+ DELETESHARD,
+ (req, rsp, h) -> {
+ Map<String, Object> map =
+ copy(req.getParams().required(), null, COLLECTION_PROP, SHARD_ID_PROP);
+ copy(
+ req.getParams(),
+ map,
+ DELETE_INDEX,
+ DELETE_DATA_DIR,
+ DELETE_INSTANCE_DIR,
+ FOLLOW_ALIASES);
+ return map;
+ }),
+ FORCELEADER_OP(
+ FORCELEADER,
+ (req, rsp, h) -> {
+ forceLeaderElection(req, h);
+ return null;
+ }),
+ CREATESHARD_OP(
+ CREATESHARD,
+ (req, rsp, h) -> {
+ Map<String, Object> map =
+ copy(req.getParams().required(), null, COLLECTION_PROP, SHARD_ID_PROP);
+ ClusterState clusterState = h.coreContainer.getZkController().getClusterState();
+ final String newShardName =
+ SolrIdentifierValidator.validateShardName(req.getParams().get(SHARD_ID_PROP));
+ boolean followAliases = req.getParams().getBool(FOLLOW_ALIASES, false);
+ String extCollectionName = req.getParams().get(COLLECTION_PROP);
+ String collectionName =
+ followAliases
+ ? h.coreContainer
+ .getZkController()
+ .getZkStateReader()
+ .getAliases()
+ .resolveSimpleAlias(extCollectionName)
+ : extCollectionName;
+ if (!ImplicitDocRouter.NAME.equals(
+ ((Map<?, ?>) clusterState.getCollection(collectionName).get(DOC_ROUTER)).get(NAME)))
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST, "shards can be added only to 'implicit' collections");
+ copy(
+ req.getParams(),
+ map,
+ REPLICATION_FACTOR,
+ NRT_REPLICAS,
+ TLOG_REPLICAS,
+ PULL_REPLICAS,
+ CREATE_NODE_SET,
+ WAIT_FOR_FINAL_STATE,
+ FOLLOW_ALIASES);
+ return copyPropertiesWithPrefix(req.getParams(), map, PROPERTY_PREFIX);
+ }),
+ DELETEREPLICA_OP(
+ DELETEREPLICA,
+ (req, rsp, h) -> {
+ Map<String, Object> map = copy(req.getParams().required(), null, COLLECTION_PROP);
+
+ return copy(
+ req.getParams(),
+ map,
+ DELETE_INDEX,
+ DELETE_DATA_DIR,
+ DELETE_INSTANCE_DIR,
+ COUNT_PROP,
+ REPLICA_PROP,
+ SHARD_ID_PROP,
+ ONLY_IF_DOWN,
+ FOLLOW_ALIASES);
+ }),
+ MIGRATE_OP(
+ MIGRATE,
+ (req, rsp, h) -> {
+ Map<String, Object> map =
+ copy(
+ req.getParams().required(),
+ null,
+ COLLECTION_PROP,
+ "split.key",
+ "target.collection");
+ return copy(req.getParams(), map, "forward.timeout", FOLLOW_ALIASES);
+ }),
+ ADDROLE_OP(
+ ADDROLE,
+ (req, rsp, h) -> {
+ Map<String, Object> map = copy(req.getParams().required(), null, "role", "node");
+ if (!KNOWN_ROLES.contains(map.get("role")))
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST, "Unknown role. Supported roles are ," + KNOWN_ROLES);
+ return map;
+ }),
+ REMOVEROLE_OP(
+ REMOVEROLE,
+ (req, rsp, h) -> {
+ Map<String, Object> map = copy(req.getParams().required(), null, "role", "node");
+ if (!KNOWN_ROLES.contains(map.get("role")))
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST, "Unknown role. Supported roles are ," + KNOWN_ROLES);
+ return map;
+ }),
+ CLUSTERPROP_OP(
+ CLUSTERPROP,
+ (req, rsp, h) -> {
+ String name = req.getParams().required().get(NAME);
+ String val = req.getParams().get(VALUE_LONG);
+ ClusterProperties cp =
+ new ClusterProperties(h.coreContainer.getZkController().getZkClient());
+ cp.setClusterProperty(name, val);
+ return null;
+ }),
+ COLLECTIONPROP_OP(
+ COLLECTIONPROP,
+ (req, rsp, h) -> {
+ String extCollection = req.getParams().required().get(NAME);
+ String collection =
+ h.coreContainer
+ .getZkController()
+ .getZkStateReader()
+ .getAliases()
+ .resolveSimpleAlias(extCollection);
+ String name = req.getParams().required().get(PROPERTY_NAME);
+ String val = req.getParams().get(PROPERTY_VALUE);
+ CollectionProperties cp =
+ new CollectionProperties(h.coreContainer.getZkController().getZkClient());
+ cp.setCollectionProperty(collection, name, val);
+ return null;
+ }),
@SuppressWarnings({"unchecked"})
- REQUESTSTATUS_OP(REQUESTSTATUS, (req, rsp, h) -> {
- req.getParams().required().check(REQUESTID);
-
- final CoreContainer coreContainer = h.coreContainer;
- final String requestId = req.getParams().get(REQUESTID);
- final ZkController zkController = coreContainer.getZkController();
-
- final NamedList<Object> status = new NamedList<>();
- if (coreContainer.getDistributedCollectionCommandRunner().isEmpty()) {
- if (zkController.getOverseerCompletedMap().contains(requestId)) {
- final byte[] mapEntry = zkController.getOverseerCompletedMap().get(requestId);
- rsp.getValues().addAll(OverseerSolrResponseSerializer.deserialize(mapEntry).getResponse());
- addStatusToResponse(status, COMPLETED, "found [" + requestId + "] in completed tasks");
- } else if (zkController.getOverseerFailureMap().contains(requestId)) {
- final byte[] mapEntry = zkController.getOverseerFailureMap().get(requestId);
- rsp.getValues().addAll(OverseerSolrResponseSerializer.deserialize(mapEntry).getResponse());
- addStatusToResponse(status, FAILED, "found [" + requestId + "] in failed tasks");
- } else if (zkController.getOverseerRunningMap().contains(requestId)) {
- addStatusToResponse(status, RUNNING, "found [" + requestId + "] in running tasks");
- } else if (h.overseerCollectionQueueContains(requestId)) {
- addStatusToResponse(status, SUBMITTED, "found [" + requestId + "] in submitted tasks");
- } else {
- addStatusToResponse(status, NOT_FOUND, "Did not find [" + requestId + "] in any tasks queue");
- }
- } else {
- Pair<RequestStatusState, OverseerSolrResponse> sr =
- coreContainer.getDistributedCollectionCommandRunner().get().getAsyncTaskRequestStatus(requestId);
- final String message;
- switch (sr.first()) {
- case COMPLETED:
- message = "found [" + requestId + "] in completed tasks";
- rsp.getValues().addAll(sr.second().getResponse());
- break;
- case FAILED:
- message = "found [" + requestId + "] in failed tasks";
- rsp.getValues().addAll(sr.second().getResponse());
- break;
- case RUNNING:
- message = "found [" + requestId + "] in running tasks";
- break;
- case SUBMITTED:
- message = "found [" + requestId + "] in submitted tasks";
- break;
- default:
- message = "Did not find [" + requestId + "] in any tasks queue";
- }
- addStatusToResponse(status, sr.first(), message);
- }
-
- rsp.getValues().addAll(status);
- return null;
- }),
- DELETESTATUS_OP(DELETESTATUS, new CollectionOp() {
- @SuppressWarnings("unchecked")
- @Override
- public Map<String, Object> execute(SolrQueryRequest req, SolrQueryResponse rsp, CollectionsHandler h) throws Exception {
- final CoreContainer coreContainer = h.coreContainer;
- final String requestId = req.getParams().get(REQUESTID);
- final ZkController zkController = coreContainer.getZkController();
- Boolean flush = req.getParams().getBool(CollectionAdminParams.FLUSH, false);
-
- if (requestId == null && !flush) {
- throw new SolrException(ErrorCode.BAD_REQUEST, "Either requestid or flush parameter must be specified.");
- }
-
- if (requestId != null && flush) {
- throw new SolrException(ErrorCode.BAD_REQUEST,
- "Both requestid and flush parameters can not be specified together.");
- }
-
- if (coreContainer.getDistributedCollectionCommandRunner().isEmpty()) {
- if (flush) {
- Collection<String> completed = zkController.getOverseerCompletedMap().keys();
- Collection<String> failed = zkController.getOverseerFailureMap().keys();
- for (String asyncId : completed) {
- zkController.getOverseerCompletedMap().remove(asyncId);
- zkController.clearAsyncId(asyncId);
- }
- for (String asyncId : failed) {
- zkController.getOverseerFailureMap().remove(asyncId);
- zkController.clearAsyncId(asyncId);
+ REQUESTSTATUS_OP(
+ REQUESTSTATUS,
+ (req, rsp, h) -> {
+ req.getParams().required().check(REQUESTID);
+
+ final CoreContainer coreContainer = h.coreContainer;
+ final String requestId = req.getParams().get(REQUESTID);
+ final ZkController zkController = coreContainer.getZkController();
+
+ final NamedList<Object> status = new NamedList<>();
+ if (coreContainer.getDistributedCollectionCommandRunner().isEmpty()) {
+ if (zkController.getOverseerCompletedMap().contains(requestId)) {
+ final byte[] mapEntry = zkController.getOverseerCompletedMap().get(requestId);
+ rsp.getValues()
+ .addAll(OverseerSolrResponseSerializer.deserialize(mapEntry).getResponse());
+ addStatusToResponse(
+ status, COMPLETED, "found [" + requestId + "] in completed tasks");
+ } else if (zkController.getOverseerFailureMap().contains(requestId)) {
+ final byte[] mapEntry = zkController.getOverseerFailureMap().get(requestId);
+ rsp.getValues()
+ .addAll(OverseerSolrResponseSerializer.deserialize(mapEntry).getResponse());
+ addStatusToResponse(status, FAILED, "found [" + requestId + "] in failed tasks");
+ } else if (zkController.getOverseerRunningMap().contains(requestId)) {
+ addStatusToResponse(status, RUNNING, "found [" + requestId + "] in running tasks");
+ } else if (h.overseerCollectionQueueContains(requestId)) {
+ addStatusToResponse(
+ status, SUBMITTED, "found [" + requestId + "] in submitted tasks");
+ } else {
+ addStatusToResponse(
+ status, NOT_FOUND, "Did not find [" + requestId + "] in any tasks queue");
}
- rsp.getValues().add("status", "successfully cleared stored collection api responses");
} else {
- // Request to cleanup
- if (zkController.getOverseerCompletedMap().remove(requestId)) {
- zkController.clearAsyncId(requestId);
- rsp.getValues().add("status", "successfully removed stored response for [" + requestId + "]");
- } else if (zkController.getOverseerFailureMap().remove(requestId)) {
- zkController.clearAsyncId(requestId);
- rsp.getValues().add("status", "successfully removed stored response for [" + requestId + "]");
- } else {
- rsp.getValues().add("status", "[" + requestId + "] not found in stored responses");
- // Don't call zkController.clearAsyncId for this, since it could be a running/pending task
+ Pair<RequestStatusState, OverseerSolrResponse> sr =
+ coreContainer
+ .getDistributedCollectionCommandRunner()
+ .get()
+ .getAsyncTaskRequestStatus(requestId);
+ final String message;
+ switch (sr.first()) {
+ case COMPLETED:
+ message = "found [" + requestId + "] in completed tasks";
+ rsp.getValues().addAll(sr.second().getResponse());
+ break;
+ case FAILED:
+ message = "found [" + requestId + "] in failed tasks";
+ rsp.getValues().addAll(sr.second().getResponse());
+ break;
+ case RUNNING:
+ message = "found [" + requestId + "] in running tasks";
+ break;
+ case SUBMITTED:
+ message = "found [" + requestId + "] in submitted tasks";
+ break;
+ default:
+ message = "Did not find [" + requestId + "] in any tasks queue";
}
+ addStatusToResponse(status, sr.first(), message);
}
- } else {
- if (flush) {
- coreContainer.getDistributedCollectionCommandRunner().get().deleteAllAsyncIds();
- rsp.getValues().add("status", "successfully cleared stored collection api responses");
- } else {
- if (coreContainer.getDistributedCollectionCommandRunner().get().deleteSingleAsyncId(requestId)) {
- rsp.getValues().add("status", "successfully removed stored response for [" + requestId + "]");
+
+ rsp.getValues().addAll(status);
+ return null;
+ }),
+ DELETESTATUS_OP(
+ DELETESTATUS,
+ new CollectionOp() {
+ @SuppressWarnings("unchecked")
+ @Override
+ public Map<String, Object> execute(
+ SolrQueryRequest req, SolrQueryResponse rsp, CollectionsHandler h) throws Exception {
+ final CoreContainer coreContainer = h.coreContainer;
+ final String requestId = req.getParams().get(REQUESTID);
+ final ZkController zkController = coreContainer.getZkController();
+ Boolean flush = req.getParams().getBool(CollectionAdminParams.FLUSH, false);
+
+ if (requestId == null && !flush) {
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST, "Either requestid or flush parameter must be specified.");
+ }
+
+ if (requestId != null && flush) {
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST,
+ "Both requestid and flush parameters can not be specified together.");
+ }
+
+ if (coreContainer.getDistributedCollectionCommandRunner().isEmpty()) {
+ if (flush) {
+ Collection<String> completed = zkController.getOverseerCompletedMap().keys();
+ Collection<String> failed = zkController.getOverseerFailureMap().keys();
+ for (String asyncId : completed) {
+ zkController.getOverseerCompletedMap().remove(asyncId);
+ zkController.clearAsyncId(asyncId);
+ }
+ for (String asyncId : failed) {
+ zkController.getOverseerFailureMap().remove(asyncId);
+ zkController.clearAsyncId(asyncId);
+ }
+ rsp.getValues()
+ .add("status", "successfully cleared stored collection api responses");
+ } else {
+ // Request to cleanup
+ if (zkController.getOverseerCompletedMap().remove(requestId)) {
+ zkController.clearAsyncId(requestId);
+ rsp.getValues()
+ .add(
+ "status", "successfully removed stored response for [" + requestId + "]");
+ } else if (zkController.getOverseerFailureMap().remove(requestId)) {
+ zkController.clearAsyncId(requestId);
+ rsp.getValues()
+ .add(
+ "status", "successfully removed stored response for [" + requestId + "]");
+ } else {
+ rsp.getValues()
+ .add("status", "[" + requestId + "] not found in stored responses");
+ // Don't call zkController.clearAsyncId for this, since it could be a
+ // running/pending task
+ }
+ }
} else {
- rsp.getValues().add("status", "[" + requestId + "] not found in stored responses");
+ if (flush) {
+ coreContainer.getDistributedCollectionCommandRunner().get().deleteAllAsyncIds();
+ rsp.getValues()
+ .add("status", "successfully cleared stored collection api responses");
+ } else {
+ if (coreContainer
+ .getDistributedCollectionCommandRunner()
+ .get()
+ .deleteSingleAsyncId(requestId)) {
+ rsp.getValues()
+ .add(
+ "status", "successfully removed stored response for [" + requestId + "]");
+ } else {
+ rsp.getValues()
+ .add("status", "[" + requestId + "] not found in stored responses");
+ }
+ }
}
+ return null;
}
- }
- return null;
- }
- }),
- ADDREPLICA_OP(ADDREPLICA, (req, rsp, h) -> {
- Map<String, Object> props = copy(req.getParams(), null,
- COLLECTION_PROP,
- "node",
- SHARD_ID_PROP,
- _ROUTE_,
- CoreAdminParams.NAME,
- INSTANCE_DIR,
- DATA_DIR,
- ULOG_DIR,
- REPLICA_TYPE,
- WAIT_FOR_FINAL_STATE,
- NRT_REPLICAS,
- TLOG_REPLICAS,
- PULL_REPLICAS,
- CREATE_NODE_SET,
- FOLLOW_ALIASES,
- SKIP_NODE_ASSIGNMENT);
- return copyPropertiesWithPrefix(req.getParams(), props, PROPERTY_PREFIX);
- }),
+ }),
+ ADDREPLICA_OP(
+ ADDREPLICA,
+ (req, rsp, h) -> {
+ Map<String, Object> props =
+ copy(
+ req.getParams(),
+ null,
+ COLLECTION_PROP,
+ "node",
+ SHARD_ID_PROP,
+ _ROUTE_,
+ CoreAdminParams.NAME,
+ INSTANCE_DIR,
+ DATA_DIR,
+ ULOG_DIR,
+ REPLICA_TYPE,
+ WAIT_FOR_FINAL_STATE,
+ NRT_REPLICAS,
+ TLOG_REPLICAS,
+ PULL_REPLICAS,
+ CREATE_NODE_SET,
+ FOLLOW_ALIASES,
+ SKIP_NODE_ASSIGNMENT);
+ return copyPropertiesWithPrefix(req.getParams(), props, PROPERTY_PREFIX);
+ }),
OVERSEERSTATUS_OP(OVERSEERSTATUS, (req, rsp, h) -> new LinkedHashMap<>()),
@SuppressWarnings({"unchecked"})
- DISTRIBUTEDAPIPROCESSING_OP(DISTRIBUTEDAPIPROCESSING, (req, rsp, h) -> {
- NamedList<Object> results = new NamedList<>();
- boolean isDistributedApi = h.coreContainer.getDistributedCollectionCommandRunner().isPresent();
- results.add("isDistributedApi", isDistributedApi);
- rsp.getValues().addAll(results);
- return null;
- }),
- /**
- * Handle list collection request.
- * Do list collection request to zk host
- */
+ DISTRIBUTEDAPIPROCESSING_OP(
+ DISTRIBUTEDAPIPROCESSING,
+ (req, rsp, h) -> {
+ NamedList<Object> results = new NamedList<>();
+ boolean isDistributedApi =
+ h.coreContainer.getDistributedCollectionCommandRunner().isPresent();
+ results.add("isDistributedApi", isDistributedApi);
+ rsp.getValues().addAll(results);
+ return null;
+ }),
+ /** Handle list collection request. Do list collection request to zk host */
@SuppressWarnings({"unchecked"})
- LIST_OP(LIST, (req, rsp, h) -> {
- NamedList<Object> results = new NamedList<>();
- Map<String, DocCollection> collections = h.coreContainer.getZkController().getZkStateReader().getClusterState().getCollectionsMap();
- List<String> collectionList = new ArrayList<>(collections.keySet());
- Collections.sort(collectionList);
- // XXX should we add aliases here?
- results.add("collections", collectionList);
- SolrResponse response = new OverseerSolrResponse(results);
- rsp.getValues().addAll(response.getResponse());
- return null;
- }),
+ LIST_OP(
+ LIST,
+ (req, rsp, h) -> {
+ NamedList<Object> results = new NamedList<>();
+ Map<String, DocCollection> collections =
+ h.coreContainer
+ .getZkController()
+ .getZkStateReader()
+ .getClusterState()
+ .getCollectionsMap();
+ List<String> collectionList = new ArrayList<>(collections.keySet());
+ Collections.sort(collectionList);
+ // XXX should we add aliases here?
+ results.add("collections", collectionList);
+ SolrResponse response = new OverseerSolrResponse(results);
+ rsp.getValues().addAll(response.getResponse());
+ return null;
+ }),
/**
- * Handle cluster status request.
- * Can return status per specific collection/shard or per all collections.
+ * Handle cluster status request. Can return status per specific collection/shard or per all
+ * collections.
*/
- CLUSTERSTATUS_OP(CLUSTERSTATUS, (req, rsp, h) -> {
- Map<String, Object> all = copy(req.getParams(), null,
- COLLECTION_PROP,
- SHARD_ID_PROP,
- _ROUTE_);
- new ClusterStatus(h.coreContainer.getZkController().getZkStateReader(),
- new ZkNodeProps(all)).getClusterStatus(rsp.getValues());
- return null;
- }),
- ADDREPLICAPROP_OP(ADDREPLICAPROP, (req, rsp, h) -> {
- Map<String, Object> map = copy(req.getParams().required(), null,
- COLLECTION_PROP,
- PROPERTY_PROP,
- SHARD_ID_PROP,
- REPLICA_PROP,
- PROPERTY_VALUE_PROP);
- copy(req.getParams(), map, SHARD_UNIQUE);
- String property = (String) map.get(PROPERTY_PROP);
- if (!property.startsWith(PROPERTY_PREFIX)) {
- property = PROPERTY_PREFIX + property;
- }
+ CLUSTERSTATUS_OP(
+ CLUSTERSTATUS,
+ (req, rsp, h) -> {
+ Map<String, Object> all =
+ copy(req.getParams(), null, COLLECTION_PROP, SHARD_ID_PROP, _ROUTE_);
+ new ClusterStatus(
+ h.coreContainer.getZkController().getZkStateReader(), new ZkNodeProps(all))
+ .getClusterStatus(rsp.getValues());
+ return null;
+ }),
+ ADDREPLICAPROP_OP(
+ ADDREPLICAPROP,
+ (req, rsp, h) -> {
+ Map<String, Object> map =
+ copy(
+ req.getParams().required(),
+ null,
+ COLLECTION_PROP,
+ PROPERTY_PROP,
+ SHARD_ID_PROP,
+ REPLICA_PROP,
+ PROPERTY_VALUE_PROP);
+ copy(req.getParams(), map, SHARD_UNIQUE);
+ String property = (String) map.get(PROPERTY_PROP);
+ if (!property.startsWith(PROPERTY_PREFIX)) {
+ property = PROPERTY_PREFIX + property;
+ }
- boolean uniquePerSlice = Boolean.parseBoolean((String) map.get(SHARD_UNIQUE));
-
- // Check if we're trying to set a property with parameters that allow us to set the property on multiple replicas
- // in a slice on properties that are known to only be one-per-slice and error out if so.
- if (StringUtils.isNotBlank((String) map.get(SHARD_UNIQUE)) &&
- SliceMutator.SLICE_UNIQUE_BOOLEAN_PROPERTIES.contains(property.toLowerCase(Locale.ROOT)) &&
- uniquePerSlice == false) {
- throw new SolrException(ErrorCode.BAD_REQUEST,
- "Overseer replica property command received for property " + property +
- " with the " + SHARD_UNIQUE +
- " parameter set to something other than 'true'. No action taken.");
- }
- return map;
- }),
+ boolean uniquePerSlice = Boolean.parseBoolean((String) map.get(SHARD_UNIQUE));
+
+ // Check if we're trying to set a property with parameters that allow us to set the
+ // property on multiple replicas
+ // in a slice on properties that are known to only be one-per-slice and error out if so.
+ if (StringUtils.isNotBlank((String) map.get(SHARD_UNIQUE))
+ && SliceMutator.SLICE_UNIQUE_BOOLEAN_PROPERTIES.contains(
+ property.toLowerCase(Locale.ROOT))
+ && uniquePerSlice == false) {
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST,
+ "Overseer replica property command received for property "
+ + property
+ + " with the "
+ + SHARD_UNIQUE
+ + " parameter set to something other than 'true'. No action taken.");
+ }
+ return map;
+ }),
// XXX should this command support followAliases?
- DELETEREPLICAPROP_OP(DELETEREPLICAPROP, (req, rsp, h) -> {
- Map<String, Object> map = copy(req.getParams().required(), null,
- COLLECTION_PROP,
- PROPERTY_PROP,
- SHARD_ID_PROP,
- REPLICA_PROP);
- return copy(req.getParams(), map, PROPERTY_PROP);
- }),
+ DELETEREPLICAPROP_OP(
+ DELETEREPLICAPROP,
+ (req, rsp, h) -> {
+ Map<String, Object> map =
+ copy(
+ req.getParams().required(),
+ null,
+ COLLECTION_PROP,
+ PROPERTY_PROP,
+ SHARD_ID_PROP,
+ REPLICA_PROP);
+ return copy(req.getParams(), map, PROPERTY_PROP);
+ }),
// XXX should this command support followAliases?
- BALANCESHARDUNIQUE_OP(BALANCESHARDUNIQUE, (req, rsp, h) -> {
- Map<String, Object> map = copy(req.getParams().required(), null,
- COLLECTION_PROP,
- PROPERTY_PROP);
- Boolean shardUnique = Boolean.parseBoolean(req.getParams().get(SHARD_UNIQUE));
- String prop = req.getParams().get(PROPERTY_PROP).toLowerCase(Locale.ROOT);
- if (!StringUtils.startsWith(prop, PROPERTY_PREFIX)) {
- prop = PROPERTY_PREFIX + prop;
- }
+ BALANCESHARDUNIQUE_OP(
+ BALANCESHARDUNIQUE,
+ (req, rsp, h) -> {
+ Map<String, Object> map =
+ copy(req.getParams().required(), null, COLLECTION_PROP, PROPERTY_PROP);
+ Boolean shardUnique = Boolean.parseBoolean(req.getParams().get(SHARD_UNIQUE));
+ String prop = req.getParams().get(PROPERTY_PROP).toLowerCase(Locale.ROOT);
+ if (!StringUtils.startsWith(prop, PROPERTY_PREFIX)) {
+ prop = PROPERTY_PREFIX + prop;
+ }
- if (!shardUnique && !SliceMutator.SLICE_UNIQUE_BOOLEAN_PROPERTIES.contains(prop)) {
- throw new SolrException(ErrorCode.BAD_REQUEST, "Balancing properties amongst replicas in a slice requires that"
- + " the property be pre-defined as a unique property (e.g. 'preferredLeader') or that 'shardUnique' be set to 'true'. " +
- " Property: " + prop + " shardUnique: " + shardUnique);
- }
+ if (!shardUnique && !SliceMutator.SLICE_UNIQUE_BOOLEAN_PROPERTIES.contains(prop)) {
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST,
+ "Balancing properties amongst replicas in a slice requires that"
+ + " the property be pre-defined as a unique property (e.g. 'preferredLeader') or that 'shardUnique' be set to 'true'. "
+ + " Property: "
+ + prop
+ + " shardUnique: "
+ + shardUnique);
+ }
- return copy(req.getParams(), map, ONLY_ACTIVE_NODES, SHARD_UNIQUE);
- }),
- REBALANCELEADERS_OP(REBALANCELEADERS, (req, rsp, h) -> {
- new RebalanceLeaders(req, rsp, h).execute();
- return null;
- }),
+ return copy(req.getParams(), map, ONLY_ACTIVE_NODES, SHARD_UNIQUE);
+ }),
+ REBALANCELEADERS_OP(
+ REBALANCELEADERS,
+ (req, rsp, h) -> {
+ new RebalanceLeaders(req, rsp, h).execute();
+ return null;
+ }),
// XXX should this command support followAliases?
- MODIFYCOLLECTION_OP(MODIFYCOLLECTION, (req, rsp, h) -> {
- Map<String, Object> m = copy(req.getParams(), null, CollectionAdminRequest.MODIFIABLE_COLLECTION_PROPERTIES);
- copyPropertiesWithPrefix(req.getParams(), m, PROPERTY_PREFIX);
- if (m.isEmpty()) {
- throw new SolrException(ErrorCode.BAD_REQUEST,
- formatString("no supported values provided {0}", CollectionAdminRequest.MODIFIABLE_COLLECTION_PROPERTIES.toString()));
- }
- copy(req.getParams().required(), m, COLLECTION_PROP);
- for (Map.Entry<String, Object> entry : m.entrySet()) {
- String prop = entry.getKey();
- if ("".equals(entry.getValue())) {
- // set to an empty string is equivalent to removing the property, see SOLR-12507
- entry.setValue(null);
- }
- DocCollection.verifyProp(m, prop);
- }
- if (m.get(REPLICATION_FACTOR) != null) {
- m.put(NRT_REPLICAS, m.get(REPLICATION_FACTOR));
- }
- return m;
- }),
- BACKUP_OP(BACKUP, (req, rsp, h) -> {
- req.getParams().required().check(NAME, COLLECTION_PROP);
-
- final String extCollectionName = req.getParams().get(COLLECTION_PROP);
- final boolean followAliases = req.getParams().getBool(FOLLOW_ALIASES, false);
- final String collectionName = followAliases ? h.coreContainer.getZkController().getZkStateReader()
- .getAliases().resolveSimpleAlias(extCollectionName) : extCollectionName;
- final ClusterState clusterState = h.coreContainer.getZkController().getClusterState();
- if (!clusterState.hasCollection(collectionName)) {
- throw new SolrException(ErrorCode.BAD_REQUEST, "Collection '" + collectionName + "' does not exist, no action taken.");
- }
-
- CoreContainer cc = h.coreContainer;
- String repo = req.getParams().get(CoreAdminParams.BACKUP_REPOSITORY);
- BackupRepository repository = cc.newBackupRepository(repo);
-
- String location = repository.getBackupLocation(req.getParams().get(CoreAdminParams.BACKUP_LOCATION));
- if (location == null) {
- //Refresh the cluster property file to make sure the value set for location is the latest
- // Check if the location is specified in the cluster property.
- location = new ClusterProperties(h.coreContainer.getZkController().getZkClient()).getClusterProperty(CoreAdminParams.BACKUP_LOCATION, null);
- if (location == null) {
- throw new SolrException(ErrorCode.BAD_REQUEST, "'location' is not specified as a query"
- + " parameter or as a default repository property or as a cluster property.");
- }
- }
- boolean incremental = req.getParams().getBool(CoreAdminParams.BACKUP_INCREMENTAL, true);
-
- // Check if the specified location is valid for this repository.
- final URI uri = repository.createDirectoryURI(location);
- try {
- if (!repository.exists(uri)) {
- throw new SolrException(ErrorCode.SERVER_ERROR, "specified location " + uri + " does not exist.");
- }
- } catch (IOException ex) {
- throw new SolrException(ErrorCode.SERVER_ERROR, "Failed to check the existence of " + uri + ". Is it valid?", ex);
- }
+ MODIFYCOLLECTION_OP(
+ MODIFYCOLLECTION,
+ (req, rsp, h) -> {
+ Map<String, Object> m =
+ copy(req.getParams(), null, CollectionAdminRequest.MODIFIABLE_COLLECTION_PROPERTIES);
+ copyPropertiesWithPrefix(req.getParams(), m, PROPERTY_PREFIX);
+ if (m.isEmpty()) {
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST,
+ formatString(
+ "no supported values provided {0}",
+ CollectionAdminRequest.MODIFIABLE_COLLECTION_PROPERTIES.toString()));
+ }
+ copy(req.getParams().required(), m, COLLECTION_PROP);
+ for (Map.Entry<String, Object> entry : m.entrySet()) {
+ String prop = entry.getKey();
+ if ("".equals(entry.getValue())) {
+ // set to an empty string is equivalent to removing the property, see SOLR-12507
+ entry.setValue(null);
+ }
+ DocCollection.verifyProp(m, prop);
+ }
+ if (m.get(REPLICATION_FACTOR) != null) {
+ m.put(NRT_REPLICAS, m.get(REPLICATION_FACTOR));
+ }
+ return m;
+ }),
+ BACKUP_OP(
+ BACKUP,
+ (req, rsp, h) -> {
+ req.getParams().required().check(NAME, COLLECTION_PROP);
+
+ final String extCollectionName = req.getParams().get(COLLECTION_PROP);
+ final boolean followAliases = req.getParams().getBool(FOLLOW_ALIASES, false);
+ final String collectionName =
+ followAliases
+ ? h.coreContainer
+ .getZkController()
+ .getZkStateReader()
+ .getAliases()
+ .resolveSimpleAlias(extCollectionName)
+ : extCollectionName;
+ final ClusterState clusterState = h.coreContainer.getZkController().getClusterState();
+ if (!clusterState.hasCollection(collectionName)) {
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST,
+ "Collection '" + collectionName + "' does not exist, no action taken.");
+ }
- String strategy = req.getParams().get(CollectionAdminParams.INDEX_BACKUP_STRATEGY, CollectionAdminParams.COPY_FILES_STRATEGY);
- if (!CollectionAdminParams.INDEX_BACKUP_STRATEGIES.contains(strategy)) {
- throw new SolrException(ErrorCode.BAD_REQUEST, "Unknown index backup strategy " + strategy);
- }
+ CoreContainer cc = h.coreContainer;
+ String repo = req.getParams().get(CoreAdminParams.BACKUP_REPOSITORY);
+ BackupRepository repository = cc.newBackupRepository(repo);
- Map<String, Object> params = copy(req.getParams(), null, NAME, COLLECTION_PROP,
- FOLLOW_ALIASES, CoreAdminParams.COMMIT_NAME, CoreAdminParams.MAX_NUM_BACKUP_POINTS);
- params.put(CoreAdminParams.BACKUP_LOCATION, location);
- if (repo != null) {
- params.put(CoreAdminParams.BACKUP_REPOSITORY, repo);
- }
+ String location =
+ repository.getBackupLocation(req.getParams().get(CoreAdminParams.BACKUP_LOCATION));
+ if (location == null) {
+ // Refresh the cluster property file to make sure the value set for location is the
+ // latest
+ // Check if the location is specified in the cluster property.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/core/SolrCore.java
##########
@@ -2548,30 +2777,33 @@ private void registerSearcher(RefCounted<SolrIndexSearcher> newSearcherHolder) {
synchronized (searcherLock) {
try {
if (_searcher == newSearcherHolder) {
- // trying to re-register the same searcher... this can now happen when a commit has been done but
+ // trying to re-register the same searcher... this can now happen when a commit has been
+ // done but
// there were no changes to the index.
- newSearcherHolder.decref(); // decref since the caller should have still incref'd (since they didn't know the searcher was the same)
- return; // still execute the finally block to notify anyone waiting.
+ newSearcherHolder
+ .decref(); // decref since the caller should have still incref'd (since they didn't
+ // know the searcher was the same)
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/loader/JsonLoader.java
##########
@@ -428,29 +451,32 @@ RollbackUpdateCommand parseRollback() throws IOException {
void parseCommitOptions(CommitUpdateCommand cmd) throws IOException {
assertNextEvent(JSONParser.OBJECT_START);
@SuppressWarnings({"unchecked"})
- final Map<String, Object> map = (Map<String,Object>) ObjectBuilder.getVal(parser);
+ final Map<String, Object> map = (Map<String, Object>) ObjectBuilder.getVal(parser);
// SolrParams currently expects string values...
- SolrParams p = new SolrParams() {
- @Override
- public String get(String param) {
- Object o = map.get(param);
- return o == null ? null : o.toString();
- }
+ SolrParams p =
+ new SolrParams() {
+ @Override
+ public String get(String param) {
+ Object o = map.get(param);
+ return o == null ? null : o.toString();
+ }
- @Override
- public String[] getParams(String param) {
- return new String[]{get(param)};
- }
+ @Override
+ public String[] getParams(String param) {
+ return new String[] {get(param)};
+ }
- @Override
- public Iterator<String> getParameterNamesIterator() {
- return map.keySet().iterator();
- }
- };
+ @Override
+ public Iterator<String> getParameterNamesIterator() {
+ return map.keySet().iterator();
+ }
+ };
RequestHandlerUtils.validateCommitParams(p);
- p = SolrParams.wrapDefaults(p, req.getParams()); // default to the normal request params for commit options
+ p =
+ SolrParams.wrapDefaults(
+ p, req.getParams()); // default to the normal request params for commit options
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/core/SolrCore.java
##########
@@ -2548,30 +2777,33 @@ private void registerSearcher(RefCounted<SolrIndexSearcher> newSearcherHolder) {
synchronized (searcherLock) {
try {
if (_searcher == newSearcherHolder) {
- // trying to re-register the same searcher... this can now happen when a commit has been done but
+ // trying to re-register the same searcher... this can now happen when a commit has been
+ // done but
// there were no changes to the index.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/loader/CSVLoaderBase.java
##########
@@ -16,75 +16,73 @@
*/
package org.apache.solr.handler.loader;
-import org.apache.solr.request.SolrQueryRequest;
-import org.apache.solr.response.SolrQueryResponse;
+import java.io.*;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.regex.Pattern;
+import org.apache.commons.io.IOUtils;
import org.apache.solr.common.SolrException;
import org.apache.solr.common.SolrInputDocument;
import org.apache.solr.common.params.SolrParams;
import org.apache.solr.common.params.UpdateParams;
-import org.apache.solr.common.util.StrUtils;
import org.apache.solr.common.util.ContentStream;
+import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.internal.csv.CSVParser;
+import org.apache.solr.internal.csv.CSVStrategy;
+import org.apache.solr.request.SolrQueryRequest;
+import org.apache.solr.response.SolrQueryResponse;
import org.apache.solr.update.*;
import org.apache.solr.update.processor.UpdateRequestProcessor;
-import org.apache.solr.internal.csv.CSVStrategy;
-import org.apache.solr.internal.csv.CSVParser;
-import org.apache.commons.io.IOUtils;
-
-import java.util.Map;
-import java.util.regex.Pattern;
-import java.util.List;
-import java.util.HashMap;
-import java.util.Iterator;
-import java.io.*;
public abstract class CSVLoaderBase extends ContentStreamLoader {
- public static final String SEPARATOR="separator";
- public static final String FIELDNAMES="fieldnames";
- public static final String HEADER="header";
- public static final String SKIP="skip";
- public static final String SKIPLINES="skipLines";
- public static final String MAP="map";
- public static final String TRIM="trim";
- public static final String EMPTY="keepEmpty";
- public static final String SPLIT="split";
- public static final String ENCAPSULATOR="encapsulator";
- public static final String ESCAPE="escape";
- public static final String OVERWRITE="overwrite";
+ public static final String SEPARATOR = "separator";
+ public static final String FIELDNAMES = "fieldnames";
+ public static final String HEADER = "header";
+ public static final String SKIP = "skip";
+ public static final String SKIPLINES = "skipLines";
+ public static final String MAP = "map";
+ public static final String TRIM = "trim";
+ public static final String EMPTY = "keepEmpty";
+ public static final String SPLIT = "split";
+ public static final String ENCAPSULATOR = "encapsulator";
+ public static final String ESCAPE = "escape";
+ public static final String OVERWRITE = "overwrite";
public static final String LITERALS_PREFIX = "literal.";
public static final String ROW_ID = "rowid";
public static final String ROW_ID_OFFSET = "rowidOffset";
private static Pattern colonSplit = Pattern.compile(":");
private static Pattern commaSplit = Pattern.compile(",");
-
+
final SolrParams params;
final CSVStrategy strategy;
protected final UpdateRequestProcessor processor;
// hashmap to save any literal fields and their values
- HashMap <String, String> literals;
+ HashMap<String, String> literals;
String[] fieldnames;
CSVLoaderBase.FieldAdder[] adders;
- String rowId = null;// if not null, add a special field by the name given with the line number/row id as the value
- int rowIdOffset = 0; //add to line/rowid before creating the field
+ String rowId =
+ null; // if not null, add a special field by the name given with the line number/row id as the
+ // value
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/core/SolrCore.java
##########
@@ -2491,45 +2718,47 @@ public IndexFingerprint getIndexFingerprint(SolrIndexSearcher searcher, LeafRead
}
if (searchHolder != null) {
- searchHolder.decref(); // decrement 1 for _searcher (searchHolder will never become _searcher now)
+ searchHolder
+ .decref(); // decrement 1 for _searcher (searchHolder will never become _searcher now)
if (returnSearcher) {
- searchHolder.decref(); // decrement 1 because we won't be returning the searcher to the user
+ searchHolder
+ .decref(); // decrement 1 because we won't be returning the searcher to the user
}
}
}
// we want to do this after we decrement onDeckSearchers so another thread
// doesn't increment first and throw a false warning.
openSearcherLock.unlock();
-
}
-
}
-
- private RefCounted<SolrIndexSearcher> newHolder(SolrIndexSearcher newSearcher, final List<RefCounted<SolrIndexSearcher>> searcherList) {
- RefCounted<SolrIndexSearcher> holder = new RefCounted<SolrIndexSearcher>(newSearcher) {
- @Override
- public void close() {
- try {
- synchronized (searcherLock) {
- // it's possible for someone to get a reference via the _searchers queue
- // and increment the refcount while RefCounted.close() is being called.
- // we check the refcount again to see if this has happened and abort the close.
- // This relies on the RefCounted class allowing close() to be called every
- // time the counter hits zero.
- if (refcount.get() > 0) return;
- searcherList.remove(this);
+ private RefCounted<SolrIndexSearcher> newHolder(
+ SolrIndexSearcher newSearcher, final List<RefCounted<SolrIndexSearcher>> searcherList) {
+ RefCounted<SolrIndexSearcher> holder =
+ new RefCounted<SolrIndexSearcher>(newSearcher) {
+ @Override
+ public void close() {
+ try {
+ synchronized (searcherLock) {
+ // it's possible for someone to get a reference via the _searchers queue
+ // and increment the refcount while RefCounted.close() is being called.
+ // we check the refcount again to see if this has happened and abort the close.
+ // This relies on the RefCounted class allowing close() to be called every
+ // time the counter hits zero.
+ if (refcount.get() > 0) return;
+ searcherList.remove(this);
+ }
+ resource.close();
+ } catch (Exception e) {
+ // do not allow decref() operations to fail since they are typically called in finally
+ // blocks
+ // and throwing another exception would be very unexpected.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/admin/CollectionsHandler.java
##########
@@ -388,982 +426,1324 @@ public Category getCategory() {
return Category.ADMIN;
}
- private static void createSysConfigSet(CoreContainer coreContainer) throws KeeperException, InterruptedException {
+ private static void createSysConfigSet(CoreContainer coreContainer)
+ throws KeeperException, InterruptedException {
SolrZkClient zk = coreContainer.getZkController().getZkStateReader().getZkClient();
ZkCmdExecutor cmdExecutor = new ZkCmdExecutor(zk.getZkClientTimeout());
cmdExecutor.ensureExists(ZkStateReader.CONFIGS_ZKNODE, zk);
- cmdExecutor.ensureExists(ZkStateReader.CONFIGS_ZKNODE + "/" + CollectionAdminParams.SYSTEM_COLL, zk);
+ cmdExecutor.ensureExists(
+ ZkStateReader.CONFIGS_ZKNODE + "/" + CollectionAdminParams.SYSTEM_COLL, zk);
try {
- String path = ZkStateReader.CONFIGS_ZKNODE + "/" + CollectionAdminParams.SYSTEM_COLL + "/schema.xml";
- byte[] data = IOUtils.toByteArray(CollectionsHandler.class.getResourceAsStream("/SystemCollectionSchema.xml"));
+ String path =
+ ZkStateReader.CONFIGS_ZKNODE + "/" + CollectionAdminParams.SYSTEM_COLL + "/schema.xml";
+ byte[] data =
+ IOUtils.toByteArray(
+ CollectionsHandler.class.getResourceAsStream("/SystemCollectionSchema.xml"));
assert data != null && data.length > 0;
cmdExecutor.ensureExists(path, data, CreateMode.PERSISTENT, zk);
- path = ZkStateReader.CONFIGS_ZKNODE + "/" + CollectionAdminParams.SYSTEM_COLL + "/solrconfig.xml";
- data = IOUtils.toByteArray(CollectionsHandler.class.getResourceAsStream("/SystemCollectionSolrConfig.xml"));
+ path =
+ ZkStateReader.CONFIGS_ZKNODE
+ + "/"
+ + CollectionAdminParams.SYSTEM_COLL
+ + "/solrconfig.xml";
+ data =
+ IOUtils.toByteArray(
+ CollectionsHandler.class.getResourceAsStream("/SystemCollectionSolrConfig.xml"));
assert data != null && data.length > 0;
cmdExecutor.ensureExists(path, data, CreateMode.PERSISTENT, zk);
} catch (IOException e) {
throw new SolrException(ErrorCode.SERVER_ERROR, e);
}
-
-
}
- private static void addStatusToResponse(NamedList<Object> results, RequestStatusState state, String msg) {
+ private static void addStatusToResponse(
+ NamedList<Object> results, RequestStatusState state, String msg) {
SimpleOrderedMap<String> status = new SimpleOrderedMap<>();
status.add("state", state.getKey());
status.add("msg", msg);
results.add("status", status);
}
public enum CollectionOperation implements CollectionOp {
- CREATE_OP(CREATE, (req, rsp, h) -> {
- Map<String, Object> props = copy(req.getParams().required(), null, NAME);
- props.put("fromApi", "true");
- copy(req.getParams(), props,
- REPLICATION_FACTOR,
- COLL_CONF,
- NUM_SLICES,
- CREATE_NODE_SET,
- CREATE_NODE_SET_SHUFFLE,
- SHARDS_PROP,
- PULL_REPLICAS,
- TLOG_REPLICAS,
- NRT_REPLICAS,
- WAIT_FOR_FINAL_STATE,
- PER_REPLICA_STATE,
- ALIAS);
-
- if (props.get(REPLICATION_FACTOR) != null && props.get(NRT_REPLICAS) != null) {
- //TODO: Remove this in 8.0 . Keep this for SolrJ client back-compat. See SOLR-11676 for more details
- int replicationFactor = Integer.parseInt((String) props.get(REPLICATION_FACTOR));
- int nrtReplicas = Integer.parseInt((String) props.get(NRT_REPLICAS));
- if (replicationFactor != nrtReplicas) {
- throw new SolrException(ErrorCode.BAD_REQUEST,
- "Cannot specify both replicationFactor and nrtReplicas as they mean the same thing");
- }
- }
- if (props.get(REPLICATION_FACTOR) != null) {
- props.put(NRT_REPLICAS, props.get(REPLICATION_FACTOR));
- } else if (props.get(NRT_REPLICAS) != null) {
- props.put(REPLICATION_FACTOR, props.get(NRT_REPLICAS));
- }
-
- final String collectionName = SolrIdentifierValidator.validateCollectionName((String) props.get(NAME));
- final String shardsParam = (String) props.get(SHARDS_PROP);
- if (StringUtils.isNotEmpty(shardsParam)) {
- verifyShardsParam(shardsParam);
- }
- if (CollectionAdminParams.SYSTEM_COLL.equals(collectionName)) {
- //We must always create a .system collection with only a single shard
- props.put(NUM_SLICES, 1);
- props.remove(SHARDS_PROP);
- createSysConfigSet(h.coreContainer);
-
- }
- if (shardsParam == null) h.copyFromClusterProp(props, NUM_SLICES);
- for (String prop : ImmutableSet.of(NRT_REPLICAS, PULL_REPLICAS, TLOG_REPLICAS))
- h.copyFromClusterProp(props, prop);
- copyPropertiesWithPrefix(req.getParams(), props, PROPERTY_PREFIX);
- return copyPropertiesWithPrefix(req.getParams(), props, "router.");
+ CREATE_OP(
+ CREATE,
+ (req, rsp, h) -> {
+ Map<String, Object> props = copy(req.getParams().required(), null, NAME);
+ props.put("fromApi", "true");
+ copy(
+ req.getParams(),
+ props,
+ REPLICATION_FACTOR,
+ COLL_CONF,
+ NUM_SLICES,
+ CREATE_NODE_SET,
+ CREATE_NODE_SET_SHUFFLE,
+ SHARDS_PROP,
+ PULL_REPLICAS,
+ TLOG_REPLICAS,
+ NRT_REPLICAS,
+ WAIT_FOR_FINAL_STATE,
+ PER_REPLICA_STATE,
+ ALIAS);
+
+ if (props.get(REPLICATION_FACTOR) != null && props.get(NRT_REPLICAS) != null) {
+ // TODO: Remove this in 8.0 . Keep this for SolrJ client back-compat. See SOLR-11676 for
+ // more details
+ int replicationFactor = Integer.parseInt((String) props.get(REPLICATION_FACTOR));
+ int nrtReplicas = Integer.parseInt((String) props.get(NRT_REPLICAS));
+ if (replicationFactor != nrtReplicas) {
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST,
+ "Cannot specify both replicationFactor and nrtReplicas as they mean the same thing");
+ }
+ }
+ if (props.get(REPLICATION_FACTOR) != null) {
+ props.put(NRT_REPLICAS, props.get(REPLICATION_FACTOR));
+ } else if (props.get(NRT_REPLICAS) != null) {
+ props.put(REPLICATION_FACTOR, props.get(NRT_REPLICAS));
+ }
- }),
+ final String collectionName =
+ SolrIdentifierValidator.validateCollectionName((String) props.get(NAME));
+ final String shardsParam = (String) props.get(SHARDS_PROP);
+ if (StringUtils.isNotEmpty(shardsParam)) {
+ verifyShardsParam(shardsParam);
+ }
+ if (CollectionAdminParams.SYSTEM_COLL.equals(collectionName)) {
+ // We must always create a .system collection with only a single shard
+ props.put(NUM_SLICES, 1);
+ props.remove(SHARDS_PROP);
+ createSysConfigSet(h.coreContainer);
+ }
+ if (shardsParam == null) h.copyFromClusterProp(props, NUM_SLICES);
+ for (String prop : ImmutableSet.of(NRT_REPLICAS, PULL_REPLICAS, TLOG_REPLICAS))
+ h.copyFromClusterProp(props, prop);
+ copyPropertiesWithPrefix(req.getParams(), props, PROPERTY_PREFIX);
+ return copyPropertiesWithPrefix(req.getParams(), props, "router.");
+ }),
@SuppressWarnings({"unchecked"})
- COLSTATUS_OP(COLSTATUS, (req, rsp, h) -> {
- Map<String, Object> props = copy(req.getParams(), null,
- COLLECTION_PROP,
- ColStatus.CORE_INFO_PROP,
- ColStatus.SEGMENTS_PROP,
- ColStatus.FIELD_INFO_PROP,
- ColStatus.RAW_SIZE_PROP,
- ColStatus.RAW_SIZE_SUMMARY_PROP,
- ColStatus.RAW_SIZE_DETAILS_PROP,
- ColStatus.RAW_SIZE_SAMPLING_PERCENT_PROP,
- ColStatus.SIZE_INFO_PROP);
-
- new ColStatus(h.coreContainer.getSolrClientCache(),
- h.coreContainer.getZkController().getZkStateReader().getClusterState(), new ZkNodeProps(props))
- .getColStatus(rsp.getValues());
- return null;
- }),
- DELETE_OP(DELETE, (req, rsp, h) -> {
- Map<String, Object> map = copy(req.getParams().required(), null, NAME);
- return copy(req.getParams(), map, FOLLOW_ALIASES);
- }),
+ COLSTATUS_OP(
+ COLSTATUS,
+ (req, rsp, h) -> {
+ Map<String, Object> props =
+ copy(
+ req.getParams(),
+ null,
+ COLLECTION_PROP,
+ ColStatus.CORE_INFO_PROP,
+ ColStatus.SEGMENTS_PROP,
+ ColStatus.FIELD_INFO_PROP,
+ ColStatus.RAW_SIZE_PROP,
+ ColStatus.RAW_SIZE_SUMMARY_PROP,
+ ColStatus.RAW_SIZE_DETAILS_PROP,
+ ColStatus.RAW_SIZE_SAMPLING_PERCENT_PROP,
+ ColStatus.SIZE_INFO_PROP);
+
+ new ColStatus(
+ h.coreContainer.getSolrClientCache(),
+ h.coreContainer.getZkController().getZkStateReader().getClusterState(),
+ new ZkNodeProps(props))
+ .getColStatus(rsp.getValues());
+ return null;
+ }),
+ DELETE_OP(
+ DELETE,
+ (req, rsp, h) -> {
+ Map<String, Object> map = copy(req.getParams().required(), null, NAME);
+ return copy(req.getParams(), map, FOLLOW_ALIASES);
+ }),
// XXX should this command support followAliases?
- RELOAD_OP(RELOAD, (req, rsp, h) -> {
- Map<String, Object> map = copy(req.getParams().required(), null, NAME);
- return copy(req.getParams(), map);
- }),
-
- RENAME_OP(RENAME, (req, rsp, h) -> {
- Map<String, Object> map = copy(req.getParams().required(), null, NAME, CollectionAdminParams.TARGET);
- return copy(req.getParams(), map, FOLLOW_ALIASES);
- }),
-
- REINDEXCOLLECTION_OP(REINDEXCOLLECTION, (req, rsp, h) -> {
- Map<String, Object> m = copy(req.getParams().required(), null, NAME);
- copy(req.getParams(), m,
- ReindexCollectionCmd.COMMAND,
- ReindexCollectionCmd.REMOVE_SOURCE,
- ReindexCollectionCmd.TARGET,
- ZkStateReader.CONFIGNAME_PROP,
- NUM_SLICES,
- NRT_REPLICAS,
- PULL_REPLICAS,
- TLOG_REPLICAS,
- REPLICATION_FACTOR,
- CREATE_NODE_SET,
- CREATE_NODE_SET_SHUFFLE,
- "shards",
- CommonParams.ROWS,
- CommonParams.Q,
- CommonParams.FL,
- FOLLOW_ALIASES);
- if (req.getParams().get("collection." + ZkStateReader.CONFIGNAME_PROP) != null) {
- m.put(ZkStateReader.CONFIGNAME_PROP, req.getParams().get("collection." + ZkStateReader.CONFIGNAME_PROP));
- }
- copyPropertiesWithPrefix(req.getParams(), m, "router.");
- return m;
- }),
-
- SYNCSHARD_OP(SYNCSHARD, (req, rsp, h) -> {
- String extCollection = req.getParams().required().get("collection");
- String collection = h.coreContainer.getZkController().getZkStateReader().getAliases().resolveSimpleAlias(extCollection);
- String shard = req.getParams().required().get("shard");
-
- ClusterState clusterState = h.coreContainer.getZkController().getClusterState();
-
- DocCollection docCollection = clusterState.getCollection(collection);
- ZkNodeProps leaderProps = docCollection.getLeader(shard);
- ZkCoreNodeProps nodeProps = new ZkCoreNodeProps(leaderProps);
-
- try (HttpSolrClient client = new Builder(nodeProps.getBaseUrl())
- .withConnectionTimeout(15000)
- .withSocketTimeout(60000)
- .build()) {
- RequestSyncShard reqSyncShard = new RequestSyncShard();
- reqSyncShard.setCollection(collection);
- reqSyncShard.setShard(shard);
- reqSyncShard.setCoreName(nodeProps.getCoreName());
- client.request(reqSyncShard);
- }
- return null;
- }),
-
- CREATEALIAS_OP(CREATEALIAS, (req, rsp, h) -> {
- String alias = req.getParams().get(NAME);
- SolrIdentifierValidator.validateAliasName(alias);
- String collections = req.getParams().get("collections");
- RoutedAlias routedAlias = null;
- Exception ex = null;
- HashMap<String,Object> possiblyModifiedParams = new HashMap<>();
- try {
- // note that RA specific validation occurs here.
- req.getParams().toMap(possiblyModifiedParams);
- @SuppressWarnings({"unchecked", "rawtypes"})
- // This is awful because RoutedAlias lies about what types it wants
- Map<String, String> temp = (Map<String, String>) (Map) possiblyModifiedParams;
- routedAlias = RoutedAlias.fromProps(alias, temp);
- } catch (SolrException e) {
- // we'll throw this later if we are in fact creating a routed alias.
- ex = e;
- }
- ModifiableSolrParams finalParams = new ModifiableSolrParams();
- for (Map.Entry<String, Object> entry : possiblyModifiedParams.entrySet()) {
- if (entry.getValue().getClass().isArray() ) {
- // v2 api hits this case
- for (Object o : (Object[]) entry.getValue()) {
- finalParams.add(entry.getKey(),o.toString());
+ RELOAD_OP(
+ RELOAD,
+ (req, rsp, h) -> {
+ Map<String, Object> map = copy(req.getParams().required(), null, NAME);
+ return copy(req.getParams(), map);
+ }),
+
+ RENAME_OP(
+ RENAME,
+ (req, rsp, h) -> {
+ Map<String, Object> map =
+ copy(req.getParams().required(), null, NAME, CollectionAdminParams.TARGET);
+ return copy(req.getParams(), map, FOLLOW_ALIASES);
+ }),
+
+ REINDEXCOLLECTION_OP(
+ REINDEXCOLLECTION,
+ (req, rsp, h) -> {
+ Map<String, Object> m = copy(req.getParams().required(), null, NAME);
+ copy(
+ req.getParams(),
+ m,
+ ReindexCollectionCmd.COMMAND,
+ ReindexCollectionCmd.REMOVE_SOURCE,
+ ReindexCollectionCmd.TARGET,
+ ZkStateReader.CONFIGNAME_PROP,
+ NUM_SLICES,
+ NRT_REPLICAS,
+ PULL_REPLICAS,
+ TLOG_REPLICAS,
+ REPLICATION_FACTOR,
+ CREATE_NODE_SET,
+ CREATE_NODE_SET_SHUFFLE,
+ "shards",
+ CommonParams.ROWS,
+ CommonParams.Q,
+ CommonParams.FL,
+ FOLLOW_ALIASES);
+ if (req.getParams().get("collection." + ZkStateReader.CONFIGNAME_PROP) != null) {
+ m.put(
+ ZkStateReader.CONFIGNAME_PROP,
+ req.getParams().get("collection." + ZkStateReader.CONFIGNAME_PROP));
+ }
+ copyPropertiesWithPrefix(req.getParams(), m, "router.");
+ return m;
+ }),
+
+ SYNCSHARD_OP(
+ SYNCSHARD,
+ (req, rsp, h) -> {
+ String extCollection = req.getParams().required().get("collection");
+ String collection =
+ h.coreContainer
+ .getZkController()
+ .getZkStateReader()
+ .getAliases()
+ .resolveSimpleAlias(extCollection);
+ String shard = req.getParams().required().get("shard");
+
+ ClusterState clusterState = h.coreContainer.getZkController().getClusterState();
+
+ DocCollection docCollection = clusterState.getCollection(collection);
+ ZkNodeProps leaderProps = docCollection.getLeader(shard);
+ ZkCoreNodeProps nodeProps = new ZkCoreNodeProps(leaderProps);
+
+ try (HttpSolrClient client =
+ new Builder(nodeProps.getBaseUrl())
+ .withConnectionTimeout(15000)
+ .withSocketTimeout(60000)
+ .build()) {
+ RequestSyncShard reqSyncShard = new RequestSyncShard();
+ reqSyncShard.setCollection(collection);
+ reqSyncShard.setShard(shard);
+ reqSyncShard.setCoreName(nodeProps.getCoreName());
+ client.request(reqSyncShard);
+ }
+ return null;
+ }),
+
+ CREATEALIAS_OP(
+ CREATEALIAS,
+ (req, rsp, h) -> {
+ String alias = req.getParams().get(NAME);
+ SolrIdentifierValidator.validateAliasName(alias);
+ String collections = req.getParams().get("collections");
+ RoutedAlias routedAlias = null;
+ Exception ex = null;
+ HashMap<String, Object> possiblyModifiedParams = new HashMap<>();
+ try {
+ // note that RA specific validation occurs here.
+ req.getParams().toMap(possiblyModifiedParams);
+ @SuppressWarnings({"unchecked", "rawtypes"})
+ // This is awful because RoutedAlias lies about what types it wants
+ Map<String, String> temp = (Map<String, String>) (Map) possiblyModifiedParams;
+ routedAlias = RoutedAlias.fromProps(alias, temp);
+ } catch (SolrException e) {
+ // we'll throw this later if we are in fact creating a routed alias.
+ ex = e;
+ }
+ ModifiableSolrParams finalParams = new ModifiableSolrParams();
+ for (Map.Entry<String, Object> entry : possiblyModifiedParams.entrySet()) {
+ if (entry.getValue().getClass().isArray()) {
+ // v2 api hits this case
+ for (Object o : (Object[]) entry.getValue()) {
+ finalParams.add(entry.getKey(), o.toString());
+ }
+ } else {
+ finalParams.add(entry.getKey(), entry.getValue().toString());
+ }
}
- } else {
- finalParams.add(entry.getKey(),entry.getValue().toString());
- }
- }
- if (collections != null) {
- if (routedAlias != null) {
- throw new SolrException(BAD_REQUEST, "Collections cannot be specified when creating a routed alias.");
- } else {
- //////////////////////////////////////
- // Regular alias creation indicated //
- //////////////////////////////////////
- return copy(finalParams.required(), null, NAME, "collections");
- }
- }
+ if (collections != null) {
+ if (routedAlias != null) {
+ throw new SolrException(
+ BAD_REQUEST, "Collections cannot be specified when creating a routed alias.");
+ } else {
+ //////////////////////////////////////
+ // Regular alias creation indicated //
+ //////////////////////////////////////
+ return copy(finalParams.required(), null, NAME, "collections");
+ }
+ }
- /////////////////////////////////////////////////
- // We are creating a routed alias from here on //
- /////////////////////////////////////////////////
+ /////////////////////////////////////////////////
+ // We are creating a routed alias from here on //
+ /////////////////////////////////////////////////
- // If our prior creation attempt had issues expose them now.
- if (ex != null) {
- throw ex;
- }
+ // If our prior creation attempt had issues expose them now.
+ if (ex != null) {
+ throw ex;
+ }
- // Now filter out just the parameters we care about from the request
- assert routedAlias != null;
- Map<String, Object> result = copy(finalParams, null, routedAlias.getRequiredParams());
- copy(finalParams, result, routedAlias.getOptionalParams());
-
- ModifiableSolrParams createCollParams = new ModifiableSolrParams(); // without prefix
-
- // add to result params that start with "create-collection.".
- // Additionally, save these without the prefix to createCollParams
- for (Map.Entry<String, String[]> entry : finalParams) {
- final String p = entry.getKey();
- if (p.startsWith(CREATE_COLLECTION_PREFIX)) {
- // This is what SolrParams#getAll(Map, Collection)} does
- final String[] v = entry.getValue();
- if (v.length == 1) {
- result.put(p, v[0]);
- } else {
- result.put(p, v);
+ // Now filter out just the parameters we care about from the request
+ assert routedAlias != null;
+ Map<String, Object> result = copy(finalParams, null, routedAlias.getRequiredParams());
+ copy(finalParams, result, routedAlias.getOptionalParams());
+
+ ModifiableSolrParams createCollParams = new ModifiableSolrParams(); // without prefix
+
+ // add to result params that start with "create-collection.".
+ // Additionally, save these without the prefix to createCollParams
+ for (Map.Entry<String, String[]> entry : finalParams) {
+ final String p = entry.getKey();
+ if (p.startsWith(CREATE_COLLECTION_PREFIX)) {
+ // This is what SolrParams#getAll(Map, Collection)} does
+ final String[] v = entry.getValue();
+ if (v.length == 1) {
+ result.put(p, v[0]);
+ } else {
+ result.put(p, v);
+ }
+ createCollParams.set(p.substring(CREATE_COLLECTION_PREFIX.length()), v);
+ }
}
- createCollParams.set(p.substring(CREATE_COLLECTION_PREFIX.length()), v);
- }
- }
- // Verify that the create-collection prefix'ed params appear to be valid.
- if (createCollParams.get(NAME) != null) {
- throw new SolrException(BAD_REQUEST, "routed aliases calculate names for their " +
- "dependent collections, you cannot specify the name.");
- }
- if (createCollParams.get(COLL_CONF) == null) {
- throw new SolrException(SolrException.ErrorCode.BAD_REQUEST,
- "We require an explicit " + COLL_CONF);
- }
- // note: could insist on a config name here as well.... or wait to throw at overseer
- createCollParams.add(NAME, "TMP_name_TMP_name_TMP"); // just to pass validation
- CREATE_OP.execute(new LocalSolrQueryRequest(null, createCollParams), rsp, h); // ignore results
+ // Verify that the create-collection prefix'ed params appear to be valid.
+ if (createCollParams.get(NAME) != null) {
+ throw new SolrException(
+ BAD_REQUEST,
+ "routed aliases calculate names for their "
+ + "dependent collections, you cannot specify the name.");
+ }
+ if (createCollParams.get(COLL_CONF) == null) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST, "We require an explicit " + COLL_CONF);
+ }
+ // note: could insist on a config name here as well.... or wait to throw at overseer
+ createCollParams.add(NAME, "TMP_name_TMP_name_TMP"); // just to pass validation
+ CREATE_OP.execute(
+ new LocalSolrQueryRequest(null, createCollParams), rsp, h); // ignore results
- return result;
- }),
+ return result;
+ }),
DELETEALIAS_OP(DELETEALIAS, (req, rsp, h) -> copy(req.getParams().required(), null, NAME)),
/**
* Change properties for an alias (use CREATEALIAS_OP to change the actual value of the alias)
*/
- ALIASPROP_OP(ALIASPROP, (req, rsp, h) -> {
- Map<String, Object> params = copy(req.getParams().required(), null, NAME);
-
- // Note: success/no-op in the event of no properties supplied is intentional. Keeps code simple and one less case
- // for api-callers to check for.
- return convertPrefixToMap(req.getParams(), params, "property");
- }),
-
- /**
- * List the aliases and associated properties.
- */
+ ALIASPROP_OP(
+ ALIASPROP,
+ (req, rsp, h) -> {
+ Map<String, Object> params = copy(req.getParams().required(), null, NAME);
+
+ // Note: success/no-op in the event of no properties supplied is intentional. Keeps code
+ // simple and one less case
+ // for api-callers to check for.
+ return convertPrefixToMap(req.getParams(), params, "property");
+ }),
+
+ /** List the aliases and associated properties. */
@SuppressWarnings({"unchecked"})
- LISTALIASES_OP(LISTALIASES, (req, rsp, h) -> {
- ZkStateReader zkStateReader = h.coreContainer.getZkController().getZkStateReader();
- // if someone calls listAliases, lets ensure we return an up to date response
- zkStateReader.aliasesManager.update();
- Aliases aliases = zkStateReader.getAliases();
- if (aliases != null) {
- // the aliases themselves...
- rsp.getValues().add("aliases", aliases.getCollectionAliasMap());
- // Any properties for the above aliases.
- Map<String, Map<String, String>> meta = new LinkedHashMap<>();
- for (String alias : aliases.getCollectionAliasListMap().keySet()) {
- Map<String, String> collectionAliasProperties = aliases.getCollectionAliasProperties(alias);
- if (!collectionAliasProperties.isEmpty()) {
- meta.put(alias, collectionAliasProperties);
+ LISTALIASES_OP(
+ LISTALIASES,
+ (req, rsp, h) -> {
+ ZkStateReader zkStateReader = h.coreContainer.getZkController().getZkStateReader();
+ // if someone calls listAliases, lets ensure we return an up to date response
+ zkStateReader.aliasesManager.update();
+ Aliases aliases = zkStateReader.getAliases();
+ if (aliases != null) {
+ // the aliases themselves...
+ rsp.getValues().add("aliases", aliases.getCollectionAliasMap());
+ // Any properties for the above aliases.
+ Map<String, Map<String, String>> meta = new LinkedHashMap<>();
+ for (String alias : aliases.getCollectionAliasListMap().keySet()) {
+ Map<String, String> collectionAliasProperties =
+ aliases.getCollectionAliasProperties(alias);
+ if (!collectionAliasProperties.isEmpty()) {
+ meta.put(alias, collectionAliasProperties);
+ }
+ }
+ rsp.getValues().add("properties", meta);
+ }
+ return null;
+ }),
+ SPLITSHARD_OP(
+ SPLITSHARD,
+ DEFAULT_COLLECTION_OP_TIMEOUT * 5,
+ (req, rsp, h) -> {
+ String name = req.getParams().required().get(COLLECTION_PROP);
+ // TODO : add support for multiple shards
+ String shard = req.getParams().get(SHARD_ID_PROP);
+ String rangesStr = req.getParams().get(CoreAdminParams.RANGES);
+ String splitKey = req.getParams().get("split.key");
+ String numSubShards = req.getParams().get(NUM_SUB_SHARDS);
+ String fuzz = req.getParams().get(SPLIT_FUZZ);
+
+ if (splitKey == null && shard == null) {
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST, "At least one of shard, or split.key should be specified.");
+ }
+ if (splitKey != null && shard != null) {
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST, "Only one of 'shard' or 'split.key' should be specified");
+ }
+ if (splitKey != null && rangesStr != null) {
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST, "Only one of 'ranges' or 'split.key' should be specified");
+ }
+ if (numSubShards != null && (splitKey != null || rangesStr != null)) {
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST,
+ "numSubShards can not be specified with split.key or ranges parameters");
+ }
+ if (fuzz != null && (splitKey != null || rangesStr != null)) {
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST,
+ "fuzz can not be specified with split.key or ranges parameters");
}
- }
- rsp.getValues().add("properties", meta);
- }
- return null;
- }),
- SPLITSHARD_OP(SPLITSHARD, DEFAULT_COLLECTION_OP_TIMEOUT * 5, (req, rsp, h) -> {
- String name = req.getParams().required().get(COLLECTION_PROP);
- // TODO : add support for multiple shards
- String shard = req.getParams().get(SHARD_ID_PROP);
- String rangesStr = req.getParams().get(CoreAdminParams.RANGES);
- String splitKey = req.getParams().get("split.key");
- String numSubShards = req.getParams().get(NUM_SUB_SHARDS);
- String fuzz = req.getParams().get(SPLIT_FUZZ);
-
- if (splitKey == null && shard == null) {
- throw new SolrException(ErrorCode.BAD_REQUEST, "At least one of shard, or split.key should be specified.");
- }
- if (splitKey != null && shard != null) {
- throw new SolrException(ErrorCode.BAD_REQUEST,
- "Only one of 'shard' or 'split.key' should be specified");
- }
- if (splitKey != null && rangesStr != null) {
- throw new SolrException(ErrorCode.BAD_REQUEST,
- "Only one of 'ranges' or 'split.key' should be specified");
- }
- if (numSubShards != null && (splitKey != null || rangesStr != null)) {
- throw new SolrException(ErrorCode.BAD_REQUEST,
- "numSubShards can not be specified with split.key or ranges parameters");
- }
- if (fuzz != null && (splitKey != null || rangesStr != null)) {
- throw new SolrException(ErrorCode.BAD_REQUEST,
- "fuzz can not be specified with split.key or ranges parameters");
- }
- Map<String, Object> map = copy(req.getParams(), null,
- COLLECTION_PROP,
- SHARD_ID_PROP,
- "split.key",
- CoreAdminParams.RANGES,
- WAIT_FOR_FINAL_STATE,
- TIMING,
- SPLIT_METHOD,
- NUM_SUB_SHARDS,
- SPLIT_FUZZ,
- SPLIT_BY_PREFIX,
- FOLLOW_ALIASES);
- return copyPropertiesWithPrefix(req.getParams(), map, PROPERTY_PREFIX);
- }),
- DELETESHARD_OP(DELETESHARD, (req, rsp, h) -> {
- Map<String, Object> map = copy(req.getParams().required(), null,
- COLLECTION_PROP,
- SHARD_ID_PROP);
- copy(req.getParams(), map,
- DELETE_INDEX,
- DELETE_DATA_DIR,
- DELETE_INSTANCE_DIR,
- FOLLOW_ALIASES);
- return map;
- }),
- FORCELEADER_OP(FORCELEADER, (req, rsp, h) -> {
- forceLeaderElection(req, h);
- return null;
- }),
- CREATESHARD_OP(CREATESHARD, (req, rsp, h) -> {
- Map<String, Object> map = copy(req.getParams().required(), null,
- COLLECTION_PROP,
- SHARD_ID_PROP);
- ClusterState clusterState = h.coreContainer.getZkController().getClusterState();
- final String newShardName = SolrIdentifierValidator.validateShardName(req.getParams().get(SHARD_ID_PROP));
- boolean followAliases = req.getParams().getBool(FOLLOW_ALIASES, false);
- String extCollectionName = req.getParams().get(COLLECTION_PROP);
- String collectionName = followAliases ? h.coreContainer.getZkController().getZkStateReader()
- .getAliases().resolveSimpleAlias(extCollectionName) : extCollectionName;
- if (!ImplicitDocRouter.NAME.equals(((Map<?,?>) clusterState.getCollection(collectionName).get(DOC_ROUTER)).get(NAME)))
- throw new SolrException(ErrorCode.BAD_REQUEST, "shards can be added only to 'implicit' collections");
- copy(req.getParams(), map,
- REPLICATION_FACTOR,
- NRT_REPLICAS,
- TLOG_REPLICAS,
- PULL_REPLICAS,
- CREATE_NODE_SET,
- WAIT_FOR_FINAL_STATE,
- FOLLOW_ALIASES);
- return copyPropertiesWithPrefix(req.getParams(), map, PROPERTY_PREFIX);
- }),
- DELETEREPLICA_OP(DELETEREPLICA, (req, rsp, h) -> {
- Map<String, Object> map = copy(req.getParams().required(), null,
- COLLECTION_PROP);
-
- return copy(req.getParams(), map,
- DELETE_INDEX,
- DELETE_DATA_DIR,
- DELETE_INSTANCE_DIR,
- COUNT_PROP, REPLICA_PROP,
- SHARD_ID_PROP,
- ONLY_IF_DOWN,
- FOLLOW_ALIASES);
- }),
- MIGRATE_OP(MIGRATE, (req, rsp, h) -> {
- Map<String, Object> map = copy(req.getParams().required(), null, COLLECTION_PROP, "split.key", "target.collection");
- return copy(req.getParams(), map, "forward.timeout", FOLLOW_ALIASES);
- }),
- ADDROLE_OP(ADDROLE, (req, rsp, h) -> {
- Map<String, Object> map = copy(req.getParams().required(), null, "role", "node");
- if (!KNOWN_ROLES.contains(map.get("role")))
- throw new SolrException(ErrorCode.BAD_REQUEST, "Unknown role. Supported roles are ," + KNOWN_ROLES);
- return map;
- }),
- REMOVEROLE_OP(REMOVEROLE, (req, rsp, h) -> {
- Map<String, Object> map = copy(req.getParams().required(), null, "role", "node");
- if (!KNOWN_ROLES.contains(map.get("role")))
- throw new SolrException(ErrorCode.BAD_REQUEST, "Unknown role. Supported roles are ," + KNOWN_ROLES);
- return map;
- }),
- CLUSTERPROP_OP(CLUSTERPROP, (req, rsp, h) -> {
- String name = req.getParams().required().get(NAME);
- String val = req.getParams().get(VALUE_LONG);
- ClusterProperties cp = new ClusterProperties(h.coreContainer.getZkController().getZkClient());
- cp.setClusterProperty(name, val);
- return null;
- }),
- COLLECTIONPROP_OP(COLLECTIONPROP, (req, rsp, h) -> {
- String extCollection = req.getParams().required().get(NAME);
- String collection = h.coreContainer.getZkController().getZkStateReader().getAliases().resolveSimpleAlias(extCollection);
- String name = req.getParams().required().get(PROPERTY_NAME);
- String val = req.getParams().get(PROPERTY_VALUE);
- CollectionProperties cp = new CollectionProperties(h.coreContainer.getZkController().getZkClient());
- cp.setCollectionProperty(collection, name, val);
- return null;
- }),
+ Map<String, Object> map =
+ copy(
+ req.getParams(),
+ null,
+ COLLECTION_PROP,
+ SHARD_ID_PROP,
+ "split.key",
+ CoreAdminParams.RANGES,
+ WAIT_FOR_FINAL_STATE,
+ TIMING,
+ SPLIT_METHOD,
+ NUM_SUB_SHARDS,
+ SPLIT_FUZZ,
+ SPLIT_BY_PREFIX,
+ FOLLOW_ALIASES);
+ return copyPropertiesWithPrefix(req.getParams(), map, PROPERTY_PREFIX);
+ }),
+ DELETESHARD_OP(
+ DELETESHARD,
+ (req, rsp, h) -> {
+ Map<String, Object> map =
+ copy(req.getParams().required(), null, COLLECTION_PROP, SHARD_ID_PROP);
+ copy(
+ req.getParams(),
+ map,
+ DELETE_INDEX,
+ DELETE_DATA_DIR,
+ DELETE_INSTANCE_DIR,
+ FOLLOW_ALIASES);
+ return map;
+ }),
+ FORCELEADER_OP(
+ FORCELEADER,
+ (req, rsp, h) -> {
+ forceLeaderElection(req, h);
+ return null;
+ }),
+ CREATESHARD_OP(
+ CREATESHARD,
+ (req, rsp, h) -> {
+ Map<String, Object> map =
+ copy(req.getParams().required(), null, COLLECTION_PROP, SHARD_ID_PROP);
+ ClusterState clusterState = h.coreContainer.getZkController().getClusterState();
+ final String newShardName =
+ SolrIdentifierValidator.validateShardName(req.getParams().get(SHARD_ID_PROP));
+ boolean followAliases = req.getParams().getBool(FOLLOW_ALIASES, false);
+ String extCollectionName = req.getParams().get(COLLECTION_PROP);
+ String collectionName =
+ followAliases
+ ? h.coreContainer
+ .getZkController()
+ .getZkStateReader()
+ .getAliases()
+ .resolveSimpleAlias(extCollectionName)
+ : extCollectionName;
+ if (!ImplicitDocRouter.NAME.equals(
+ ((Map<?, ?>) clusterState.getCollection(collectionName).get(DOC_ROUTER)).get(NAME)))
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST, "shards can be added only to 'implicit' collections");
+ copy(
+ req.getParams(),
+ map,
+ REPLICATION_FACTOR,
+ NRT_REPLICAS,
+ TLOG_REPLICAS,
+ PULL_REPLICAS,
+ CREATE_NODE_SET,
+ WAIT_FOR_FINAL_STATE,
+ FOLLOW_ALIASES);
+ return copyPropertiesWithPrefix(req.getParams(), map, PROPERTY_PREFIX);
+ }),
+ DELETEREPLICA_OP(
+ DELETEREPLICA,
+ (req, rsp, h) -> {
+ Map<String, Object> map = copy(req.getParams().required(), null, COLLECTION_PROP);
+
+ return copy(
+ req.getParams(),
+ map,
+ DELETE_INDEX,
+ DELETE_DATA_DIR,
+ DELETE_INSTANCE_DIR,
+ COUNT_PROP,
+ REPLICA_PROP,
+ SHARD_ID_PROP,
+ ONLY_IF_DOWN,
+ FOLLOW_ALIASES);
+ }),
+ MIGRATE_OP(
+ MIGRATE,
+ (req, rsp, h) -> {
+ Map<String, Object> map =
+ copy(
+ req.getParams().required(),
+ null,
+ COLLECTION_PROP,
+ "split.key",
+ "target.collection");
+ return copy(req.getParams(), map, "forward.timeout", FOLLOW_ALIASES);
+ }),
+ ADDROLE_OP(
+ ADDROLE,
+ (req, rsp, h) -> {
+ Map<String, Object> map = copy(req.getParams().required(), null, "role", "node");
+ if (!KNOWN_ROLES.contains(map.get("role")))
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST, "Unknown role. Supported roles are ," + KNOWN_ROLES);
+ return map;
+ }),
+ REMOVEROLE_OP(
+ REMOVEROLE,
+ (req, rsp, h) -> {
+ Map<String, Object> map = copy(req.getParams().required(), null, "role", "node");
+ if (!KNOWN_ROLES.contains(map.get("role")))
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST, "Unknown role. Supported roles are ," + KNOWN_ROLES);
+ return map;
+ }),
+ CLUSTERPROP_OP(
+ CLUSTERPROP,
+ (req, rsp, h) -> {
+ String name = req.getParams().required().get(NAME);
+ String val = req.getParams().get(VALUE_LONG);
+ ClusterProperties cp =
+ new ClusterProperties(h.coreContainer.getZkController().getZkClient());
+ cp.setClusterProperty(name, val);
+ return null;
+ }),
+ COLLECTIONPROP_OP(
+ COLLECTIONPROP,
+ (req, rsp, h) -> {
+ String extCollection = req.getParams().required().get(NAME);
+ String collection =
+ h.coreContainer
+ .getZkController()
+ .getZkStateReader()
+ .getAliases()
+ .resolveSimpleAlias(extCollection);
+ String name = req.getParams().required().get(PROPERTY_NAME);
+ String val = req.getParams().get(PROPERTY_VALUE);
+ CollectionProperties cp =
+ new CollectionProperties(h.coreContainer.getZkController().getZkClient());
+ cp.setCollectionProperty(collection, name, val);
+ return null;
+ }),
@SuppressWarnings({"unchecked"})
- REQUESTSTATUS_OP(REQUESTSTATUS, (req, rsp, h) -> {
- req.getParams().required().check(REQUESTID);
-
- final CoreContainer coreContainer = h.coreContainer;
- final String requestId = req.getParams().get(REQUESTID);
- final ZkController zkController = coreContainer.getZkController();
-
- final NamedList<Object> status = new NamedList<>();
- if (coreContainer.getDistributedCollectionCommandRunner().isEmpty()) {
- if (zkController.getOverseerCompletedMap().contains(requestId)) {
- final byte[] mapEntry = zkController.getOverseerCompletedMap().get(requestId);
- rsp.getValues().addAll(OverseerSolrResponseSerializer.deserialize(mapEntry).getResponse());
- addStatusToResponse(status, COMPLETED, "found [" + requestId + "] in completed tasks");
- } else if (zkController.getOverseerFailureMap().contains(requestId)) {
- final byte[] mapEntry = zkController.getOverseerFailureMap().get(requestId);
- rsp.getValues().addAll(OverseerSolrResponseSerializer.deserialize(mapEntry).getResponse());
- addStatusToResponse(status, FAILED, "found [" + requestId + "] in failed tasks");
- } else if (zkController.getOverseerRunningMap().contains(requestId)) {
- addStatusToResponse(status, RUNNING, "found [" + requestId + "] in running tasks");
- } else if (h.overseerCollectionQueueContains(requestId)) {
- addStatusToResponse(status, SUBMITTED, "found [" + requestId + "] in submitted tasks");
- } else {
- addStatusToResponse(status, NOT_FOUND, "Did not find [" + requestId + "] in any tasks queue");
- }
- } else {
- Pair<RequestStatusState, OverseerSolrResponse> sr =
- coreContainer.getDistributedCollectionCommandRunner().get().getAsyncTaskRequestStatus(requestId);
- final String message;
- switch (sr.first()) {
- case COMPLETED:
- message = "found [" + requestId + "] in completed tasks";
- rsp.getValues().addAll(sr.second().getResponse());
- break;
- case FAILED:
- message = "found [" + requestId + "] in failed tasks";
- rsp.getValues().addAll(sr.second().getResponse());
- break;
- case RUNNING:
- message = "found [" + requestId + "] in running tasks";
- break;
- case SUBMITTED:
- message = "found [" + requestId + "] in submitted tasks";
- break;
- default:
- message = "Did not find [" + requestId + "] in any tasks queue";
- }
- addStatusToResponse(status, sr.first(), message);
- }
-
- rsp.getValues().addAll(status);
- return null;
- }),
- DELETESTATUS_OP(DELETESTATUS, new CollectionOp() {
- @SuppressWarnings("unchecked")
- @Override
- public Map<String, Object> execute(SolrQueryRequest req, SolrQueryResponse rsp, CollectionsHandler h) throws Exception {
- final CoreContainer coreContainer = h.coreContainer;
- final String requestId = req.getParams().get(REQUESTID);
- final ZkController zkController = coreContainer.getZkController();
- Boolean flush = req.getParams().getBool(CollectionAdminParams.FLUSH, false);
-
- if (requestId == null && !flush) {
- throw new SolrException(ErrorCode.BAD_REQUEST, "Either requestid or flush parameter must be specified.");
- }
-
- if (requestId != null && flush) {
- throw new SolrException(ErrorCode.BAD_REQUEST,
- "Both requestid and flush parameters can not be specified together.");
- }
-
- if (coreContainer.getDistributedCollectionCommandRunner().isEmpty()) {
- if (flush) {
- Collection<String> completed = zkController.getOverseerCompletedMap().keys();
- Collection<String> failed = zkController.getOverseerFailureMap().keys();
- for (String asyncId : completed) {
- zkController.getOverseerCompletedMap().remove(asyncId);
- zkController.clearAsyncId(asyncId);
- }
- for (String asyncId : failed) {
- zkController.getOverseerFailureMap().remove(asyncId);
- zkController.clearAsyncId(asyncId);
+ REQUESTSTATUS_OP(
+ REQUESTSTATUS,
+ (req, rsp, h) -> {
+ req.getParams().required().check(REQUESTID);
+
+ final CoreContainer coreContainer = h.coreContainer;
+ final String requestId = req.getParams().get(REQUESTID);
+ final ZkController zkController = coreContainer.getZkController();
+
+ final NamedList<Object> status = new NamedList<>();
+ if (coreContainer.getDistributedCollectionCommandRunner().isEmpty()) {
+ if (zkController.getOverseerCompletedMap().contains(requestId)) {
+ final byte[] mapEntry = zkController.getOverseerCompletedMap().get(requestId);
+ rsp.getValues()
+ .addAll(OverseerSolrResponseSerializer.deserialize(mapEntry).getResponse());
+ addStatusToResponse(
+ status, COMPLETED, "found [" + requestId + "] in completed tasks");
+ } else if (zkController.getOverseerFailureMap().contains(requestId)) {
+ final byte[] mapEntry = zkController.getOverseerFailureMap().get(requestId);
+ rsp.getValues()
+ .addAll(OverseerSolrResponseSerializer.deserialize(mapEntry).getResponse());
+ addStatusToResponse(status, FAILED, "found [" + requestId + "] in failed tasks");
+ } else if (zkController.getOverseerRunningMap().contains(requestId)) {
+ addStatusToResponse(status, RUNNING, "found [" + requestId + "] in running tasks");
+ } else if (h.overseerCollectionQueueContains(requestId)) {
+ addStatusToResponse(
+ status, SUBMITTED, "found [" + requestId + "] in submitted tasks");
+ } else {
+ addStatusToResponse(
+ status, NOT_FOUND, "Did not find [" + requestId + "] in any tasks queue");
}
- rsp.getValues().add("status", "successfully cleared stored collection api responses");
} else {
- // Request to cleanup
- if (zkController.getOverseerCompletedMap().remove(requestId)) {
- zkController.clearAsyncId(requestId);
- rsp.getValues().add("status", "successfully removed stored response for [" + requestId + "]");
- } else if (zkController.getOverseerFailureMap().remove(requestId)) {
- zkController.clearAsyncId(requestId);
- rsp.getValues().add("status", "successfully removed stored response for [" + requestId + "]");
- } else {
- rsp.getValues().add("status", "[" + requestId + "] not found in stored responses");
- // Don't call zkController.clearAsyncId for this, since it could be a running/pending task
+ Pair<RequestStatusState, OverseerSolrResponse> sr =
+ coreContainer
+ .getDistributedCollectionCommandRunner()
+ .get()
+ .getAsyncTaskRequestStatus(requestId);
+ final String message;
+ switch (sr.first()) {
+ case COMPLETED:
+ message = "found [" + requestId + "] in completed tasks";
+ rsp.getValues().addAll(sr.second().getResponse());
+ break;
+ case FAILED:
+ message = "found [" + requestId + "] in failed tasks";
+ rsp.getValues().addAll(sr.second().getResponse());
+ break;
+ case RUNNING:
+ message = "found [" + requestId + "] in running tasks";
+ break;
+ case SUBMITTED:
+ message = "found [" + requestId + "] in submitted tasks";
+ break;
+ default:
+ message = "Did not find [" + requestId + "] in any tasks queue";
}
+ addStatusToResponse(status, sr.first(), message);
}
- } else {
- if (flush) {
- coreContainer.getDistributedCollectionCommandRunner().get().deleteAllAsyncIds();
- rsp.getValues().add("status", "successfully cleared stored collection api responses");
- } else {
- if (coreContainer.getDistributedCollectionCommandRunner().get().deleteSingleAsyncId(requestId)) {
- rsp.getValues().add("status", "successfully removed stored response for [" + requestId + "]");
+
+ rsp.getValues().addAll(status);
+ return null;
+ }),
+ DELETESTATUS_OP(
+ DELETESTATUS,
+ new CollectionOp() {
+ @SuppressWarnings("unchecked")
+ @Override
+ public Map<String, Object> execute(
+ SolrQueryRequest req, SolrQueryResponse rsp, CollectionsHandler h) throws Exception {
+ final CoreContainer coreContainer = h.coreContainer;
+ final String requestId = req.getParams().get(REQUESTID);
+ final ZkController zkController = coreContainer.getZkController();
+ Boolean flush = req.getParams().getBool(CollectionAdminParams.FLUSH, false);
+
+ if (requestId == null && !flush) {
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST, "Either requestid or flush parameter must be specified.");
+ }
+
+ if (requestId != null && flush) {
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST,
+ "Both requestid and flush parameters can not be specified together.");
+ }
+
+ if (coreContainer.getDistributedCollectionCommandRunner().isEmpty()) {
+ if (flush) {
+ Collection<String> completed = zkController.getOverseerCompletedMap().keys();
+ Collection<String> failed = zkController.getOverseerFailureMap().keys();
+ for (String asyncId : completed) {
+ zkController.getOverseerCompletedMap().remove(asyncId);
+ zkController.clearAsyncId(asyncId);
+ }
+ for (String asyncId : failed) {
+ zkController.getOverseerFailureMap().remove(asyncId);
+ zkController.clearAsyncId(asyncId);
+ }
+ rsp.getValues()
+ .add("status", "successfully cleared stored collection api responses");
+ } else {
+ // Request to cleanup
+ if (zkController.getOverseerCompletedMap().remove(requestId)) {
+ zkController.clearAsyncId(requestId);
+ rsp.getValues()
+ .add(
+ "status", "successfully removed stored response for [" + requestId + "]");
+ } else if (zkController.getOverseerFailureMap().remove(requestId)) {
+ zkController.clearAsyncId(requestId);
+ rsp.getValues()
+ .add(
+ "status", "successfully removed stored response for [" + requestId + "]");
+ } else {
+ rsp.getValues()
+ .add("status", "[" + requestId + "] not found in stored responses");
+ // Don't call zkController.clearAsyncId for this, since it could be a
+ // running/pending task
+ }
+ }
} else {
- rsp.getValues().add("status", "[" + requestId + "] not found in stored responses");
+ if (flush) {
+ coreContainer.getDistributedCollectionCommandRunner().get().deleteAllAsyncIds();
+ rsp.getValues()
+ .add("status", "successfully cleared stored collection api responses");
+ } else {
+ if (coreContainer
+ .getDistributedCollectionCommandRunner()
+ .get()
+ .deleteSingleAsyncId(requestId)) {
+ rsp.getValues()
+ .add(
+ "status", "successfully removed stored response for [" + requestId + "]");
+ } else {
+ rsp.getValues()
+ .add("status", "[" + requestId + "] not found in stored responses");
+ }
+ }
}
+ return null;
}
- }
- return null;
- }
- }),
- ADDREPLICA_OP(ADDREPLICA, (req, rsp, h) -> {
- Map<String, Object> props = copy(req.getParams(), null,
- COLLECTION_PROP,
- "node",
- SHARD_ID_PROP,
- _ROUTE_,
- CoreAdminParams.NAME,
- INSTANCE_DIR,
- DATA_DIR,
- ULOG_DIR,
- REPLICA_TYPE,
- WAIT_FOR_FINAL_STATE,
- NRT_REPLICAS,
- TLOG_REPLICAS,
- PULL_REPLICAS,
- CREATE_NODE_SET,
- FOLLOW_ALIASES,
- SKIP_NODE_ASSIGNMENT);
- return copyPropertiesWithPrefix(req.getParams(), props, PROPERTY_PREFIX);
- }),
+ }),
+ ADDREPLICA_OP(
+ ADDREPLICA,
+ (req, rsp, h) -> {
+ Map<String, Object> props =
+ copy(
+ req.getParams(),
+ null,
+ COLLECTION_PROP,
+ "node",
+ SHARD_ID_PROP,
+ _ROUTE_,
+ CoreAdminParams.NAME,
+ INSTANCE_DIR,
+ DATA_DIR,
+ ULOG_DIR,
+ REPLICA_TYPE,
+ WAIT_FOR_FINAL_STATE,
+ NRT_REPLICAS,
+ TLOG_REPLICAS,
+ PULL_REPLICAS,
+ CREATE_NODE_SET,
+ FOLLOW_ALIASES,
+ SKIP_NODE_ASSIGNMENT);
+ return copyPropertiesWithPrefix(req.getParams(), props, PROPERTY_PREFIX);
+ }),
OVERSEERSTATUS_OP(OVERSEERSTATUS, (req, rsp, h) -> new LinkedHashMap<>()),
@SuppressWarnings({"unchecked"})
- DISTRIBUTEDAPIPROCESSING_OP(DISTRIBUTEDAPIPROCESSING, (req, rsp, h) -> {
- NamedList<Object> results = new NamedList<>();
- boolean isDistributedApi = h.coreContainer.getDistributedCollectionCommandRunner().isPresent();
- results.add("isDistributedApi", isDistributedApi);
- rsp.getValues().addAll(results);
- return null;
- }),
- /**
- * Handle list collection request.
- * Do list collection request to zk host
- */
+ DISTRIBUTEDAPIPROCESSING_OP(
+ DISTRIBUTEDAPIPROCESSING,
+ (req, rsp, h) -> {
+ NamedList<Object> results = new NamedList<>();
+ boolean isDistributedApi =
+ h.coreContainer.getDistributedCollectionCommandRunner().isPresent();
+ results.add("isDistributedApi", isDistributedApi);
+ rsp.getValues().addAll(results);
+ return null;
+ }),
+ /** Handle list collection request. Do list collection request to zk host */
@SuppressWarnings({"unchecked"})
- LIST_OP(LIST, (req, rsp, h) -> {
- NamedList<Object> results = new NamedList<>();
- Map<String, DocCollection> collections = h.coreContainer.getZkController().getZkStateReader().getClusterState().getCollectionsMap();
- List<String> collectionList = new ArrayList<>(collections.keySet());
- Collections.sort(collectionList);
- // XXX should we add aliases here?
- results.add("collections", collectionList);
- SolrResponse response = new OverseerSolrResponse(results);
- rsp.getValues().addAll(response.getResponse());
- return null;
- }),
+ LIST_OP(
+ LIST,
+ (req, rsp, h) -> {
+ NamedList<Object> results = new NamedList<>();
+ Map<String, DocCollection> collections =
+ h.coreContainer
+ .getZkController()
+ .getZkStateReader()
+ .getClusterState()
+ .getCollectionsMap();
+ List<String> collectionList = new ArrayList<>(collections.keySet());
+ Collections.sort(collectionList);
+ // XXX should we add aliases here?
+ results.add("collections", collectionList);
+ SolrResponse response = new OverseerSolrResponse(results);
+ rsp.getValues().addAll(response.getResponse());
+ return null;
+ }),
/**
- * Handle cluster status request.
- * Can return status per specific collection/shard or per all collections.
+ * Handle cluster status request. Can return status per specific collection/shard or per all
+ * collections.
*/
- CLUSTERSTATUS_OP(CLUSTERSTATUS, (req, rsp, h) -> {
- Map<String, Object> all = copy(req.getParams(), null,
- COLLECTION_PROP,
- SHARD_ID_PROP,
- _ROUTE_);
- new ClusterStatus(h.coreContainer.getZkController().getZkStateReader(),
- new ZkNodeProps(all)).getClusterStatus(rsp.getValues());
- return null;
- }),
- ADDREPLICAPROP_OP(ADDREPLICAPROP, (req, rsp, h) -> {
- Map<String, Object> map = copy(req.getParams().required(), null,
- COLLECTION_PROP,
- PROPERTY_PROP,
- SHARD_ID_PROP,
- REPLICA_PROP,
- PROPERTY_VALUE_PROP);
- copy(req.getParams(), map, SHARD_UNIQUE);
- String property = (String) map.get(PROPERTY_PROP);
- if (!property.startsWith(PROPERTY_PREFIX)) {
- property = PROPERTY_PREFIX + property;
- }
+ CLUSTERSTATUS_OP(
+ CLUSTERSTATUS,
+ (req, rsp, h) -> {
+ Map<String, Object> all =
+ copy(req.getParams(), null, COLLECTION_PROP, SHARD_ID_PROP, _ROUTE_);
+ new ClusterStatus(
+ h.coreContainer.getZkController().getZkStateReader(), new ZkNodeProps(all))
+ .getClusterStatus(rsp.getValues());
+ return null;
+ }),
+ ADDREPLICAPROP_OP(
+ ADDREPLICAPROP,
+ (req, rsp, h) -> {
+ Map<String, Object> map =
+ copy(
+ req.getParams().required(),
+ null,
+ COLLECTION_PROP,
+ PROPERTY_PROP,
+ SHARD_ID_PROP,
+ REPLICA_PROP,
+ PROPERTY_VALUE_PROP);
+ copy(req.getParams(), map, SHARD_UNIQUE);
+ String property = (String) map.get(PROPERTY_PROP);
+ if (!property.startsWith(PROPERTY_PREFIX)) {
+ property = PROPERTY_PREFIX + property;
+ }
- boolean uniquePerSlice = Boolean.parseBoolean((String) map.get(SHARD_UNIQUE));
-
- // Check if we're trying to set a property with parameters that allow us to set the property on multiple replicas
- // in a slice on properties that are known to only be one-per-slice and error out if so.
- if (StringUtils.isNotBlank((String) map.get(SHARD_UNIQUE)) &&
- SliceMutator.SLICE_UNIQUE_BOOLEAN_PROPERTIES.contains(property.toLowerCase(Locale.ROOT)) &&
- uniquePerSlice == false) {
- throw new SolrException(ErrorCode.BAD_REQUEST,
- "Overseer replica property command received for property " + property +
- " with the " + SHARD_UNIQUE +
- " parameter set to something other than 'true'. No action taken.");
- }
- return map;
- }),
+ boolean uniquePerSlice = Boolean.parseBoolean((String) map.get(SHARD_UNIQUE));
+
+ // Check if we're trying to set a property with parameters that allow us to set the
+ // property on multiple replicas
+ // in a slice on properties that are known to only be one-per-slice and error out if so.
+ if (StringUtils.isNotBlank((String) map.get(SHARD_UNIQUE))
+ && SliceMutator.SLICE_UNIQUE_BOOLEAN_PROPERTIES.contains(
+ property.toLowerCase(Locale.ROOT))
+ && uniquePerSlice == false) {
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST,
+ "Overseer replica property command received for property "
+ + property
+ + " with the "
+ + SHARD_UNIQUE
+ + " parameter set to something other than 'true'. No action taken.");
+ }
+ return map;
+ }),
// XXX should this command support followAliases?
- DELETEREPLICAPROP_OP(DELETEREPLICAPROP, (req, rsp, h) -> {
- Map<String, Object> map = copy(req.getParams().required(), null,
- COLLECTION_PROP,
- PROPERTY_PROP,
- SHARD_ID_PROP,
- REPLICA_PROP);
- return copy(req.getParams(), map, PROPERTY_PROP);
- }),
+ DELETEREPLICAPROP_OP(
+ DELETEREPLICAPROP,
+ (req, rsp, h) -> {
+ Map<String, Object> map =
+ copy(
+ req.getParams().required(),
+ null,
+ COLLECTION_PROP,
+ PROPERTY_PROP,
+ SHARD_ID_PROP,
+ REPLICA_PROP);
+ return copy(req.getParams(), map, PROPERTY_PROP);
+ }),
// XXX should this command support followAliases?
- BALANCESHARDUNIQUE_OP(BALANCESHARDUNIQUE, (req, rsp, h) -> {
- Map<String, Object> map = copy(req.getParams().required(), null,
- COLLECTION_PROP,
- PROPERTY_PROP);
- Boolean shardUnique = Boolean.parseBoolean(req.getParams().get(SHARD_UNIQUE));
- String prop = req.getParams().get(PROPERTY_PROP).toLowerCase(Locale.ROOT);
- if (!StringUtils.startsWith(prop, PROPERTY_PREFIX)) {
- prop = PROPERTY_PREFIX + prop;
- }
+ BALANCESHARDUNIQUE_OP(
+ BALANCESHARDUNIQUE,
+ (req, rsp, h) -> {
+ Map<String, Object> map =
+ copy(req.getParams().required(), null, COLLECTION_PROP, PROPERTY_PROP);
+ Boolean shardUnique = Boolean.parseBoolean(req.getParams().get(SHARD_UNIQUE));
+ String prop = req.getParams().get(PROPERTY_PROP).toLowerCase(Locale.ROOT);
+ if (!StringUtils.startsWith(prop, PROPERTY_PREFIX)) {
+ prop = PROPERTY_PREFIX + prop;
+ }
- if (!shardUnique && !SliceMutator.SLICE_UNIQUE_BOOLEAN_PROPERTIES.contains(prop)) {
- throw new SolrException(ErrorCode.BAD_REQUEST, "Balancing properties amongst replicas in a slice requires that"
- + " the property be pre-defined as a unique property (e.g. 'preferredLeader') or that 'shardUnique' be set to 'true'. " +
- " Property: " + prop + " shardUnique: " + shardUnique);
- }
+ if (!shardUnique && !SliceMutator.SLICE_UNIQUE_BOOLEAN_PROPERTIES.contains(prop)) {
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST,
+ "Balancing properties amongst replicas in a slice requires that"
+ + " the property be pre-defined as a unique property (e.g. 'preferredLeader') or that 'shardUnique' be set to 'true'. "
+ + " Property: "
+ + prop
+ + " shardUnique: "
+ + shardUnique);
+ }
- return copy(req.getParams(), map, ONLY_ACTIVE_NODES, SHARD_UNIQUE);
- }),
- REBALANCELEADERS_OP(REBALANCELEADERS, (req, rsp, h) -> {
- new RebalanceLeaders(req, rsp, h).execute();
- return null;
- }),
+ return copy(req.getParams(), map, ONLY_ACTIVE_NODES, SHARD_UNIQUE);
+ }),
+ REBALANCELEADERS_OP(
+ REBALANCELEADERS,
+ (req, rsp, h) -> {
+ new RebalanceLeaders(req, rsp, h).execute();
+ return null;
+ }),
// XXX should this command support followAliases?
- MODIFYCOLLECTION_OP(MODIFYCOLLECTION, (req, rsp, h) -> {
- Map<String, Object> m = copy(req.getParams(), null, CollectionAdminRequest.MODIFIABLE_COLLECTION_PROPERTIES);
- copyPropertiesWithPrefix(req.getParams(), m, PROPERTY_PREFIX);
- if (m.isEmpty()) {
- throw new SolrException(ErrorCode.BAD_REQUEST,
- formatString("no supported values provided {0}", CollectionAdminRequest.MODIFIABLE_COLLECTION_PROPERTIES.toString()));
- }
- copy(req.getParams().required(), m, COLLECTION_PROP);
- for (Map.Entry<String, Object> entry : m.entrySet()) {
- String prop = entry.getKey();
- if ("".equals(entry.getValue())) {
- // set to an empty string is equivalent to removing the property, see SOLR-12507
- entry.setValue(null);
- }
- DocCollection.verifyProp(m, prop);
- }
- if (m.get(REPLICATION_FACTOR) != null) {
- m.put(NRT_REPLICAS, m.get(REPLICATION_FACTOR));
- }
- return m;
- }),
- BACKUP_OP(BACKUP, (req, rsp, h) -> {
- req.getParams().required().check(NAME, COLLECTION_PROP);
-
- final String extCollectionName = req.getParams().get(COLLECTION_PROP);
- final boolean followAliases = req.getParams().getBool(FOLLOW_ALIASES, false);
- final String collectionName = followAliases ? h.coreContainer.getZkController().getZkStateReader()
- .getAliases().resolveSimpleAlias(extCollectionName) : extCollectionName;
- final ClusterState clusterState = h.coreContainer.getZkController().getClusterState();
- if (!clusterState.hasCollection(collectionName)) {
- throw new SolrException(ErrorCode.BAD_REQUEST, "Collection '" + collectionName + "' does not exist, no action taken.");
- }
-
- CoreContainer cc = h.coreContainer;
- String repo = req.getParams().get(CoreAdminParams.BACKUP_REPOSITORY);
- BackupRepository repository = cc.newBackupRepository(repo);
-
- String location = repository.getBackupLocation(req.getParams().get(CoreAdminParams.BACKUP_LOCATION));
- if (location == null) {
- //Refresh the cluster property file to make sure the value set for location is the latest
- // Check if the location is specified in the cluster property.
- location = new ClusterProperties(h.coreContainer.getZkController().getZkClient()).getClusterProperty(CoreAdminParams.BACKUP_LOCATION, null);
- if (location == null) {
- throw new SolrException(ErrorCode.BAD_REQUEST, "'location' is not specified as a query"
- + " parameter or as a default repository property or as a cluster property.");
- }
- }
- boolean incremental = req.getParams().getBool(CoreAdminParams.BACKUP_INCREMENTAL, true);
-
- // Check if the specified location is valid for this repository.
- final URI uri = repository.createDirectoryURI(location);
- try {
- if (!repository.exists(uri)) {
- throw new SolrException(ErrorCode.SERVER_ERROR, "specified location " + uri + " does not exist.");
- }
- } catch (IOException ex) {
- throw new SolrException(ErrorCode.SERVER_ERROR, "Failed to check the existence of " + uri + ". Is it valid?", ex);
- }
+ MODIFYCOLLECTION_OP(
+ MODIFYCOLLECTION,
+ (req, rsp, h) -> {
+ Map<String, Object> m =
+ copy(req.getParams(), null, CollectionAdminRequest.MODIFIABLE_COLLECTION_PROPERTIES);
+ copyPropertiesWithPrefix(req.getParams(), m, PROPERTY_PREFIX);
+ if (m.isEmpty()) {
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST,
+ formatString(
+ "no supported values provided {0}",
+ CollectionAdminRequest.MODIFIABLE_COLLECTION_PROPERTIES.toString()));
+ }
+ copy(req.getParams().required(), m, COLLECTION_PROP);
+ for (Map.Entry<String, Object> entry : m.entrySet()) {
+ String prop = entry.getKey();
+ if ("".equals(entry.getValue())) {
+ // set to an empty string is equivalent to removing the property, see SOLR-12507
+ entry.setValue(null);
+ }
+ DocCollection.verifyProp(m, prop);
+ }
+ if (m.get(REPLICATION_FACTOR) != null) {
+ m.put(NRT_REPLICAS, m.get(REPLICATION_FACTOR));
+ }
+ return m;
+ }),
+ BACKUP_OP(
+ BACKUP,
+ (req, rsp, h) -> {
+ req.getParams().required().check(NAME, COLLECTION_PROP);
+
+ final String extCollectionName = req.getParams().get(COLLECTION_PROP);
+ final boolean followAliases = req.getParams().getBool(FOLLOW_ALIASES, false);
+ final String collectionName =
+ followAliases
+ ? h.coreContainer
+ .getZkController()
+ .getZkStateReader()
+ .getAliases()
+ .resolveSimpleAlias(extCollectionName)
+ : extCollectionName;
+ final ClusterState clusterState = h.coreContainer.getZkController().getClusterState();
+ if (!clusterState.hasCollection(collectionName)) {
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST,
+ "Collection '" + collectionName + "' does not exist, no action taken.");
+ }
- String strategy = req.getParams().get(CollectionAdminParams.INDEX_BACKUP_STRATEGY, CollectionAdminParams.COPY_FILES_STRATEGY);
- if (!CollectionAdminParams.INDEX_BACKUP_STRATEGIES.contains(strategy)) {
- throw new SolrException(ErrorCode.BAD_REQUEST, "Unknown index backup strategy " + strategy);
- }
+ CoreContainer cc = h.coreContainer;
+ String repo = req.getParams().get(CoreAdminParams.BACKUP_REPOSITORY);
+ BackupRepository repository = cc.newBackupRepository(repo);
- Map<String, Object> params = copy(req.getParams(), null, NAME, COLLECTION_PROP,
- FOLLOW_ALIASES, CoreAdminParams.COMMIT_NAME, CoreAdminParams.MAX_NUM_BACKUP_POINTS);
- params.put(CoreAdminParams.BACKUP_LOCATION, location);
- if (repo != null) {
- params.put(CoreAdminParams.BACKUP_REPOSITORY, repo);
- }
+ String location =
+ repository.getBackupLocation(req.getParams().get(CoreAdminParams.BACKUP_LOCATION));
+ if (location == null) {
+ // Refresh the cluster property file to make sure the value set for location is the
+ // latest
+ // Check if the location is specified in the cluster property.
+ location =
+ new ClusterProperties(h.coreContainer.getZkController().getZkClient())
+ .getClusterProperty(CoreAdminParams.BACKUP_LOCATION, null);
+ if (location == null) {
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST,
+ "'location' is not specified as a query"
+ + " parameter or as a default repository property or as a cluster property.");
+ }
+ }
+ boolean incremental = req.getParams().getBool(CoreAdminParams.BACKUP_INCREMENTAL, true);
- params.put(CollectionAdminParams.INDEX_BACKUP_STRATEGY, strategy);
- params.put(CoreAdminParams.BACKUP_INCREMENTAL, incremental);
- return params;
- }),
- RESTORE_OP(RESTORE, (req, rsp, h) -> {
- req.getParams().required().check(NAME, COLLECTION_PROP);
+ // Check if the specified location is valid for this repository.
+ final URI uri = repository.createDirectoryURI(location);
+ try {
+ if (!repository.exists(uri)) {
+ throw new SolrException(
+ ErrorCode.SERVER_ERROR, "specified location " + uri + " does not exist.");
+ }
+ } catch (IOException ex) {
+ throw new SolrException(
+ ErrorCode.SERVER_ERROR,
+ "Failed to check the existence of " + uri + ". Is it valid?",
+ ex);
+ }
- final String collectionName = SolrIdentifierValidator.validateCollectionName(req.getParams().get(COLLECTION_PROP));
- if (h.coreContainer.getZkController().getZkStateReader().getAliases().hasAlias(collectionName)) {
- throw new SolrException(ErrorCode.BAD_REQUEST, "Collection '" + collectionName + "' is an existing alias, no action taken.");
- }
+ String strategy =
+ req.getParams()
+ .get(
+ CollectionAdminParams.INDEX_BACKUP_STRATEGY,
+ CollectionAdminParams.COPY_FILES_STRATEGY);
+ if (!CollectionAdminParams.INDEX_BACKUP_STRATEGIES.contains(strategy)) {
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST, "Unknown index backup strategy " + strategy);
+ }
- final CoreContainer cc = h.coreContainer;
- final String repo = req.getParams().get(CoreAdminParams.BACKUP_REPOSITORY);
- final BackupRepository repository = cc.newBackupRepository(repo);
-
- String location = repository.getBackupLocation(req.getParams().get(CoreAdminParams.BACKUP_LOCATION));
- if (location == null) {
- //Refresh the cluster property file to make sure the value set for location is the latest
- // Check if the location is specified in the cluster property.
- location = new ClusterProperties(h.coreContainer.getZkController().getZkClient()).getClusterProperty("location", null);
- if (location == null) {
- throw new SolrException(ErrorCode.BAD_REQUEST, "'location' is not specified as a query"
- + " parameter or as a default repository property or as a cluster property.");
- }
- }
+ Map<String, Object> params =
+ copy(
+ req.getParams(),
+ null,
+ NAME,
+ COLLECTION_PROP,
+ FOLLOW_ALIASES,
+ CoreAdminParams.COMMIT_NAME,
+ CoreAdminParams.MAX_NUM_BACKUP_POINTS);
+ params.put(CoreAdminParams.BACKUP_LOCATION, location);
+ if (repo != null) {
+ params.put(CoreAdminParams.BACKUP_REPOSITORY, repo);
+ }
- // Check if the specified location is valid for this repository.
- final URI uri = repository.createDirectoryURI(location);
- try {
- if (!repository.exists(uri)) {
- throw new SolrException(ErrorCode.SERVER_ERROR, "specified location " + uri + " does not exist.");
- }
- } catch (IOException ex) {
- throw new SolrException(ErrorCode.SERVER_ERROR, "Failed to check the existence of " + uri + ". Is it valid?", ex);
- }
+ params.put(CollectionAdminParams.INDEX_BACKUP_STRATEGY, strategy);
+ params.put(CoreAdminParams.BACKUP_INCREMENTAL, incremental);
+ return params;
+ }),
+ RESTORE_OP(
+ RESTORE,
+ (req, rsp, h) -> {
+ req.getParams().required().check(NAME, COLLECTION_PROP);
+
+ final String collectionName =
+ SolrIdentifierValidator.validateCollectionName(req.getParams().get(COLLECTION_PROP));
+ if (h.coreContainer
+ .getZkController()
+ .getZkStateReader()
+ .getAliases()
+ .hasAlias(collectionName)) {
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST,
+ "Collection '" + collectionName + "' is an existing alias, no action taken.");
+ }
- final String createNodeArg = req.getParams().get(CREATE_NODE_SET);
- if (CREATE_NODE_SET_EMPTY.equals(createNodeArg)) {
- throw new SolrException(
- SolrException.ErrorCode.BAD_REQUEST,
- "Cannot restore with a CREATE_NODE_SET of CREATE_NODE_SET_EMPTY."
- );
- }
- if (req.getParams().get(NRT_REPLICAS) != null && req.getParams().get(REPLICATION_FACTOR) != null) {
- throw new SolrException(SolrException.ErrorCode.BAD_REQUEST,
- "Cannot set both replicationFactor and nrtReplicas as they mean the same thing");
- }
+ final CoreContainer cc = h.coreContainer;
+ final String repo = req.getParams().get(CoreAdminParams.BACKUP_REPOSITORY);
+ final BackupRepository repository = cc.newBackupRepository(repo);
- final Map<String, Object> params = copy(req.getParams(), null, NAME, COLLECTION_PROP);
- params.put(CoreAdminParams.BACKUP_LOCATION, location);
- if (repo != null) {
- params.put(CoreAdminParams.BACKUP_REPOSITORY, repo);
- }
- // from CREATE_OP:
- copy(req.getParams(), params, COLL_CONF, REPLICATION_FACTOR, NRT_REPLICAS, TLOG_REPLICAS,
- PULL_REPLICAS, CREATE_NODE_SET, CREATE_NODE_SET_SHUFFLE, BACKUP_ID);
- copyPropertiesWithPrefix(req.getParams(), params, PROPERTY_PREFIX);
- return params;
- }),
- DELETEBACKUP_OP(DELETEBACKUP, (req, rsp, h) -> {
- req.getParams().required().check(NAME);
-
- CoreContainer cc = h.coreContainer;
- String repo = req.getParams().get(CoreAdminParams.BACKUP_REPOSITORY);
- try (BackupRepository repository = cc.newBackupRepository(repo)) {
-
- String location = repository.getBackupLocation(req.getParams().get(CoreAdminParams.BACKUP_LOCATION));
- if (location == null) {
- //Refresh the cluster property file to make sure the value set for location is the latest
- // Check if the location is specified in the cluster property.
- location = new ClusterProperties(h.coreContainer.getZkController().getZkClient()).getClusterProperty("location", null);
+ String location =
+ repository.getBackupLocation(req.getParams().get(CoreAdminParams.BACKUP_LOCATION));
if (location == null) {
- throw new SolrException(ErrorCode.BAD_REQUEST, "'location' is not specified as a query"
- + " parameter or as a default repository property or as a cluster property.");
+ // Refresh the cluster property file to make sure the value set for location is the
+ // latest
+ // Check if the location is specified in the cluster property.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/admin/CollectionsHandler.java
##########
@@ -388,982 +426,1324 @@ public Category getCategory() {
return Category.ADMIN;
}
- private static void createSysConfigSet(CoreContainer coreContainer) throws KeeperException, InterruptedException {
+ private static void createSysConfigSet(CoreContainer coreContainer)
+ throws KeeperException, InterruptedException {
SolrZkClient zk = coreContainer.getZkController().getZkStateReader().getZkClient();
ZkCmdExecutor cmdExecutor = new ZkCmdExecutor(zk.getZkClientTimeout());
cmdExecutor.ensureExists(ZkStateReader.CONFIGS_ZKNODE, zk);
- cmdExecutor.ensureExists(ZkStateReader.CONFIGS_ZKNODE + "/" + CollectionAdminParams.SYSTEM_COLL, zk);
+ cmdExecutor.ensureExists(
+ ZkStateReader.CONFIGS_ZKNODE + "/" + CollectionAdminParams.SYSTEM_COLL, zk);
try {
- String path = ZkStateReader.CONFIGS_ZKNODE + "/" + CollectionAdminParams.SYSTEM_COLL + "/schema.xml";
- byte[] data = IOUtils.toByteArray(CollectionsHandler.class.getResourceAsStream("/SystemCollectionSchema.xml"));
+ String path =
+ ZkStateReader.CONFIGS_ZKNODE + "/" + CollectionAdminParams.SYSTEM_COLL + "/schema.xml";
+ byte[] data =
+ IOUtils.toByteArray(
+ CollectionsHandler.class.getResourceAsStream("/SystemCollectionSchema.xml"));
assert data != null && data.length > 0;
cmdExecutor.ensureExists(path, data, CreateMode.PERSISTENT, zk);
- path = ZkStateReader.CONFIGS_ZKNODE + "/" + CollectionAdminParams.SYSTEM_COLL + "/solrconfig.xml";
- data = IOUtils.toByteArray(CollectionsHandler.class.getResourceAsStream("/SystemCollectionSolrConfig.xml"));
+ path =
+ ZkStateReader.CONFIGS_ZKNODE
+ + "/"
+ + CollectionAdminParams.SYSTEM_COLL
+ + "/solrconfig.xml";
+ data =
+ IOUtils.toByteArray(
+ CollectionsHandler.class.getResourceAsStream("/SystemCollectionSolrConfig.xml"));
assert data != null && data.length > 0;
cmdExecutor.ensureExists(path, data, CreateMode.PERSISTENT, zk);
} catch (IOException e) {
throw new SolrException(ErrorCode.SERVER_ERROR, e);
}
-
-
}
- private static void addStatusToResponse(NamedList<Object> results, RequestStatusState state, String msg) {
+ private static void addStatusToResponse(
+ NamedList<Object> results, RequestStatusState state, String msg) {
SimpleOrderedMap<String> status = new SimpleOrderedMap<>();
status.add("state", state.getKey());
status.add("msg", msg);
results.add("status", status);
}
public enum CollectionOperation implements CollectionOp {
- CREATE_OP(CREATE, (req, rsp, h) -> {
- Map<String, Object> props = copy(req.getParams().required(), null, NAME);
- props.put("fromApi", "true");
- copy(req.getParams(), props,
- REPLICATION_FACTOR,
- COLL_CONF,
- NUM_SLICES,
- CREATE_NODE_SET,
- CREATE_NODE_SET_SHUFFLE,
- SHARDS_PROP,
- PULL_REPLICAS,
- TLOG_REPLICAS,
- NRT_REPLICAS,
- WAIT_FOR_FINAL_STATE,
- PER_REPLICA_STATE,
- ALIAS);
-
- if (props.get(REPLICATION_FACTOR) != null && props.get(NRT_REPLICAS) != null) {
- //TODO: Remove this in 8.0 . Keep this for SolrJ client back-compat. See SOLR-11676 for more details
- int replicationFactor = Integer.parseInt((String) props.get(REPLICATION_FACTOR));
- int nrtReplicas = Integer.parseInt((String) props.get(NRT_REPLICAS));
- if (replicationFactor != nrtReplicas) {
- throw new SolrException(ErrorCode.BAD_REQUEST,
- "Cannot specify both replicationFactor and nrtReplicas as they mean the same thing");
- }
- }
- if (props.get(REPLICATION_FACTOR) != null) {
- props.put(NRT_REPLICAS, props.get(REPLICATION_FACTOR));
- } else if (props.get(NRT_REPLICAS) != null) {
- props.put(REPLICATION_FACTOR, props.get(NRT_REPLICAS));
- }
-
- final String collectionName = SolrIdentifierValidator.validateCollectionName((String) props.get(NAME));
- final String shardsParam = (String) props.get(SHARDS_PROP);
- if (StringUtils.isNotEmpty(shardsParam)) {
- verifyShardsParam(shardsParam);
- }
- if (CollectionAdminParams.SYSTEM_COLL.equals(collectionName)) {
- //We must always create a .system collection with only a single shard
- props.put(NUM_SLICES, 1);
- props.remove(SHARDS_PROP);
- createSysConfigSet(h.coreContainer);
-
- }
- if (shardsParam == null) h.copyFromClusterProp(props, NUM_SLICES);
- for (String prop : ImmutableSet.of(NRT_REPLICAS, PULL_REPLICAS, TLOG_REPLICAS))
- h.copyFromClusterProp(props, prop);
- copyPropertiesWithPrefix(req.getParams(), props, PROPERTY_PREFIX);
- return copyPropertiesWithPrefix(req.getParams(), props, "router.");
+ CREATE_OP(
+ CREATE,
+ (req, rsp, h) -> {
+ Map<String, Object> props = copy(req.getParams().required(), null, NAME);
+ props.put("fromApi", "true");
+ copy(
+ req.getParams(),
+ props,
+ REPLICATION_FACTOR,
+ COLL_CONF,
+ NUM_SLICES,
+ CREATE_NODE_SET,
+ CREATE_NODE_SET_SHUFFLE,
+ SHARDS_PROP,
+ PULL_REPLICAS,
+ TLOG_REPLICAS,
+ NRT_REPLICAS,
+ WAIT_FOR_FINAL_STATE,
+ PER_REPLICA_STATE,
+ ALIAS);
+
+ if (props.get(REPLICATION_FACTOR) != null && props.get(NRT_REPLICAS) != null) {
+ // TODO: Remove this in 8.0 . Keep this for SolrJ client back-compat. See SOLR-11676 for
+ // more details
+ int replicationFactor = Integer.parseInt((String) props.get(REPLICATION_FACTOR));
+ int nrtReplicas = Integer.parseInt((String) props.get(NRT_REPLICAS));
+ if (replicationFactor != nrtReplicas) {
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST,
+ "Cannot specify both replicationFactor and nrtReplicas as they mean the same thing");
+ }
+ }
+ if (props.get(REPLICATION_FACTOR) != null) {
+ props.put(NRT_REPLICAS, props.get(REPLICATION_FACTOR));
+ } else if (props.get(NRT_REPLICAS) != null) {
+ props.put(REPLICATION_FACTOR, props.get(NRT_REPLICAS));
+ }
- }),
+ final String collectionName =
+ SolrIdentifierValidator.validateCollectionName((String) props.get(NAME));
+ final String shardsParam = (String) props.get(SHARDS_PROP);
+ if (StringUtils.isNotEmpty(shardsParam)) {
+ verifyShardsParam(shardsParam);
+ }
+ if (CollectionAdminParams.SYSTEM_COLL.equals(collectionName)) {
+ // We must always create a .system collection with only a single shard
+ props.put(NUM_SLICES, 1);
+ props.remove(SHARDS_PROP);
+ createSysConfigSet(h.coreContainer);
+ }
+ if (shardsParam == null) h.copyFromClusterProp(props, NUM_SLICES);
+ for (String prop : ImmutableSet.of(NRT_REPLICAS, PULL_REPLICAS, TLOG_REPLICAS))
+ h.copyFromClusterProp(props, prop);
+ copyPropertiesWithPrefix(req.getParams(), props, PROPERTY_PREFIX);
+ return copyPropertiesWithPrefix(req.getParams(), props, "router.");
+ }),
@SuppressWarnings({"unchecked"})
- COLSTATUS_OP(COLSTATUS, (req, rsp, h) -> {
- Map<String, Object> props = copy(req.getParams(), null,
- COLLECTION_PROP,
- ColStatus.CORE_INFO_PROP,
- ColStatus.SEGMENTS_PROP,
- ColStatus.FIELD_INFO_PROP,
- ColStatus.RAW_SIZE_PROP,
- ColStatus.RAW_SIZE_SUMMARY_PROP,
- ColStatus.RAW_SIZE_DETAILS_PROP,
- ColStatus.RAW_SIZE_SAMPLING_PERCENT_PROP,
- ColStatus.SIZE_INFO_PROP);
-
- new ColStatus(h.coreContainer.getSolrClientCache(),
- h.coreContainer.getZkController().getZkStateReader().getClusterState(), new ZkNodeProps(props))
- .getColStatus(rsp.getValues());
- return null;
- }),
- DELETE_OP(DELETE, (req, rsp, h) -> {
- Map<String, Object> map = copy(req.getParams().required(), null, NAME);
- return copy(req.getParams(), map, FOLLOW_ALIASES);
- }),
+ COLSTATUS_OP(
+ COLSTATUS,
+ (req, rsp, h) -> {
+ Map<String, Object> props =
+ copy(
+ req.getParams(),
+ null,
+ COLLECTION_PROP,
+ ColStatus.CORE_INFO_PROP,
+ ColStatus.SEGMENTS_PROP,
+ ColStatus.FIELD_INFO_PROP,
+ ColStatus.RAW_SIZE_PROP,
+ ColStatus.RAW_SIZE_SUMMARY_PROP,
+ ColStatus.RAW_SIZE_DETAILS_PROP,
+ ColStatus.RAW_SIZE_SAMPLING_PERCENT_PROP,
+ ColStatus.SIZE_INFO_PROP);
+
+ new ColStatus(
+ h.coreContainer.getSolrClientCache(),
+ h.coreContainer.getZkController().getZkStateReader().getClusterState(),
+ new ZkNodeProps(props))
+ .getColStatus(rsp.getValues());
+ return null;
+ }),
+ DELETE_OP(
+ DELETE,
+ (req, rsp, h) -> {
+ Map<String, Object> map = copy(req.getParams().required(), null, NAME);
+ return copy(req.getParams(), map, FOLLOW_ALIASES);
+ }),
// XXX should this command support followAliases?
- RELOAD_OP(RELOAD, (req, rsp, h) -> {
- Map<String, Object> map = copy(req.getParams().required(), null, NAME);
- return copy(req.getParams(), map);
- }),
-
- RENAME_OP(RENAME, (req, rsp, h) -> {
- Map<String, Object> map = copy(req.getParams().required(), null, NAME, CollectionAdminParams.TARGET);
- return copy(req.getParams(), map, FOLLOW_ALIASES);
- }),
-
- REINDEXCOLLECTION_OP(REINDEXCOLLECTION, (req, rsp, h) -> {
- Map<String, Object> m = copy(req.getParams().required(), null, NAME);
- copy(req.getParams(), m,
- ReindexCollectionCmd.COMMAND,
- ReindexCollectionCmd.REMOVE_SOURCE,
- ReindexCollectionCmd.TARGET,
- ZkStateReader.CONFIGNAME_PROP,
- NUM_SLICES,
- NRT_REPLICAS,
- PULL_REPLICAS,
- TLOG_REPLICAS,
- REPLICATION_FACTOR,
- CREATE_NODE_SET,
- CREATE_NODE_SET_SHUFFLE,
- "shards",
- CommonParams.ROWS,
- CommonParams.Q,
- CommonParams.FL,
- FOLLOW_ALIASES);
- if (req.getParams().get("collection." + ZkStateReader.CONFIGNAME_PROP) != null) {
- m.put(ZkStateReader.CONFIGNAME_PROP, req.getParams().get("collection." + ZkStateReader.CONFIGNAME_PROP));
- }
- copyPropertiesWithPrefix(req.getParams(), m, "router.");
- return m;
- }),
-
- SYNCSHARD_OP(SYNCSHARD, (req, rsp, h) -> {
- String extCollection = req.getParams().required().get("collection");
- String collection = h.coreContainer.getZkController().getZkStateReader().getAliases().resolveSimpleAlias(extCollection);
- String shard = req.getParams().required().get("shard");
-
- ClusterState clusterState = h.coreContainer.getZkController().getClusterState();
-
- DocCollection docCollection = clusterState.getCollection(collection);
- ZkNodeProps leaderProps = docCollection.getLeader(shard);
- ZkCoreNodeProps nodeProps = new ZkCoreNodeProps(leaderProps);
-
- try (HttpSolrClient client = new Builder(nodeProps.getBaseUrl())
- .withConnectionTimeout(15000)
- .withSocketTimeout(60000)
- .build()) {
- RequestSyncShard reqSyncShard = new RequestSyncShard();
- reqSyncShard.setCollection(collection);
- reqSyncShard.setShard(shard);
- reqSyncShard.setCoreName(nodeProps.getCoreName());
- client.request(reqSyncShard);
- }
- return null;
- }),
-
- CREATEALIAS_OP(CREATEALIAS, (req, rsp, h) -> {
- String alias = req.getParams().get(NAME);
- SolrIdentifierValidator.validateAliasName(alias);
- String collections = req.getParams().get("collections");
- RoutedAlias routedAlias = null;
- Exception ex = null;
- HashMap<String,Object> possiblyModifiedParams = new HashMap<>();
- try {
- // note that RA specific validation occurs here.
- req.getParams().toMap(possiblyModifiedParams);
- @SuppressWarnings({"unchecked", "rawtypes"})
- // This is awful because RoutedAlias lies about what types it wants
- Map<String, String> temp = (Map<String, String>) (Map) possiblyModifiedParams;
- routedAlias = RoutedAlias.fromProps(alias, temp);
- } catch (SolrException e) {
- // we'll throw this later if we are in fact creating a routed alias.
- ex = e;
- }
- ModifiableSolrParams finalParams = new ModifiableSolrParams();
- for (Map.Entry<String, Object> entry : possiblyModifiedParams.entrySet()) {
- if (entry.getValue().getClass().isArray() ) {
- // v2 api hits this case
- for (Object o : (Object[]) entry.getValue()) {
- finalParams.add(entry.getKey(),o.toString());
+ RELOAD_OP(
+ RELOAD,
+ (req, rsp, h) -> {
+ Map<String, Object> map = copy(req.getParams().required(), null, NAME);
+ return copy(req.getParams(), map);
+ }),
+
+ RENAME_OP(
+ RENAME,
+ (req, rsp, h) -> {
+ Map<String, Object> map =
+ copy(req.getParams().required(), null, NAME, CollectionAdminParams.TARGET);
+ return copy(req.getParams(), map, FOLLOW_ALIASES);
+ }),
+
+ REINDEXCOLLECTION_OP(
+ REINDEXCOLLECTION,
+ (req, rsp, h) -> {
+ Map<String, Object> m = copy(req.getParams().required(), null, NAME);
+ copy(
+ req.getParams(),
+ m,
+ ReindexCollectionCmd.COMMAND,
+ ReindexCollectionCmd.REMOVE_SOURCE,
+ ReindexCollectionCmd.TARGET,
+ ZkStateReader.CONFIGNAME_PROP,
+ NUM_SLICES,
+ NRT_REPLICAS,
+ PULL_REPLICAS,
+ TLOG_REPLICAS,
+ REPLICATION_FACTOR,
+ CREATE_NODE_SET,
+ CREATE_NODE_SET_SHUFFLE,
+ "shards",
+ CommonParams.ROWS,
+ CommonParams.Q,
+ CommonParams.FL,
+ FOLLOW_ALIASES);
+ if (req.getParams().get("collection." + ZkStateReader.CONFIGNAME_PROP) != null) {
+ m.put(
+ ZkStateReader.CONFIGNAME_PROP,
+ req.getParams().get("collection." + ZkStateReader.CONFIGNAME_PROP));
+ }
+ copyPropertiesWithPrefix(req.getParams(), m, "router.");
+ return m;
+ }),
+
+ SYNCSHARD_OP(
+ SYNCSHARD,
+ (req, rsp, h) -> {
+ String extCollection = req.getParams().required().get("collection");
+ String collection =
+ h.coreContainer
+ .getZkController()
+ .getZkStateReader()
+ .getAliases()
+ .resolveSimpleAlias(extCollection);
+ String shard = req.getParams().required().get("shard");
+
+ ClusterState clusterState = h.coreContainer.getZkController().getClusterState();
+
+ DocCollection docCollection = clusterState.getCollection(collection);
+ ZkNodeProps leaderProps = docCollection.getLeader(shard);
+ ZkCoreNodeProps nodeProps = new ZkCoreNodeProps(leaderProps);
+
+ try (HttpSolrClient client =
+ new Builder(nodeProps.getBaseUrl())
+ .withConnectionTimeout(15000)
+ .withSocketTimeout(60000)
+ .build()) {
+ RequestSyncShard reqSyncShard = new RequestSyncShard();
+ reqSyncShard.setCollection(collection);
+ reqSyncShard.setShard(shard);
+ reqSyncShard.setCoreName(nodeProps.getCoreName());
+ client.request(reqSyncShard);
+ }
+ return null;
+ }),
+
+ CREATEALIAS_OP(
+ CREATEALIAS,
+ (req, rsp, h) -> {
+ String alias = req.getParams().get(NAME);
+ SolrIdentifierValidator.validateAliasName(alias);
+ String collections = req.getParams().get("collections");
+ RoutedAlias routedAlias = null;
+ Exception ex = null;
+ HashMap<String, Object> possiblyModifiedParams = new HashMap<>();
+ try {
+ // note that RA specific validation occurs here.
+ req.getParams().toMap(possiblyModifiedParams);
+ @SuppressWarnings({"unchecked", "rawtypes"})
+ // This is awful because RoutedAlias lies about what types it wants
+ Map<String, String> temp = (Map<String, String>) (Map) possiblyModifiedParams;
+ routedAlias = RoutedAlias.fromProps(alias, temp);
+ } catch (SolrException e) {
+ // we'll throw this later if we are in fact creating a routed alias.
+ ex = e;
+ }
+ ModifiableSolrParams finalParams = new ModifiableSolrParams();
+ for (Map.Entry<String, Object> entry : possiblyModifiedParams.entrySet()) {
+ if (entry.getValue().getClass().isArray()) {
+ // v2 api hits this case
+ for (Object o : (Object[]) entry.getValue()) {
+ finalParams.add(entry.getKey(), o.toString());
+ }
+ } else {
+ finalParams.add(entry.getKey(), entry.getValue().toString());
+ }
}
- } else {
- finalParams.add(entry.getKey(),entry.getValue().toString());
- }
- }
- if (collections != null) {
- if (routedAlias != null) {
- throw new SolrException(BAD_REQUEST, "Collections cannot be specified when creating a routed alias.");
- } else {
- //////////////////////////////////////
- // Regular alias creation indicated //
- //////////////////////////////////////
- return copy(finalParams.required(), null, NAME, "collections");
- }
- }
+ if (collections != null) {
+ if (routedAlias != null) {
+ throw new SolrException(
+ BAD_REQUEST, "Collections cannot be specified when creating a routed alias.");
+ } else {
+ //////////////////////////////////////
+ // Regular alias creation indicated //
+ //////////////////////////////////////
+ return copy(finalParams.required(), null, NAME, "collections");
+ }
+ }
- /////////////////////////////////////////////////
- // We are creating a routed alias from here on //
- /////////////////////////////////////////////////
+ /////////////////////////////////////////////////
+ // We are creating a routed alias from here on //
+ /////////////////////////////////////////////////
- // If our prior creation attempt had issues expose them now.
- if (ex != null) {
- throw ex;
- }
+ // If our prior creation attempt had issues expose them now.
+ if (ex != null) {
+ throw ex;
+ }
- // Now filter out just the parameters we care about from the request
- assert routedAlias != null;
- Map<String, Object> result = copy(finalParams, null, routedAlias.getRequiredParams());
- copy(finalParams, result, routedAlias.getOptionalParams());
-
- ModifiableSolrParams createCollParams = new ModifiableSolrParams(); // without prefix
-
- // add to result params that start with "create-collection.".
- // Additionally, save these without the prefix to createCollParams
- for (Map.Entry<String, String[]> entry : finalParams) {
- final String p = entry.getKey();
- if (p.startsWith(CREATE_COLLECTION_PREFIX)) {
- // This is what SolrParams#getAll(Map, Collection)} does
- final String[] v = entry.getValue();
- if (v.length == 1) {
- result.put(p, v[0]);
- } else {
- result.put(p, v);
+ // Now filter out just the parameters we care about from the request
+ assert routedAlias != null;
+ Map<String, Object> result = copy(finalParams, null, routedAlias.getRequiredParams());
+ copy(finalParams, result, routedAlias.getOptionalParams());
+
+ ModifiableSolrParams createCollParams = new ModifiableSolrParams(); // without prefix
+
+ // add to result params that start with "create-collection.".
+ // Additionally, save these without the prefix to createCollParams
+ for (Map.Entry<String, String[]> entry : finalParams) {
+ final String p = entry.getKey();
+ if (p.startsWith(CREATE_COLLECTION_PREFIX)) {
+ // This is what SolrParams#getAll(Map, Collection)} does
+ final String[] v = entry.getValue();
+ if (v.length == 1) {
+ result.put(p, v[0]);
+ } else {
+ result.put(p, v);
+ }
+ createCollParams.set(p.substring(CREATE_COLLECTION_PREFIX.length()), v);
+ }
}
- createCollParams.set(p.substring(CREATE_COLLECTION_PREFIX.length()), v);
- }
- }
- // Verify that the create-collection prefix'ed params appear to be valid.
- if (createCollParams.get(NAME) != null) {
- throw new SolrException(BAD_REQUEST, "routed aliases calculate names for their " +
- "dependent collections, you cannot specify the name.");
- }
- if (createCollParams.get(COLL_CONF) == null) {
- throw new SolrException(SolrException.ErrorCode.BAD_REQUEST,
- "We require an explicit " + COLL_CONF);
- }
- // note: could insist on a config name here as well.... or wait to throw at overseer
- createCollParams.add(NAME, "TMP_name_TMP_name_TMP"); // just to pass validation
- CREATE_OP.execute(new LocalSolrQueryRequest(null, createCollParams), rsp, h); // ignore results
+ // Verify that the create-collection prefix'ed params appear to be valid.
+ if (createCollParams.get(NAME) != null) {
+ throw new SolrException(
+ BAD_REQUEST,
+ "routed aliases calculate names for their "
+ + "dependent collections, you cannot specify the name.");
+ }
+ if (createCollParams.get(COLL_CONF) == null) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST, "We require an explicit " + COLL_CONF);
+ }
+ // note: could insist on a config name here as well.... or wait to throw at overseer
+ createCollParams.add(NAME, "TMP_name_TMP_name_TMP"); // just to pass validation
+ CREATE_OP.execute(
+ new LocalSolrQueryRequest(null, createCollParams), rsp, h); // ignore results
- return result;
- }),
+ return result;
+ }),
DELETEALIAS_OP(DELETEALIAS, (req, rsp, h) -> copy(req.getParams().required(), null, NAME)),
/**
* Change properties for an alias (use CREATEALIAS_OP to change the actual value of the alias)
*/
- ALIASPROP_OP(ALIASPROP, (req, rsp, h) -> {
- Map<String, Object> params = copy(req.getParams().required(), null, NAME);
-
- // Note: success/no-op in the event of no properties supplied is intentional. Keeps code simple and one less case
- // for api-callers to check for.
- return convertPrefixToMap(req.getParams(), params, "property");
- }),
-
- /**
- * List the aliases and associated properties.
- */
+ ALIASPROP_OP(
+ ALIASPROP,
+ (req, rsp, h) -> {
+ Map<String, Object> params = copy(req.getParams().required(), null, NAME);
+
+ // Note: success/no-op in the event of no properties supplied is intentional. Keeps code
+ // simple and one less case
+ // for api-callers to check for.
+ return convertPrefixToMap(req.getParams(), params, "property");
+ }),
+
+ /** List the aliases and associated properties. */
@SuppressWarnings({"unchecked"})
- LISTALIASES_OP(LISTALIASES, (req, rsp, h) -> {
- ZkStateReader zkStateReader = h.coreContainer.getZkController().getZkStateReader();
- // if someone calls listAliases, lets ensure we return an up to date response
- zkStateReader.aliasesManager.update();
- Aliases aliases = zkStateReader.getAliases();
- if (aliases != null) {
- // the aliases themselves...
- rsp.getValues().add("aliases", aliases.getCollectionAliasMap());
- // Any properties for the above aliases.
- Map<String, Map<String, String>> meta = new LinkedHashMap<>();
- for (String alias : aliases.getCollectionAliasListMap().keySet()) {
- Map<String, String> collectionAliasProperties = aliases.getCollectionAliasProperties(alias);
- if (!collectionAliasProperties.isEmpty()) {
- meta.put(alias, collectionAliasProperties);
+ LISTALIASES_OP(
+ LISTALIASES,
+ (req, rsp, h) -> {
+ ZkStateReader zkStateReader = h.coreContainer.getZkController().getZkStateReader();
+ // if someone calls listAliases, lets ensure we return an up to date response
+ zkStateReader.aliasesManager.update();
+ Aliases aliases = zkStateReader.getAliases();
+ if (aliases != null) {
+ // the aliases themselves...
+ rsp.getValues().add("aliases", aliases.getCollectionAliasMap());
+ // Any properties for the above aliases.
+ Map<String, Map<String, String>> meta = new LinkedHashMap<>();
+ for (String alias : aliases.getCollectionAliasListMap().keySet()) {
+ Map<String, String> collectionAliasProperties =
+ aliases.getCollectionAliasProperties(alias);
+ if (!collectionAliasProperties.isEmpty()) {
+ meta.put(alias, collectionAliasProperties);
+ }
+ }
+ rsp.getValues().add("properties", meta);
+ }
+ return null;
+ }),
+ SPLITSHARD_OP(
+ SPLITSHARD,
+ DEFAULT_COLLECTION_OP_TIMEOUT * 5,
+ (req, rsp, h) -> {
+ String name = req.getParams().required().get(COLLECTION_PROP);
+ // TODO : add support for multiple shards
+ String shard = req.getParams().get(SHARD_ID_PROP);
+ String rangesStr = req.getParams().get(CoreAdminParams.RANGES);
+ String splitKey = req.getParams().get("split.key");
+ String numSubShards = req.getParams().get(NUM_SUB_SHARDS);
+ String fuzz = req.getParams().get(SPLIT_FUZZ);
+
+ if (splitKey == null && shard == null) {
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST, "At least one of shard, or split.key should be specified.");
+ }
+ if (splitKey != null && shard != null) {
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST, "Only one of 'shard' or 'split.key' should be specified");
+ }
+ if (splitKey != null && rangesStr != null) {
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST, "Only one of 'ranges' or 'split.key' should be specified");
+ }
+ if (numSubShards != null && (splitKey != null || rangesStr != null)) {
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST,
+ "numSubShards can not be specified with split.key or ranges parameters");
+ }
+ if (fuzz != null && (splitKey != null || rangesStr != null)) {
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST,
+ "fuzz can not be specified with split.key or ranges parameters");
}
- }
- rsp.getValues().add("properties", meta);
- }
- return null;
- }),
- SPLITSHARD_OP(SPLITSHARD, DEFAULT_COLLECTION_OP_TIMEOUT * 5, (req, rsp, h) -> {
- String name = req.getParams().required().get(COLLECTION_PROP);
- // TODO : add support for multiple shards
- String shard = req.getParams().get(SHARD_ID_PROP);
- String rangesStr = req.getParams().get(CoreAdminParams.RANGES);
- String splitKey = req.getParams().get("split.key");
- String numSubShards = req.getParams().get(NUM_SUB_SHARDS);
- String fuzz = req.getParams().get(SPLIT_FUZZ);
-
- if (splitKey == null && shard == null) {
- throw new SolrException(ErrorCode.BAD_REQUEST, "At least one of shard, or split.key should be specified.");
- }
- if (splitKey != null && shard != null) {
- throw new SolrException(ErrorCode.BAD_REQUEST,
- "Only one of 'shard' or 'split.key' should be specified");
- }
- if (splitKey != null && rangesStr != null) {
- throw new SolrException(ErrorCode.BAD_REQUEST,
- "Only one of 'ranges' or 'split.key' should be specified");
- }
- if (numSubShards != null && (splitKey != null || rangesStr != null)) {
- throw new SolrException(ErrorCode.BAD_REQUEST,
- "numSubShards can not be specified with split.key or ranges parameters");
- }
- if (fuzz != null && (splitKey != null || rangesStr != null)) {
- throw new SolrException(ErrorCode.BAD_REQUEST,
- "fuzz can not be specified with split.key or ranges parameters");
- }
- Map<String, Object> map = copy(req.getParams(), null,
- COLLECTION_PROP,
- SHARD_ID_PROP,
- "split.key",
- CoreAdminParams.RANGES,
- WAIT_FOR_FINAL_STATE,
- TIMING,
- SPLIT_METHOD,
- NUM_SUB_SHARDS,
- SPLIT_FUZZ,
- SPLIT_BY_PREFIX,
- FOLLOW_ALIASES);
- return copyPropertiesWithPrefix(req.getParams(), map, PROPERTY_PREFIX);
- }),
- DELETESHARD_OP(DELETESHARD, (req, rsp, h) -> {
- Map<String, Object> map = copy(req.getParams().required(), null,
- COLLECTION_PROP,
- SHARD_ID_PROP);
- copy(req.getParams(), map,
- DELETE_INDEX,
- DELETE_DATA_DIR,
- DELETE_INSTANCE_DIR,
- FOLLOW_ALIASES);
- return map;
- }),
- FORCELEADER_OP(FORCELEADER, (req, rsp, h) -> {
- forceLeaderElection(req, h);
- return null;
- }),
- CREATESHARD_OP(CREATESHARD, (req, rsp, h) -> {
- Map<String, Object> map = copy(req.getParams().required(), null,
- COLLECTION_PROP,
- SHARD_ID_PROP);
- ClusterState clusterState = h.coreContainer.getZkController().getClusterState();
- final String newShardName = SolrIdentifierValidator.validateShardName(req.getParams().get(SHARD_ID_PROP));
- boolean followAliases = req.getParams().getBool(FOLLOW_ALIASES, false);
- String extCollectionName = req.getParams().get(COLLECTION_PROP);
- String collectionName = followAliases ? h.coreContainer.getZkController().getZkStateReader()
- .getAliases().resolveSimpleAlias(extCollectionName) : extCollectionName;
- if (!ImplicitDocRouter.NAME.equals(((Map<?,?>) clusterState.getCollection(collectionName).get(DOC_ROUTER)).get(NAME)))
- throw new SolrException(ErrorCode.BAD_REQUEST, "shards can be added only to 'implicit' collections");
- copy(req.getParams(), map,
- REPLICATION_FACTOR,
- NRT_REPLICAS,
- TLOG_REPLICAS,
- PULL_REPLICAS,
- CREATE_NODE_SET,
- WAIT_FOR_FINAL_STATE,
- FOLLOW_ALIASES);
- return copyPropertiesWithPrefix(req.getParams(), map, PROPERTY_PREFIX);
- }),
- DELETEREPLICA_OP(DELETEREPLICA, (req, rsp, h) -> {
- Map<String, Object> map = copy(req.getParams().required(), null,
- COLLECTION_PROP);
-
- return copy(req.getParams(), map,
- DELETE_INDEX,
- DELETE_DATA_DIR,
- DELETE_INSTANCE_DIR,
- COUNT_PROP, REPLICA_PROP,
- SHARD_ID_PROP,
- ONLY_IF_DOWN,
- FOLLOW_ALIASES);
- }),
- MIGRATE_OP(MIGRATE, (req, rsp, h) -> {
- Map<String, Object> map = copy(req.getParams().required(), null, COLLECTION_PROP, "split.key", "target.collection");
- return copy(req.getParams(), map, "forward.timeout", FOLLOW_ALIASES);
- }),
- ADDROLE_OP(ADDROLE, (req, rsp, h) -> {
- Map<String, Object> map = copy(req.getParams().required(), null, "role", "node");
- if (!KNOWN_ROLES.contains(map.get("role")))
- throw new SolrException(ErrorCode.BAD_REQUEST, "Unknown role. Supported roles are ," + KNOWN_ROLES);
- return map;
- }),
- REMOVEROLE_OP(REMOVEROLE, (req, rsp, h) -> {
- Map<String, Object> map = copy(req.getParams().required(), null, "role", "node");
- if (!KNOWN_ROLES.contains(map.get("role")))
- throw new SolrException(ErrorCode.BAD_REQUEST, "Unknown role. Supported roles are ," + KNOWN_ROLES);
- return map;
- }),
- CLUSTERPROP_OP(CLUSTERPROP, (req, rsp, h) -> {
- String name = req.getParams().required().get(NAME);
- String val = req.getParams().get(VALUE_LONG);
- ClusterProperties cp = new ClusterProperties(h.coreContainer.getZkController().getZkClient());
- cp.setClusterProperty(name, val);
- return null;
- }),
- COLLECTIONPROP_OP(COLLECTIONPROP, (req, rsp, h) -> {
- String extCollection = req.getParams().required().get(NAME);
- String collection = h.coreContainer.getZkController().getZkStateReader().getAliases().resolveSimpleAlias(extCollection);
- String name = req.getParams().required().get(PROPERTY_NAME);
- String val = req.getParams().get(PROPERTY_VALUE);
- CollectionProperties cp = new CollectionProperties(h.coreContainer.getZkController().getZkClient());
- cp.setCollectionProperty(collection, name, val);
- return null;
- }),
+ Map<String, Object> map =
+ copy(
+ req.getParams(),
+ null,
+ COLLECTION_PROP,
+ SHARD_ID_PROP,
+ "split.key",
+ CoreAdminParams.RANGES,
+ WAIT_FOR_FINAL_STATE,
+ TIMING,
+ SPLIT_METHOD,
+ NUM_SUB_SHARDS,
+ SPLIT_FUZZ,
+ SPLIT_BY_PREFIX,
+ FOLLOW_ALIASES);
+ return copyPropertiesWithPrefix(req.getParams(), map, PROPERTY_PREFIX);
+ }),
+ DELETESHARD_OP(
+ DELETESHARD,
+ (req, rsp, h) -> {
+ Map<String, Object> map =
+ copy(req.getParams().required(), null, COLLECTION_PROP, SHARD_ID_PROP);
+ copy(
+ req.getParams(),
+ map,
+ DELETE_INDEX,
+ DELETE_DATA_DIR,
+ DELETE_INSTANCE_DIR,
+ FOLLOW_ALIASES);
+ return map;
+ }),
+ FORCELEADER_OP(
+ FORCELEADER,
+ (req, rsp, h) -> {
+ forceLeaderElection(req, h);
+ return null;
+ }),
+ CREATESHARD_OP(
+ CREATESHARD,
+ (req, rsp, h) -> {
+ Map<String, Object> map =
+ copy(req.getParams().required(), null, COLLECTION_PROP, SHARD_ID_PROP);
+ ClusterState clusterState = h.coreContainer.getZkController().getClusterState();
+ final String newShardName =
+ SolrIdentifierValidator.validateShardName(req.getParams().get(SHARD_ID_PROP));
+ boolean followAliases = req.getParams().getBool(FOLLOW_ALIASES, false);
+ String extCollectionName = req.getParams().get(COLLECTION_PROP);
+ String collectionName =
+ followAliases
+ ? h.coreContainer
+ .getZkController()
+ .getZkStateReader()
+ .getAliases()
+ .resolveSimpleAlias(extCollectionName)
+ : extCollectionName;
+ if (!ImplicitDocRouter.NAME.equals(
+ ((Map<?, ?>) clusterState.getCollection(collectionName).get(DOC_ROUTER)).get(NAME)))
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST, "shards can be added only to 'implicit' collections");
+ copy(
+ req.getParams(),
+ map,
+ REPLICATION_FACTOR,
+ NRT_REPLICAS,
+ TLOG_REPLICAS,
+ PULL_REPLICAS,
+ CREATE_NODE_SET,
+ WAIT_FOR_FINAL_STATE,
+ FOLLOW_ALIASES);
+ return copyPropertiesWithPrefix(req.getParams(), map, PROPERTY_PREFIX);
+ }),
+ DELETEREPLICA_OP(
+ DELETEREPLICA,
+ (req, rsp, h) -> {
+ Map<String, Object> map = copy(req.getParams().required(), null, COLLECTION_PROP);
+
+ return copy(
+ req.getParams(),
+ map,
+ DELETE_INDEX,
+ DELETE_DATA_DIR,
+ DELETE_INSTANCE_DIR,
+ COUNT_PROP,
+ REPLICA_PROP,
+ SHARD_ID_PROP,
+ ONLY_IF_DOWN,
+ FOLLOW_ALIASES);
+ }),
+ MIGRATE_OP(
+ MIGRATE,
+ (req, rsp, h) -> {
+ Map<String, Object> map =
+ copy(
+ req.getParams().required(),
+ null,
+ COLLECTION_PROP,
+ "split.key",
+ "target.collection");
+ return copy(req.getParams(), map, "forward.timeout", FOLLOW_ALIASES);
+ }),
+ ADDROLE_OP(
+ ADDROLE,
+ (req, rsp, h) -> {
+ Map<String, Object> map = copy(req.getParams().required(), null, "role", "node");
+ if (!KNOWN_ROLES.contains(map.get("role")))
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST, "Unknown role. Supported roles are ," + KNOWN_ROLES);
+ return map;
+ }),
+ REMOVEROLE_OP(
+ REMOVEROLE,
+ (req, rsp, h) -> {
+ Map<String, Object> map = copy(req.getParams().required(), null, "role", "node");
+ if (!KNOWN_ROLES.contains(map.get("role")))
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST, "Unknown role. Supported roles are ," + KNOWN_ROLES);
+ return map;
+ }),
+ CLUSTERPROP_OP(
+ CLUSTERPROP,
+ (req, rsp, h) -> {
+ String name = req.getParams().required().get(NAME);
+ String val = req.getParams().get(VALUE_LONG);
+ ClusterProperties cp =
+ new ClusterProperties(h.coreContainer.getZkController().getZkClient());
+ cp.setClusterProperty(name, val);
+ return null;
+ }),
+ COLLECTIONPROP_OP(
+ COLLECTIONPROP,
+ (req, rsp, h) -> {
+ String extCollection = req.getParams().required().get(NAME);
+ String collection =
+ h.coreContainer
+ .getZkController()
+ .getZkStateReader()
+ .getAliases()
+ .resolveSimpleAlias(extCollection);
+ String name = req.getParams().required().get(PROPERTY_NAME);
+ String val = req.getParams().get(PROPERTY_VALUE);
+ CollectionProperties cp =
+ new CollectionProperties(h.coreContainer.getZkController().getZkClient());
+ cp.setCollectionProperty(collection, name, val);
+ return null;
+ }),
@SuppressWarnings({"unchecked"})
- REQUESTSTATUS_OP(REQUESTSTATUS, (req, rsp, h) -> {
- req.getParams().required().check(REQUESTID);
-
- final CoreContainer coreContainer = h.coreContainer;
- final String requestId = req.getParams().get(REQUESTID);
- final ZkController zkController = coreContainer.getZkController();
-
- final NamedList<Object> status = new NamedList<>();
- if (coreContainer.getDistributedCollectionCommandRunner().isEmpty()) {
- if (zkController.getOverseerCompletedMap().contains(requestId)) {
- final byte[] mapEntry = zkController.getOverseerCompletedMap().get(requestId);
- rsp.getValues().addAll(OverseerSolrResponseSerializer.deserialize(mapEntry).getResponse());
- addStatusToResponse(status, COMPLETED, "found [" + requestId + "] in completed tasks");
- } else if (zkController.getOverseerFailureMap().contains(requestId)) {
- final byte[] mapEntry = zkController.getOverseerFailureMap().get(requestId);
- rsp.getValues().addAll(OverseerSolrResponseSerializer.deserialize(mapEntry).getResponse());
- addStatusToResponse(status, FAILED, "found [" + requestId + "] in failed tasks");
- } else if (zkController.getOverseerRunningMap().contains(requestId)) {
- addStatusToResponse(status, RUNNING, "found [" + requestId + "] in running tasks");
- } else if (h.overseerCollectionQueueContains(requestId)) {
- addStatusToResponse(status, SUBMITTED, "found [" + requestId + "] in submitted tasks");
- } else {
- addStatusToResponse(status, NOT_FOUND, "Did not find [" + requestId + "] in any tasks queue");
- }
- } else {
- Pair<RequestStatusState, OverseerSolrResponse> sr =
- coreContainer.getDistributedCollectionCommandRunner().get().getAsyncTaskRequestStatus(requestId);
- final String message;
- switch (sr.first()) {
- case COMPLETED:
- message = "found [" + requestId + "] in completed tasks";
- rsp.getValues().addAll(sr.second().getResponse());
- break;
- case FAILED:
- message = "found [" + requestId + "] in failed tasks";
- rsp.getValues().addAll(sr.second().getResponse());
- break;
- case RUNNING:
- message = "found [" + requestId + "] in running tasks";
- break;
- case SUBMITTED:
- message = "found [" + requestId + "] in submitted tasks";
- break;
- default:
- message = "Did not find [" + requestId + "] in any tasks queue";
- }
- addStatusToResponse(status, sr.first(), message);
- }
-
- rsp.getValues().addAll(status);
- return null;
- }),
- DELETESTATUS_OP(DELETESTATUS, new CollectionOp() {
- @SuppressWarnings("unchecked")
- @Override
- public Map<String, Object> execute(SolrQueryRequest req, SolrQueryResponse rsp, CollectionsHandler h) throws Exception {
- final CoreContainer coreContainer = h.coreContainer;
- final String requestId = req.getParams().get(REQUESTID);
- final ZkController zkController = coreContainer.getZkController();
- Boolean flush = req.getParams().getBool(CollectionAdminParams.FLUSH, false);
-
- if (requestId == null && !flush) {
- throw new SolrException(ErrorCode.BAD_REQUEST, "Either requestid or flush parameter must be specified.");
- }
-
- if (requestId != null && flush) {
- throw new SolrException(ErrorCode.BAD_REQUEST,
- "Both requestid and flush parameters can not be specified together.");
- }
-
- if (coreContainer.getDistributedCollectionCommandRunner().isEmpty()) {
- if (flush) {
- Collection<String> completed = zkController.getOverseerCompletedMap().keys();
- Collection<String> failed = zkController.getOverseerFailureMap().keys();
- for (String asyncId : completed) {
- zkController.getOverseerCompletedMap().remove(asyncId);
- zkController.clearAsyncId(asyncId);
- }
- for (String asyncId : failed) {
- zkController.getOverseerFailureMap().remove(asyncId);
- zkController.clearAsyncId(asyncId);
+ REQUESTSTATUS_OP(
+ REQUESTSTATUS,
+ (req, rsp, h) -> {
+ req.getParams().required().check(REQUESTID);
+
+ final CoreContainer coreContainer = h.coreContainer;
+ final String requestId = req.getParams().get(REQUESTID);
+ final ZkController zkController = coreContainer.getZkController();
+
+ final NamedList<Object> status = new NamedList<>();
+ if (coreContainer.getDistributedCollectionCommandRunner().isEmpty()) {
+ if (zkController.getOverseerCompletedMap().contains(requestId)) {
+ final byte[] mapEntry = zkController.getOverseerCompletedMap().get(requestId);
+ rsp.getValues()
+ .addAll(OverseerSolrResponseSerializer.deserialize(mapEntry).getResponse());
+ addStatusToResponse(
+ status, COMPLETED, "found [" + requestId + "] in completed tasks");
+ } else if (zkController.getOverseerFailureMap().contains(requestId)) {
+ final byte[] mapEntry = zkController.getOverseerFailureMap().get(requestId);
+ rsp.getValues()
+ .addAll(OverseerSolrResponseSerializer.deserialize(mapEntry).getResponse());
+ addStatusToResponse(status, FAILED, "found [" + requestId + "] in failed tasks");
+ } else if (zkController.getOverseerRunningMap().contains(requestId)) {
+ addStatusToResponse(status, RUNNING, "found [" + requestId + "] in running tasks");
+ } else if (h.overseerCollectionQueueContains(requestId)) {
+ addStatusToResponse(
+ status, SUBMITTED, "found [" + requestId + "] in submitted tasks");
+ } else {
+ addStatusToResponse(
+ status, NOT_FOUND, "Did not find [" + requestId + "] in any tasks queue");
}
- rsp.getValues().add("status", "successfully cleared stored collection api responses");
} else {
- // Request to cleanup
- if (zkController.getOverseerCompletedMap().remove(requestId)) {
- zkController.clearAsyncId(requestId);
- rsp.getValues().add("status", "successfully removed stored response for [" + requestId + "]");
- } else if (zkController.getOverseerFailureMap().remove(requestId)) {
- zkController.clearAsyncId(requestId);
- rsp.getValues().add("status", "successfully removed stored response for [" + requestId + "]");
- } else {
- rsp.getValues().add("status", "[" + requestId + "] not found in stored responses");
- // Don't call zkController.clearAsyncId for this, since it could be a running/pending task
+ Pair<RequestStatusState, OverseerSolrResponse> sr =
+ coreContainer
+ .getDistributedCollectionCommandRunner()
+ .get()
+ .getAsyncTaskRequestStatus(requestId);
+ final String message;
+ switch (sr.first()) {
+ case COMPLETED:
+ message = "found [" + requestId + "] in completed tasks";
+ rsp.getValues().addAll(sr.second().getResponse());
+ break;
+ case FAILED:
+ message = "found [" + requestId + "] in failed tasks";
+ rsp.getValues().addAll(sr.second().getResponse());
+ break;
+ case RUNNING:
+ message = "found [" + requestId + "] in running tasks";
+ break;
+ case SUBMITTED:
+ message = "found [" + requestId + "] in submitted tasks";
+ break;
+ default:
+ message = "Did not find [" + requestId + "] in any tasks queue";
}
+ addStatusToResponse(status, sr.first(), message);
}
- } else {
- if (flush) {
- coreContainer.getDistributedCollectionCommandRunner().get().deleteAllAsyncIds();
- rsp.getValues().add("status", "successfully cleared stored collection api responses");
- } else {
- if (coreContainer.getDistributedCollectionCommandRunner().get().deleteSingleAsyncId(requestId)) {
- rsp.getValues().add("status", "successfully removed stored response for [" + requestId + "]");
+
+ rsp.getValues().addAll(status);
+ return null;
+ }),
+ DELETESTATUS_OP(
+ DELETESTATUS,
+ new CollectionOp() {
+ @SuppressWarnings("unchecked")
+ @Override
+ public Map<String, Object> execute(
+ SolrQueryRequest req, SolrQueryResponse rsp, CollectionsHandler h) throws Exception {
+ final CoreContainer coreContainer = h.coreContainer;
+ final String requestId = req.getParams().get(REQUESTID);
+ final ZkController zkController = coreContainer.getZkController();
+ Boolean flush = req.getParams().getBool(CollectionAdminParams.FLUSH, false);
+
+ if (requestId == null && !flush) {
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST, "Either requestid or flush parameter must be specified.");
+ }
+
+ if (requestId != null && flush) {
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST,
+ "Both requestid and flush parameters can not be specified together.");
+ }
+
+ if (coreContainer.getDistributedCollectionCommandRunner().isEmpty()) {
+ if (flush) {
+ Collection<String> completed = zkController.getOverseerCompletedMap().keys();
+ Collection<String> failed = zkController.getOverseerFailureMap().keys();
+ for (String asyncId : completed) {
+ zkController.getOverseerCompletedMap().remove(asyncId);
+ zkController.clearAsyncId(asyncId);
+ }
+ for (String asyncId : failed) {
+ zkController.getOverseerFailureMap().remove(asyncId);
+ zkController.clearAsyncId(asyncId);
+ }
+ rsp.getValues()
+ .add("status", "successfully cleared stored collection api responses");
+ } else {
+ // Request to cleanup
+ if (zkController.getOverseerCompletedMap().remove(requestId)) {
+ zkController.clearAsyncId(requestId);
+ rsp.getValues()
+ .add(
+ "status", "successfully removed stored response for [" + requestId + "]");
+ } else if (zkController.getOverseerFailureMap().remove(requestId)) {
+ zkController.clearAsyncId(requestId);
+ rsp.getValues()
+ .add(
+ "status", "successfully removed stored response for [" + requestId + "]");
+ } else {
+ rsp.getValues()
+ .add("status", "[" + requestId + "] not found in stored responses");
+ // Don't call zkController.clearAsyncId for this, since it could be a
+ // running/pending task
+ }
+ }
} else {
- rsp.getValues().add("status", "[" + requestId + "] not found in stored responses");
+ if (flush) {
+ coreContainer.getDistributedCollectionCommandRunner().get().deleteAllAsyncIds();
+ rsp.getValues()
+ .add("status", "successfully cleared stored collection api responses");
+ } else {
+ if (coreContainer
+ .getDistributedCollectionCommandRunner()
+ .get()
+ .deleteSingleAsyncId(requestId)) {
+ rsp.getValues()
+ .add(
+ "status", "successfully removed stored response for [" + requestId + "]");
+ } else {
+ rsp.getValues()
+ .add("status", "[" + requestId + "] not found in stored responses");
+ }
+ }
}
+ return null;
}
- }
- return null;
- }
- }),
- ADDREPLICA_OP(ADDREPLICA, (req, rsp, h) -> {
- Map<String, Object> props = copy(req.getParams(), null,
- COLLECTION_PROP,
- "node",
- SHARD_ID_PROP,
- _ROUTE_,
- CoreAdminParams.NAME,
- INSTANCE_DIR,
- DATA_DIR,
- ULOG_DIR,
- REPLICA_TYPE,
- WAIT_FOR_FINAL_STATE,
- NRT_REPLICAS,
- TLOG_REPLICAS,
- PULL_REPLICAS,
- CREATE_NODE_SET,
- FOLLOW_ALIASES,
- SKIP_NODE_ASSIGNMENT);
- return copyPropertiesWithPrefix(req.getParams(), props, PROPERTY_PREFIX);
- }),
+ }),
+ ADDREPLICA_OP(
+ ADDREPLICA,
+ (req, rsp, h) -> {
+ Map<String, Object> props =
+ copy(
+ req.getParams(),
+ null,
+ COLLECTION_PROP,
+ "node",
+ SHARD_ID_PROP,
+ _ROUTE_,
+ CoreAdminParams.NAME,
+ INSTANCE_DIR,
+ DATA_DIR,
+ ULOG_DIR,
+ REPLICA_TYPE,
+ WAIT_FOR_FINAL_STATE,
+ NRT_REPLICAS,
+ TLOG_REPLICAS,
+ PULL_REPLICAS,
+ CREATE_NODE_SET,
+ FOLLOW_ALIASES,
+ SKIP_NODE_ASSIGNMENT);
+ return copyPropertiesWithPrefix(req.getParams(), props, PROPERTY_PREFIX);
+ }),
OVERSEERSTATUS_OP(OVERSEERSTATUS, (req, rsp, h) -> new LinkedHashMap<>()),
@SuppressWarnings({"unchecked"})
- DISTRIBUTEDAPIPROCESSING_OP(DISTRIBUTEDAPIPROCESSING, (req, rsp, h) -> {
- NamedList<Object> results = new NamedList<>();
- boolean isDistributedApi = h.coreContainer.getDistributedCollectionCommandRunner().isPresent();
- results.add("isDistributedApi", isDistributedApi);
- rsp.getValues().addAll(results);
- return null;
- }),
- /**
- * Handle list collection request.
- * Do list collection request to zk host
- */
+ DISTRIBUTEDAPIPROCESSING_OP(
+ DISTRIBUTEDAPIPROCESSING,
+ (req, rsp, h) -> {
+ NamedList<Object> results = new NamedList<>();
+ boolean isDistributedApi =
+ h.coreContainer.getDistributedCollectionCommandRunner().isPresent();
+ results.add("isDistributedApi", isDistributedApi);
+ rsp.getValues().addAll(results);
+ return null;
+ }),
+ /** Handle list collection request. Do list collection request to zk host */
@SuppressWarnings({"unchecked"})
- LIST_OP(LIST, (req, rsp, h) -> {
- NamedList<Object> results = new NamedList<>();
- Map<String, DocCollection> collections = h.coreContainer.getZkController().getZkStateReader().getClusterState().getCollectionsMap();
- List<String> collectionList = new ArrayList<>(collections.keySet());
- Collections.sort(collectionList);
- // XXX should we add aliases here?
- results.add("collections", collectionList);
- SolrResponse response = new OverseerSolrResponse(results);
- rsp.getValues().addAll(response.getResponse());
- return null;
- }),
+ LIST_OP(
+ LIST,
+ (req, rsp, h) -> {
+ NamedList<Object> results = new NamedList<>();
+ Map<String, DocCollection> collections =
+ h.coreContainer
+ .getZkController()
+ .getZkStateReader()
+ .getClusterState()
+ .getCollectionsMap();
+ List<String> collectionList = new ArrayList<>(collections.keySet());
+ Collections.sort(collectionList);
+ // XXX should we add aliases here?
+ results.add("collections", collectionList);
+ SolrResponse response = new OverseerSolrResponse(results);
+ rsp.getValues().addAll(response.getResponse());
+ return null;
+ }),
/**
- * Handle cluster status request.
- * Can return status per specific collection/shard or per all collections.
+ * Handle cluster status request. Can return status per specific collection/shard or per all
+ * collections.
*/
- CLUSTERSTATUS_OP(CLUSTERSTATUS, (req, rsp, h) -> {
- Map<String, Object> all = copy(req.getParams(), null,
- COLLECTION_PROP,
- SHARD_ID_PROP,
- _ROUTE_);
- new ClusterStatus(h.coreContainer.getZkController().getZkStateReader(),
- new ZkNodeProps(all)).getClusterStatus(rsp.getValues());
- return null;
- }),
- ADDREPLICAPROP_OP(ADDREPLICAPROP, (req, rsp, h) -> {
- Map<String, Object> map = copy(req.getParams().required(), null,
- COLLECTION_PROP,
- PROPERTY_PROP,
- SHARD_ID_PROP,
- REPLICA_PROP,
- PROPERTY_VALUE_PROP);
- copy(req.getParams(), map, SHARD_UNIQUE);
- String property = (String) map.get(PROPERTY_PROP);
- if (!property.startsWith(PROPERTY_PREFIX)) {
- property = PROPERTY_PREFIX + property;
- }
+ CLUSTERSTATUS_OP(
+ CLUSTERSTATUS,
+ (req, rsp, h) -> {
+ Map<String, Object> all =
+ copy(req.getParams(), null, COLLECTION_PROP, SHARD_ID_PROP, _ROUTE_);
+ new ClusterStatus(
+ h.coreContainer.getZkController().getZkStateReader(), new ZkNodeProps(all))
+ .getClusterStatus(rsp.getValues());
+ return null;
+ }),
+ ADDREPLICAPROP_OP(
+ ADDREPLICAPROP,
+ (req, rsp, h) -> {
+ Map<String, Object> map =
+ copy(
+ req.getParams().required(),
+ null,
+ COLLECTION_PROP,
+ PROPERTY_PROP,
+ SHARD_ID_PROP,
+ REPLICA_PROP,
+ PROPERTY_VALUE_PROP);
+ copy(req.getParams(), map, SHARD_UNIQUE);
+ String property = (String) map.get(PROPERTY_PROP);
+ if (!property.startsWith(PROPERTY_PREFIX)) {
+ property = PROPERTY_PREFIX + property;
+ }
- boolean uniquePerSlice = Boolean.parseBoolean((String) map.get(SHARD_UNIQUE));
-
- // Check if we're trying to set a property with parameters that allow us to set the property on multiple replicas
- // in a slice on properties that are known to only be one-per-slice and error out if so.
- if (StringUtils.isNotBlank((String) map.get(SHARD_UNIQUE)) &&
- SliceMutator.SLICE_UNIQUE_BOOLEAN_PROPERTIES.contains(property.toLowerCase(Locale.ROOT)) &&
- uniquePerSlice == false) {
- throw new SolrException(ErrorCode.BAD_REQUEST,
- "Overseer replica property command received for property " + property +
- " with the " + SHARD_UNIQUE +
- " parameter set to something other than 'true'. No action taken.");
- }
- return map;
- }),
+ boolean uniquePerSlice = Boolean.parseBoolean((String) map.get(SHARD_UNIQUE));
+
+ // Check if we're trying to set a property with parameters that allow us to set the
+ // property on multiple replicas
+ // in a slice on properties that are known to only be one-per-slice and error out if so.
+ if (StringUtils.isNotBlank((String) map.get(SHARD_UNIQUE))
+ && SliceMutator.SLICE_UNIQUE_BOOLEAN_PROPERTIES.contains(
+ property.toLowerCase(Locale.ROOT))
+ && uniquePerSlice == false) {
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST,
+ "Overseer replica property command received for property "
+ + property
+ + " with the "
+ + SHARD_UNIQUE
+ + " parameter set to something other than 'true'. No action taken.");
+ }
+ return map;
+ }),
// XXX should this command support followAliases?
- DELETEREPLICAPROP_OP(DELETEREPLICAPROP, (req, rsp, h) -> {
- Map<String, Object> map = copy(req.getParams().required(), null,
- COLLECTION_PROP,
- PROPERTY_PROP,
- SHARD_ID_PROP,
- REPLICA_PROP);
- return copy(req.getParams(), map, PROPERTY_PROP);
- }),
+ DELETEREPLICAPROP_OP(
+ DELETEREPLICAPROP,
+ (req, rsp, h) -> {
+ Map<String, Object> map =
+ copy(
+ req.getParams().required(),
+ null,
+ COLLECTION_PROP,
+ PROPERTY_PROP,
+ SHARD_ID_PROP,
+ REPLICA_PROP);
+ return copy(req.getParams(), map, PROPERTY_PROP);
+ }),
// XXX should this command support followAliases?
- BALANCESHARDUNIQUE_OP(BALANCESHARDUNIQUE, (req, rsp, h) -> {
- Map<String, Object> map = copy(req.getParams().required(), null,
- COLLECTION_PROP,
- PROPERTY_PROP);
- Boolean shardUnique = Boolean.parseBoolean(req.getParams().get(SHARD_UNIQUE));
- String prop = req.getParams().get(PROPERTY_PROP).toLowerCase(Locale.ROOT);
- if (!StringUtils.startsWith(prop, PROPERTY_PREFIX)) {
- prop = PROPERTY_PREFIX + prop;
- }
+ BALANCESHARDUNIQUE_OP(
+ BALANCESHARDUNIQUE,
+ (req, rsp, h) -> {
+ Map<String, Object> map =
+ copy(req.getParams().required(), null, COLLECTION_PROP, PROPERTY_PROP);
+ Boolean shardUnique = Boolean.parseBoolean(req.getParams().get(SHARD_UNIQUE));
+ String prop = req.getParams().get(PROPERTY_PROP).toLowerCase(Locale.ROOT);
+ if (!StringUtils.startsWith(prop, PROPERTY_PREFIX)) {
+ prop = PROPERTY_PREFIX + prop;
+ }
- if (!shardUnique && !SliceMutator.SLICE_UNIQUE_BOOLEAN_PROPERTIES.contains(prop)) {
- throw new SolrException(ErrorCode.BAD_REQUEST, "Balancing properties amongst replicas in a slice requires that"
- + " the property be pre-defined as a unique property (e.g. 'preferredLeader') or that 'shardUnique' be set to 'true'. " +
- " Property: " + prop + " shardUnique: " + shardUnique);
- }
+ if (!shardUnique && !SliceMutator.SLICE_UNIQUE_BOOLEAN_PROPERTIES.contains(prop)) {
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST,
+ "Balancing properties amongst replicas in a slice requires that"
+ + " the property be pre-defined as a unique property (e.g. 'preferredLeader') or that 'shardUnique' be set to 'true'. "
+ + " Property: "
+ + prop
+ + " shardUnique: "
+ + shardUnique);
+ }
- return copy(req.getParams(), map, ONLY_ACTIVE_NODES, SHARD_UNIQUE);
- }),
- REBALANCELEADERS_OP(REBALANCELEADERS, (req, rsp, h) -> {
- new RebalanceLeaders(req, rsp, h).execute();
- return null;
- }),
+ return copy(req.getParams(), map, ONLY_ACTIVE_NODES, SHARD_UNIQUE);
+ }),
+ REBALANCELEADERS_OP(
+ REBALANCELEADERS,
+ (req, rsp, h) -> {
+ new RebalanceLeaders(req, rsp, h).execute();
+ return null;
+ }),
// XXX should this command support followAliases?
- MODIFYCOLLECTION_OP(MODIFYCOLLECTION, (req, rsp, h) -> {
- Map<String, Object> m = copy(req.getParams(), null, CollectionAdminRequest.MODIFIABLE_COLLECTION_PROPERTIES);
- copyPropertiesWithPrefix(req.getParams(), m, PROPERTY_PREFIX);
- if (m.isEmpty()) {
- throw new SolrException(ErrorCode.BAD_REQUEST,
- formatString("no supported values provided {0}", CollectionAdminRequest.MODIFIABLE_COLLECTION_PROPERTIES.toString()));
- }
- copy(req.getParams().required(), m, COLLECTION_PROP);
- for (Map.Entry<String, Object> entry : m.entrySet()) {
- String prop = entry.getKey();
- if ("".equals(entry.getValue())) {
- // set to an empty string is equivalent to removing the property, see SOLR-12507
- entry.setValue(null);
- }
- DocCollection.verifyProp(m, prop);
- }
- if (m.get(REPLICATION_FACTOR) != null) {
- m.put(NRT_REPLICAS, m.get(REPLICATION_FACTOR));
- }
- return m;
- }),
- BACKUP_OP(BACKUP, (req, rsp, h) -> {
- req.getParams().required().check(NAME, COLLECTION_PROP);
-
- final String extCollectionName = req.getParams().get(COLLECTION_PROP);
- final boolean followAliases = req.getParams().getBool(FOLLOW_ALIASES, false);
- final String collectionName = followAliases ? h.coreContainer.getZkController().getZkStateReader()
- .getAliases().resolveSimpleAlias(extCollectionName) : extCollectionName;
- final ClusterState clusterState = h.coreContainer.getZkController().getClusterState();
- if (!clusterState.hasCollection(collectionName)) {
- throw new SolrException(ErrorCode.BAD_REQUEST, "Collection '" + collectionName + "' does not exist, no action taken.");
- }
-
- CoreContainer cc = h.coreContainer;
- String repo = req.getParams().get(CoreAdminParams.BACKUP_REPOSITORY);
- BackupRepository repository = cc.newBackupRepository(repo);
-
- String location = repository.getBackupLocation(req.getParams().get(CoreAdminParams.BACKUP_LOCATION));
- if (location == null) {
- //Refresh the cluster property file to make sure the value set for location is the latest
- // Check if the location is specified in the cluster property.
- location = new ClusterProperties(h.coreContainer.getZkController().getZkClient()).getClusterProperty(CoreAdminParams.BACKUP_LOCATION, null);
- if (location == null) {
- throw new SolrException(ErrorCode.BAD_REQUEST, "'location' is not specified as a query"
- + " parameter or as a default repository property or as a cluster property.");
- }
- }
- boolean incremental = req.getParams().getBool(CoreAdminParams.BACKUP_INCREMENTAL, true);
-
- // Check if the specified location is valid for this repository.
- final URI uri = repository.createDirectoryURI(location);
- try {
- if (!repository.exists(uri)) {
- throw new SolrException(ErrorCode.SERVER_ERROR, "specified location " + uri + " does not exist.");
- }
- } catch (IOException ex) {
- throw new SolrException(ErrorCode.SERVER_ERROR, "Failed to check the existence of " + uri + ". Is it valid?", ex);
- }
+ MODIFYCOLLECTION_OP(
+ MODIFYCOLLECTION,
+ (req, rsp, h) -> {
+ Map<String, Object> m =
+ copy(req.getParams(), null, CollectionAdminRequest.MODIFIABLE_COLLECTION_PROPERTIES);
+ copyPropertiesWithPrefix(req.getParams(), m, PROPERTY_PREFIX);
+ if (m.isEmpty()) {
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST,
+ formatString(
+ "no supported values provided {0}",
+ CollectionAdminRequest.MODIFIABLE_COLLECTION_PROPERTIES.toString()));
+ }
+ copy(req.getParams().required(), m, COLLECTION_PROP);
+ for (Map.Entry<String, Object> entry : m.entrySet()) {
+ String prop = entry.getKey();
+ if ("".equals(entry.getValue())) {
+ // set to an empty string is equivalent to removing the property, see SOLR-12507
+ entry.setValue(null);
+ }
+ DocCollection.verifyProp(m, prop);
+ }
+ if (m.get(REPLICATION_FACTOR) != null) {
+ m.put(NRT_REPLICAS, m.get(REPLICATION_FACTOR));
+ }
+ return m;
+ }),
+ BACKUP_OP(
+ BACKUP,
+ (req, rsp, h) -> {
+ req.getParams().required().check(NAME, COLLECTION_PROP);
+
+ final String extCollectionName = req.getParams().get(COLLECTION_PROP);
+ final boolean followAliases = req.getParams().getBool(FOLLOW_ALIASES, false);
+ final String collectionName =
+ followAliases
+ ? h.coreContainer
+ .getZkController()
+ .getZkStateReader()
+ .getAliases()
+ .resolveSimpleAlias(extCollectionName)
+ : extCollectionName;
+ final ClusterState clusterState = h.coreContainer.getZkController().getClusterState();
+ if (!clusterState.hasCollection(collectionName)) {
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST,
+ "Collection '" + collectionName + "' does not exist, no action taken.");
+ }
- String strategy = req.getParams().get(CollectionAdminParams.INDEX_BACKUP_STRATEGY, CollectionAdminParams.COPY_FILES_STRATEGY);
- if (!CollectionAdminParams.INDEX_BACKUP_STRATEGIES.contains(strategy)) {
- throw new SolrException(ErrorCode.BAD_REQUEST, "Unknown index backup strategy " + strategy);
- }
+ CoreContainer cc = h.coreContainer;
+ String repo = req.getParams().get(CoreAdminParams.BACKUP_REPOSITORY);
+ BackupRepository repository = cc.newBackupRepository(repo);
- Map<String, Object> params = copy(req.getParams(), null, NAME, COLLECTION_PROP,
- FOLLOW_ALIASES, CoreAdminParams.COMMIT_NAME, CoreAdminParams.MAX_NUM_BACKUP_POINTS);
- params.put(CoreAdminParams.BACKUP_LOCATION, location);
- if (repo != null) {
- params.put(CoreAdminParams.BACKUP_REPOSITORY, repo);
- }
+ String location =
+ repository.getBackupLocation(req.getParams().get(CoreAdminParams.BACKUP_LOCATION));
+ if (location == null) {
+ // Refresh the cluster property file to make sure the value set for location is the
+ // latest
+ // Check if the location is specified in the cluster property.
+ location =
+ new ClusterProperties(h.coreContainer.getZkController().getZkClient())
+ .getClusterProperty(CoreAdminParams.BACKUP_LOCATION, null);
+ if (location == null) {
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST,
+ "'location' is not specified as a query"
+ + " parameter or as a default repository property or as a cluster property.");
+ }
+ }
+ boolean incremental = req.getParams().getBool(CoreAdminParams.BACKUP_INCREMENTAL, true);
- params.put(CollectionAdminParams.INDEX_BACKUP_STRATEGY, strategy);
- params.put(CoreAdminParams.BACKUP_INCREMENTAL, incremental);
- return params;
- }),
- RESTORE_OP(RESTORE, (req, rsp, h) -> {
- req.getParams().required().check(NAME, COLLECTION_PROP);
+ // Check if the specified location is valid for this repository.
+ final URI uri = repository.createDirectoryURI(location);
+ try {
+ if (!repository.exists(uri)) {
+ throw new SolrException(
+ ErrorCode.SERVER_ERROR, "specified location " + uri + " does not exist.");
+ }
+ } catch (IOException ex) {
+ throw new SolrException(
+ ErrorCode.SERVER_ERROR,
+ "Failed to check the existence of " + uri + ". Is it valid?",
+ ex);
+ }
- final String collectionName = SolrIdentifierValidator.validateCollectionName(req.getParams().get(COLLECTION_PROP));
- if (h.coreContainer.getZkController().getZkStateReader().getAliases().hasAlias(collectionName)) {
- throw new SolrException(ErrorCode.BAD_REQUEST, "Collection '" + collectionName + "' is an existing alias, no action taken.");
- }
+ String strategy =
+ req.getParams()
+ .get(
+ CollectionAdminParams.INDEX_BACKUP_STRATEGY,
+ CollectionAdminParams.COPY_FILES_STRATEGY);
+ if (!CollectionAdminParams.INDEX_BACKUP_STRATEGIES.contains(strategy)) {
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST, "Unknown index backup strategy " + strategy);
+ }
- final CoreContainer cc = h.coreContainer;
- final String repo = req.getParams().get(CoreAdminParams.BACKUP_REPOSITORY);
- final BackupRepository repository = cc.newBackupRepository(repo);
-
- String location = repository.getBackupLocation(req.getParams().get(CoreAdminParams.BACKUP_LOCATION));
- if (location == null) {
- //Refresh the cluster property file to make sure the value set for location is the latest
- // Check if the location is specified in the cluster property.
- location = new ClusterProperties(h.coreContainer.getZkController().getZkClient()).getClusterProperty("location", null);
- if (location == null) {
- throw new SolrException(ErrorCode.BAD_REQUEST, "'location' is not specified as a query"
- + " parameter or as a default repository property or as a cluster property.");
- }
- }
+ Map<String, Object> params =
+ copy(
+ req.getParams(),
+ null,
+ NAME,
+ COLLECTION_PROP,
+ FOLLOW_ALIASES,
+ CoreAdminParams.COMMIT_NAME,
+ CoreAdminParams.MAX_NUM_BACKUP_POINTS);
+ params.put(CoreAdminParams.BACKUP_LOCATION, location);
+ if (repo != null) {
+ params.put(CoreAdminParams.BACKUP_REPOSITORY, repo);
+ }
- // Check if the specified location is valid for this repository.
- final URI uri = repository.createDirectoryURI(location);
- try {
- if (!repository.exists(uri)) {
- throw new SolrException(ErrorCode.SERVER_ERROR, "specified location " + uri + " does not exist.");
- }
- } catch (IOException ex) {
- throw new SolrException(ErrorCode.SERVER_ERROR, "Failed to check the existence of " + uri + ". Is it valid?", ex);
- }
+ params.put(CollectionAdminParams.INDEX_BACKUP_STRATEGY, strategy);
+ params.put(CoreAdminParams.BACKUP_INCREMENTAL, incremental);
+ return params;
+ }),
+ RESTORE_OP(
+ RESTORE,
+ (req, rsp, h) -> {
+ req.getParams().required().check(NAME, COLLECTION_PROP);
+
+ final String collectionName =
+ SolrIdentifierValidator.validateCollectionName(req.getParams().get(COLLECTION_PROP));
+ if (h.coreContainer
+ .getZkController()
+ .getZkStateReader()
+ .getAliases()
+ .hasAlias(collectionName)) {
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST,
+ "Collection '" + collectionName + "' is an existing alias, no action taken.");
+ }
- final String createNodeArg = req.getParams().get(CREATE_NODE_SET);
- if (CREATE_NODE_SET_EMPTY.equals(createNodeArg)) {
- throw new SolrException(
- SolrException.ErrorCode.BAD_REQUEST,
- "Cannot restore with a CREATE_NODE_SET of CREATE_NODE_SET_EMPTY."
- );
- }
- if (req.getParams().get(NRT_REPLICAS) != null && req.getParams().get(REPLICATION_FACTOR) != null) {
- throw new SolrException(SolrException.ErrorCode.BAD_REQUEST,
- "Cannot set both replicationFactor and nrtReplicas as they mean the same thing");
- }
+ final CoreContainer cc = h.coreContainer;
+ final String repo = req.getParams().get(CoreAdminParams.BACKUP_REPOSITORY);
+ final BackupRepository repository = cc.newBackupRepository(repo);
- final Map<String, Object> params = copy(req.getParams(), null, NAME, COLLECTION_PROP);
- params.put(CoreAdminParams.BACKUP_LOCATION, location);
- if (repo != null) {
- params.put(CoreAdminParams.BACKUP_REPOSITORY, repo);
- }
- // from CREATE_OP:
- copy(req.getParams(), params, COLL_CONF, REPLICATION_FACTOR, NRT_REPLICAS, TLOG_REPLICAS,
- PULL_REPLICAS, CREATE_NODE_SET, CREATE_NODE_SET_SHUFFLE, BACKUP_ID);
- copyPropertiesWithPrefix(req.getParams(), params, PROPERTY_PREFIX);
- return params;
- }),
- DELETEBACKUP_OP(DELETEBACKUP, (req, rsp, h) -> {
- req.getParams().required().check(NAME);
-
- CoreContainer cc = h.coreContainer;
- String repo = req.getParams().get(CoreAdminParams.BACKUP_REPOSITORY);
- try (BackupRepository repository = cc.newBackupRepository(repo)) {
-
- String location = repository.getBackupLocation(req.getParams().get(CoreAdminParams.BACKUP_LOCATION));
- if (location == null) {
- //Refresh the cluster property file to make sure the value set for location is the latest
- // Check if the location is specified in the cluster property.
- location = new ClusterProperties(h.coreContainer.getZkController().getZkClient()).getClusterProperty("location", null);
+ String location =
+ repository.getBackupLocation(req.getParams().get(CoreAdminParams.BACKUP_LOCATION));
if (location == null) {
- throw new SolrException(ErrorCode.BAD_REQUEST, "'location' is not specified as a query"
- + " parameter or as a default repository property or as a cluster property.");
+ // Refresh the cluster property file to make sure the value set for location is the
+ // latest
+ // Check if the location is specified in the cluster property.
+ location =
+ new ClusterProperties(h.coreContainer.getZkController().getZkClient())
+ .getClusterProperty("location", null);
+ if (location == null) {
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST,
+ "'location' is not specified as a query"
+ + " parameter or as a default repository property or as a cluster property.");
+ }
}
- }
- // Check if the specified location is valid for this repository.
- URI uri = repository.createDirectoryURI(location);
- try {
- if (!repository.exists(uri)) {
- throw new SolrException(ErrorCode.BAD_REQUEST, "specified location " + uri + " does not exist.");
+ // Check if the specified location is valid for this repository.
+ final URI uri = repository.createDirectoryURI(location);
+ try {
+ if (!repository.exists(uri)) {
+ throw new SolrException(
+ ErrorCode.SERVER_ERROR, "specified location " + uri + " does not exist.");
+ }
+ } catch (IOException ex) {
+ throw new SolrException(
+ ErrorCode.SERVER_ERROR,
+ "Failed to check the existence of " + uri + ". Is it valid?",
+ ex);
}
- } catch (IOException ex) {
- throw new SolrException(ErrorCode.SERVER_ERROR, "Failed to check the existence of " + uri + ". Is it valid?", ex);
- }
- int deletionModesProvided = 0;
- if (req.getParams().get(MAX_NUM_BACKUP_POINTS) != null) deletionModesProvided++;
- if (req.getParams().get(BACKUP_PURGE_UNUSED) != null) deletionModesProvided++;
- if (req.getParams().get(BACKUP_ID) != null) deletionModesProvided++;
- if (deletionModesProvided != 1) {
- throw new SolrException(BAD_REQUEST,
- String.format(Locale.ROOT, "Exactly one of %s, %s, and %s parameters must be provided",
- MAX_NUM_BACKUP_POINTS, BACKUP_PURGE_UNUSED, BACKUP_ID));
- }
-
- final Map<String, Object> params = copy(req.getParams(), null, NAME, BACKUP_REPOSITORY,
- BACKUP_LOCATION, BACKUP_ID, MAX_NUM_BACKUP_POINTS, BACKUP_PURGE_UNUSED);
- params.put(BACKUP_LOCATION, location);
- if (repo != null) {
- params.put(CoreAdminParams.BACKUP_REPOSITORY, repo);
- }
- return params;
- }
- }),
- LISTBACKUP_OP(LISTBACKUP, (req, rsp, h) -> {
- req.getParams().required().check(NAME);
-
- CoreContainer cc = h.coreContainer;
- String repo = req.getParams().get(CoreAdminParams.BACKUP_REPOSITORY);
- try (BackupRepository repository = cc.newBackupRepository(repo)) {
-
- String location = repository.getBackupLocation(req.getParams().get(CoreAdminParams.BACKUP_LOCATION));
- if (location == null) {
- //Refresh the cluster property file to make sure the value set for location is the latest
- // Check if the location is specified in the cluster property.
- location = new ClusterProperties(h.coreContainer.getZkController().getZkClient()).getClusterProperty(CoreAdminParams.BACKUP_LOCATION, null);
- if (location == null) {
- throw new SolrException(ErrorCode.BAD_REQUEST, "'location' is not specified as a query"
- + " parameter or as a default repository property or as a cluster property.");
+ final String createNodeArg = req.getParams().get(CREATE_NODE_SET);
+ if (CREATE_NODE_SET_EMPTY.equals(createNodeArg)) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Cannot restore with a CREATE_NODE_SET of CREATE_NODE_SET_EMPTY.");
+ }
+ if (req.getParams().get(NRT_REPLICAS) != null
+ && req.getParams().get(REPLICATION_FACTOR) != null) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Cannot set both replicationFactor and nrtReplicas as they mean the same thing");
}
- }
- String backupName = req.getParams().get(NAME);
- final URI locationURI = repository.createDirectoryURI(location);
- try {
- if (!repository.exists(locationURI)) {
- throw new SolrException(ErrorCode.BAD_REQUEST, "specified location " + locationURI + " does not exist.");
+ final Map<String, Object> params = copy(req.getParams(), null, NAME, COLLECTION_PROP);
+ params.put(CoreAdminParams.BACKUP_LOCATION, location);
+ if (repo != null) {
+ params.put(CoreAdminParams.BACKUP_REPOSITORY, repo);
}
- } catch (IOException ex) {
- throw new SolrException(ErrorCode.SERVER_ERROR, "Failed to check the existence of " + locationURI + ". Is it valid?", ex);
- }
- URI backupLocation = BackupFilePaths.buildExistingBackupLocationURI(repository, locationURI, backupName);
- if (repository.exists(repository.resolve(backupLocation, BackupManager.TRADITIONAL_BACKUP_PROPS_FILE))) {
- throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "The backup name [" + backupName + "] at " +
- "location [" + location + "] holds a non-incremental (legacy) backup, but " +
- "backup-listing is only supported on incremental backups");
- }
+ // from CREATE_OP:
+ copy(
+ req.getParams(),
+ params,
+ COLL_CONF,
+ REPLICATION_FACTOR,
+ NRT_REPLICAS,
+ TLOG_REPLICAS,
+ PULL_REPLICAS,
+ CREATE_NODE_SET,
+ CREATE_NODE_SET_SHUFFLE,
+ BACKUP_ID);
+ copyPropertiesWithPrefix(req.getParams(), params, PROPERTY_PREFIX);
+ return params;
+ }),
+ DELETEBACKUP_OP(
+ DELETEBACKUP,
+ (req, rsp, h) -> {
+ req.getParams().required().check(NAME);
+
+ CoreContainer cc = h.coreContainer;
+ String repo = req.getParams().get(CoreAdminParams.BACKUP_REPOSITORY);
+ try (BackupRepository repository = cc.newBackupRepository(repo)) {
+
+ String location =
+ repository.getBackupLocation(req.getParams().get(CoreAdminParams.BACKUP_LOCATION));
+ if (location == null) {
+ // Refresh the cluster property file to make sure the value set for location is the
+ // latest
+ // Check if the location is specified in the cluster property.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/admin/CollectionsHandler.java
##########
@@ -388,982 +426,1324 @@ public Category getCategory() {
return Category.ADMIN;
}
- private static void createSysConfigSet(CoreContainer coreContainer) throws KeeperException, InterruptedException {
+ private static void createSysConfigSet(CoreContainer coreContainer)
+ throws KeeperException, InterruptedException {
SolrZkClient zk = coreContainer.getZkController().getZkStateReader().getZkClient();
ZkCmdExecutor cmdExecutor = new ZkCmdExecutor(zk.getZkClientTimeout());
cmdExecutor.ensureExists(ZkStateReader.CONFIGS_ZKNODE, zk);
- cmdExecutor.ensureExists(ZkStateReader.CONFIGS_ZKNODE + "/" + CollectionAdminParams.SYSTEM_COLL, zk);
+ cmdExecutor.ensureExists(
+ ZkStateReader.CONFIGS_ZKNODE + "/" + CollectionAdminParams.SYSTEM_COLL, zk);
try {
- String path = ZkStateReader.CONFIGS_ZKNODE + "/" + CollectionAdminParams.SYSTEM_COLL + "/schema.xml";
- byte[] data = IOUtils.toByteArray(CollectionsHandler.class.getResourceAsStream("/SystemCollectionSchema.xml"));
+ String path =
+ ZkStateReader.CONFIGS_ZKNODE + "/" + CollectionAdminParams.SYSTEM_COLL + "/schema.xml";
+ byte[] data =
+ IOUtils.toByteArray(
+ CollectionsHandler.class.getResourceAsStream("/SystemCollectionSchema.xml"));
assert data != null && data.length > 0;
cmdExecutor.ensureExists(path, data, CreateMode.PERSISTENT, zk);
- path = ZkStateReader.CONFIGS_ZKNODE + "/" + CollectionAdminParams.SYSTEM_COLL + "/solrconfig.xml";
- data = IOUtils.toByteArray(CollectionsHandler.class.getResourceAsStream("/SystemCollectionSolrConfig.xml"));
+ path =
+ ZkStateReader.CONFIGS_ZKNODE
+ + "/"
+ + CollectionAdminParams.SYSTEM_COLL
+ + "/solrconfig.xml";
+ data =
+ IOUtils.toByteArray(
+ CollectionsHandler.class.getResourceAsStream("/SystemCollectionSolrConfig.xml"));
assert data != null && data.length > 0;
cmdExecutor.ensureExists(path, data, CreateMode.PERSISTENT, zk);
} catch (IOException e) {
throw new SolrException(ErrorCode.SERVER_ERROR, e);
}
-
-
}
- private static void addStatusToResponse(NamedList<Object> results, RequestStatusState state, String msg) {
+ private static void addStatusToResponse(
+ NamedList<Object> results, RequestStatusState state, String msg) {
SimpleOrderedMap<String> status = new SimpleOrderedMap<>();
status.add("state", state.getKey());
status.add("msg", msg);
results.add("status", status);
}
public enum CollectionOperation implements CollectionOp {
- CREATE_OP(CREATE, (req, rsp, h) -> {
- Map<String, Object> props = copy(req.getParams().required(), null, NAME);
- props.put("fromApi", "true");
- copy(req.getParams(), props,
- REPLICATION_FACTOR,
- COLL_CONF,
- NUM_SLICES,
- CREATE_NODE_SET,
- CREATE_NODE_SET_SHUFFLE,
- SHARDS_PROP,
- PULL_REPLICAS,
- TLOG_REPLICAS,
- NRT_REPLICAS,
- WAIT_FOR_FINAL_STATE,
- PER_REPLICA_STATE,
- ALIAS);
-
- if (props.get(REPLICATION_FACTOR) != null && props.get(NRT_REPLICAS) != null) {
- //TODO: Remove this in 8.0 . Keep this for SolrJ client back-compat. See SOLR-11676 for more details
- int replicationFactor = Integer.parseInt((String) props.get(REPLICATION_FACTOR));
- int nrtReplicas = Integer.parseInt((String) props.get(NRT_REPLICAS));
- if (replicationFactor != nrtReplicas) {
- throw new SolrException(ErrorCode.BAD_REQUEST,
- "Cannot specify both replicationFactor and nrtReplicas as they mean the same thing");
- }
- }
- if (props.get(REPLICATION_FACTOR) != null) {
- props.put(NRT_REPLICAS, props.get(REPLICATION_FACTOR));
- } else if (props.get(NRT_REPLICAS) != null) {
- props.put(REPLICATION_FACTOR, props.get(NRT_REPLICAS));
- }
-
- final String collectionName = SolrIdentifierValidator.validateCollectionName((String) props.get(NAME));
- final String shardsParam = (String) props.get(SHARDS_PROP);
- if (StringUtils.isNotEmpty(shardsParam)) {
- verifyShardsParam(shardsParam);
- }
- if (CollectionAdminParams.SYSTEM_COLL.equals(collectionName)) {
- //We must always create a .system collection with only a single shard
- props.put(NUM_SLICES, 1);
- props.remove(SHARDS_PROP);
- createSysConfigSet(h.coreContainer);
-
- }
- if (shardsParam == null) h.copyFromClusterProp(props, NUM_SLICES);
- for (String prop : ImmutableSet.of(NRT_REPLICAS, PULL_REPLICAS, TLOG_REPLICAS))
- h.copyFromClusterProp(props, prop);
- copyPropertiesWithPrefix(req.getParams(), props, PROPERTY_PREFIX);
- return copyPropertiesWithPrefix(req.getParams(), props, "router.");
+ CREATE_OP(
+ CREATE,
+ (req, rsp, h) -> {
+ Map<String, Object> props = copy(req.getParams().required(), null, NAME);
+ props.put("fromApi", "true");
+ copy(
+ req.getParams(),
+ props,
+ REPLICATION_FACTOR,
+ COLL_CONF,
+ NUM_SLICES,
+ CREATE_NODE_SET,
+ CREATE_NODE_SET_SHUFFLE,
+ SHARDS_PROP,
+ PULL_REPLICAS,
+ TLOG_REPLICAS,
+ NRT_REPLICAS,
+ WAIT_FOR_FINAL_STATE,
+ PER_REPLICA_STATE,
+ ALIAS);
+
+ if (props.get(REPLICATION_FACTOR) != null && props.get(NRT_REPLICAS) != null) {
+ // TODO: Remove this in 8.0 . Keep this for SolrJ client back-compat. See SOLR-11676 for
+ // more details
+ int replicationFactor = Integer.parseInt((String) props.get(REPLICATION_FACTOR));
+ int nrtReplicas = Integer.parseInt((String) props.get(NRT_REPLICAS));
+ if (replicationFactor != nrtReplicas) {
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST,
+ "Cannot specify both replicationFactor and nrtReplicas as they mean the same thing");
+ }
+ }
+ if (props.get(REPLICATION_FACTOR) != null) {
+ props.put(NRT_REPLICAS, props.get(REPLICATION_FACTOR));
+ } else if (props.get(NRT_REPLICAS) != null) {
+ props.put(REPLICATION_FACTOR, props.get(NRT_REPLICAS));
+ }
- }),
+ final String collectionName =
+ SolrIdentifierValidator.validateCollectionName((String) props.get(NAME));
+ final String shardsParam = (String) props.get(SHARDS_PROP);
+ if (StringUtils.isNotEmpty(shardsParam)) {
+ verifyShardsParam(shardsParam);
+ }
+ if (CollectionAdminParams.SYSTEM_COLL.equals(collectionName)) {
+ // We must always create a .system collection with only a single shard
+ props.put(NUM_SLICES, 1);
+ props.remove(SHARDS_PROP);
+ createSysConfigSet(h.coreContainer);
+ }
+ if (shardsParam == null) h.copyFromClusterProp(props, NUM_SLICES);
+ for (String prop : ImmutableSet.of(NRT_REPLICAS, PULL_REPLICAS, TLOG_REPLICAS))
+ h.copyFromClusterProp(props, prop);
+ copyPropertiesWithPrefix(req.getParams(), props, PROPERTY_PREFIX);
+ return copyPropertiesWithPrefix(req.getParams(), props, "router.");
+ }),
@SuppressWarnings({"unchecked"})
- COLSTATUS_OP(COLSTATUS, (req, rsp, h) -> {
- Map<String, Object> props = copy(req.getParams(), null,
- COLLECTION_PROP,
- ColStatus.CORE_INFO_PROP,
- ColStatus.SEGMENTS_PROP,
- ColStatus.FIELD_INFO_PROP,
- ColStatus.RAW_SIZE_PROP,
- ColStatus.RAW_SIZE_SUMMARY_PROP,
- ColStatus.RAW_SIZE_DETAILS_PROP,
- ColStatus.RAW_SIZE_SAMPLING_PERCENT_PROP,
- ColStatus.SIZE_INFO_PROP);
-
- new ColStatus(h.coreContainer.getSolrClientCache(),
- h.coreContainer.getZkController().getZkStateReader().getClusterState(), new ZkNodeProps(props))
- .getColStatus(rsp.getValues());
- return null;
- }),
- DELETE_OP(DELETE, (req, rsp, h) -> {
- Map<String, Object> map = copy(req.getParams().required(), null, NAME);
- return copy(req.getParams(), map, FOLLOW_ALIASES);
- }),
+ COLSTATUS_OP(
+ COLSTATUS,
+ (req, rsp, h) -> {
+ Map<String, Object> props =
+ copy(
+ req.getParams(),
+ null,
+ COLLECTION_PROP,
+ ColStatus.CORE_INFO_PROP,
+ ColStatus.SEGMENTS_PROP,
+ ColStatus.FIELD_INFO_PROP,
+ ColStatus.RAW_SIZE_PROP,
+ ColStatus.RAW_SIZE_SUMMARY_PROP,
+ ColStatus.RAW_SIZE_DETAILS_PROP,
+ ColStatus.RAW_SIZE_SAMPLING_PERCENT_PROP,
+ ColStatus.SIZE_INFO_PROP);
+
+ new ColStatus(
+ h.coreContainer.getSolrClientCache(),
+ h.coreContainer.getZkController().getZkStateReader().getClusterState(),
+ new ZkNodeProps(props))
+ .getColStatus(rsp.getValues());
+ return null;
+ }),
+ DELETE_OP(
+ DELETE,
+ (req, rsp, h) -> {
+ Map<String, Object> map = copy(req.getParams().required(), null, NAME);
+ return copy(req.getParams(), map, FOLLOW_ALIASES);
+ }),
// XXX should this command support followAliases?
- RELOAD_OP(RELOAD, (req, rsp, h) -> {
- Map<String, Object> map = copy(req.getParams().required(), null, NAME);
- return copy(req.getParams(), map);
- }),
-
- RENAME_OP(RENAME, (req, rsp, h) -> {
- Map<String, Object> map = copy(req.getParams().required(), null, NAME, CollectionAdminParams.TARGET);
- return copy(req.getParams(), map, FOLLOW_ALIASES);
- }),
-
- REINDEXCOLLECTION_OP(REINDEXCOLLECTION, (req, rsp, h) -> {
- Map<String, Object> m = copy(req.getParams().required(), null, NAME);
- copy(req.getParams(), m,
- ReindexCollectionCmd.COMMAND,
- ReindexCollectionCmd.REMOVE_SOURCE,
- ReindexCollectionCmd.TARGET,
- ZkStateReader.CONFIGNAME_PROP,
- NUM_SLICES,
- NRT_REPLICAS,
- PULL_REPLICAS,
- TLOG_REPLICAS,
- REPLICATION_FACTOR,
- CREATE_NODE_SET,
- CREATE_NODE_SET_SHUFFLE,
- "shards",
- CommonParams.ROWS,
- CommonParams.Q,
- CommonParams.FL,
- FOLLOW_ALIASES);
- if (req.getParams().get("collection." + ZkStateReader.CONFIGNAME_PROP) != null) {
- m.put(ZkStateReader.CONFIGNAME_PROP, req.getParams().get("collection." + ZkStateReader.CONFIGNAME_PROP));
- }
- copyPropertiesWithPrefix(req.getParams(), m, "router.");
- return m;
- }),
-
- SYNCSHARD_OP(SYNCSHARD, (req, rsp, h) -> {
- String extCollection = req.getParams().required().get("collection");
- String collection = h.coreContainer.getZkController().getZkStateReader().getAliases().resolveSimpleAlias(extCollection);
- String shard = req.getParams().required().get("shard");
-
- ClusterState clusterState = h.coreContainer.getZkController().getClusterState();
-
- DocCollection docCollection = clusterState.getCollection(collection);
- ZkNodeProps leaderProps = docCollection.getLeader(shard);
- ZkCoreNodeProps nodeProps = new ZkCoreNodeProps(leaderProps);
-
- try (HttpSolrClient client = new Builder(nodeProps.getBaseUrl())
- .withConnectionTimeout(15000)
- .withSocketTimeout(60000)
- .build()) {
- RequestSyncShard reqSyncShard = new RequestSyncShard();
- reqSyncShard.setCollection(collection);
- reqSyncShard.setShard(shard);
- reqSyncShard.setCoreName(nodeProps.getCoreName());
- client.request(reqSyncShard);
- }
- return null;
- }),
-
- CREATEALIAS_OP(CREATEALIAS, (req, rsp, h) -> {
- String alias = req.getParams().get(NAME);
- SolrIdentifierValidator.validateAliasName(alias);
- String collections = req.getParams().get("collections");
- RoutedAlias routedAlias = null;
- Exception ex = null;
- HashMap<String,Object> possiblyModifiedParams = new HashMap<>();
- try {
- // note that RA specific validation occurs here.
- req.getParams().toMap(possiblyModifiedParams);
- @SuppressWarnings({"unchecked", "rawtypes"})
- // This is awful because RoutedAlias lies about what types it wants
- Map<String, String> temp = (Map<String, String>) (Map) possiblyModifiedParams;
- routedAlias = RoutedAlias.fromProps(alias, temp);
- } catch (SolrException e) {
- // we'll throw this later if we are in fact creating a routed alias.
- ex = e;
- }
- ModifiableSolrParams finalParams = new ModifiableSolrParams();
- for (Map.Entry<String, Object> entry : possiblyModifiedParams.entrySet()) {
- if (entry.getValue().getClass().isArray() ) {
- // v2 api hits this case
- for (Object o : (Object[]) entry.getValue()) {
- finalParams.add(entry.getKey(),o.toString());
+ RELOAD_OP(
+ RELOAD,
+ (req, rsp, h) -> {
+ Map<String, Object> map = copy(req.getParams().required(), null, NAME);
+ return copy(req.getParams(), map);
+ }),
+
+ RENAME_OP(
+ RENAME,
+ (req, rsp, h) -> {
+ Map<String, Object> map =
+ copy(req.getParams().required(), null, NAME, CollectionAdminParams.TARGET);
+ return copy(req.getParams(), map, FOLLOW_ALIASES);
+ }),
+
+ REINDEXCOLLECTION_OP(
+ REINDEXCOLLECTION,
+ (req, rsp, h) -> {
+ Map<String, Object> m = copy(req.getParams().required(), null, NAME);
+ copy(
+ req.getParams(),
+ m,
+ ReindexCollectionCmd.COMMAND,
+ ReindexCollectionCmd.REMOVE_SOURCE,
+ ReindexCollectionCmd.TARGET,
+ ZkStateReader.CONFIGNAME_PROP,
+ NUM_SLICES,
+ NRT_REPLICAS,
+ PULL_REPLICAS,
+ TLOG_REPLICAS,
+ REPLICATION_FACTOR,
+ CREATE_NODE_SET,
+ CREATE_NODE_SET_SHUFFLE,
+ "shards",
+ CommonParams.ROWS,
+ CommonParams.Q,
+ CommonParams.FL,
+ FOLLOW_ALIASES);
+ if (req.getParams().get("collection." + ZkStateReader.CONFIGNAME_PROP) != null) {
+ m.put(
+ ZkStateReader.CONFIGNAME_PROP,
+ req.getParams().get("collection." + ZkStateReader.CONFIGNAME_PROP));
+ }
+ copyPropertiesWithPrefix(req.getParams(), m, "router.");
+ return m;
+ }),
+
+ SYNCSHARD_OP(
+ SYNCSHARD,
+ (req, rsp, h) -> {
+ String extCollection = req.getParams().required().get("collection");
+ String collection =
+ h.coreContainer
+ .getZkController()
+ .getZkStateReader()
+ .getAliases()
+ .resolveSimpleAlias(extCollection);
+ String shard = req.getParams().required().get("shard");
+
+ ClusterState clusterState = h.coreContainer.getZkController().getClusterState();
+
+ DocCollection docCollection = clusterState.getCollection(collection);
+ ZkNodeProps leaderProps = docCollection.getLeader(shard);
+ ZkCoreNodeProps nodeProps = new ZkCoreNodeProps(leaderProps);
+
+ try (HttpSolrClient client =
+ new Builder(nodeProps.getBaseUrl())
+ .withConnectionTimeout(15000)
+ .withSocketTimeout(60000)
+ .build()) {
+ RequestSyncShard reqSyncShard = new RequestSyncShard();
+ reqSyncShard.setCollection(collection);
+ reqSyncShard.setShard(shard);
+ reqSyncShard.setCoreName(nodeProps.getCoreName());
+ client.request(reqSyncShard);
+ }
+ return null;
+ }),
+
+ CREATEALIAS_OP(
+ CREATEALIAS,
+ (req, rsp, h) -> {
+ String alias = req.getParams().get(NAME);
+ SolrIdentifierValidator.validateAliasName(alias);
+ String collections = req.getParams().get("collections");
+ RoutedAlias routedAlias = null;
+ Exception ex = null;
+ HashMap<String, Object> possiblyModifiedParams = new HashMap<>();
+ try {
+ // note that RA specific validation occurs here.
+ req.getParams().toMap(possiblyModifiedParams);
+ @SuppressWarnings({"unchecked", "rawtypes"})
+ // This is awful because RoutedAlias lies about what types it wants
+ Map<String, String> temp = (Map<String, String>) (Map) possiblyModifiedParams;
+ routedAlias = RoutedAlias.fromProps(alias, temp);
+ } catch (SolrException e) {
+ // we'll throw this later if we are in fact creating a routed alias.
+ ex = e;
+ }
+ ModifiableSolrParams finalParams = new ModifiableSolrParams();
+ for (Map.Entry<String, Object> entry : possiblyModifiedParams.entrySet()) {
+ if (entry.getValue().getClass().isArray()) {
+ // v2 api hits this case
+ for (Object o : (Object[]) entry.getValue()) {
+ finalParams.add(entry.getKey(), o.toString());
+ }
+ } else {
+ finalParams.add(entry.getKey(), entry.getValue().toString());
+ }
}
- } else {
- finalParams.add(entry.getKey(),entry.getValue().toString());
- }
- }
- if (collections != null) {
- if (routedAlias != null) {
- throw new SolrException(BAD_REQUEST, "Collections cannot be specified when creating a routed alias.");
- } else {
- //////////////////////////////////////
- // Regular alias creation indicated //
- //////////////////////////////////////
- return copy(finalParams.required(), null, NAME, "collections");
- }
- }
+ if (collections != null) {
+ if (routedAlias != null) {
+ throw new SolrException(
+ BAD_REQUEST, "Collections cannot be specified when creating a routed alias.");
+ } else {
+ //////////////////////////////////////
+ // Regular alias creation indicated //
+ //////////////////////////////////////
+ return copy(finalParams.required(), null, NAME, "collections");
+ }
+ }
- /////////////////////////////////////////////////
- // We are creating a routed alias from here on //
- /////////////////////////////////////////////////
+ /////////////////////////////////////////////////
+ // We are creating a routed alias from here on //
+ /////////////////////////////////////////////////
- // If our prior creation attempt had issues expose them now.
- if (ex != null) {
- throw ex;
- }
+ // If our prior creation attempt had issues expose them now.
+ if (ex != null) {
+ throw ex;
+ }
- // Now filter out just the parameters we care about from the request
- assert routedAlias != null;
- Map<String, Object> result = copy(finalParams, null, routedAlias.getRequiredParams());
- copy(finalParams, result, routedAlias.getOptionalParams());
-
- ModifiableSolrParams createCollParams = new ModifiableSolrParams(); // without prefix
-
- // add to result params that start with "create-collection.".
- // Additionally, save these without the prefix to createCollParams
- for (Map.Entry<String, String[]> entry : finalParams) {
- final String p = entry.getKey();
- if (p.startsWith(CREATE_COLLECTION_PREFIX)) {
- // This is what SolrParams#getAll(Map, Collection)} does
- final String[] v = entry.getValue();
- if (v.length == 1) {
- result.put(p, v[0]);
- } else {
- result.put(p, v);
+ // Now filter out just the parameters we care about from the request
+ assert routedAlias != null;
+ Map<String, Object> result = copy(finalParams, null, routedAlias.getRequiredParams());
+ copy(finalParams, result, routedAlias.getOptionalParams());
+
+ ModifiableSolrParams createCollParams = new ModifiableSolrParams(); // without prefix
+
+ // add to result params that start with "create-collection.".
+ // Additionally, save these without the prefix to createCollParams
+ for (Map.Entry<String, String[]> entry : finalParams) {
+ final String p = entry.getKey();
+ if (p.startsWith(CREATE_COLLECTION_PREFIX)) {
+ // This is what SolrParams#getAll(Map, Collection)} does
+ final String[] v = entry.getValue();
+ if (v.length == 1) {
+ result.put(p, v[0]);
+ } else {
+ result.put(p, v);
+ }
+ createCollParams.set(p.substring(CREATE_COLLECTION_PREFIX.length()), v);
+ }
}
- createCollParams.set(p.substring(CREATE_COLLECTION_PREFIX.length()), v);
- }
- }
- // Verify that the create-collection prefix'ed params appear to be valid.
- if (createCollParams.get(NAME) != null) {
- throw new SolrException(BAD_REQUEST, "routed aliases calculate names for their " +
- "dependent collections, you cannot specify the name.");
- }
- if (createCollParams.get(COLL_CONF) == null) {
- throw new SolrException(SolrException.ErrorCode.BAD_REQUEST,
- "We require an explicit " + COLL_CONF);
- }
- // note: could insist on a config name here as well.... or wait to throw at overseer
- createCollParams.add(NAME, "TMP_name_TMP_name_TMP"); // just to pass validation
- CREATE_OP.execute(new LocalSolrQueryRequest(null, createCollParams), rsp, h); // ignore results
+ // Verify that the create-collection prefix'ed params appear to be valid.
+ if (createCollParams.get(NAME) != null) {
+ throw new SolrException(
+ BAD_REQUEST,
+ "routed aliases calculate names for their "
+ + "dependent collections, you cannot specify the name.");
+ }
+ if (createCollParams.get(COLL_CONF) == null) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST, "We require an explicit " + COLL_CONF);
+ }
+ // note: could insist on a config name here as well.... or wait to throw at overseer
+ createCollParams.add(NAME, "TMP_name_TMP_name_TMP"); // just to pass validation
+ CREATE_OP.execute(
+ new LocalSolrQueryRequest(null, createCollParams), rsp, h); // ignore results
- return result;
- }),
+ return result;
+ }),
DELETEALIAS_OP(DELETEALIAS, (req, rsp, h) -> copy(req.getParams().required(), null, NAME)),
/**
* Change properties for an alias (use CREATEALIAS_OP to change the actual value of the alias)
*/
- ALIASPROP_OP(ALIASPROP, (req, rsp, h) -> {
- Map<String, Object> params = copy(req.getParams().required(), null, NAME);
-
- // Note: success/no-op in the event of no properties supplied is intentional. Keeps code simple and one less case
- // for api-callers to check for.
- return convertPrefixToMap(req.getParams(), params, "property");
- }),
-
- /**
- * List the aliases and associated properties.
- */
+ ALIASPROP_OP(
+ ALIASPROP,
+ (req, rsp, h) -> {
+ Map<String, Object> params = copy(req.getParams().required(), null, NAME);
+
+ // Note: success/no-op in the event of no properties supplied is intentional. Keeps code
+ // simple and one less case
+ // for api-callers to check for.
+ return convertPrefixToMap(req.getParams(), params, "property");
+ }),
+
+ /** List the aliases and associated properties. */
@SuppressWarnings({"unchecked"})
- LISTALIASES_OP(LISTALIASES, (req, rsp, h) -> {
- ZkStateReader zkStateReader = h.coreContainer.getZkController().getZkStateReader();
- // if someone calls listAliases, lets ensure we return an up to date response
- zkStateReader.aliasesManager.update();
- Aliases aliases = zkStateReader.getAliases();
- if (aliases != null) {
- // the aliases themselves...
- rsp.getValues().add("aliases", aliases.getCollectionAliasMap());
- // Any properties for the above aliases.
- Map<String, Map<String, String>> meta = new LinkedHashMap<>();
- for (String alias : aliases.getCollectionAliasListMap().keySet()) {
- Map<String, String> collectionAliasProperties = aliases.getCollectionAliasProperties(alias);
- if (!collectionAliasProperties.isEmpty()) {
- meta.put(alias, collectionAliasProperties);
+ LISTALIASES_OP(
+ LISTALIASES,
+ (req, rsp, h) -> {
+ ZkStateReader zkStateReader = h.coreContainer.getZkController().getZkStateReader();
+ // if someone calls listAliases, lets ensure we return an up to date response
+ zkStateReader.aliasesManager.update();
+ Aliases aliases = zkStateReader.getAliases();
+ if (aliases != null) {
+ // the aliases themselves...
+ rsp.getValues().add("aliases", aliases.getCollectionAliasMap());
+ // Any properties for the above aliases.
+ Map<String, Map<String, String>> meta = new LinkedHashMap<>();
+ for (String alias : aliases.getCollectionAliasListMap().keySet()) {
+ Map<String, String> collectionAliasProperties =
+ aliases.getCollectionAliasProperties(alias);
+ if (!collectionAliasProperties.isEmpty()) {
+ meta.put(alias, collectionAliasProperties);
+ }
+ }
+ rsp.getValues().add("properties", meta);
+ }
+ return null;
+ }),
+ SPLITSHARD_OP(
+ SPLITSHARD,
+ DEFAULT_COLLECTION_OP_TIMEOUT * 5,
+ (req, rsp, h) -> {
+ String name = req.getParams().required().get(COLLECTION_PROP);
+ // TODO : add support for multiple shards
+ String shard = req.getParams().get(SHARD_ID_PROP);
+ String rangesStr = req.getParams().get(CoreAdminParams.RANGES);
+ String splitKey = req.getParams().get("split.key");
+ String numSubShards = req.getParams().get(NUM_SUB_SHARDS);
+ String fuzz = req.getParams().get(SPLIT_FUZZ);
+
+ if (splitKey == null && shard == null) {
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST, "At least one of shard, or split.key should be specified.");
+ }
+ if (splitKey != null && shard != null) {
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST, "Only one of 'shard' or 'split.key' should be specified");
+ }
+ if (splitKey != null && rangesStr != null) {
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST, "Only one of 'ranges' or 'split.key' should be specified");
+ }
+ if (numSubShards != null && (splitKey != null || rangesStr != null)) {
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST,
+ "numSubShards can not be specified with split.key or ranges parameters");
+ }
+ if (fuzz != null && (splitKey != null || rangesStr != null)) {
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST,
+ "fuzz can not be specified with split.key or ranges parameters");
}
- }
- rsp.getValues().add("properties", meta);
- }
- return null;
- }),
- SPLITSHARD_OP(SPLITSHARD, DEFAULT_COLLECTION_OP_TIMEOUT * 5, (req, rsp, h) -> {
- String name = req.getParams().required().get(COLLECTION_PROP);
- // TODO : add support for multiple shards
- String shard = req.getParams().get(SHARD_ID_PROP);
- String rangesStr = req.getParams().get(CoreAdminParams.RANGES);
- String splitKey = req.getParams().get("split.key");
- String numSubShards = req.getParams().get(NUM_SUB_SHARDS);
- String fuzz = req.getParams().get(SPLIT_FUZZ);
-
- if (splitKey == null && shard == null) {
- throw new SolrException(ErrorCode.BAD_REQUEST, "At least one of shard, or split.key should be specified.");
- }
- if (splitKey != null && shard != null) {
- throw new SolrException(ErrorCode.BAD_REQUEST,
- "Only one of 'shard' or 'split.key' should be specified");
- }
- if (splitKey != null && rangesStr != null) {
- throw new SolrException(ErrorCode.BAD_REQUEST,
- "Only one of 'ranges' or 'split.key' should be specified");
- }
- if (numSubShards != null && (splitKey != null || rangesStr != null)) {
- throw new SolrException(ErrorCode.BAD_REQUEST,
- "numSubShards can not be specified with split.key or ranges parameters");
- }
- if (fuzz != null && (splitKey != null || rangesStr != null)) {
- throw new SolrException(ErrorCode.BAD_REQUEST,
- "fuzz can not be specified with split.key or ranges parameters");
- }
- Map<String, Object> map = copy(req.getParams(), null,
- COLLECTION_PROP,
- SHARD_ID_PROP,
- "split.key",
- CoreAdminParams.RANGES,
- WAIT_FOR_FINAL_STATE,
- TIMING,
- SPLIT_METHOD,
- NUM_SUB_SHARDS,
- SPLIT_FUZZ,
- SPLIT_BY_PREFIX,
- FOLLOW_ALIASES);
- return copyPropertiesWithPrefix(req.getParams(), map, PROPERTY_PREFIX);
- }),
- DELETESHARD_OP(DELETESHARD, (req, rsp, h) -> {
- Map<String, Object> map = copy(req.getParams().required(), null,
- COLLECTION_PROP,
- SHARD_ID_PROP);
- copy(req.getParams(), map,
- DELETE_INDEX,
- DELETE_DATA_DIR,
- DELETE_INSTANCE_DIR,
- FOLLOW_ALIASES);
- return map;
- }),
- FORCELEADER_OP(FORCELEADER, (req, rsp, h) -> {
- forceLeaderElection(req, h);
- return null;
- }),
- CREATESHARD_OP(CREATESHARD, (req, rsp, h) -> {
- Map<String, Object> map = copy(req.getParams().required(), null,
- COLLECTION_PROP,
- SHARD_ID_PROP);
- ClusterState clusterState = h.coreContainer.getZkController().getClusterState();
- final String newShardName = SolrIdentifierValidator.validateShardName(req.getParams().get(SHARD_ID_PROP));
- boolean followAliases = req.getParams().getBool(FOLLOW_ALIASES, false);
- String extCollectionName = req.getParams().get(COLLECTION_PROP);
- String collectionName = followAliases ? h.coreContainer.getZkController().getZkStateReader()
- .getAliases().resolveSimpleAlias(extCollectionName) : extCollectionName;
- if (!ImplicitDocRouter.NAME.equals(((Map<?,?>) clusterState.getCollection(collectionName).get(DOC_ROUTER)).get(NAME)))
- throw new SolrException(ErrorCode.BAD_REQUEST, "shards can be added only to 'implicit' collections");
- copy(req.getParams(), map,
- REPLICATION_FACTOR,
- NRT_REPLICAS,
- TLOG_REPLICAS,
- PULL_REPLICAS,
- CREATE_NODE_SET,
- WAIT_FOR_FINAL_STATE,
- FOLLOW_ALIASES);
- return copyPropertiesWithPrefix(req.getParams(), map, PROPERTY_PREFIX);
- }),
- DELETEREPLICA_OP(DELETEREPLICA, (req, rsp, h) -> {
- Map<String, Object> map = copy(req.getParams().required(), null,
- COLLECTION_PROP);
-
- return copy(req.getParams(), map,
- DELETE_INDEX,
- DELETE_DATA_DIR,
- DELETE_INSTANCE_DIR,
- COUNT_PROP, REPLICA_PROP,
- SHARD_ID_PROP,
- ONLY_IF_DOWN,
- FOLLOW_ALIASES);
- }),
- MIGRATE_OP(MIGRATE, (req, rsp, h) -> {
- Map<String, Object> map = copy(req.getParams().required(), null, COLLECTION_PROP, "split.key", "target.collection");
- return copy(req.getParams(), map, "forward.timeout", FOLLOW_ALIASES);
- }),
- ADDROLE_OP(ADDROLE, (req, rsp, h) -> {
- Map<String, Object> map = copy(req.getParams().required(), null, "role", "node");
- if (!KNOWN_ROLES.contains(map.get("role")))
- throw new SolrException(ErrorCode.BAD_REQUEST, "Unknown role. Supported roles are ," + KNOWN_ROLES);
- return map;
- }),
- REMOVEROLE_OP(REMOVEROLE, (req, rsp, h) -> {
- Map<String, Object> map = copy(req.getParams().required(), null, "role", "node");
- if (!KNOWN_ROLES.contains(map.get("role")))
- throw new SolrException(ErrorCode.BAD_REQUEST, "Unknown role. Supported roles are ," + KNOWN_ROLES);
- return map;
- }),
- CLUSTERPROP_OP(CLUSTERPROP, (req, rsp, h) -> {
- String name = req.getParams().required().get(NAME);
- String val = req.getParams().get(VALUE_LONG);
- ClusterProperties cp = new ClusterProperties(h.coreContainer.getZkController().getZkClient());
- cp.setClusterProperty(name, val);
- return null;
- }),
- COLLECTIONPROP_OP(COLLECTIONPROP, (req, rsp, h) -> {
- String extCollection = req.getParams().required().get(NAME);
- String collection = h.coreContainer.getZkController().getZkStateReader().getAliases().resolveSimpleAlias(extCollection);
- String name = req.getParams().required().get(PROPERTY_NAME);
- String val = req.getParams().get(PROPERTY_VALUE);
- CollectionProperties cp = new CollectionProperties(h.coreContainer.getZkController().getZkClient());
- cp.setCollectionProperty(collection, name, val);
- return null;
- }),
+ Map<String, Object> map =
+ copy(
+ req.getParams(),
+ null,
+ COLLECTION_PROP,
+ SHARD_ID_PROP,
+ "split.key",
+ CoreAdminParams.RANGES,
+ WAIT_FOR_FINAL_STATE,
+ TIMING,
+ SPLIT_METHOD,
+ NUM_SUB_SHARDS,
+ SPLIT_FUZZ,
+ SPLIT_BY_PREFIX,
+ FOLLOW_ALIASES);
+ return copyPropertiesWithPrefix(req.getParams(), map, PROPERTY_PREFIX);
+ }),
+ DELETESHARD_OP(
+ DELETESHARD,
+ (req, rsp, h) -> {
+ Map<String, Object> map =
+ copy(req.getParams().required(), null, COLLECTION_PROP, SHARD_ID_PROP);
+ copy(
+ req.getParams(),
+ map,
+ DELETE_INDEX,
+ DELETE_DATA_DIR,
+ DELETE_INSTANCE_DIR,
+ FOLLOW_ALIASES);
+ return map;
+ }),
+ FORCELEADER_OP(
+ FORCELEADER,
+ (req, rsp, h) -> {
+ forceLeaderElection(req, h);
+ return null;
+ }),
+ CREATESHARD_OP(
+ CREATESHARD,
+ (req, rsp, h) -> {
+ Map<String, Object> map =
+ copy(req.getParams().required(), null, COLLECTION_PROP, SHARD_ID_PROP);
+ ClusterState clusterState = h.coreContainer.getZkController().getClusterState();
+ final String newShardName =
+ SolrIdentifierValidator.validateShardName(req.getParams().get(SHARD_ID_PROP));
+ boolean followAliases = req.getParams().getBool(FOLLOW_ALIASES, false);
+ String extCollectionName = req.getParams().get(COLLECTION_PROP);
+ String collectionName =
+ followAliases
+ ? h.coreContainer
+ .getZkController()
+ .getZkStateReader()
+ .getAliases()
+ .resolveSimpleAlias(extCollectionName)
+ : extCollectionName;
+ if (!ImplicitDocRouter.NAME.equals(
+ ((Map<?, ?>) clusterState.getCollection(collectionName).get(DOC_ROUTER)).get(NAME)))
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST, "shards can be added only to 'implicit' collections");
+ copy(
+ req.getParams(),
+ map,
+ REPLICATION_FACTOR,
+ NRT_REPLICAS,
+ TLOG_REPLICAS,
+ PULL_REPLICAS,
+ CREATE_NODE_SET,
+ WAIT_FOR_FINAL_STATE,
+ FOLLOW_ALIASES);
+ return copyPropertiesWithPrefix(req.getParams(), map, PROPERTY_PREFIX);
+ }),
+ DELETEREPLICA_OP(
+ DELETEREPLICA,
+ (req, rsp, h) -> {
+ Map<String, Object> map = copy(req.getParams().required(), null, COLLECTION_PROP);
+
+ return copy(
+ req.getParams(),
+ map,
+ DELETE_INDEX,
+ DELETE_DATA_DIR,
+ DELETE_INSTANCE_DIR,
+ COUNT_PROP,
+ REPLICA_PROP,
+ SHARD_ID_PROP,
+ ONLY_IF_DOWN,
+ FOLLOW_ALIASES);
+ }),
+ MIGRATE_OP(
+ MIGRATE,
+ (req, rsp, h) -> {
+ Map<String, Object> map =
+ copy(
+ req.getParams().required(),
+ null,
+ COLLECTION_PROP,
+ "split.key",
+ "target.collection");
+ return copy(req.getParams(), map, "forward.timeout", FOLLOW_ALIASES);
+ }),
+ ADDROLE_OP(
+ ADDROLE,
+ (req, rsp, h) -> {
+ Map<String, Object> map = copy(req.getParams().required(), null, "role", "node");
+ if (!KNOWN_ROLES.contains(map.get("role")))
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST, "Unknown role. Supported roles are ," + KNOWN_ROLES);
+ return map;
+ }),
+ REMOVEROLE_OP(
+ REMOVEROLE,
+ (req, rsp, h) -> {
+ Map<String, Object> map = copy(req.getParams().required(), null, "role", "node");
+ if (!KNOWN_ROLES.contains(map.get("role")))
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST, "Unknown role. Supported roles are ," + KNOWN_ROLES);
+ return map;
+ }),
+ CLUSTERPROP_OP(
+ CLUSTERPROP,
+ (req, rsp, h) -> {
+ String name = req.getParams().required().get(NAME);
+ String val = req.getParams().get(VALUE_LONG);
+ ClusterProperties cp =
+ new ClusterProperties(h.coreContainer.getZkController().getZkClient());
+ cp.setClusterProperty(name, val);
+ return null;
+ }),
+ COLLECTIONPROP_OP(
+ COLLECTIONPROP,
+ (req, rsp, h) -> {
+ String extCollection = req.getParams().required().get(NAME);
+ String collection =
+ h.coreContainer
+ .getZkController()
+ .getZkStateReader()
+ .getAliases()
+ .resolveSimpleAlias(extCollection);
+ String name = req.getParams().required().get(PROPERTY_NAME);
+ String val = req.getParams().get(PROPERTY_VALUE);
+ CollectionProperties cp =
+ new CollectionProperties(h.coreContainer.getZkController().getZkClient());
+ cp.setCollectionProperty(collection, name, val);
+ return null;
+ }),
@SuppressWarnings({"unchecked"})
- REQUESTSTATUS_OP(REQUESTSTATUS, (req, rsp, h) -> {
- req.getParams().required().check(REQUESTID);
-
- final CoreContainer coreContainer = h.coreContainer;
- final String requestId = req.getParams().get(REQUESTID);
- final ZkController zkController = coreContainer.getZkController();
-
- final NamedList<Object> status = new NamedList<>();
- if (coreContainer.getDistributedCollectionCommandRunner().isEmpty()) {
- if (zkController.getOverseerCompletedMap().contains(requestId)) {
- final byte[] mapEntry = zkController.getOverseerCompletedMap().get(requestId);
- rsp.getValues().addAll(OverseerSolrResponseSerializer.deserialize(mapEntry).getResponse());
- addStatusToResponse(status, COMPLETED, "found [" + requestId + "] in completed tasks");
- } else if (zkController.getOverseerFailureMap().contains(requestId)) {
- final byte[] mapEntry = zkController.getOverseerFailureMap().get(requestId);
- rsp.getValues().addAll(OverseerSolrResponseSerializer.deserialize(mapEntry).getResponse());
- addStatusToResponse(status, FAILED, "found [" + requestId + "] in failed tasks");
- } else if (zkController.getOverseerRunningMap().contains(requestId)) {
- addStatusToResponse(status, RUNNING, "found [" + requestId + "] in running tasks");
- } else if (h.overseerCollectionQueueContains(requestId)) {
- addStatusToResponse(status, SUBMITTED, "found [" + requestId + "] in submitted tasks");
- } else {
- addStatusToResponse(status, NOT_FOUND, "Did not find [" + requestId + "] in any tasks queue");
- }
- } else {
- Pair<RequestStatusState, OverseerSolrResponse> sr =
- coreContainer.getDistributedCollectionCommandRunner().get().getAsyncTaskRequestStatus(requestId);
- final String message;
- switch (sr.first()) {
- case COMPLETED:
- message = "found [" + requestId + "] in completed tasks";
- rsp.getValues().addAll(sr.second().getResponse());
- break;
- case FAILED:
- message = "found [" + requestId + "] in failed tasks";
- rsp.getValues().addAll(sr.second().getResponse());
- break;
- case RUNNING:
- message = "found [" + requestId + "] in running tasks";
- break;
- case SUBMITTED:
- message = "found [" + requestId + "] in submitted tasks";
- break;
- default:
- message = "Did not find [" + requestId + "] in any tasks queue";
- }
- addStatusToResponse(status, sr.first(), message);
- }
-
- rsp.getValues().addAll(status);
- return null;
- }),
- DELETESTATUS_OP(DELETESTATUS, new CollectionOp() {
- @SuppressWarnings("unchecked")
- @Override
- public Map<String, Object> execute(SolrQueryRequest req, SolrQueryResponse rsp, CollectionsHandler h) throws Exception {
- final CoreContainer coreContainer = h.coreContainer;
- final String requestId = req.getParams().get(REQUESTID);
- final ZkController zkController = coreContainer.getZkController();
- Boolean flush = req.getParams().getBool(CollectionAdminParams.FLUSH, false);
-
- if (requestId == null && !flush) {
- throw new SolrException(ErrorCode.BAD_REQUEST, "Either requestid or flush parameter must be specified.");
- }
-
- if (requestId != null && flush) {
- throw new SolrException(ErrorCode.BAD_REQUEST,
- "Both requestid and flush parameters can not be specified together.");
- }
-
- if (coreContainer.getDistributedCollectionCommandRunner().isEmpty()) {
- if (flush) {
- Collection<String> completed = zkController.getOverseerCompletedMap().keys();
- Collection<String> failed = zkController.getOverseerFailureMap().keys();
- for (String asyncId : completed) {
- zkController.getOverseerCompletedMap().remove(asyncId);
- zkController.clearAsyncId(asyncId);
- }
- for (String asyncId : failed) {
- zkController.getOverseerFailureMap().remove(asyncId);
- zkController.clearAsyncId(asyncId);
+ REQUESTSTATUS_OP(
+ REQUESTSTATUS,
+ (req, rsp, h) -> {
+ req.getParams().required().check(REQUESTID);
+
+ final CoreContainer coreContainer = h.coreContainer;
+ final String requestId = req.getParams().get(REQUESTID);
+ final ZkController zkController = coreContainer.getZkController();
+
+ final NamedList<Object> status = new NamedList<>();
+ if (coreContainer.getDistributedCollectionCommandRunner().isEmpty()) {
+ if (zkController.getOverseerCompletedMap().contains(requestId)) {
+ final byte[] mapEntry = zkController.getOverseerCompletedMap().get(requestId);
+ rsp.getValues()
+ .addAll(OverseerSolrResponseSerializer.deserialize(mapEntry).getResponse());
+ addStatusToResponse(
+ status, COMPLETED, "found [" + requestId + "] in completed tasks");
+ } else if (zkController.getOverseerFailureMap().contains(requestId)) {
+ final byte[] mapEntry = zkController.getOverseerFailureMap().get(requestId);
+ rsp.getValues()
+ .addAll(OverseerSolrResponseSerializer.deserialize(mapEntry).getResponse());
+ addStatusToResponse(status, FAILED, "found [" + requestId + "] in failed tasks");
+ } else if (zkController.getOverseerRunningMap().contains(requestId)) {
+ addStatusToResponse(status, RUNNING, "found [" + requestId + "] in running tasks");
+ } else if (h.overseerCollectionQueueContains(requestId)) {
+ addStatusToResponse(
+ status, SUBMITTED, "found [" + requestId + "] in submitted tasks");
+ } else {
+ addStatusToResponse(
+ status, NOT_FOUND, "Did not find [" + requestId + "] in any tasks queue");
}
- rsp.getValues().add("status", "successfully cleared stored collection api responses");
} else {
- // Request to cleanup
- if (zkController.getOverseerCompletedMap().remove(requestId)) {
- zkController.clearAsyncId(requestId);
- rsp.getValues().add("status", "successfully removed stored response for [" + requestId + "]");
- } else if (zkController.getOverseerFailureMap().remove(requestId)) {
- zkController.clearAsyncId(requestId);
- rsp.getValues().add("status", "successfully removed stored response for [" + requestId + "]");
- } else {
- rsp.getValues().add("status", "[" + requestId + "] not found in stored responses");
- // Don't call zkController.clearAsyncId for this, since it could be a running/pending task
+ Pair<RequestStatusState, OverseerSolrResponse> sr =
+ coreContainer
+ .getDistributedCollectionCommandRunner()
+ .get()
+ .getAsyncTaskRequestStatus(requestId);
+ final String message;
+ switch (sr.first()) {
+ case COMPLETED:
+ message = "found [" + requestId + "] in completed tasks";
+ rsp.getValues().addAll(sr.second().getResponse());
+ break;
+ case FAILED:
+ message = "found [" + requestId + "] in failed tasks";
+ rsp.getValues().addAll(sr.second().getResponse());
+ break;
+ case RUNNING:
+ message = "found [" + requestId + "] in running tasks";
+ break;
+ case SUBMITTED:
+ message = "found [" + requestId + "] in submitted tasks";
+ break;
+ default:
+ message = "Did not find [" + requestId + "] in any tasks queue";
}
+ addStatusToResponse(status, sr.first(), message);
}
- } else {
- if (flush) {
- coreContainer.getDistributedCollectionCommandRunner().get().deleteAllAsyncIds();
- rsp.getValues().add("status", "successfully cleared stored collection api responses");
- } else {
- if (coreContainer.getDistributedCollectionCommandRunner().get().deleteSingleAsyncId(requestId)) {
- rsp.getValues().add("status", "successfully removed stored response for [" + requestId + "]");
+
+ rsp.getValues().addAll(status);
+ return null;
+ }),
+ DELETESTATUS_OP(
+ DELETESTATUS,
+ new CollectionOp() {
+ @SuppressWarnings("unchecked")
+ @Override
+ public Map<String, Object> execute(
+ SolrQueryRequest req, SolrQueryResponse rsp, CollectionsHandler h) throws Exception {
+ final CoreContainer coreContainer = h.coreContainer;
+ final String requestId = req.getParams().get(REQUESTID);
+ final ZkController zkController = coreContainer.getZkController();
+ Boolean flush = req.getParams().getBool(CollectionAdminParams.FLUSH, false);
+
+ if (requestId == null && !flush) {
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST, "Either requestid or flush parameter must be specified.");
+ }
+
+ if (requestId != null && flush) {
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST,
+ "Both requestid and flush parameters can not be specified together.");
+ }
+
+ if (coreContainer.getDistributedCollectionCommandRunner().isEmpty()) {
+ if (flush) {
+ Collection<String> completed = zkController.getOverseerCompletedMap().keys();
+ Collection<String> failed = zkController.getOverseerFailureMap().keys();
+ for (String asyncId : completed) {
+ zkController.getOverseerCompletedMap().remove(asyncId);
+ zkController.clearAsyncId(asyncId);
+ }
+ for (String asyncId : failed) {
+ zkController.getOverseerFailureMap().remove(asyncId);
+ zkController.clearAsyncId(asyncId);
+ }
+ rsp.getValues()
+ .add("status", "successfully cleared stored collection api responses");
+ } else {
+ // Request to cleanup
+ if (zkController.getOverseerCompletedMap().remove(requestId)) {
+ zkController.clearAsyncId(requestId);
+ rsp.getValues()
+ .add(
+ "status", "successfully removed stored response for [" + requestId + "]");
+ } else if (zkController.getOverseerFailureMap().remove(requestId)) {
+ zkController.clearAsyncId(requestId);
+ rsp.getValues()
+ .add(
+ "status", "successfully removed stored response for [" + requestId + "]");
+ } else {
+ rsp.getValues()
+ .add("status", "[" + requestId + "] not found in stored responses");
+ // Don't call zkController.clearAsyncId for this, since it could be a
+ // running/pending task
+ }
+ }
} else {
- rsp.getValues().add("status", "[" + requestId + "] not found in stored responses");
+ if (flush) {
+ coreContainer.getDistributedCollectionCommandRunner().get().deleteAllAsyncIds();
+ rsp.getValues()
+ .add("status", "successfully cleared stored collection api responses");
+ } else {
+ if (coreContainer
+ .getDistributedCollectionCommandRunner()
+ .get()
+ .deleteSingleAsyncId(requestId)) {
+ rsp.getValues()
+ .add(
+ "status", "successfully removed stored response for [" + requestId + "]");
+ } else {
+ rsp.getValues()
+ .add("status", "[" + requestId + "] not found in stored responses");
+ }
+ }
}
+ return null;
}
- }
- return null;
- }
- }),
- ADDREPLICA_OP(ADDREPLICA, (req, rsp, h) -> {
- Map<String, Object> props = copy(req.getParams(), null,
- COLLECTION_PROP,
- "node",
- SHARD_ID_PROP,
- _ROUTE_,
- CoreAdminParams.NAME,
- INSTANCE_DIR,
- DATA_DIR,
- ULOG_DIR,
- REPLICA_TYPE,
- WAIT_FOR_FINAL_STATE,
- NRT_REPLICAS,
- TLOG_REPLICAS,
- PULL_REPLICAS,
- CREATE_NODE_SET,
- FOLLOW_ALIASES,
- SKIP_NODE_ASSIGNMENT);
- return copyPropertiesWithPrefix(req.getParams(), props, PROPERTY_PREFIX);
- }),
+ }),
+ ADDREPLICA_OP(
+ ADDREPLICA,
+ (req, rsp, h) -> {
+ Map<String, Object> props =
+ copy(
+ req.getParams(),
+ null,
+ COLLECTION_PROP,
+ "node",
+ SHARD_ID_PROP,
+ _ROUTE_,
+ CoreAdminParams.NAME,
+ INSTANCE_DIR,
+ DATA_DIR,
+ ULOG_DIR,
+ REPLICA_TYPE,
+ WAIT_FOR_FINAL_STATE,
+ NRT_REPLICAS,
+ TLOG_REPLICAS,
+ PULL_REPLICAS,
+ CREATE_NODE_SET,
+ FOLLOW_ALIASES,
+ SKIP_NODE_ASSIGNMENT);
+ return copyPropertiesWithPrefix(req.getParams(), props, PROPERTY_PREFIX);
+ }),
OVERSEERSTATUS_OP(OVERSEERSTATUS, (req, rsp, h) -> new LinkedHashMap<>()),
@SuppressWarnings({"unchecked"})
- DISTRIBUTEDAPIPROCESSING_OP(DISTRIBUTEDAPIPROCESSING, (req, rsp, h) -> {
- NamedList<Object> results = new NamedList<>();
- boolean isDistributedApi = h.coreContainer.getDistributedCollectionCommandRunner().isPresent();
- results.add("isDistributedApi", isDistributedApi);
- rsp.getValues().addAll(results);
- return null;
- }),
- /**
- * Handle list collection request.
- * Do list collection request to zk host
- */
+ DISTRIBUTEDAPIPROCESSING_OP(
+ DISTRIBUTEDAPIPROCESSING,
+ (req, rsp, h) -> {
+ NamedList<Object> results = new NamedList<>();
+ boolean isDistributedApi =
+ h.coreContainer.getDistributedCollectionCommandRunner().isPresent();
+ results.add("isDistributedApi", isDistributedApi);
+ rsp.getValues().addAll(results);
+ return null;
+ }),
+ /** Handle list collection request. Do list collection request to zk host */
@SuppressWarnings({"unchecked"})
- LIST_OP(LIST, (req, rsp, h) -> {
- NamedList<Object> results = new NamedList<>();
- Map<String, DocCollection> collections = h.coreContainer.getZkController().getZkStateReader().getClusterState().getCollectionsMap();
- List<String> collectionList = new ArrayList<>(collections.keySet());
- Collections.sort(collectionList);
- // XXX should we add aliases here?
- results.add("collections", collectionList);
- SolrResponse response = new OverseerSolrResponse(results);
- rsp.getValues().addAll(response.getResponse());
- return null;
- }),
+ LIST_OP(
+ LIST,
+ (req, rsp, h) -> {
+ NamedList<Object> results = new NamedList<>();
+ Map<String, DocCollection> collections =
+ h.coreContainer
+ .getZkController()
+ .getZkStateReader()
+ .getClusterState()
+ .getCollectionsMap();
+ List<String> collectionList = new ArrayList<>(collections.keySet());
+ Collections.sort(collectionList);
+ // XXX should we add aliases here?
+ results.add("collections", collectionList);
+ SolrResponse response = new OverseerSolrResponse(results);
+ rsp.getValues().addAll(response.getResponse());
+ return null;
+ }),
/**
- * Handle cluster status request.
- * Can return status per specific collection/shard or per all collections.
+ * Handle cluster status request. Can return status per specific collection/shard or per all
+ * collections.
*/
- CLUSTERSTATUS_OP(CLUSTERSTATUS, (req, rsp, h) -> {
- Map<String, Object> all = copy(req.getParams(), null,
- COLLECTION_PROP,
- SHARD_ID_PROP,
- _ROUTE_);
- new ClusterStatus(h.coreContainer.getZkController().getZkStateReader(),
- new ZkNodeProps(all)).getClusterStatus(rsp.getValues());
- return null;
- }),
- ADDREPLICAPROP_OP(ADDREPLICAPROP, (req, rsp, h) -> {
- Map<String, Object> map = copy(req.getParams().required(), null,
- COLLECTION_PROP,
- PROPERTY_PROP,
- SHARD_ID_PROP,
- REPLICA_PROP,
- PROPERTY_VALUE_PROP);
- copy(req.getParams(), map, SHARD_UNIQUE);
- String property = (String) map.get(PROPERTY_PROP);
- if (!property.startsWith(PROPERTY_PREFIX)) {
- property = PROPERTY_PREFIX + property;
- }
+ CLUSTERSTATUS_OP(
+ CLUSTERSTATUS,
+ (req, rsp, h) -> {
+ Map<String, Object> all =
+ copy(req.getParams(), null, COLLECTION_PROP, SHARD_ID_PROP, _ROUTE_);
+ new ClusterStatus(
+ h.coreContainer.getZkController().getZkStateReader(), new ZkNodeProps(all))
+ .getClusterStatus(rsp.getValues());
+ return null;
+ }),
+ ADDREPLICAPROP_OP(
+ ADDREPLICAPROP,
+ (req, rsp, h) -> {
+ Map<String, Object> map =
+ copy(
+ req.getParams().required(),
+ null,
+ COLLECTION_PROP,
+ PROPERTY_PROP,
+ SHARD_ID_PROP,
+ REPLICA_PROP,
+ PROPERTY_VALUE_PROP);
+ copy(req.getParams(), map, SHARD_UNIQUE);
+ String property = (String) map.get(PROPERTY_PROP);
+ if (!property.startsWith(PROPERTY_PREFIX)) {
+ property = PROPERTY_PREFIX + property;
+ }
- boolean uniquePerSlice = Boolean.parseBoolean((String) map.get(SHARD_UNIQUE));
-
- // Check if we're trying to set a property with parameters that allow us to set the property on multiple replicas
- // in a slice on properties that are known to only be one-per-slice and error out if so.
- if (StringUtils.isNotBlank((String) map.get(SHARD_UNIQUE)) &&
- SliceMutator.SLICE_UNIQUE_BOOLEAN_PROPERTIES.contains(property.toLowerCase(Locale.ROOT)) &&
- uniquePerSlice == false) {
- throw new SolrException(ErrorCode.BAD_REQUEST,
- "Overseer replica property command received for property " + property +
- " with the " + SHARD_UNIQUE +
- " parameter set to something other than 'true'. No action taken.");
- }
- return map;
- }),
+ boolean uniquePerSlice = Boolean.parseBoolean((String) map.get(SHARD_UNIQUE));
+
+ // Check if we're trying to set a property with parameters that allow us to set the
+ // property on multiple replicas
+ // in a slice on properties that are known to only be one-per-slice and error out if so.
+ if (StringUtils.isNotBlank((String) map.get(SHARD_UNIQUE))
+ && SliceMutator.SLICE_UNIQUE_BOOLEAN_PROPERTIES.contains(
+ property.toLowerCase(Locale.ROOT))
+ && uniquePerSlice == false) {
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST,
+ "Overseer replica property command received for property "
+ + property
+ + " with the "
+ + SHARD_UNIQUE
+ + " parameter set to something other than 'true'. No action taken.");
+ }
+ return map;
+ }),
// XXX should this command support followAliases?
- DELETEREPLICAPROP_OP(DELETEREPLICAPROP, (req, rsp, h) -> {
- Map<String, Object> map = copy(req.getParams().required(), null,
- COLLECTION_PROP,
- PROPERTY_PROP,
- SHARD_ID_PROP,
- REPLICA_PROP);
- return copy(req.getParams(), map, PROPERTY_PROP);
- }),
+ DELETEREPLICAPROP_OP(
+ DELETEREPLICAPROP,
+ (req, rsp, h) -> {
+ Map<String, Object> map =
+ copy(
+ req.getParams().required(),
+ null,
+ COLLECTION_PROP,
+ PROPERTY_PROP,
+ SHARD_ID_PROP,
+ REPLICA_PROP);
+ return copy(req.getParams(), map, PROPERTY_PROP);
+ }),
// XXX should this command support followAliases?
- BALANCESHARDUNIQUE_OP(BALANCESHARDUNIQUE, (req, rsp, h) -> {
- Map<String, Object> map = copy(req.getParams().required(), null,
- COLLECTION_PROP,
- PROPERTY_PROP);
- Boolean shardUnique = Boolean.parseBoolean(req.getParams().get(SHARD_UNIQUE));
- String prop = req.getParams().get(PROPERTY_PROP).toLowerCase(Locale.ROOT);
- if (!StringUtils.startsWith(prop, PROPERTY_PREFIX)) {
- prop = PROPERTY_PREFIX + prop;
- }
+ BALANCESHARDUNIQUE_OP(
+ BALANCESHARDUNIQUE,
+ (req, rsp, h) -> {
+ Map<String, Object> map =
+ copy(req.getParams().required(), null, COLLECTION_PROP, PROPERTY_PROP);
+ Boolean shardUnique = Boolean.parseBoolean(req.getParams().get(SHARD_UNIQUE));
+ String prop = req.getParams().get(PROPERTY_PROP).toLowerCase(Locale.ROOT);
+ if (!StringUtils.startsWith(prop, PROPERTY_PREFIX)) {
+ prop = PROPERTY_PREFIX + prop;
+ }
- if (!shardUnique && !SliceMutator.SLICE_UNIQUE_BOOLEAN_PROPERTIES.contains(prop)) {
- throw new SolrException(ErrorCode.BAD_REQUEST, "Balancing properties amongst replicas in a slice requires that"
- + " the property be pre-defined as a unique property (e.g. 'preferredLeader') or that 'shardUnique' be set to 'true'. " +
- " Property: " + prop + " shardUnique: " + shardUnique);
- }
+ if (!shardUnique && !SliceMutator.SLICE_UNIQUE_BOOLEAN_PROPERTIES.contains(prop)) {
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST,
+ "Balancing properties amongst replicas in a slice requires that"
+ + " the property be pre-defined as a unique property (e.g. 'preferredLeader') or that 'shardUnique' be set to 'true'. "
+ + " Property: "
+ + prop
+ + " shardUnique: "
+ + shardUnique);
+ }
- return copy(req.getParams(), map, ONLY_ACTIVE_NODES, SHARD_UNIQUE);
- }),
- REBALANCELEADERS_OP(REBALANCELEADERS, (req, rsp, h) -> {
- new RebalanceLeaders(req, rsp, h).execute();
- return null;
- }),
+ return copy(req.getParams(), map, ONLY_ACTIVE_NODES, SHARD_UNIQUE);
+ }),
+ REBALANCELEADERS_OP(
+ REBALANCELEADERS,
+ (req, rsp, h) -> {
+ new RebalanceLeaders(req, rsp, h).execute();
+ return null;
+ }),
// XXX should this command support followAliases?
- MODIFYCOLLECTION_OP(MODIFYCOLLECTION, (req, rsp, h) -> {
- Map<String, Object> m = copy(req.getParams(), null, CollectionAdminRequest.MODIFIABLE_COLLECTION_PROPERTIES);
- copyPropertiesWithPrefix(req.getParams(), m, PROPERTY_PREFIX);
- if (m.isEmpty()) {
- throw new SolrException(ErrorCode.BAD_REQUEST,
- formatString("no supported values provided {0}", CollectionAdminRequest.MODIFIABLE_COLLECTION_PROPERTIES.toString()));
- }
- copy(req.getParams().required(), m, COLLECTION_PROP);
- for (Map.Entry<String, Object> entry : m.entrySet()) {
- String prop = entry.getKey();
- if ("".equals(entry.getValue())) {
- // set to an empty string is equivalent to removing the property, see SOLR-12507
- entry.setValue(null);
- }
- DocCollection.verifyProp(m, prop);
- }
- if (m.get(REPLICATION_FACTOR) != null) {
- m.put(NRT_REPLICAS, m.get(REPLICATION_FACTOR));
- }
- return m;
- }),
- BACKUP_OP(BACKUP, (req, rsp, h) -> {
- req.getParams().required().check(NAME, COLLECTION_PROP);
-
- final String extCollectionName = req.getParams().get(COLLECTION_PROP);
- final boolean followAliases = req.getParams().getBool(FOLLOW_ALIASES, false);
- final String collectionName = followAliases ? h.coreContainer.getZkController().getZkStateReader()
- .getAliases().resolveSimpleAlias(extCollectionName) : extCollectionName;
- final ClusterState clusterState = h.coreContainer.getZkController().getClusterState();
- if (!clusterState.hasCollection(collectionName)) {
- throw new SolrException(ErrorCode.BAD_REQUEST, "Collection '" + collectionName + "' does not exist, no action taken.");
- }
-
- CoreContainer cc = h.coreContainer;
- String repo = req.getParams().get(CoreAdminParams.BACKUP_REPOSITORY);
- BackupRepository repository = cc.newBackupRepository(repo);
-
- String location = repository.getBackupLocation(req.getParams().get(CoreAdminParams.BACKUP_LOCATION));
- if (location == null) {
- //Refresh the cluster property file to make sure the value set for location is the latest
- // Check if the location is specified in the cluster property.
- location = new ClusterProperties(h.coreContainer.getZkController().getZkClient()).getClusterProperty(CoreAdminParams.BACKUP_LOCATION, null);
- if (location == null) {
- throw new SolrException(ErrorCode.BAD_REQUEST, "'location' is not specified as a query"
- + " parameter or as a default repository property or as a cluster property.");
- }
- }
- boolean incremental = req.getParams().getBool(CoreAdminParams.BACKUP_INCREMENTAL, true);
-
- // Check if the specified location is valid for this repository.
- final URI uri = repository.createDirectoryURI(location);
- try {
- if (!repository.exists(uri)) {
- throw new SolrException(ErrorCode.SERVER_ERROR, "specified location " + uri + " does not exist.");
- }
- } catch (IOException ex) {
- throw new SolrException(ErrorCode.SERVER_ERROR, "Failed to check the existence of " + uri + ". Is it valid?", ex);
- }
+ MODIFYCOLLECTION_OP(
+ MODIFYCOLLECTION,
+ (req, rsp, h) -> {
+ Map<String, Object> m =
+ copy(req.getParams(), null, CollectionAdminRequest.MODIFIABLE_COLLECTION_PROPERTIES);
+ copyPropertiesWithPrefix(req.getParams(), m, PROPERTY_PREFIX);
+ if (m.isEmpty()) {
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST,
+ formatString(
+ "no supported values provided {0}",
+ CollectionAdminRequest.MODIFIABLE_COLLECTION_PROPERTIES.toString()));
+ }
+ copy(req.getParams().required(), m, COLLECTION_PROP);
+ for (Map.Entry<String, Object> entry : m.entrySet()) {
+ String prop = entry.getKey();
+ if ("".equals(entry.getValue())) {
+ // set to an empty string is equivalent to removing the property, see SOLR-12507
+ entry.setValue(null);
+ }
+ DocCollection.verifyProp(m, prop);
+ }
+ if (m.get(REPLICATION_FACTOR) != null) {
+ m.put(NRT_REPLICAS, m.get(REPLICATION_FACTOR));
+ }
+ return m;
+ }),
+ BACKUP_OP(
+ BACKUP,
+ (req, rsp, h) -> {
+ req.getParams().required().check(NAME, COLLECTION_PROP);
+
+ final String extCollectionName = req.getParams().get(COLLECTION_PROP);
+ final boolean followAliases = req.getParams().getBool(FOLLOW_ALIASES, false);
+ final String collectionName =
+ followAliases
+ ? h.coreContainer
+ .getZkController()
+ .getZkStateReader()
+ .getAliases()
+ .resolveSimpleAlias(extCollectionName)
+ : extCollectionName;
+ final ClusterState clusterState = h.coreContainer.getZkController().getClusterState();
+ if (!clusterState.hasCollection(collectionName)) {
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST,
+ "Collection '" + collectionName + "' does not exist, no action taken.");
+ }
- String strategy = req.getParams().get(CollectionAdminParams.INDEX_BACKUP_STRATEGY, CollectionAdminParams.COPY_FILES_STRATEGY);
- if (!CollectionAdminParams.INDEX_BACKUP_STRATEGIES.contains(strategy)) {
- throw new SolrException(ErrorCode.BAD_REQUEST, "Unknown index backup strategy " + strategy);
- }
+ CoreContainer cc = h.coreContainer;
+ String repo = req.getParams().get(CoreAdminParams.BACKUP_REPOSITORY);
+ BackupRepository repository = cc.newBackupRepository(repo);
- Map<String, Object> params = copy(req.getParams(), null, NAME, COLLECTION_PROP,
- FOLLOW_ALIASES, CoreAdminParams.COMMIT_NAME, CoreAdminParams.MAX_NUM_BACKUP_POINTS);
- params.put(CoreAdminParams.BACKUP_LOCATION, location);
- if (repo != null) {
- params.put(CoreAdminParams.BACKUP_REPOSITORY, repo);
- }
+ String location =
+ repository.getBackupLocation(req.getParams().get(CoreAdminParams.BACKUP_LOCATION));
+ if (location == null) {
+ // Refresh the cluster property file to make sure the value set for location is the
+ // latest
+ // Check if the location is specified in the cluster property.
+ location =
+ new ClusterProperties(h.coreContainer.getZkController().getZkClient())
+ .getClusterProperty(CoreAdminParams.BACKUP_LOCATION, null);
+ if (location == null) {
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST,
+ "'location' is not specified as a query"
+ + " parameter or as a default repository property or as a cluster property.");
+ }
+ }
+ boolean incremental = req.getParams().getBool(CoreAdminParams.BACKUP_INCREMENTAL, true);
- params.put(CollectionAdminParams.INDEX_BACKUP_STRATEGY, strategy);
- params.put(CoreAdminParams.BACKUP_INCREMENTAL, incremental);
- return params;
- }),
- RESTORE_OP(RESTORE, (req, rsp, h) -> {
- req.getParams().required().check(NAME, COLLECTION_PROP);
+ // Check if the specified location is valid for this repository.
+ final URI uri = repository.createDirectoryURI(location);
+ try {
+ if (!repository.exists(uri)) {
+ throw new SolrException(
+ ErrorCode.SERVER_ERROR, "specified location " + uri + " does not exist.");
+ }
+ } catch (IOException ex) {
+ throw new SolrException(
+ ErrorCode.SERVER_ERROR,
+ "Failed to check the existence of " + uri + ". Is it valid?",
+ ex);
+ }
- final String collectionName = SolrIdentifierValidator.validateCollectionName(req.getParams().get(COLLECTION_PROP));
- if (h.coreContainer.getZkController().getZkStateReader().getAliases().hasAlias(collectionName)) {
- throw new SolrException(ErrorCode.BAD_REQUEST, "Collection '" + collectionName + "' is an existing alias, no action taken.");
- }
+ String strategy =
+ req.getParams()
+ .get(
+ CollectionAdminParams.INDEX_BACKUP_STRATEGY,
+ CollectionAdminParams.COPY_FILES_STRATEGY);
+ if (!CollectionAdminParams.INDEX_BACKUP_STRATEGIES.contains(strategy)) {
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST, "Unknown index backup strategy " + strategy);
+ }
- final CoreContainer cc = h.coreContainer;
- final String repo = req.getParams().get(CoreAdminParams.BACKUP_REPOSITORY);
- final BackupRepository repository = cc.newBackupRepository(repo);
-
- String location = repository.getBackupLocation(req.getParams().get(CoreAdminParams.BACKUP_LOCATION));
- if (location == null) {
- //Refresh the cluster property file to make sure the value set for location is the latest
- // Check if the location is specified in the cluster property.
- location = new ClusterProperties(h.coreContainer.getZkController().getZkClient()).getClusterProperty("location", null);
- if (location == null) {
- throw new SolrException(ErrorCode.BAD_REQUEST, "'location' is not specified as a query"
- + " parameter or as a default repository property or as a cluster property.");
- }
- }
+ Map<String, Object> params =
+ copy(
+ req.getParams(),
+ null,
+ NAME,
+ COLLECTION_PROP,
+ FOLLOW_ALIASES,
+ CoreAdminParams.COMMIT_NAME,
+ CoreAdminParams.MAX_NUM_BACKUP_POINTS);
+ params.put(CoreAdminParams.BACKUP_LOCATION, location);
+ if (repo != null) {
+ params.put(CoreAdminParams.BACKUP_REPOSITORY, repo);
+ }
- // Check if the specified location is valid for this repository.
- final URI uri = repository.createDirectoryURI(location);
- try {
- if (!repository.exists(uri)) {
- throw new SolrException(ErrorCode.SERVER_ERROR, "specified location " + uri + " does not exist.");
- }
- } catch (IOException ex) {
- throw new SolrException(ErrorCode.SERVER_ERROR, "Failed to check the existence of " + uri + ". Is it valid?", ex);
- }
+ params.put(CollectionAdminParams.INDEX_BACKUP_STRATEGY, strategy);
+ params.put(CoreAdminParams.BACKUP_INCREMENTAL, incremental);
+ return params;
+ }),
+ RESTORE_OP(
+ RESTORE,
+ (req, rsp, h) -> {
+ req.getParams().required().check(NAME, COLLECTION_PROP);
+
+ final String collectionName =
+ SolrIdentifierValidator.validateCollectionName(req.getParams().get(COLLECTION_PROP));
+ if (h.coreContainer
+ .getZkController()
+ .getZkStateReader()
+ .getAliases()
+ .hasAlias(collectionName)) {
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST,
+ "Collection '" + collectionName + "' is an existing alias, no action taken.");
+ }
- final String createNodeArg = req.getParams().get(CREATE_NODE_SET);
- if (CREATE_NODE_SET_EMPTY.equals(createNodeArg)) {
- throw new SolrException(
- SolrException.ErrorCode.BAD_REQUEST,
- "Cannot restore with a CREATE_NODE_SET of CREATE_NODE_SET_EMPTY."
- );
- }
- if (req.getParams().get(NRT_REPLICAS) != null && req.getParams().get(REPLICATION_FACTOR) != null) {
- throw new SolrException(SolrException.ErrorCode.BAD_REQUEST,
- "Cannot set both replicationFactor and nrtReplicas as they mean the same thing");
- }
+ final CoreContainer cc = h.coreContainer;
+ final String repo = req.getParams().get(CoreAdminParams.BACKUP_REPOSITORY);
+ final BackupRepository repository = cc.newBackupRepository(repo);
- final Map<String, Object> params = copy(req.getParams(), null, NAME, COLLECTION_PROP);
- params.put(CoreAdminParams.BACKUP_LOCATION, location);
- if (repo != null) {
- params.put(CoreAdminParams.BACKUP_REPOSITORY, repo);
- }
- // from CREATE_OP:
- copy(req.getParams(), params, COLL_CONF, REPLICATION_FACTOR, NRT_REPLICAS, TLOG_REPLICAS,
- PULL_REPLICAS, CREATE_NODE_SET, CREATE_NODE_SET_SHUFFLE, BACKUP_ID);
- copyPropertiesWithPrefix(req.getParams(), params, PROPERTY_PREFIX);
- return params;
- }),
- DELETEBACKUP_OP(DELETEBACKUP, (req, rsp, h) -> {
- req.getParams().required().check(NAME);
-
- CoreContainer cc = h.coreContainer;
- String repo = req.getParams().get(CoreAdminParams.BACKUP_REPOSITORY);
- try (BackupRepository repository = cc.newBackupRepository(repo)) {
-
- String location = repository.getBackupLocation(req.getParams().get(CoreAdminParams.BACKUP_LOCATION));
- if (location == null) {
- //Refresh the cluster property file to make sure the value set for location is the latest
- // Check if the location is specified in the cluster property.
- location = new ClusterProperties(h.coreContainer.getZkController().getZkClient()).getClusterProperty("location", null);
+ String location =
+ repository.getBackupLocation(req.getParams().get(CoreAdminParams.BACKUP_LOCATION));
if (location == null) {
- throw new SolrException(ErrorCode.BAD_REQUEST, "'location' is not specified as a query"
- + " parameter or as a default repository property or as a cluster property.");
+ // Refresh the cluster property file to make sure the value set for location is the
+ // latest
+ // Check if the location is specified in the cluster property.
+ location =
+ new ClusterProperties(h.coreContainer.getZkController().getZkClient())
+ .getClusterProperty("location", null);
+ if (location == null) {
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST,
+ "'location' is not specified as a query"
+ + " parameter or as a default repository property or as a cluster property.");
+ }
}
- }
- // Check if the specified location is valid for this repository.
- URI uri = repository.createDirectoryURI(location);
- try {
- if (!repository.exists(uri)) {
- throw new SolrException(ErrorCode.BAD_REQUEST, "specified location " + uri + " does not exist.");
+ // Check if the specified location is valid for this repository.
+ final URI uri = repository.createDirectoryURI(location);
+ try {
+ if (!repository.exists(uri)) {
+ throw new SolrException(
+ ErrorCode.SERVER_ERROR, "specified location " + uri + " does not exist.");
+ }
+ } catch (IOException ex) {
+ throw new SolrException(
+ ErrorCode.SERVER_ERROR,
+ "Failed to check the existence of " + uri + ". Is it valid?",
+ ex);
}
- } catch (IOException ex) {
- throw new SolrException(ErrorCode.SERVER_ERROR, "Failed to check the existence of " + uri + ". Is it valid?", ex);
- }
- int deletionModesProvided = 0;
- if (req.getParams().get(MAX_NUM_BACKUP_POINTS) != null) deletionModesProvided++;
- if (req.getParams().get(BACKUP_PURGE_UNUSED) != null) deletionModesProvided++;
- if (req.getParams().get(BACKUP_ID) != null) deletionModesProvided++;
- if (deletionModesProvided != 1) {
- throw new SolrException(BAD_REQUEST,
- String.format(Locale.ROOT, "Exactly one of %s, %s, and %s parameters must be provided",
- MAX_NUM_BACKUP_POINTS, BACKUP_PURGE_UNUSED, BACKUP_ID));
- }
-
- final Map<String, Object> params = copy(req.getParams(), null, NAME, BACKUP_REPOSITORY,
- BACKUP_LOCATION, BACKUP_ID, MAX_NUM_BACKUP_POINTS, BACKUP_PURGE_UNUSED);
- params.put(BACKUP_LOCATION, location);
- if (repo != null) {
- params.put(CoreAdminParams.BACKUP_REPOSITORY, repo);
- }
- return params;
- }
- }),
- LISTBACKUP_OP(LISTBACKUP, (req, rsp, h) -> {
- req.getParams().required().check(NAME);
-
- CoreContainer cc = h.coreContainer;
- String repo = req.getParams().get(CoreAdminParams.BACKUP_REPOSITORY);
- try (BackupRepository repository = cc.newBackupRepository(repo)) {
-
- String location = repository.getBackupLocation(req.getParams().get(CoreAdminParams.BACKUP_LOCATION));
- if (location == null) {
- //Refresh the cluster property file to make sure the value set for location is the latest
- // Check if the location is specified in the cluster property.
- location = new ClusterProperties(h.coreContainer.getZkController().getZkClient()).getClusterProperty(CoreAdminParams.BACKUP_LOCATION, null);
- if (location == null) {
- throw new SolrException(ErrorCode.BAD_REQUEST, "'location' is not specified as a query"
- + " parameter or as a default repository property or as a cluster property.");
+ final String createNodeArg = req.getParams().get(CREATE_NODE_SET);
+ if (CREATE_NODE_SET_EMPTY.equals(createNodeArg)) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Cannot restore with a CREATE_NODE_SET of CREATE_NODE_SET_EMPTY.");
+ }
+ if (req.getParams().get(NRT_REPLICAS) != null
+ && req.getParams().get(REPLICATION_FACTOR) != null) {
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Cannot set both replicationFactor and nrtReplicas as they mean the same thing");
}
- }
- String backupName = req.getParams().get(NAME);
- final URI locationURI = repository.createDirectoryURI(location);
- try {
- if (!repository.exists(locationURI)) {
- throw new SolrException(ErrorCode.BAD_REQUEST, "specified location " + locationURI + " does not exist.");
+ final Map<String, Object> params = copy(req.getParams(), null, NAME, COLLECTION_PROP);
+ params.put(CoreAdminParams.BACKUP_LOCATION, location);
+ if (repo != null) {
+ params.put(CoreAdminParams.BACKUP_REPOSITORY, repo);
}
- } catch (IOException ex) {
- throw new SolrException(ErrorCode.SERVER_ERROR, "Failed to check the existence of " + locationURI + ". Is it valid?", ex);
- }
- URI backupLocation = BackupFilePaths.buildExistingBackupLocationURI(repository, locationURI, backupName);
- if (repository.exists(repository.resolve(backupLocation, BackupManager.TRADITIONAL_BACKUP_PROPS_FILE))) {
- throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "The backup name [" + backupName + "] at " +
- "location [" + location + "] holds a non-incremental (legacy) backup, but " +
- "backup-listing is only supported on incremental backups");
- }
+ // from CREATE_OP:
+ copy(
+ req.getParams(),
+ params,
+ COLL_CONF,
+ REPLICATION_FACTOR,
+ NRT_REPLICAS,
+ TLOG_REPLICAS,
+ PULL_REPLICAS,
+ CREATE_NODE_SET,
+ CREATE_NODE_SET_SHUFFLE,
+ BACKUP_ID);
+ copyPropertiesWithPrefix(req.getParams(), params, PROPERTY_PREFIX);
+ return params;
+ }),
+ DELETEBACKUP_OP(
+ DELETEBACKUP,
+ (req, rsp, h) -> {
+ req.getParams().required().check(NAME);
+
+ CoreContainer cc = h.coreContainer;
+ String repo = req.getParams().get(CoreAdminParams.BACKUP_REPOSITORY);
+ try (BackupRepository repository = cc.newBackupRepository(repo)) {
+
+ String location =
+ repository.getBackupLocation(req.getParams().get(CoreAdminParams.BACKUP_LOCATION));
+ if (location == null) {
+ // Refresh the cluster property file to make sure the value set for location is the
+ // latest
+ // Check if the location is specified in the cluster property.
+ location =
+ new ClusterProperties(h.coreContainer.getZkController().getZkClient())
+ .getClusterProperty("location", null);
+ if (location == null) {
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST,
+ "'location' is not specified as a query"
+ + " parameter or as a default repository property or as a cluster property.");
+ }
+ }
+
+ // Check if the specified location is valid for this repository.
+ URI uri = repository.createDirectoryURI(location);
+ try {
+ if (!repository.exists(uri)) {
+ throw new SolrException(
+ ErrorCode.BAD_REQUEST, "specified location " + uri + " does not exist.");
+ }
+ } catch (IOException ex) {
+ throw new SolrException(
+ ErrorCode.SERVER_ERROR,
+ "Failed to check the existence of " + uri + ". Is it valid?",
+ ex);
+ }
+
+ int deletionModesProvided = 0;
+ if (req.getParams().get(MAX_NUM_BACKUP_POINTS) != null) deletionModesProvided++;
+ if (req.getParams().get(BACKUP_PURGE_UNUSED) != null) deletionModesProvided++;
+ if (req.getParams().get(BACKUP_ID) != null) deletionModesProvided++;
+ if (deletionModesProvided != 1) {
+ throw new SolrException(
+ BAD_REQUEST,
+ String.format(
+ Locale.ROOT,
+ "Exactly one of %s, %s, and %s parameters must be provided",
+ MAX_NUM_BACKUP_POINTS,
+ BACKUP_PURGE_UNUSED,
+ BACKUP_ID));
+ }
- String[] subFiles = repository.listAllOrEmpty(backupLocation);
- List<BackupId> propsFiles = BackupFilePaths.findAllBackupIdsFromFileListing(subFiles);
-
- NamedList<Object> results = new NamedList<>();
- ArrayList<Map<Object,Object>> backups = new ArrayList<>();
- String collectionName = null;
- for (BackupId backupId: propsFiles) {
- BackupProperties properties = BackupProperties.readFrom(repository, backupLocation, BackupFilePaths.getBackupPropsName(backupId));
- if (collectionName == null) {
- collectionName = properties.getCollection();
- results.add(BackupManager.COLLECTION_NAME_PROP, collectionName);
+ final Map<String, Object> params =
+ copy(
+ req.getParams(),
+ null,
+ NAME,
+ BACKUP_REPOSITORY,
+ BACKUP_LOCATION,
+ BACKUP_ID,
+ MAX_NUM_BACKUP_POINTS,
+ BACKUP_PURGE_UNUSED);
+ params.put(BACKUP_LOCATION, location);
+ if (repo != null) {
+ params.put(CoreAdminParams.BACKUP_REPOSITORY, repo);
+ }
+ return params;
}
+ }),
+ LISTBACKUP_OP(
+ LISTBACKUP,
+ (req, rsp, h) -> {
+ req.getParams().required().check(NAME);
+
+ CoreContainer cc = h.coreContainer;
+ String repo = req.getParams().get(CoreAdminParams.BACKUP_REPOSITORY);
+ try (BackupRepository repository = cc.newBackupRepository(repo)) {
+
+ String location =
+ repository.getBackupLocation(req.getParams().get(CoreAdminParams.BACKUP_LOCATION));
+ if (location == null) {
+ // Refresh the cluster property file to make sure the value set for location is the
+ // latest
+ // Check if the location is specified in the cluster property.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/handler/loader/XMLLoader.java
##########
@@ -171,28 +182,34 @@ protected void processUpdate(SolrQueryRequest req, UpdateRequestProcessor proces
}
} else if ("doc".equals(currTag)) {
- if(addCmd != null) {
+ if (addCmd != null) {
log.trace("adding doc...");
addCmd.clear();
addCmd.solrDoc = readDoc(parser);
processor.processAdd(addCmd);
} else {
- throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "Unexpected <doc> tag without an <add> tag surrounding it.");
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
+ "Unexpected <doc> tag without an <add> tag surrounding it.");
}
- } else if (UpdateRequestHandler.COMMIT.equals(currTag) || UpdateRequestHandler.OPTIMIZE.equals(currTag)) {
+ } else if (UpdateRequestHandler.COMMIT.equals(currTag)
+ || UpdateRequestHandler.OPTIMIZE.equals(currTag)) {
log.trace("parsing {}", currTag);
- CommitUpdateCommand cmd = new CommitUpdateCommand(req, UpdateRequestHandler.OPTIMIZE.equals(currTag));
+ CommitUpdateCommand cmd =
+ new CommitUpdateCommand(req, UpdateRequestHandler.OPTIMIZE.equals(currTag));
ModifiableSolrParams mp = new ModifiableSolrParams();
-
+
for (int i = 0; i < parser.getAttributeCount(); i++) {
String attrName = parser.getAttributeLocalName(i);
String attrVal = parser.getAttributeValue(i);
mp.set(attrName, attrVal);
}
RequestHandlerUtils.validateCommitParams(mp);
- SolrParams p = SolrParams.wrapDefaults(mp, req.getParams()); // default to the normal request params for commit options
+ SolrParams p =
+ SolrParams.wrapDefaults(
+ mp, req.getParams()); // default to the normal request params for commit options
Review comment:
Fix this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] risdenk commented on a change in pull request #705: SOLR-14920: Spotless formatting for core - non-test only
Posted by GitBox <gi...@apache.org>.
risdenk commented on a change in pull request #705:
URL: https://github.com/apache/solr/pull/705#discussion_r816411492
##########
File path: solr/core/src/java/org/apache/solr/update/AddUpdateCommand.java
##########
@@ -225,11 +224,13 @@ public void setIndexedId(BytesRef indexedId) {
if (versionSif != null) {
addVersionField(sdoc, versionSif);
}
- // TODO: if possible concurrent modification exception (if SolrInputDocument not cloned and is being forwarded to replicas)
+ // TODO: if possible concurrent modification exception (if SolrInputDocument not cloned and is
+ // being forwarded to replicas)
// then we could add this field to the generated lucene document instead.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/DefaultSolrCoreState.java
##########
@@ -69,25 +69,26 @@
private volatile boolean lastReplicationSuccess = true;
- // will we attempt recovery as if we just started up (i.e. use starting versions rather than recent versions for peersync
+ // will we attempt recovery as if we just started up (i.e. use starting versions rather than
+ // recent versions for peersync
// so we aren't looking at update versions that have started buffering since we came up.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/DefaultSolrCoreState.java
##########
@@ -128,7 +128,8 @@ private void closeIndexWriter(IndexWriterCloser closer) {
}
refCntWriter.incref();
- succeeded = true; // the returned RefCounted<IndexWriter> will release the readLock on a decref()
+ succeeded =
+ true; // the returned RefCounted<IndexWriter> will release the readLock on a decref()
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/DefaultSolrCoreState.java
##########
@@ -138,27 +139,32 @@ private void closeIndexWriter(IndexWriterCloser closer) {
iwLock.readLock().unlock();
}
}
-
}
private void initRefCntWriter() {
- // TODO: since we moved to a read-write lock, and don't rely on the count to close the writer, we don't really
- // need this class any more. It could also be a singleton created at the same time as SolrCoreState
- // or we could change the API of SolrCoreState to just return the writer and then add a releaseWriter() call.
+ // TODO: since we moved to a read-write lock, and don't rely on the count to close the writer,
+ // we don't really
+ // need this class any more. It could also be a singleton created at the same time as
+ // SolrCoreState
+ // or we could change the API of SolrCoreState to just return the writer and then add a
+ // releaseWriter() call.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/DefaultSolrCoreState.java
##########
@@ -138,27 +139,32 @@ private void closeIndexWriter(IndexWriterCloser closer) {
iwLock.readLock().unlock();
}
}
-
}
private void initRefCntWriter() {
- // TODO: since we moved to a read-write lock, and don't rely on the count to close the writer, we don't really
- // need this class any more. It could also be a singleton created at the same time as SolrCoreState
- // or we could change the API of SolrCoreState to just return the writer and then add a releaseWriter() call.
+ // TODO: since we moved to a read-write lock, and don't rely on the count to close the writer,
+ // we don't really
+ // need this class any more. It could also be a singleton created at the same time as
+ // SolrCoreState
+ // or we could change the API of SolrCoreState to just return the writer and then add a
+ // releaseWriter() call.
if (refCntWriter == null && indexWriter != null) {
- refCntWriter = new RefCounted<IndexWriter>(indexWriter) {
-
- @Override
- public void decref() {
- iwLock.readLock().unlock();
- super.decref(); // This is now redundant (since we switched to read-write locks), we don't really need to maintain our own reference count.
- }
+ refCntWriter =
+ new RefCounted<IndexWriter>(indexWriter) {
+
+ @Override
+ public void decref() {
+ iwLock.readLock().unlock();
+ super.decref(); // This is now redundant (since we switched to read-write locks), we
+ // don't really need to maintain our own reference count.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/DirectUpdateHandler2.java
##########
@@ -646,14 +736,15 @@ public void commit(CommitUpdateCommand cmd) throws IOException {
} else if (cmd.expungeDeletes) {
writer.forceMergeDeletes();
}
-
+
if (!cmd.softCommit) {
- synchronized (solrCoreState.getUpdateLock()) { // sync is currently needed to prevent preCommit
- // from being called between preSoft and
- // postSoft... see postSoft comments.
+ synchronized (
+ solrCoreState.getUpdateLock()) { // sync is currently needed to prevent preCommit
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/PeerSync.java
##########
@@ -353,36 +359,60 @@ private boolean handleResponse(ShardResponse srsp) {
// If the replica went down between asking for versions and asking for specific updates, that
// shouldn't be treated as success since we counted on getting those updates back (and avoided
// redundantly asking other replicas for them).
- if (cantReachIsSuccess && sreq.purpose == SHARD_REQUEST_PURPOSE_GET_VERSIONS && srsp.getException() instanceof SolrServerException) {
- Throwable solrException = ((SolrServerException) srsp.getException())
- .getRootCause();
- boolean connectTimeoutExceptionInChain = connectTimeoutExceptionInChain(srsp.getException());
- if (connectTimeoutExceptionInChain || solrException instanceof ConnectTimeoutException || solrException instanceof SocketTimeoutException
- || solrException instanceof NoHttpResponseException || solrException instanceof SocketException) {
-
- log.warn("{} couldn't connect to {}, counting as success ", msg(), srsp.getShardAddress(), srsp.getException());
+ if (cantReachIsSuccess
+ && sreq.purpose == SHARD_REQUEST_PURPOSE_GET_VERSIONS
+ && srsp.getException() instanceof SolrServerException) {
+ Throwable solrException = ((SolrServerException) srsp.getException()).getRootCause();
+ boolean connectTimeoutExceptionInChain =
+ connectTimeoutExceptionInChain(srsp.getException());
+ if (connectTimeoutExceptionInChain
+ || solrException instanceof ConnectTimeoutException
+ || solrException instanceof SocketTimeoutException
+ || solrException instanceof NoHttpResponseException
+ || solrException instanceof SocketException) {
+
+ log.warn(
+ "{} couldn't connect to {}, counting as success ",
+ msg(),
+ srsp.getShardAddress(),
+ srsp.getException());
return true;
}
}
-
- if (cantReachIsSuccess && sreq.purpose == SHARD_REQUEST_PURPOSE_GET_VERSIONS && srsp.getException() instanceof SolrException && ((SolrException) srsp.getException()).code() == 503) {
- log.warn("{} got a 503 from {}, counting as success ", msg(), srsp.getShardAddress(), srsp.getException());
+
+ if (cantReachIsSuccess
+ && sreq.purpose == SHARD_REQUEST_PURPOSE_GET_VERSIONS
+ && srsp.getException() instanceof SolrException
+ && ((SolrException) srsp.getException()).code() == 503) {
+ log.warn(
+ "{} got a 503 from {}, counting as success ",
+ msg(),
+ srsp.getShardAddress(),
+ srsp.getException());
return true;
}
-
- if (cantReachIsSuccess && sreq.purpose == SHARD_REQUEST_PURPOSE_GET_VERSIONS && srsp.getException() instanceof SolrException && ((SolrException) srsp.getException()).code() == 404) {
- log.warn("{} got a 404 from {}, counting as success. {} Perhaps /get is not registered?"
- , msg(), srsp.getShardAddress(), srsp.getException());
+
+ if (cantReachIsSuccess
+ && sreq.purpose == SHARD_REQUEST_PURPOSE_GET_VERSIONS
+ && srsp.getException() instanceof SolrException
+ && ((SolrException) srsp.getException()).code() == 404) {
+ log.warn(
+ "{} got a 404 from {}, counting as success. {} Perhaps /get is not registered?",
+ msg(),
+ srsp.getShardAddress(),
+ srsp.getException());
return true;
}
-
- // TODO: we should return the above information so that when we can request a recovery through zookeeper, we do
+
+ // TODO: we should return the above information so that when we can request a recovery through
+ // zookeeper, we do
// that for these nodes
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/SolrCmdDistributor.java
##########
@@ -392,18 +439,24 @@ public String toString() {
}
// Called whenever we get results back from a sub-request.
- // The only ambiguity is if I have _both_ a rollup tracker and a leader tracker. In that case we need to handle
- // both requests returning from leaders of other shards _and_ from my followers. This happens if a leader happens
+ // The only ambiguity is if I have _both_ a rollup tracker and a leader tracker. In that case we
+ // need to handle
+ // both requests returning from leaders of other shards _and_ from my followers. This happens if
+ // a leader happens
// to be the aggregator too.
//
- // This isn't really a problem because only responses _from_ some leader will have the "rf" parameter, in which case
+ // This isn't really a problem because only responses _from_ some leader will have the "rf"
+ // parameter, in which case
// we need to add the data to the rollup tracker.
//
- // In the case of a leaderTracker and rollupTracker both being present, then we need to take care when assembling
+ // In the case of a leaderTracker and rollupTracker both being present, then we need to take
+ // care when assembling
// the final response to check both the rollup and leader trackers on the aggregator node.
- public void trackRequestResult(org.eclipse.jetty.client.api.Response resp, InputStream respBody, boolean success) {
+ public void trackRequestResult(
+ org.eclipse.jetty.client.api.Response resp, InputStream respBody, boolean success) {
- // Returning Integer.MAX_VALUE here means there was no "rf" on the response, therefore we just need to increment
+ // Returning Integer.MAX_VALUE here means there was no "rf" on the response, therefore we just
+ // need to increment
// our achieved rf if we are a leader, i.e. have a leaderTracker.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/SolrIndexSplitter.java
##########
@@ -615,9 +715,11 @@ public void visit(QueryVisitor visitor) {
docSets[currentPartition.get()].set(doc);
}
currentPartition.set((currentPartition.get() + 1) % numPieces);
- } else {
+ } else {
int matchingRangesCount = 0;
- for (int i=0; i < rangesArr.length; i++) { // inner-loop: use array here for extra speed.
+ for (int i = 0;
+ i < rangesArr.length;
+ i++) { // inner-loop: use array here for extra speed.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/TransactionLog.java
##########
@@ -75,30 +71,37 @@
Path tlog;
FileChannel channel;
OutputStream os;
- protected FastOutputStream fos; // all accesses to this stream should be synchronized on "this" (The TransactionLog)
+ protected FastOutputStream
+ fos; // all accesses to this stream should be synchronized on "this" (The TransactionLog)
int numRecords;
public boolean isBuffer;
- protected volatile boolean deleteOnClose = true; // we can delete old tlogs since they are currently only used for real-time-get (and in the future, recovery)
+ protected volatile boolean deleteOnClose =
+ true; // we can delete old tlogs since they are currently only used for real-time-get (and in
+ // the future, recovery)
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/DirectUpdateHandler2.java
##########
@@ -101,64 +101,96 @@
// tracks when auto-commit should occur
protected final CommitTracker commitTracker;
protected final CommitTracker softCommitTracker;
-
+
protected boolean commitWithinSoftCommit;
/**
* package access for testing
+ *
* @lucene.internal
*/
void setCommitWithinSoftCommit(boolean value) {
this.commitWithinSoftCommit = value;
}
private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
-
+
public DirectUpdateHandler2(SolrCore core) {
super(core);
-
+
solrCoreState = core.getSolrCoreState();
-
- UpdateHandlerInfo updateHandlerInfo = core.getSolrConfig()
- .getUpdateHandlerInfo();
+
+ UpdateHandlerInfo updateHandlerInfo = core.getSolrConfig().getUpdateHandlerInfo();
int docsUpperBound = updateHandlerInfo.autoCommmitMaxDocs;
int timeUpperBound = updateHandlerInfo.autoCommmitMaxTime;
long fileSizeUpperBound = updateHandlerInfo.autoCommitMaxSizeBytes;
- commitTracker = new CommitTracker("Hard", core, docsUpperBound, timeUpperBound, fileSizeUpperBound, updateHandlerInfo.openSearcher, false);
-
+ commitTracker =
+ new CommitTracker(
+ "Hard",
+ core,
+ docsUpperBound,
+ timeUpperBound,
+ fileSizeUpperBound,
+ updateHandlerInfo.openSearcher,
+ false);
+
int softCommitDocsUpperBound = updateHandlerInfo.autoSoftCommmitMaxDocs;
int softCommitTimeUpperBound = updateHandlerInfo.autoSoftCommmitMaxTime;
- softCommitTracker = new CommitTracker("Soft", core, softCommitDocsUpperBound, softCommitTimeUpperBound, NO_FILE_SIZE_UPPER_BOUND_PLACEHOLDER, true, true);
-
+ softCommitTracker =
+ new CommitTracker(
+ "Soft",
+ core,
+ softCommitDocsUpperBound,
+ softCommitTimeUpperBound,
+ NO_FILE_SIZE_UPPER_BOUND_PLACEHOLDER,
+ true,
+ true);
+
commitWithinSoftCommit = updateHandlerInfo.commitWithinSoftCommit;
ZkController zkController = core.getCoreContainer().getZkController();
- if (zkController != null && core.getCoreDescriptor().getCloudDescriptor().getReplicaType() == Replica.Type.TLOG) {
+ if (zkController != null
+ && core.getCoreDescriptor().getCloudDescriptor().getReplicaType() == Replica.Type.TLOG) {
commitWithinSoftCommit = false;
commitTracker.setOpenSearcher(true);
}
-
}
-
+
public DirectUpdateHandler2(SolrCore core, UpdateHandler updateHandler) {
super(core, updateHandler.getUpdateLog());
solrCoreState = core.getSolrCoreState();
-
- UpdateHandlerInfo updateHandlerInfo = core.getSolrConfig()
- .getUpdateHandlerInfo();
+
+ UpdateHandlerInfo updateHandlerInfo = core.getSolrConfig().getUpdateHandlerInfo();
int docsUpperBound = updateHandlerInfo.autoCommmitMaxDocs;
int timeUpperBound = updateHandlerInfo.autoCommmitMaxTime;
long fileSizeUpperBound = updateHandlerInfo.autoCommitMaxSizeBytes;
- commitTracker = new CommitTracker("Hard", core, docsUpperBound, timeUpperBound, fileSizeUpperBound, updateHandlerInfo.openSearcher, false);
-
+ commitTracker =
+ new CommitTracker(
+ "Hard",
+ core,
+ docsUpperBound,
+ timeUpperBound,
+ fileSizeUpperBound,
+ updateHandlerInfo.openSearcher,
+ false);
+
int softCommitDocsUpperBound = updateHandlerInfo.autoSoftCommmitMaxDocs;
int softCommitTimeUpperBound = updateHandlerInfo.autoSoftCommmitMaxTime;
- softCommitTracker = new CommitTracker("Soft", core, softCommitDocsUpperBound, softCommitTimeUpperBound, NO_FILE_SIZE_UPPER_BOUND_PLACEHOLDER, updateHandlerInfo.openSearcher, true);
-
+ softCommitTracker =
+ new CommitTracker(
+ "Soft",
+ core,
+ softCommitDocsUpperBound,
+ softCommitTimeUpperBound,
+ NO_FILE_SIZE_UPPER_BOUND_PLACEHOLDER,
+ updateHandlerInfo.openSearcher,
+ true);
+
commitWithinSoftCommit = updateHandlerInfo.commitWithinSoftCommit;
UpdateLog existingLog = updateHandler.getUpdateLog();
if (this.ulog != null && this.ulog == existingLog) {
- // If we are reusing the existing update log, inform the log that its update handler has changed.
+ // If we are reusing the existing update log, inform the log that its update handler has
+ // changed.
// We do this as late as possible.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/TransactionLog.java
##########
@@ -852,26 +877,31 @@ public Object next() throws IOException {
// Position buffer so that this record is at the end.
// For small records, this will cause subsequent calls to next() to be within the buffer.
long seekPos = endOfThisRecord - fis.getBufferSize();
- seekPos = Math.min(seekPos, prevPos); // seek to the start of the record if it's larger then the block size.
+ seekPos =
+ Math.min(
+ seekPos,
+ prevPos); // seek to the start of the record if it's larger then the block size.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/TransactionLog.java
##########
@@ -792,22 +815,23 @@ public Object next() throws IOException, InterruptedException {
@Override
public abstract String toString();
-
}
public class FSReverseReader extends ReverseReader {
ChannelFastInputStream fis;
- private LogCodec codec = new LogCodec(resolver) {
- @Override
- public SolrInputDocument readSolrInputDocument(DataInputInputStream dis) {
- // Given that the SolrInputDocument is last in an add record, it's OK to just skip
- // reading it completely.
- return null;
- }
- };
+ private LogCodec codec =
+ new LogCodec(resolver) {
+ @Override
+ public SolrInputDocument readSolrInputDocument(DataInputInputStream dis) {
+ // Given that the SolrInputDocument is last in an add record, it's OK to just skip
+ // reading it completely.
+ return null;
+ }
+ };
- int nextLength; // length of the next record (the next one closer to the start of the log file)
- long prevPos; // where we started reading from last time (so prevPos - nextLength == start of next record)
+ int nextLength; // length of the next record (the next one closer to the start of the log file)
+ long prevPos; // where we started reading from last time (so prevPos - nextLength == start of
+ // next record)
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/UpdateLog.java
##########
@@ -186,48 +198,53 @@ public String toString() {
protected TransactionLog tlog;
protected TransactionLog prevTlog;
protected TransactionLog prevTlogOnPrecommit;
- protected final Deque<TransactionLog> logs = new LinkedList<>(); // list of recent logs, newest first
+ protected final Deque<TransactionLog> logs =
+ new LinkedList<>(); // list of recent logs, newest first
protected LinkedList<TransactionLog> newestLogsOnStartup = new LinkedList<>();
- protected int numOldRecords; // number of records in the recent logs
+ protected int numOldRecords; // number of records in the recent logs
- protected Map<BytesRef,LogPtr> map = new HashMap<>();
- protected Map<BytesRef,LogPtr> prevMap; // used while committing/reopening is happening
- protected Map<BytesRef,LogPtr> prevMap2; // used while committing/reopening is happening
- protected TransactionLog prevMapLog; // the transaction log used to look up entries found in prevMap
- protected TransactionLog prevMapLog2; // the transaction log used to look up entries found in prevMap2
+ protected Map<BytesRef, LogPtr> map = new HashMap<>();
+ protected Map<BytesRef, LogPtr> prevMap; // used while committing/reopening is happening
+ protected Map<BytesRef, LogPtr> prevMap2; // used while committing/reopening is happening
+ protected TransactionLog
+ prevMapLog; // the transaction log used to look up entries found in prevMap
+ protected TransactionLog
+ prevMapLog2; // the transaction log used to look up entries found in prevMap2
protected final int numDeletesToKeep = 1000;
protected final int numDeletesByQueryToKeep = 100;
protected int numRecordsToKeep;
protected int maxNumLogsToKeep;
- protected int numVersionBuckets; // This should only be used to initialize VersionInfo... the actual number of buckets may be rounded up to a power of two.
+ protected int
+ numVersionBuckets; // This should only be used to initialize VersionInfo... the actual number
+ // of buckets may be rounded up to a power of two.
protected Long maxVersionFromIndex = null;
protected boolean existOldBufferLog = false;
// keep track of deletes only... this is not updated on an add
- protected LinkedHashMap<BytesRef, LogPtr> oldDeletes = new LinkedHashMap<>(numDeletesToKeep) {
- @Override
- protected boolean removeEldestEntry(Map.Entry<BytesRef, LogPtr> eldest) {
- return size() > numDeletesToKeep;
- }
- };
+ protected LinkedHashMap<BytesRef, LogPtr> oldDeletes =
+ new LinkedHashMap<>(numDeletesToKeep) {
+ @Override
+ protected boolean removeEldestEntry(Map.Entry<BytesRef, LogPtr> eldest) {
+ return size() > numDeletesToKeep;
+ }
+ };
- /**
- * Holds the query and the version for a DeleteByQuery command
- */
+ /** Holds the query and the version for a DeleteByQuery command */
public static class DBQ {
- public String q; // the query string
+ public String q; // the query string
public long version; // positive version of the DBQ
@Override
public String toString() {
- return "DBQ{version=" + version + ",q="+q+"}";
+ return "DBQ{version=" + version + ",q=" + q + "}";
}
}
protected LinkedList<DBQ> deleteByQueries = new LinkedList<>();
- protected String[] tlogFiles; // Needs to be String because hdfs.Path is incompatible with nio.Path
+ protected String[]
+ tlogFiles; // Needs to be String because hdfs.Path is incompatible with nio.Path
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/UpdateCommand.java
##########
@@ -20,22 +20,22 @@
import io.opentracing.util.GlobalTracer;
import org.apache.solr.request.SolrQueryRequest;
-
-/** An index update command encapsulated in an object (Command pattern)
- *
- *
- */
+/** An index update command encapsulated in an object (Command pattern) */
public abstract class UpdateCommand implements Cloneable {
protected SolrQueryRequest req;
protected long version;
protected String route;
protected int flags;
- public static int BUFFERING = 0x00000001; // update command is being buffered.
- public static int REPLAY = 0x00000002; // update command is from replaying a log.
- public static int PEER_SYNC = 0x00000004; // update command is a missing update being provided by a peer.
- public static int IGNORE_AUTOCOMMIT = 0x00000008; // this update should not count toward triggering of autocommits.
- public static int CLEAR_CACHES = 0x00000010; // clear caches associated with the update log. used when applying reordered DBQ updates when doing an add.
+ public static int BUFFERING = 0x00000001; // update command is being buffered.
+ public static int REPLAY = 0x00000002; // update command is from replaying a log.
+ public static int PEER_SYNC =
+ 0x00000004; // update command is a missing update being provided by a peer.
+ public static int IGNORE_AUTOCOMMIT =
+ 0x00000008; // this update should not count toward triggering of autocommits.
+ public static int CLEAR_CACHES =
+ 0x00000010; // clear caches associated with the update log. used when applying reordered DBQ
+ // updates when doing an add.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/UpdateLog.java
##########
@@ -186,48 +198,53 @@ public String toString() {
protected TransactionLog tlog;
protected TransactionLog prevTlog;
protected TransactionLog prevTlogOnPrecommit;
- protected final Deque<TransactionLog> logs = new LinkedList<>(); // list of recent logs, newest first
+ protected final Deque<TransactionLog> logs =
+ new LinkedList<>(); // list of recent logs, newest first
protected LinkedList<TransactionLog> newestLogsOnStartup = new LinkedList<>();
- protected int numOldRecords; // number of records in the recent logs
+ protected int numOldRecords; // number of records in the recent logs
- protected Map<BytesRef,LogPtr> map = new HashMap<>();
- protected Map<BytesRef,LogPtr> prevMap; // used while committing/reopening is happening
- protected Map<BytesRef,LogPtr> prevMap2; // used while committing/reopening is happening
- protected TransactionLog prevMapLog; // the transaction log used to look up entries found in prevMap
- protected TransactionLog prevMapLog2; // the transaction log used to look up entries found in prevMap2
+ protected Map<BytesRef, LogPtr> map = new HashMap<>();
+ protected Map<BytesRef, LogPtr> prevMap; // used while committing/reopening is happening
+ protected Map<BytesRef, LogPtr> prevMap2; // used while committing/reopening is happening
+ protected TransactionLog
+ prevMapLog; // the transaction log used to look up entries found in prevMap
+ protected TransactionLog
+ prevMapLog2; // the transaction log used to look up entries found in prevMap2
protected final int numDeletesToKeep = 1000;
protected final int numDeletesByQueryToKeep = 100;
protected int numRecordsToKeep;
protected int maxNumLogsToKeep;
- protected int numVersionBuckets; // This should only be used to initialize VersionInfo... the actual number of buckets may be rounded up to a power of two.
+ protected int
+ numVersionBuckets; // This should only be used to initialize VersionInfo... the actual number
+ // of buckets may be rounded up to a power of two.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/UpdateLog.java
##########
@@ -252,12 +269,14 @@ public String toString() {
public static class LogPtr {
final long pointer;
final long version;
- final long previousPointer; // used for entries that are in-place updates and need a pointer to a previous update command
+ final long
+ previousPointer; // used for entries that are in-place updates and need a pointer to a
+ // previous update command
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/UpdateLog.java
##########
@@ -1561,16 +1667,16 @@ private void update() {
numUpdates++;
}
- } catch (IOException | AssertionError e) { // catch AssertionError to handle certain test failures correctly
+ } catch (IOException
+ | AssertionError e) { // catch AssertionError to handle certain test failures correctly
// failure to read a log record isn't fatal
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/UpdateLog.java
##########
@@ -1762,7 +1868,8 @@ public void run() {
params.set(DistributedUpdateProcessor.LOG_REPLAY, "true");
req = new LocalSolrQueryRequest(uhandler.core, params);
rsp = new SolrQueryResponse();
- SolrRequestInfo.setRequestInfo(new SolrRequestInfo(req, rsp)); // setting request info will help logging
+ SolrRequestInfo.setRequestInfo(
+ new SolrRequestInfo(req, rsp)); // setting request info will help logging
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/DocumentBuilder.java
##########
@@ -316,7 +380,8 @@ private static void moveLargestFieldLast(Document doc) {
if (largestIsLast && !field.name().equals(largestField)) {
largestIsLast = false;
}
- if (field.numericValue() != null) { // just ignore these as non-competitive (avoid toString'ing their number)
+ if (field.numericValue()
+ != null) { // just ignore these as non-competitive (avoid toString'ing their number)
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/DirectUpdateHandler2.java
##########
@@ -453,9 +537,14 @@ private Query getQuery(DeleteUpdateCommand cmd) {
bq.add(q, Occur.MUST);
SchemaField sf = ulog.getVersionInfo().getVersionField();
ValueSource vs = sf.getType().getValueSource(sf, null);
- ValueSourceRangeFilter filt = new ValueSourceRangeFilter(vs, Long.toString(Math.abs(cmd.getVersion())), null, true, true);
+ ValueSourceRangeFilter filt =
+ new ValueSourceRangeFilter(
+ vs, Long.toString(Math.abs(cmd.getVersion())), null, true, true);
FunctionRangeQuery range = new FunctionRangeQuery(filt);
- bq.add(range, Occur.MUST_NOT); // formulated in the "MUST_NOT" sense so we can delete docs w/o a version (some tests depend on this...)
+ bq.add(
+ range,
+ Occur.MUST_NOT); // formulated in the "MUST_NOT" sense so we can delete docs w/o a
+ // version (some tests depend on this...)
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/DirectUpdateHandler2.java
##########
@@ -914,31 +1008,36 @@ public void split(SplitIndexCommand cmd) throws IOException {
}
/**
- * Calls either {@link IndexWriter#updateDocValues} or <code>IndexWriter#updateDocument</code>(s) as
- * needed based on {@link AddUpdateCommand#isInPlaceUpdate}.
- * <p>
- * If the this is an UPDATE_INPLACE cmd, then all fields included in
- * {@link AddUpdateCommand#makeLuceneDocForInPlaceUpdate} must either be the uniqueKey field, or be DocValue
- * only fields.
- * </p>
+ * Calls either {@link IndexWriter#updateDocValues} or <code>IndexWriter#updateDocument</code>(s)
+ * as needed based on {@link AddUpdateCommand#isInPlaceUpdate}.
+ *
+ * <p>If the this is an UPDATE_INPLACE cmd, then all fields included in {@link
+ * AddUpdateCommand#makeLuceneDocForInPlaceUpdate} must either be the uniqueKey field, or be
+ * DocValue only fields.
*
* @param cmd - cmd apply to IndexWriter
* @param writer - IndexWriter to use
*/
private void updateDocOrDocValues(AddUpdateCommand cmd, IndexWriter writer) throws IOException {
- assert idField != null; // this code path requires an idField in order to potentially replace a doc
+ assert idField
+ != null; // this code path requires an idField in order to potentially replace a doc
boolean hasUpdateTerm = cmd.updateTerm != null; // AKA dedupe
if (cmd.isInPlaceUpdate()) {
if (hasUpdateTerm) {
- throw new IllegalStateException("cmd.updateTerm/dedupe is not compatible with in-place updates");
+ throw new IllegalStateException(
+ "cmd.updateTerm/dedupe is not compatible with in-place updates");
}
- // we don't support the solrInputDoc with nested child docs either but we'll throw an exception if attempted
+ // we don't support the solrInputDoc with nested child docs either but we'll throw an
+ // exception if attempted
// can't use cmd.getIndexedId because it will be a root doc if this doc is a child
- Term updateTerm = new Term(idField.getName(),
- core.getLatestSchema().indexableUniqueKey(cmd.getSelfOrNestedDocIdStr()));
- List<IndexableField> fields = cmd.makeLuceneDocForInPlaceUpdate().getFields(); // skips uniqueKey and _root_
+ Term updateTerm =
+ new Term(
+ idField.getName(),
+ core.getLatestSchema().indexableUniqueKey(cmd.getSelfOrNestedDocIdStr()));
+ List<IndexableField> fields =
+ cmd.makeLuceneDocForInPlaceUpdate().getFields(); // skips uniqueKey and _root_
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/processor/AddSchemaFieldsUpdateProcessorFactory.java
##########
@@ -477,17 +535,18 @@ public void processAdd(AddUpdateCommand cmd) throws IOException {
super.processAdd(cmd);
}
- /**
- * Recursively find unknown fields in the given doc and its child documents, if any.
- */
- private void getUnknownFields
- (FieldNameSelector selector, SolrInputDocument doc, Map<String,List<SolrInputField>> unknownFields) {
+ /** Recursively find unknown fields in the given doc and its child documents, if any. */
+ private void getUnknownFields(
+ FieldNameSelector selector,
+ SolrInputDocument doc,
+ Map<String, List<SolrInputField>> unknownFields) {
for (final String fieldName : doc.getFieldNames()) {
- //We do a assert and a null check because even after SOLR-12710 is addressed
- //older SolrJ versions can send null values causing an NPE
+ // We do a assert and a null check because even after SOLR-12710 is addressed
+ // older SolrJ versions can send null values causing an NPE
assert fieldName != null;
if (fieldName != null) {
- if (selector.shouldMutate(fieldName)) { // returns false if the field already exists in the current schema
+ if (selector.shouldMutate(
+ fieldName)) { // returns false if the field already exists in the current schema
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/DirectUpdateHandler2.java
##########
@@ -951,11 +1050,14 @@ private void updateDocOrDocValues(AddUpdateCommand cmd, IndexWriter writer) thro
log.debug("updateDocuments({})", cmd);
writer.updateDocuments(updateTerm, nestedDocs);
- // If hasUpdateTerm, then delete any existing documents with the same ID other than the one added above
+ // If hasUpdateTerm, then delete any existing documents with the same ID other than the one
+ // added above
// (used in near-duplicate replacement)
if (hasUpdateTerm) { // rare
BooleanQuery.Builder bq = new BooleanQuery.Builder();
- bq.add(new TermQuery(updateTerm), Occur.MUST_NOT); //don't want the one we added above (will be unique)
+ bq.add(
+ new TermQuery(updateTerm),
+ Occur.MUST_NOT); // don't want the one we added above (will be unique)
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/DefaultSolrCoreState.java
##########
@@ -377,7 +394,9 @@ public void cancelRecovery() {
@Override
public void recovered() {
recoveryStrat = null;
- recoveringAfterStartup = false; // once we have successfully recovered, we no longer need to act as if we are recovering after startup
+ recoveringAfterStartup =
+ false; // once we have successfully recovered, we no longer need to act as if we are
+ // recovering after startup
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/PeerSync.java
##########
@@ -717,12 +770,14 @@ MissedUpdatesRequest handleVersionsWithRanges(List<Long> otherVersions, boolean
}
@VisibleForTesting
- /**
- * Implementation assumes the passed in lists are sorted and contain no duplicates.
- */
- static MissedUpdatesRequest handleVersionsWithRanges(List<Long> otherVersions, boolean completeList,
- List<Long> ourUpdates, long ourLowThreshold) {
- // we may endup asking for updates for too many versions, causing 2MB post payload limit. Construct a range of
+ /** Implementation assumes the passed in lists are sorted and contain no duplicates. */
+ static MissedUpdatesRequest handleVersionsWithRanges(
+ List<Long> otherVersions,
+ boolean completeList,
+ List<Long> ourUpdates,
+ long ourLowThreshold) {
+ // we may endup asking for updates for too many versions, causing 2MB post payload limit.
+ // Construct a range of
// versions to request instead of asking individual versions
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/SolrIndexSplitter.java
##########
@@ -299,28 +335,45 @@ public void doSplit() throws IOException {
if (splitMethod == SplitMethod.LINK) {
t = timings.sub("deleteDocuments");
t.resume();
- // apply deletions specific to this partition. As a side-effect on the first call this also populates
- // a cache of docsets to delete per leaf reader per partition, which is reused for subsequent partitions.
- iw.deleteDocuments(new SplittingQuery(partitionNumber, field, rangesArr, hashRouter, splitKey, docsToDeleteCache, currentPartition));
+ // apply deletions specific to this partition. As a side-effect on the first call this
+ // also populates
+ // a cache of docsets to delete per leaf reader per partition, which is reused for
+ // subsequent partitions.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/SolrCmdDistributor.java
##########
@@ -392,18 +439,24 @@ public String toString() {
}
// Called whenever we get results back from a sub-request.
- // The only ambiguity is if I have _both_ a rollup tracker and a leader tracker. In that case we need to handle
- // both requests returning from leaders of other shards _and_ from my followers. This happens if a leader happens
+ // The only ambiguity is if I have _both_ a rollup tracker and a leader tracker. In that case we
+ // need to handle
+ // both requests returning from leaders of other shards _and_ from my followers. This happens if
+ // a leader happens
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/PeerSyncWithLeader.java
##########
@@ -281,26 +295,33 @@ private boolean handleUpdates(NamedList<Object> rsp, long numRequestedUpdates, I
}
// Leader will compute its fingerprint, then retrieve its recent updates versions.
- // There are a case that some updates (gap) get into recent versions but do not exist in index (fingerprint).
- // If the gap do not contains DBQ or DBI, it is safe to use leaderFingerprint.maxVersionEncountered as a cut point.
+ // There are a case that some updates (gap) get into recent versions but do not exist in index
+ // (fingerprint).
+ // If the gap do not contains DBQ or DBI, it is safe to use
+ // leaderFingerprint.maxVersionEncountered as a cut point.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/UpdateLog.java
##########
@@ -1720,24 +1827,23 @@ public State getState() {
@Override
public String toString() {
- return "FSUpdateLog{state="+getState()+", tlog="+tlog+"}";
+ return "FSUpdateLog{state=" + getState() + ", tlog=" + tlog + "}";
}
-
- public static Runnable testing_logReplayHook; // called before each log read
- public static Runnable testing_logReplayFinishHook; // called when log replay has finished
-
-
+ public static Runnable testing_logReplayHook; // called before each log read
+ public static Runnable testing_logReplayFinishHook; // called when log replay has finished
protected RecoveryInfo recoveryInfo;
class LogReplayer implements Runnable {
- private Logger loglog = log; // set to something different?
+ private Logger loglog = log; // set to something different?
Deque<TransactionLog> translogs;
TransactionLog.LogReader tlogReader;
boolean activeLog;
- boolean finishing = false; // state where we lock out other updates and finish those updates that snuck in before we locked
+ boolean finishing =
+ false; // state where we lock out other updates and finish those updates that snuck in
+ // before we locked
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/StreamingSolrClients.java
##########
@@ -66,15 +65,18 @@ public synchronized SolrClient getSolrClient(final SolrCmdDistributor.Req req) {
String url = getFullUrl(req.node.getUrl());
ConcurrentUpdateHttp2SolrClient client = solrClients.get(url);
if (client == null) {
- // NOTE: increasing to more than 1 threadCount for the client could cause updates to be reordered
- // on a greater scale since the current behavior is to only increase the number of connections/Runners when
+ // NOTE: increasing to more than 1 threadCount for the client could cause updates to be
+ // reordered
+ // on a greater scale since the current behavior is to only increase the number of
+ // connections/Runners when
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/TransactionLog.java
##########
@@ -75,30 +71,37 @@
Path tlog;
FileChannel channel;
OutputStream os;
- protected FastOutputStream fos; // all accesses to this stream should be synchronized on "this" (The TransactionLog)
+ protected FastOutputStream
+ fos; // all accesses to this stream should be synchronized on "this" (The TransactionLog)
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/PeerSync.java
##########
@@ -187,7 +194,8 @@ public PeerSyncResult sync() {
timerContext = syncTime.time();
// Fire off the requests before getting our own recent updates (for better concurrency)
- // This also allows us to avoid getting updates we don't need... if we got our updates and then got their updates,
+ // This also allows us to avoid getting updates we don't need... if we got our updates and
+ // then got their updates,
// they would
// have newer stuff that we also had (assuming updates are going on and are being forwarded).
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/UpdateLog.java
##########
@@ -186,48 +198,53 @@ public String toString() {
protected TransactionLog tlog;
protected TransactionLog prevTlog;
protected TransactionLog prevTlogOnPrecommit;
- protected final Deque<TransactionLog> logs = new LinkedList<>(); // list of recent logs, newest first
+ protected final Deque<TransactionLog> logs =
+ new LinkedList<>(); // list of recent logs, newest first
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/SolrIndexSplitter.java
##########
@@ -106,7 +105,8 @@ public String toLower() {
final SplitIndexCommand cmd;
final SolrIndexSearcher searcher;
final SchemaField field;
- final DocRouter.Range[] rangesArr; // same as ranges list, but an array for extra speed in inner loops
+ final DocRouter.Range[]
+ rangesArr; // same as ranges list, but an array for extra speed in inner loops
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/TransactionLog.java
##########
@@ -852,26 +877,31 @@ public Object next() throws IOException {
// Position buffer so that this record is at the end.
// For small records, this will cause subsequent calls to next() to be within the buffer.
long seekPos = endOfThisRecord - fis.getBufferSize();
- seekPos = Math.min(seekPos, prevPos); // seek to the start of the record if it's larger then the block size.
+ seekPos =
+ Math.min(
+ seekPos,
+ prevPos); // seek to the start of the record if it's larger then the block size.
seekPos = Math.max(seekPos, 0);
fis.seek(seekPos);
- fis.peek(); // cause buffer to be filled
+ fis.peek(); // cause buffer to be filled
}
fis.seek(prevPos);
- nextLength = fis.readInt(); // this is the length of the *next* record (i.e. closer to the beginning)
+ nextLength =
+ fis.readInt(); // this is the length of the *next* record (i.e. closer to the beginning)
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/UpdateLog.java
##########
@@ -186,48 +198,53 @@ public String toString() {
protected TransactionLog tlog;
protected TransactionLog prevTlog;
protected TransactionLog prevTlogOnPrecommit;
- protected final Deque<TransactionLog> logs = new LinkedList<>(); // list of recent logs, newest first
+ protected final Deque<TransactionLog> logs =
+ new LinkedList<>(); // list of recent logs, newest first
protected LinkedList<TransactionLog> newestLogsOnStartup = new LinkedList<>();
- protected int numOldRecords; // number of records in the recent logs
+ protected int numOldRecords; // number of records in the recent logs
- protected Map<BytesRef,LogPtr> map = new HashMap<>();
- protected Map<BytesRef,LogPtr> prevMap; // used while committing/reopening is happening
- protected Map<BytesRef,LogPtr> prevMap2; // used while committing/reopening is happening
- protected TransactionLog prevMapLog; // the transaction log used to look up entries found in prevMap
- protected TransactionLog prevMapLog2; // the transaction log used to look up entries found in prevMap2
+ protected Map<BytesRef, LogPtr> map = new HashMap<>();
+ protected Map<BytesRef, LogPtr> prevMap; // used while committing/reopening is happening
+ protected Map<BytesRef, LogPtr> prevMap2; // used while committing/reopening is happening
+ protected TransactionLog
+ prevMapLog; // the transaction log used to look up entries found in prevMap
+ protected TransactionLog
+ prevMapLog2; // the transaction log used to look up entries found in prevMap2
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/TransactionLog.java
##########
@@ -852,26 +877,31 @@ public Object next() throws IOException {
// Position buffer so that this record is at the end.
// For small records, this will cause subsequent calls to next() to be within the buffer.
long seekPos = endOfThisRecord - fis.getBufferSize();
- seekPos = Math.min(seekPos, prevPos); // seek to the start of the record if it's larger then the block size.
+ seekPos =
+ Math.min(
+ seekPos,
+ prevPos); // seek to the start of the record if it's larger then the block size.
seekPos = Math.max(seekPos, 0);
fis.seek(seekPos);
- fis.peek(); // cause buffer to be filled
+ fis.peek(); // cause buffer to be filled
}
fis.seek(prevPos);
- nextLength = fis.readInt(); // this is the length of the *next* record (i.e. closer to the beginning)
+ nextLength =
+ fis.readInt(); // this is the length of the *next* record (i.e. closer to the beginning)
// TODO: optionally skip document data
Object o = codec.readVal(fis);
- // assert fis.position() == prevPos + 4 + thisLength; // this is only true if we read all the data (and we currently skip reading SolrInputDocument
+ // assert fis.position() == prevPos + 4 + thisLength; // this is only true if we read all the
+ // data (and we currently skip reading SolrInputDocument
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/TransactionLog.java
##########
@@ -910,15 +945,17 @@ public void seek(long position) throws IOException {
// seek within buffer
pos = (int) (position - getBufferPos());
} else {
- // long currSize = ch.size(); // not needed - underlying read should handle (unless read never done)
- // if (position > currSize) throw new EOFException("Read past EOF: seeking to " + position + " on file of size " + currSize + " file=" + ch);
+ // long currSize = ch.size(); // not needed - underlying read should handle (unless read
+ // never done)
+ // if (position > currSize) throw new EOFException("Read past EOF: seeking to " + position +
+ // " on file of size " + currSize + " file=" + ch);
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/processor/AtomicUpdateProcessorFactory.java
##########
@@ -170,11 +173,17 @@ private void processAddWithRetry(AddUpdateCommand cmd, int attempts, SolrInputDo
// if lastVersion is null then we put -1 to assert that document must not exist
lastVersion = lastVersion == null ? -1 : lastVersion;
- // The AtomicUpdateDocumentMerger modifies the AddUpdateCommand.solrDoc to populate the real values of the
- // modified fields. We don't want those absolute values because they are out-of-date due to the conflict
- // so we restore the original document created in processAdd method and set the right version on it
+ // The AtomicUpdateDocumentMerger modifies the AddUpdateCommand.solrDoc to populate the
+ // real values of the
+ // modified fields. We don't want those absolute values because they are out-of-date due
+ // to the conflict
+ // so we restore the original document created in processAdd method and set the right
+ // version on it
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/processor/ClassificationUpdateProcessorParams.java
##########
@@ -30,7 +30,8 @@
private int maxPredictedClasses; // the max number of classes to assign
- private ClassificationUpdateProcessorFactory.Algorithm algorithm; // the Classification Algorithm to use - currently 'knn' or 'bayes'
+ private ClassificationUpdateProcessorFactory.Algorithm
+ algorithm; // the Classification Algorithm to use - currently 'knn' or 'bayes'
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/processor/AtomicUpdateDocumentMerger.java
##########
@@ -330,83 +342,108 @@ private static String getRouteField(AddUpdateCommand cmd) {
}
return result;
}
-
+
/**
- *
- * @param fullDoc the full doc to be compared against
+ * @param fullDoc the full doc to be compared against
* @param partialDoc the sub document to be tested
* @return whether partialDoc is derived from fullDoc
*/
public static boolean isDerivedFromDoc(SolrInputDocument fullDoc, SolrInputDocument partialDoc) {
- for(SolrInputField subSif: partialDoc) {
+ for (SolrInputField subSif : partialDoc) {
Collection<Object> fieldValues = fullDoc.getFieldValues(subSif.getName());
if (fieldValues == null) return false;
if (fieldValues.size() < subSif.getValueCount()) return false;
Collection<Object> partialFieldValues = subSif.getValues();
- // filter all derived child docs from partial field values since they fail List#containsAll check (uses SolrInputDocument#equals which fails).
- // If a child doc exists in partialDoc but not in full doc, it will not be filtered, and therefore List#containsAll will return false
- Stream<Object> nonChildDocElements = partialFieldValues.stream().filter(x -> !(isChildDoc(x) &&
- (fieldValues.stream().anyMatch(y ->
- (isChildDoc(x) &&
- isDerivedFromDoc((SolrInputDocument) y, (SolrInputDocument) x)
- )
- )
- )));
+ // filter all derived child docs from partial field values since they fail List#containsAll
+ // check (uses SolrInputDocument#equals which fails).
+ // If a child doc exists in partialDoc but not in full doc, it will not be filtered, and
+ // therefore List#containsAll will return false
+ Stream<Object> nonChildDocElements =
+ partialFieldValues.stream()
+ .filter(
+ x ->
+ !(isChildDoc(x)
+ && (fieldValues.stream()
+ .anyMatch(
+ y ->
+ (isChildDoc(x)
+ && isDerivedFromDoc(
+ (SolrInputDocument) y, (SolrInputDocument) x))))));
if (!nonChildDocElements.allMatch(fieldValues::contains)) return false;
}
return true;
}
/**
- * Given an AddUpdateCommand containing update operations (e.g. set, inc), merge and resolve the operations into
- * a partial document that can be used for indexing the in-place updates. The AddUpdateCommand is modified to contain
- * the partial document (instead of the original document which contained the update operations) and also
- * the prevVersion that this in-place update depends on.
- * Note: updatedFields passed into the method can be changed, i.e. the version field can be added to the set.
- * @return If in-place update cannot succeed, e.g. if the old document is deleted recently, then false is returned. A false
- * return indicates that this update can be re-tried as a full atomic update. Returns true if the in-place update
- * succeeds.
+ * Given an AddUpdateCommand containing update operations (e.g. set, inc), merge and resolve the
+ * operations into a partial document that can be used for indexing the in-place updates. The
+ * AddUpdateCommand is modified to contain the partial document (instead of the original document
+ * which contained the update operations) and also the prevVersion that this in-place update
+ * depends on. Note: updatedFields passed into the method can be changed, i.e. the version field
+ * can be added to the set.
+ *
+ * @return If in-place update cannot succeed, e.g. if the old document is deleted recently, then
+ * false is returned. A false return indicates that this update can be re-tried as a full
+ * atomic update. Returns true if the in-place update succeeds.
*/
- public boolean doInPlaceUpdateMerge(AddUpdateCommand cmd, Set<String> updatedFields) throws IOException {
+ public boolean doInPlaceUpdateMerge(AddUpdateCommand cmd, Set<String> updatedFields)
+ throws IOException {
SolrInputDocument inputDoc = cmd.getSolrInputDocument();
BytesRef rootIdBytes = cmd.getIndexedId();
BytesRef idBytes = schema.indexableUniqueKey(cmd.getSelfOrNestedDocIdStr());
- updatedFields.add(CommonParams.VERSION_FIELD); // add the version field so that it is fetched too
- SolrInputDocument oldDocument = RealTimeGetComponent.getInputDocument
- (cmd.getReq().getCore(), idBytes, rootIdBytes,
- null, // don't want the version to be returned
- updatedFields, RealTimeGetComponent.Resolution.DOC);
+ updatedFields.add(
+ CommonParams.VERSION_FIELD); // add the version field so that it is fetched too
+ SolrInputDocument oldDocument =
+ RealTimeGetComponent.getInputDocument(
+ cmd.getReq().getCore(),
+ idBytes,
+ rootIdBytes,
+ null, // don't want the version to be returned
+ updatedFields,
+ RealTimeGetComponent.Resolution.DOC);
if (oldDocument == RealTimeGetComponent.DELETED || oldDocument == null) {
- // This doc was deleted recently. In-place update cannot work, hence a full atomic update should be tried.
+ // This doc was deleted recently. In-place update cannot work, hence a full atomic update
+ // should be tried.
return false;
}
if (oldDocument.containsKey(CommonParams.VERSION_FIELD) == false) {
- throw new SolrException (ErrorCode.INVALID_STATE, "There is no _version_ in previous document. id=" +
- cmd.getPrintableId());
+ throw new SolrException(
+ ErrorCode.INVALID_STATE,
+ "There is no _version_ in previous document. id=" + cmd.getPrintableId());
}
Long oldVersion = (Long) oldDocument.remove(CommonParams.VERSION_FIELD).getValue();
- // If the oldDocument contains any other field apart from updatedFields (or id/version field), then remove them.
- // This can happen, despite requesting for these fields in the call to RTGC.getInputDocument, if the document was
- // fetched from the tlog and had all these fields (possibly because it was a full document ADD operation).
+ // If the oldDocument contains any other field apart from updatedFields (or id/version field),
+ // then remove them.
+ // This can happen, despite requesting for these fields in the call to RTGC.getInputDocument, if
+ // the document was
+ // fetched from the tlog and had all these fields (possibly because it was a full document ADD
+ // operation).
if (updatedFields != null) {
Collection<String> names = new HashSet<>(oldDocument.getFieldNames());
- for (String fieldName: names) {
- if (fieldName.equals(CommonParams.VERSION_FIELD)==false && fieldName.equals(ID)==false && updatedFields.contains(fieldName)==false) {
+ for (String fieldName : names) {
+ if (fieldName.equals(CommonParams.VERSION_FIELD) == false
+ && fieldName.equals(ID) == false
+ && updatedFields.contains(fieldName) == false) {
oldDocument.remove(fieldName);
}
}
}
// Copy over all supported DVs from oldDocument to partialDoc
//
- // Assuming multiple updates to the same doc: field 'dv1' in one update, then field 'dv2' in a second
- // update, and then again 'dv1' in a third update (without commits in between), the last update would
- // fetch from the tlog the partial doc for the 2nd (dv2) update. If that doc doesn't copy over the
- // previous updates to dv1 as well, then a full resolution (by following previous pointers) would
- // need to be done to calculate the dv1 value -- so instead copy all the potentially affected DV fields.
+ // Assuming multiple updates to the same doc: field 'dv1' in one update, then field 'dv2' in a
+ // second
+ // update, and then again 'dv1' in a third update (without commits in between), the last update
+ // would
+ // fetch from the tlog the partial doc for the 2nd (dv2) update. If that doc doesn't copy over
+ // the
+ // previous updates to dv1 as well, then a full resolution (by following previous pointers)
+ // would
+ // need to be done to calculate the dv1 value -- so instead copy all the potentially affected DV
+ // fields.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/processor/AtomicUpdateProcessorFactory.java
##########
@@ -170,11 +173,17 @@ private void processAddWithRetry(AddUpdateCommand cmd, int attempts, SolrInputDo
// if lastVersion is null then we put -1 to assert that document must not exist
lastVersion = lastVersion == null ? -1 : lastVersion;
- // The AtomicUpdateDocumentMerger modifies the AddUpdateCommand.solrDoc to populate the real values of the
- // modified fields. We don't want those absolute values because they are out-of-date due to the conflict
- // so we restore the original document created in processAdd method and set the right version on it
+ // The AtomicUpdateDocumentMerger modifies the AddUpdateCommand.solrDoc to populate the
+ // real values of the
+ // modified fields. We don't want those absolute values because they are out-of-date due
+ // to the conflict
+ // so we restore the original document created in processAdd method and set the right
+ // version on it
cmd.solrDoc = clonedOriginalDoc;
- clonedOriginalDoc = clonedOriginalDoc.deepCopy(); // copy again because the old cloned ref will be modified during processAdd
+ clonedOriginalDoc =
+ clonedOriginalDoc
+ .deepCopy(); // copy again because the old cloned ref will be modified during
+ // processAdd
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/DirectUpdateHandler2.java
##########
@@ -914,31 +1008,36 @@ public void split(SplitIndexCommand cmd) throws IOException {
}
/**
- * Calls either {@link IndexWriter#updateDocValues} or <code>IndexWriter#updateDocument</code>(s) as
- * needed based on {@link AddUpdateCommand#isInPlaceUpdate}.
- * <p>
- * If the this is an UPDATE_INPLACE cmd, then all fields included in
- * {@link AddUpdateCommand#makeLuceneDocForInPlaceUpdate} must either be the uniqueKey field, or be DocValue
- * only fields.
- * </p>
+ * Calls either {@link IndexWriter#updateDocValues} or <code>IndexWriter#updateDocument</code>(s)
+ * as needed based on {@link AddUpdateCommand#isInPlaceUpdate}.
+ *
+ * <p>If the this is an UPDATE_INPLACE cmd, then all fields included in {@link
+ * AddUpdateCommand#makeLuceneDocForInPlaceUpdate} must either be the uniqueKey field, or be
+ * DocValue only fields.
*
* @param cmd - cmd apply to IndexWriter
* @param writer - IndexWriter to use
*/
private void updateDocOrDocValues(AddUpdateCommand cmd, IndexWriter writer) throws IOException {
- assert idField != null; // this code path requires an idField in order to potentially replace a doc
+ assert idField
+ != null; // this code path requires an idField in order to potentially replace a doc
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/processor/DistributedUpdateProcessor.java
##########
@@ -289,15 +295,17 @@ protected boolean versionAdd(AddUpdateCommand cmd) throws IOException {
if (vinfo == null) {
if (AtomicUpdateDocumentMerger.isAtomicUpdate(cmd)) {
- throw new SolrException(SolrException.ErrorCode.BAD_REQUEST,
+ throw new SolrException(
+ SolrException.ErrorCode.BAD_REQUEST,
"Atomic document updates are not supported unless <updateLog/> is configured");
} else {
super.processAdd(cmd);
return false;
}
}
- // This is only the hash for the bucket, and must be based only on the uniqueKey (i.e. do not use a pluggable hash
+ // This is only the hash for the bucket, and must be based only on the uniqueKey (i.e. do not
+ // use a pluggable hash
// here)
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/processor/CloneFieldUpdateProcessorFactory.java
##########
@@ -145,140 +139,173 @@
* <str name="replacement">key_feat$1</str>
* </str>
* </processor>
- *
+ *
* <!-- syntactic sugar syntax -->
* <processor class="solr.processor.CloneFieldUpdateProcessorFactory">
* <str name="pattern">^feat(.*)s$</str>
* <str name="replacement">key_feat$1</str>
* </processor>
* </pre>
*
- * <p>
- * When cloning multiple fields (or a single multivalued field) into a single valued field, one of the
- * {@link FieldValueSubsetUpdateProcessorFactory} implementations configured after the
- * <code>CloneFieldUpdateProcessorFactory</code> can be useful to reduce the list of values down to a
+ * <p>When cloning multiple fields (or a single multivalued field) into a single valued field, one
+ * of the {@link FieldValueSubsetUpdateProcessorFactory} implementations configured after the <code>
+ * CloneFieldUpdateProcessorFactory</code> can be useful to reduce the list of values down to a
* single value.
- * </p>
- *
+ *
* @see FieldValueSubsetUpdateProcessorFactory
* @since 4.0.0
*/
-public class CloneFieldUpdateProcessorFactory
- extends UpdateRequestProcessorFactory implements SolrCoreAware {
+public class CloneFieldUpdateProcessorFactory extends UpdateRequestProcessorFactory
+ implements SolrCoreAware {
private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
-
+
public static final String SOURCE_PARAM = "source";
public static final String DEST_PARAM = "dest";
public static final String PATTERN_PARAM = "pattern";
public static final String REPLACEMENT_PARAM = "replacement";
private SelectorParams srcInclusions = new SelectorParams();
- private Collection<SelectorParams> srcExclusions
- = new ArrayList<>();
+ private Collection<SelectorParams> srcExclusions = new ArrayList<>();
private FieldNameSelector srcSelector = null;
- /**
- * If pattern is null, this this is a literal field name. If pattern is non-null then this
- * is a replacement string that may contain meta-characters (ie: capture group identifiers)
+ /**
+ * If pattern is null, this this is a literal field name. If pattern is non-null then this is a
+ * replacement string that may contain meta-characters (ie: capture group identifiers)
+ *
* @see #pattern
*/
private String dest = null;
- /** @see #dest */
+ /**
+ * @see #dest
+ */
private Pattern pattern = null;
@SuppressWarnings("WeakerAccess")
protected final FieldNameSelector getSourceSelector() {
if (null != srcSelector) return srcSelector;
- throw new SolrException(SERVER_ERROR, "selector was never initialized, "+
- " inform(SolrCore) never called???");
+ throw new SolrException(
+ SERVER_ERROR, "selector was never initialized, " + " inform(SolrCore) never called???");
}
@Override
public void init(NamedList<?> args) {
// high level (loose) check for which type of config we have.
- //
+ //
// individual init methods do more strict syntax checking
- if (0 <= args.indexOf(SOURCE_PARAM, 0) && 0 <= args.indexOf(DEST_PARAM, 0) ) {
+ if (0 <= args.indexOf(SOURCE_PARAM, 0) && 0 <= args.indexOf(DEST_PARAM, 0)) {
initSourceSelectorSyntax(args);
} else if (0 <= args.indexOf(PATTERN_PARAM, 0) && 0 <= args.indexOf(REPLACEMENT_PARAM, 0)) {
initSimpleRegexReplacement(args);
} else {
- throw new SolrException(SERVER_ERROR, "A combination of either '" + SOURCE_PARAM + "' + '"+
- DEST_PARAM + "', or '" + REPLACEMENT_PARAM + "' + '" +
- PATTERN_PARAM + "' init params are mandatory");
+ throw new SolrException(
+ SERVER_ERROR,
+ "A combination of either '"
+ + SOURCE_PARAM
+ + "' + '"
+ + DEST_PARAM
+ + "', or '"
+ + REPLACEMENT_PARAM
+ + "' + '"
+ + PATTERN_PARAM
+ + "' init params are mandatory");
}
-
+
if (0 < args.size()) {
- throw new SolrException(SERVER_ERROR,
- "Unexpected init param(s): '" +
- args.getName(0) + "'");
+ throw new SolrException(SERVER_ERROR, "Unexpected init param(s): '" + args.getName(0) + "'");
}
super.init(args);
}
/**
- * init helper method that should only be called when we know for certain that both the
- * "source" and "dest" init params do <em>not</em> exist.
+ * init helper method that should only be called when we know for certain that both the "source"
+ * and "dest" init params do <em>not</em> exist.
*/
private void initSimpleRegexReplacement(NamedList<?> args) {
- // The syntactic sugar for the case where there is only one regex pattern for source and the same pattern
+ // The syntactic sugar for the case where there is only one regex pattern for source and the
+ // same pattern
// is used for the destination pattern...
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/processor/CloneFieldUpdateProcessorFactory.java
##########
@@ -145,140 +139,173 @@
* <str name="replacement">key_feat$1</str>
* </str>
* </processor>
- *
+ *
* <!-- syntactic sugar syntax -->
* <processor class="solr.processor.CloneFieldUpdateProcessorFactory">
* <str name="pattern">^feat(.*)s$</str>
* <str name="replacement">key_feat$1</str>
* </processor>
* </pre>
*
- * <p>
- * When cloning multiple fields (or a single multivalued field) into a single valued field, one of the
- * {@link FieldValueSubsetUpdateProcessorFactory} implementations configured after the
- * <code>CloneFieldUpdateProcessorFactory</code> can be useful to reduce the list of values down to a
+ * <p>When cloning multiple fields (or a single multivalued field) into a single valued field, one
+ * of the {@link FieldValueSubsetUpdateProcessorFactory} implementations configured after the <code>
+ * CloneFieldUpdateProcessorFactory</code> can be useful to reduce the list of values down to a
* single value.
- * </p>
- *
+ *
* @see FieldValueSubsetUpdateProcessorFactory
* @since 4.0.0
*/
-public class CloneFieldUpdateProcessorFactory
- extends UpdateRequestProcessorFactory implements SolrCoreAware {
+public class CloneFieldUpdateProcessorFactory extends UpdateRequestProcessorFactory
+ implements SolrCoreAware {
private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
-
+
public static final String SOURCE_PARAM = "source";
public static final String DEST_PARAM = "dest";
public static final String PATTERN_PARAM = "pattern";
public static final String REPLACEMENT_PARAM = "replacement";
private SelectorParams srcInclusions = new SelectorParams();
- private Collection<SelectorParams> srcExclusions
- = new ArrayList<>();
+ private Collection<SelectorParams> srcExclusions = new ArrayList<>();
private FieldNameSelector srcSelector = null;
- /**
- * If pattern is null, this this is a literal field name. If pattern is non-null then this
- * is a replacement string that may contain meta-characters (ie: capture group identifiers)
+ /**
+ * If pattern is null, this this is a literal field name. If pattern is non-null then this is a
+ * replacement string that may contain meta-characters (ie: capture group identifiers)
+ *
* @see #pattern
*/
private String dest = null;
- /** @see #dest */
+ /**
+ * @see #dest
+ */
private Pattern pattern = null;
@SuppressWarnings("WeakerAccess")
protected final FieldNameSelector getSourceSelector() {
if (null != srcSelector) return srcSelector;
- throw new SolrException(SERVER_ERROR, "selector was never initialized, "+
- " inform(SolrCore) never called???");
+ throw new SolrException(
+ SERVER_ERROR, "selector was never initialized, " + " inform(SolrCore) never called???");
}
@Override
public void init(NamedList<?> args) {
// high level (loose) check for which type of config we have.
- //
+ //
// individual init methods do more strict syntax checking
- if (0 <= args.indexOf(SOURCE_PARAM, 0) && 0 <= args.indexOf(DEST_PARAM, 0) ) {
+ if (0 <= args.indexOf(SOURCE_PARAM, 0) && 0 <= args.indexOf(DEST_PARAM, 0)) {
initSourceSelectorSyntax(args);
} else if (0 <= args.indexOf(PATTERN_PARAM, 0) && 0 <= args.indexOf(REPLACEMENT_PARAM, 0)) {
initSimpleRegexReplacement(args);
} else {
- throw new SolrException(SERVER_ERROR, "A combination of either '" + SOURCE_PARAM + "' + '"+
- DEST_PARAM + "', or '" + REPLACEMENT_PARAM + "' + '" +
- PATTERN_PARAM + "' init params are mandatory");
+ throw new SolrException(
+ SERVER_ERROR,
+ "A combination of either '"
+ + SOURCE_PARAM
+ + "' + '"
+ + DEST_PARAM
+ + "', or '"
+ + REPLACEMENT_PARAM
+ + "' + '"
+ + PATTERN_PARAM
+ + "' init params are mandatory");
}
-
+
if (0 < args.size()) {
- throw new SolrException(SERVER_ERROR,
- "Unexpected init param(s): '" +
- args.getName(0) + "'");
+ throw new SolrException(SERVER_ERROR, "Unexpected init param(s): '" + args.getName(0) + "'");
}
super.init(args);
}
/**
- * init helper method that should only be called when we know for certain that both the
- * "source" and "dest" init params do <em>not</em> exist.
+ * init helper method that should only be called when we know for certain that both the "source"
+ * and "dest" init params do <em>not</em> exist.
*/
private void initSimpleRegexReplacement(NamedList<?> args) {
- // The syntactic sugar for the case where there is only one regex pattern for source and the same pattern
+ // The syntactic sugar for the case where there is only one regex pattern for source and the
+ // same pattern
// is used for the destination pattern...
//
// pattern != null && replacement != null
- //
+ //
// ...as top level elements, with no other config options specified
-
- // if we got here we know we had pattern and replacement, now check for the other two so that we can give a better
+
+ // if we got here we know we had pattern and replacement, now check for the other two so that
+ // we can give a better
// message than "unexpected"
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/processor/CloneFieldUpdateProcessorFactory.java
##########
@@ -288,16 +315,25 @@ private void initSourceSelectorSyntax(NamedList<?> args) {
//
// source != null && dest != null
- // if we got here we know we had source and dest, now check for the other two so that we can give a better
+ // if we got here we know we had source and dest, now check for the other two so that we can
+ // give a better
// message than "unexpected"
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/UpdateLog.java
##########
@@ -373,27 +401,35 @@ public void init(UpdateHandler uhandler, SolrCore core) {
throw new SolrException(ErrorCode.SERVER_ERROR, "Could not set up tlogs", e);
}
tlogFiles = getLogList(tlogDir.toFile());
- id = getLastLogId() + 1; // add 1 since we will create a new log for the next update
+ id = getLastLogId() + 1; // add 1 since we will create a new log for the next update
if (debug) {
- log.debug("UpdateHandler init: tlogDir={}, existing tlogs={}, next id={}", tlogDir, Arrays.asList(tlogFiles), id);
+ log.debug(
+ "UpdateHandler init: tlogDir={}, existing tlogs={}, next id={}",
+ tlogDir,
+ Arrays.asList(tlogFiles),
+ id);
}
final String prefix = BUFFER_TLOG_NAME + '.';
try (Stream<Path> bufferedTLogs = Files.walk(tlogDir, 1)) {
- existOldBufferLog = bufferedTLogs.anyMatch(path -> path.getFileName().toString().startsWith(prefix));
+ existOldBufferLog =
+ bufferedTLogs.anyMatch(path -> path.getFileName().toString().startsWith(prefix));
} catch (IOException e) {
- // Existance of buffered t-logs indicates previous recovery failed and lets us skip peer sync as an optimization
+ // Existance of buffered t-logs indicates previous recovery failed and lets us skip peer sync
+ // as an optimization
// Failing to read them is non-fatal and almost not even worth logging about
- log.debug("Could not read {} directory searching for buffered transaction log files.", tlogDir, e);
+ log.debug(
+ "Could not read {} directory searching for buffered transaction log files.", tlogDir, e);
existOldBufferLog = false;
}
TransactionLog oldLog = null;
for (String oldLogName : tlogFiles) {
Path path = tlogDir.resolve(oldLogName);
try {
oldLog = newTransactionLog(path, null, true);
- addOldLog(oldLog, false); // don't remove old logs on startup since more than one may be uncapped.
+ addOldLog(
+ oldLog, false); // don't remove old logs on startup since more than one may be uncapped.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/UpdateLog.java
##########
@@ -875,47 +927,66 @@ public void postSoftCommit(CommitUpdateCommand cmd) {
// record what old maps were created and only remove those.
if (debug) {
- SolrCore.verbose("TLOG: postSoftCommit: disposing of prevMap="+ System.identityHashCode(prevMap) + ", prevMap2=" + System.identityHashCode(prevMap2));
+ SolrCore.verbose(
+ "TLOG: postSoftCommit: disposing of prevMap="
+ + System.identityHashCode(prevMap)
+ + ", prevMap2="
+ + System.identityHashCode(prevMap2));
}
clearOldMaps();
-
}
}
/**
- * Goes over backwards, following the prevPointer, to merge all partial updates into the passed doc. Stops at either a full
- * document, or if there are no previous entries to follow in the update log.
+ * Goes over backwards, following the prevPointer, to merge all partial updates into the passed
+ * doc. Stops at either a full document, or if there are no previous entries to follow in the
+ * update log.
*
- * @param id Binary representation of the unique key field
- * @param prevPointer Pointer to the previous entry in the ulog, based on which the current in-place update was made.
- * @param prevVersion Version of the previous entry in the ulog, based on which the current in-place update was made.
- * @param onlyTheseFields When a non-null set of field names is passed in, the resolve process only attempts to populate
- * the given fields in this set. When this set is null, it resolves all fields.
- * @param latestPartialDoc Partial document that is to be populated
- * @return Returns 0 if a full document was found in the log, -1 if no full document was found. If full document was supposed
- * to be found in the tlogs, but couldn't be found (because the logs were rotated) then the prevPointer is returned.
+ * @param id Binary representation of the unique key field
+ * @param prevPointer Pointer to the previous entry in the ulog, based on which the current
+ * in-place update was made.
+ * @param prevVersion Version of the previous entry in the ulog, based on which the current
+ * in-place update was made.
+ * @param onlyTheseFields When a non-null set of field names is passed in, the resolve process
+ * only attempts to populate the given fields in this set. When this set is null, it resolves
+ * all fields.
+ * @param latestPartialDoc Partial document that is to be populated
+ * @return Returns 0 if a full document was found in the log, -1 if no full document was found. If
+ * full document was supposed to be found in the tlogs, but couldn't be found (because the
+ * logs were rotated) then the prevPointer is returned.
*/
@SuppressWarnings({"unchecked"})
- synchronized public long applyPartialUpdates(BytesRef id, long prevPointer, long prevVersion,
- Set<String> onlyTheseFields, SolrDocumentBase<?, ?> latestPartialDoc) {
-
+ public synchronized long applyPartialUpdates(
+ BytesRef id,
+ long prevPointer,
+ long prevVersion,
+ Set<String> onlyTheseFields,
+ SolrDocumentBase<?, ?> latestPartialDoc) {
+
SolrInputDocument partialUpdateDoc = null;
List<TransactionLog> lookupLogs = Arrays.asList(tlog, prevMapLog, prevMapLog2);
while (prevPointer >= 0) {
- //go through each partial update and apply it on the incoming doc one after another
+ // go through each partial update and apply it on the incoming doc one after another
List<?> entry;
entry = getEntryFromTLog(prevPointer, prevVersion, lookupLogs);
if (entry == null) {
- return prevPointer; // a previous update was supposed to be found, but wasn't found (due to log rotation)
+ return prevPointer; // a previous update was supposed to be found, but wasn't found (due to
+ // log rotation)
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/UpdateLog.java
##########
@@ -373,27 +401,35 @@ public void init(UpdateHandler uhandler, SolrCore core) {
throw new SolrException(ErrorCode.SERVER_ERROR, "Could not set up tlogs", e);
}
tlogFiles = getLogList(tlogDir.toFile());
- id = getLastLogId() + 1; // add 1 since we will create a new log for the next update
+ id = getLastLogId() + 1; // add 1 since we will create a new log for the next update
if (debug) {
- log.debug("UpdateHandler init: tlogDir={}, existing tlogs={}, next id={}", tlogDir, Arrays.asList(tlogFiles), id);
+ log.debug(
+ "UpdateHandler init: tlogDir={}, existing tlogs={}, next id={}",
+ tlogDir,
+ Arrays.asList(tlogFiles),
+ id);
}
final String prefix = BUFFER_TLOG_NAME + '.';
try (Stream<Path> bufferedTLogs = Files.walk(tlogDir, 1)) {
- existOldBufferLog = bufferedTLogs.anyMatch(path -> path.getFileName().toString().startsWith(prefix));
+ existOldBufferLog =
+ bufferedTLogs.anyMatch(path -> path.getFileName().toString().startsWith(prefix));
} catch (IOException e) {
- // Existance of buffered t-logs indicates previous recovery failed and lets us skip peer sync as an optimization
+ // Existance of buffered t-logs indicates previous recovery failed and lets us skip peer sync
+ // as an optimization
// Failing to read them is non-fatal and almost not even worth logging about
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/UpdateLog.java
##########
@@ -875,47 +927,66 @@ public void postSoftCommit(CommitUpdateCommand cmd) {
// record what old maps were created and only remove those.
if (debug) {
- SolrCore.verbose("TLOG: postSoftCommit: disposing of prevMap="+ System.identityHashCode(prevMap) + ", prevMap2=" + System.identityHashCode(prevMap2));
+ SolrCore.verbose(
+ "TLOG: postSoftCommit: disposing of prevMap="
+ + System.identityHashCode(prevMap)
+ + ", prevMap2="
+ + System.identityHashCode(prevMap2));
}
clearOldMaps();
-
}
}
/**
- * Goes over backwards, following the prevPointer, to merge all partial updates into the passed doc. Stops at either a full
- * document, or if there are no previous entries to follow in the update log.
+ * Goes over backwards, following the prevPointer, to merge all partial updates into the passed
+ * doc. Stops at either a full document, or if there are no previous entries to follow in the
+ * update log.
*
- * @param id Binary representation of the unique key field
- * @param prevPointer Pointer to the previous entry in the ulog, based on which the current in-place update was made.
- * @param prevVersion Version of the previous entry in the ulog, based on which the current in-place update was made.
- * @param onlyTheseFields When a non-null set of field names is passed in, the resolve process only attempts to populate
- * the given fields in this set. When this set is null, it resolves all fields.
- * @param latestPartialDoc Partial document that is to be populated
- * @return Returns 0 if a full document was found in the log, -1 if no full document was found. If full document was supposed
- * to be found in the tlogs, but couldn't be found (because the logs were rotated) then the prevPointer is returned.
+ * @param id Binary representation of the unique key field
+ * @param prevPointer Pointer to the previous entry in the ulog, based on which the current
+ * in-place update was made.
+ * @param prevVersion Version of the previous entry in the ulog, based on which the current
+ * in-place update was made.
+ * @param onlyTheseFields When a non-null set of field names is passed in, the resolve process
+ * only attempts to populate the given fields in this set. When this set is null, it resolves
+ * all fields.
+ * @param latestPartialDoc Partial document that is to be populated
+ * @return Returns 0 if a full document was found in the log, -1 if no full document was found. If
+ * full document was supposed to be found in the tlogs, but couldn't be found (because the
+ * logs were rotated) then the prevPointer is returned.
*/
@SuppressWarnings({"unchecked"})
- synchronized public long applyPartialUpdates(BytesRef id, long prevPointer, long prevVersion,
- Set<String> onlyTheseFields, SolrDocumentBase<?, ?> latestPartialDoc) {
-
+ public synchronized long applyPartialUpdates(
+ BytesRef id,
+ long prevPointer,
+ long prevVersion,
+ Set<String> onlyTheseFields,
+ SolrDocumentBase<?, ?> latestPartialDoc) {
+
SolrInputDocument partialUpdateDoc = null;
List<TransactionLog> lookupLogs = Arrays.asList(tlog, prevMapLog, prevMapLog2);
while (prevPointer >= 0) {
- //go through each partial update and apply it on the incoming doc one after another
+ // go through each partial update and apply it on the incoming doc one after another
List<?> entry;
entry = getEntryFromTLog(prevPointer, prevVersion, lookupLogs);
if (entry == null) {
- return prevPointer; // a previous update was supposed to be found, but wasn't found (due to log rotation)
+ return prevPointer; // a previous update was supposed to be found, but wasn't found (due to
+ // log rotation)
}
int flags = (int) entry.get(UpdateLog.FLAGS_IDX);
-
- // since updates can depend only upon ADD updates or other UPDATE_INPLACE updates, we assert that we aren't
+
+ // since updates can depend only upon ADD updates or other UPDATE_INPLACE updates, we assert
+ // that we aren't
// getting something else
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/processor/AtomicUpdateDocumentMerger.java
##########
@@ -330,83 +342,108 @@ private static String getRouteField(AddUpdateCommand cmd) {
}
return result;
}
-
+
/**
- *
- * @param fullDoc the full doc to be compared against
+ * @param fullDoc the full doc to be compared against
* @param partialDoc the sub document to be tested
* @return whether partialDoc is derived from fullDoc
*/
public static boolean isDerivedFromDoc(SolrInputDocument fullDoc, SolrInputDocument partialDoc) {
- for(SolrInputField subSif: partialDoc) {
+ for (SolrInputField subSif : partialDoc) {
Collection<Object> fieldValues = fullDoc.getFieldValues(subSif.getName());
if (fieldValues == null) return false;
if (fieldValues.size() < subSif.getValueCount()) return false;
Collection<Object> partialFieldValues = subSif.getValues();
- // filter all derived child docs from partial field values since they fail List#containsAll check (uses SolrInputDocument#equals which fails).
- // If a child doc exists in partialDoc but not in full doc, it will not be filtered, and therefore List#containsAll will return false
- Stream<Object> nonChildDocElements = partialFieldValues.stream().filter(x -> !(isChildDoc(x) &&
- (fieldValues.stream().anyMatch(y ->
- (isChildDoc(x) &&
- isDerivedFromDoc((SolrInputDocument) y, (SolrInputDocument) x)
- )
- )
- )));
+ // filter all derived child docs from partial field values since they fail List#containsAll
+ // check (uses SolrInputDocument#equals which fails).
+ // If a child doc exists in partialDoc but not in full doc, it will not be filtered, and
+ // therefore List#containsAll will return false
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/processor/AtomicUpdateDocumentMerger.java
##########
@@ -330,83 +342,108 @@ private static String getRouteField(AddUpdateCommand cmd) {
}
return result;
}
-
+
/**
- *
- * @param fullDoc the full doc to be compared against
+ * @param fullDoc the full doc to be compared against
* @param partialDoc the sub document to be tested
* @return whether partialDoc is derived from fullDoc
*/
public static boolean isDerivedFromDoc(SolrInputDocument fullDoc, SolrInputDocument partialDoc) {
- for(SolrInputField subSif: partialDoc) {
+ for (SolrInputField subSif : partialDoc) {
Collection<Object> fieldValues = fullDoc.getFieldValues(subSif.getName());
if (fieldValues == null) return false;
if (fieldValues.size() < subSif.getValueCount()) return false;
Collection<Object> partialFieldValues = subSif.getValues();
- // filter all derived child docs from partial field values since they fail List#containsAll check (uses SolrInputDocument#equals which fails).
- // If a child doc exists in partialDoc but not in full doc, it will not be filtered, and therefore List#containsAll will return false
- Stream<Object> nonChildDocElements = partialFieldValues.stream().filter(x -> !(isChildDoc(x) &&
- (fieldValues.stream().anyMatch(y ->
- (isChildDoc(x) &&
- isDerivedFromDoc((SolrInputDocument) y, (SolrInputDocument) x)
- )
- )
- )));
+ // filter all derived child docs from partial field values since they fail List#containsAll
+ // check (uses SolrInputDocument#equals which fails).
+ // If a child doc exists in partialDoc but not in full doc, it will not be filtered, and
+ // therefore List#containsAll will return false
+ Stream<Object> nonChildDocElements =
+ partialFieldValues.stream()
+ .filter(
+ x ->
+ !(isChildDoc(x)
+ && (fieldValues.stream()
+ .anyMatch(
+ y ->
+ (isChildDoc(x)
+ && isDerivedFromDoc(
+ (SolrInputDocument) y, (SolrInputDocument) x))))));
if (!nonChildDocElements.allMatch(fieldValues::contains)) return false;
}
return true;
}
/**
- * Given an AddUpdateCommand containing update operations (e.g. set, inc), merge and resolve the operations into
- * a partial document that can be used for indexing the in-place updates. The AddUpdateCommand is modified to contain
- * the partial document (instead of the original document which contained the update operations) and also
- * the prevVersion that this in-place update depends on.
- * Note: updatedFields passed into the method can be changed, i.e. the version field can be added to the set.
- * @return If in-place update cannot succeed, e.g. if the old document is deleted recently, then false is returned. A false
- * return indicates that this update can be re-tried as a full atomic update. Returns true if the in-place update
- * succeeds.
+ * Given an AddUpdateCommand containing update operations (e.g. set, inc), merge and resolve the
+ * operations into a partial document that can be used for indexing the in-place updates. The
+ * AddUpdateCommand is modified to contain the partial document (instead of the original document
+ * which contained the update operations) and also the prevVersion that this in-place update
+ * depends on. Note: updatedFields passed into the method can be changed, i.e. the version field
+ * can be added to the set.
+ *
+ * @return If in-place update cannot succeed, e.g. if the old document is deleted recently, then
+ * false is returned. A false return indicates that this update can be re-tried as a full
+ * atomic update. Returns true if the in-place update succeeds.
*/
- public boolean doInPlaceUpdateMerge(AddUpdateCommand cmd, Set<String> updatedFields) throws IOException {
+ public boolean doInPlaceUpdateMerge(AddUpdateCommand cmd, Set<String> updatedFields)
+ throws IOException {
SolrInputDocument inputDoc = cmd.getSolrInputDocument();
BytesRef rootIdBytes = cmd.getIndexedId();
BytesRef idBytes = schema.indexableUniqueKey(cmd.getSelfOrNestedDocIdStr());
- updatedFields.add(CommonParams.VERSION_FIELD); // add the version field so that it is fetched too
- SolrInputDocument oldDocument = RealTimeGetComponent.getInputDocument
- (cmd.getReq().getCore(), idBytes, rootIdBytes,
- null, // don't want the version to be returned
- updatedFields, RealTimeGetComponent.Resolution.DOC);
+ updatedFields.add(
+ CommonParams.VERSION_FIELD); // add the version field so that it is fetched too
+ SolrInputDocument oldDocument =
+ RealTimeGetComponent.getInputDocument(
+ cmd.getReq().getCore(),
+ idBytes,
+ rootIdBytes,
+ null, // don't want the version to be returned
+ updatedFields,
+ RealTimeGetComponent.Resolution.DOC);
if (oldDocument == RealTimeGetComponent.DELETED || oldDocument == null) {
- // This doc was deleted recently. In-place update cannot work, hence a full atomic update should be tried.
+ // This doc was deleted recently. In-place update cannot work, hence a full atomic update
+ // should be tried.
return false;
}
if (oldDocument.containsKey(CommonParams.VERSION_FIELD) == false) {
- throw new SolrException (ErrorCode.INVALID_STATE, "There is no _version_ in previous document. id=" +
- cmd.getPrintableId());
+ throw new SolrException(
+ ErrorCode.INVALID_STATE,
+ "There is no _version_ in previous document. id=" + cmd.getPrintableId());
}
Long oldVersion = (Long) oldDocument.remove(CommonParams.VERSION_FIELD).getValue();
- // If the oldDocument contains any other field apart from updatedFields (or id/version field), then remove them.
- // This can happen, despite requesting for these fields in the call to RTGC.getInputDocument, if the document was
- // fetched from the tlog and had all these fields (possibly because it was a full document ADD operation).
+ // If the oldDocument contains any other field apart from updatedFields (or id/version field),
+ // then remove them.
+ // This can happen, despite requesting for these fields in the call to RTGC.getInputDocument, if
+ // the document was
+ // fetched from the tlog and had all these fields (possibly because it was a full document ADD
+ // operation).
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/UpdateLog.java
##########
@@ -1982,7 +2106,9 @@ public void doReplay(TransactionLog translog) {
cmd.setFlags(UpdateCommand.REPLAY);
try {
if (debug) log.debug("commit {}", cmd);
- uhandler.commit(cmd); // this should cause a commit to be added to the incomplete log and avoid it being replayed again after a restart.
+ uhandler.commit(
+ cmd); // this should cause a commit to be added to the incomplete log and avoid it
+ // being replayed again after a restart.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/UpdateLog.java
##########
@@ -1612,10 +1718,10 @@ public RecentUpdates getRecentUpdates() {
}
}
- // TODO: what if I hand out a list of updates, then do an update, then hand out another list (and
+ // TODO: what if I hand out a list of updates, then do an update, then hand out another list
+ // (and
// one of the updates I originally handed out fell off the list). Over-request?
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/processor/DistributedUpdateProcessor.java
##########
@@ -312,25 +320,29 @@ protected boolean versionAdd(AddUpdateCommand cmd) throws IOException {
SolrInputField versionField = cmd.getSolrInputDocument().getField(CommonParams.VERSION_FIELD);
if (versionField != null) {
Object o = versionField.getValue();
- versionOnUpdate = o instanceof Number ? ((Number) o).longValue() : Long.parseLong(o.toString());
+ versionOnUpdate =
+ o instanceof Number ? ((Number) o).longValue() : Long.parseLong(o.toString());
} else {
// Find the version
String versionOnUpdateS = req.getParams().get(CommonParams.VERSION_FIELD);
versionOnUpdate = versionOnUpdateS == null ? 0 : Long.parseLong(versionOnUpdateS);
}
}
- boolean isReplayOrPeersync = (cmd.getFlags() & (UpdateCommand.REPLAY | UpdateCommand.PEER_SYNC)) != 0;
+ boolean isReplayOrPeersync =
+ (cmd.getFlags() & (UpdateCommand.REPLAY | UpdateCommand.PEER_SYNC)) != 0;
boolean leaderLogic = isLeader && !isReplayOrPeersync;
boolean forwardedFromCollection = cmd.getReq().getParams().get(DISTRIB_FROM_COLLECTION) != null;
VersionBucket bucket = vinfo.bucket(bucketHash);
long dependentVersionFound = -1;
- // if this is an in-place update, check and wait if we should be waiting for a previous update (on which
+ // if this is an in-place update, check and wait if we should be waiting for a previous update
+ // (on which
// this update depends), before entering the synchronized block
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/processor/DistributedUpdateProcessor.java
##########
@@ -394,17 +424,26 @@ private boolean doVersionAdd(AddUpdateCommand cmd, long versionOnUpdate, boolean
if (versionOnUpdate != 0) {
Long lastVersion = vinfo.lookupVersion(cmd.getIndexedId());
long foundVersion = lastVersion == null ? -1 : lastVersion;
- if (versionOnUpdate == foundVersion || (versionOnUpdate < 0 && foundVersion < 0)
+ if (versionOnUpdate == foundVersion
+ || (versionOnUpdate < 0 && foundVersion < 0)
|| (versionOnUpdate == 1 && foundVersion > 0)) {
- // we're ok if versions match, or if both are negative (all missing docs are equal), or if cmd
+ // we're ok if versions match, or if both are negative (all missing docs are equal),
+ // or if cmd
// specified it must exist (versionOnUpdate==1) and it does.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/processor/DistributedUpdateProcessor.java
##########
@@ -427,30 +466,37 @@ private boolean doVersionAdd(AddUpdateCommand cmd, long versionOnUpdate, boolean
long prev = cmd.prevVersion;
Long lastVersion = vinfo.lookupVersion(cmd.getIndexedId());
if (lastVersion == null || Math.abs(lastVersion) < prev) {
- // this was checked for (in waitForDependentUpdates()) before entering the synchronized block.
+ // this was checked for (in waitForDependentUpdates()) before entering the
+ // synchronized block.
// So we shouldn't be here, unless what must've happened is:
- // by the time synchronization block was entered, the prev update was deleted by DBQ. Since
- // now that update is not in index, the vinfo.lookupVersion() is possibly giving us a version
+ // by the time synchronization block was entered, the prev update was deleted by DBQ.
+ // Since
+ // now that update is not in index, the vinfo.lookupVersion() is possibly giving us a
+ // version
// from the deleted list (which might be older than the prev update!)
UpdateCommand fetchedFromLeader = fetchFullUpdateFromLeader(cmd, versionOnUpdate);
if (fetchedFromLeader instanceof DeleteUpdateCommand) {
if (log.isInfoEnabled()) {
- log.info("In-place update of {} failed to find valid lastVersion to apply to, and the document was deleted at the leader subsequently."
- , idBytes.utf8ToString());
+ log.info(
+ "In-place update of {} failed to find valid lastVersion to apply to, and the document was deleted at the leader subsequently.",
+ idBytes.utf8ToString());
}
versionDelete((DeleteUpdateCommand) fetchedFromLeader);
return true;
} else {
assert fetchedFromLeader instanceof AddUpdateCommand;
- // Newer document was fetched from the leader. Apply that document instead of this current in-place
+ // Newer document was fetched from the leader. Apply that document instead of this
+ // current in-place
// update.
if (log.isInfoEnabled()) {
log.info(
"In-place update of {} failed to find valid lastVersion to apply to, forced to fetch full doc from leader: {}",
- idBytes.utf8ToString(), fetchedFromLeader);
+ idBytes.utf8ToString(),
+ fetchedFromLeader);
}
- // Make this update to become a non-inplace update containing the full document obtained from the
+ // Make this update to become a non-inplace update containing the full document
+ // obtained from the
// leader
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/processor/DistributedUpdateProcessor.java
##########
@@ -427,30 +466,37 @@ private boolean doVersionAdd(AddUpdateCommand cmd, long versionOnUpdate, boolean
long prev = cmd.prevVersion;
Long lastVersion = vinfo.lookupVersion(cmd.getIndexedId());
if (lastVersion == null || Math.abs(lastVersion) < prev) {
- // this was checked for (in waitForDependentUpdates()) before entering the synchronized block.
+ // this was checked for (in waitForDependentUpdates()) before entering the
+ // synchronized block.
// So we shouldn't be here, unless what must've happened is:
- // by the time synchronization block was entered, the prev update was deleted by DBQ. Since
- // now that update is not in index, the vinfo.lookupVersion() is possibly giving us a version
+ // by the time synchronization block was entered, the prev update was deleted by DBQ.
+ // Since
+ // now that update is not in index, the vinfo.lookupVersion() is possibly giving us a
+ // version
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/processor/DistributedUpdateProcessor.java
##########
@@ -427,30 +466,37 @@ private boolean doVersionAdd(AddUpdateCommand cmd, long versionOnUpdate, boolean
long prev = cmd.prevVersion;
Long lastVersion = vinfo.lookupVersion(cmd.getIndexedId());
if (lastVersion == null || Math.abs(lastVersion) < prev) {
- // this was checked for (in waitForDependentUpdates()) before entering the synchronized block.
+ // this was checked for (in waitForDependentUpdates()) before entering the
+ // synchronized block.
// So we shouldn't be here, unless what must've happened is:
- // by the time synchronization block was entered, the prev update was deleted by DBQ. Since
- // now that update is not in index, the vinfo.lookupVersion() is possibly giving us a version
+ // by the time synchronization block was entered, the prev update was deleted by DBQ.
+ // Since
+ // now that update is not in index, the vinfo.lookupVersion() is possibly giving us a
+ // version
// from the deleted list (which might be older than the prev update!)
UpdateCommand fetchedFromLeader = fetchFullUpdateFromLeader(cmd, versionOnUpdate);
if (fetchedFromLeader instanceof DeleteUpdateCommand) {
if (log.isInfoEnabled()) {
- log.info("In-place update of {} failed to find valid lastVersion to apply to, and the document was deleted at the leader subsequently."
- , idBytes.utf8ToString());
+ log.info(
+ "In-place update of {} failed to find valid lastVersion to apply to, and the document was deleted at the leader subsequently.",
+ idBytes.utf8ToString());
}
versionDelete((DeleteUpdateCommand) fetchedFromLeader);
return true;
} else {
assert fetchedFromLeader instanceof AddUpdateCommand;
- // Newer document was fetched from the leader. Apply that document instead of this current in-place
+ // Newer document was fetched from the leader. Apply that document instead of this
+ // current in-place
// update.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/processor/DistributedUpdateProcessor.java
##########
@@ -511,52 +562,66 @@ private boolean doVersionAdd(AddUpdateCommand cmd, long versionOnUpdate, boolean
}
/**
- *
* @return whether cmd doc should be cloned before localAdd
*/
protected boolean shouldCloneCmdDoc() {
return false;
}
@VisibleForTesting
- boolean shouldBufferUpdate(AddUpdateCommand cmd, boolean isReplayOrPeersync, UpdateLog.State state) {
+ boolean shouldBufferUpdate(
+ AddUpdateCommand cmd, boolean isReplayOrPeersync, UpdateLog.State state) {
if (state == UpdateLog.State.APPLYING_BUFFERED
&& !isReplayOrPeersync
&& !cmd.isInPlaceUpdate()) {
- // this a new update sent from the leader, it contains whole document therefore it won't depend on other updates
+ // this a new update sent from the leader, it contains whole document therefore it won't
+ // depend on other updates
return false;
}
return state != UpdateLog.State.ACTIVE && isReplayOrPeersync == false;
}
/**
- * This method checks the update/transaction logs and index to find out if the update ("previous update") that the current update
- * depends on (in the case that this current update is an in-place update) has already been completed. If not,
- * this method will wait for the missing update until it has arrived. If it doesn't arrive within a timeout threshold,
- * then this actively fetches from the leader.
- *
- * @return -1 if the current in-place should be dropped, or last found version if previous update has been indexed.
+ * This method checks the update/transaction logs and index to find out if the update ("previous
+ * update") that the current update depends on (in the case that this current update is an
+ * in-place update) has already been completed. If not, this method will wait for the missing
+ * update until it has arrived. If it doesn't arrive within a timeout threshold, then this
+ * actively fetches from the leader.
+ *
+ * @return -1 if the current in-place should be dropped, or last found version if previous update
+ * has been indexed.
*/
- private long waitForDependentUpdates(AddUpdateCommand cmd, long versionOnUpdate,
- boolean isReplayOrPeersync, VersionBucket bucket) throws IOException {
+ private long waitForDependentUpdates(
+ AddUpdateCommand cmd, long versionOnUpdate, boolean isReplayOrPeersync, VersionBucket bucket)
+ throws IOException {
long lastFoundVersion = 0;
TimeOut waitTimeout = new TimeOut(5, TimeUnit.SECONDS, TimeSource.NANO_TIME);
vinfo.lockForUpdate();
try {
- lastFoundVersion = bucket.runWithLock(vinfo.getVersionBucketLockTimeoutMs(), () -> doWaitForDependentUpdates(cmd, versionOnUpdate, isReplayOrPeersync, bucket, waitTimeout));
+ lastFoundVersion =
+ bucket.runWithLock(
+ vinfo.getVersionBucketLockTimeoutMs(),
+ () ->
+ doWaitForDependentUpdates(
+ cmd, versionOnUpdate, isReplayOrPeersync, bucket, waitTimeout));
} finally {
vinfo.unlockForUpdate();
}
if (Math.abs(lastFoundVersion) > cmd.prevVersion) {
- // This must've been the case due to a higher version full update succeeding concurrently, while we were waiting or
- // trying to index this partial update. Since a full update more recent than this partial update has succeeded,
+ // This must've been the case due to a higher version full update succeeding concurrently,
+ // while we were waiting or
+ // trying to index this partial update. Since a full update more recent than this partial
+ // update has succeeded,
// we can drop the current update.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/processor/DistributedUpdateProcessor.java
##########
@@ -459,10 +505,13 @@ private boolean doVersionAdd(AddUpdateCommand cmd, long versionOnUpdate, boolean
}
} else {
if (lastVersion != null && Math.abs(lastVersion) > prev) {
- // this means we got a newer full doc update and in that case it makes no sense to apply the older
+ // this means we got a newer full doc update and in that case it makes no sense to
+ // apply the older
// inplace update. Drop this update
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/processor/DistributedUpdateProcessor.java
##########
@@ -949,7 +1067,8 @@ protected boolean versionDelete(DeleteUpdateCommand cmd) throws IOException {
return false;
}
- // This is only the hash for the bucket, and must be based only on the uniqueKey (i.e. do not use a pluggable hash
+ // This is only the hash for the bucket, and must be based only on the uniqueKey (i.e. do not
+ // use a pluggable hash
// here)
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/processor/DistributedUpdateProcessor.java
##########
@@ -744,17 +839,20 @@ public void processDelete(DeleteUpdateCommand cmd) throws IOException {
}
}
- // Implementing min_rf here was a bit tricky. When a request comes in for a delete by id to a replica that does _not_
- // have any documents specified by those IDs, the request is not forwarded to any other replicas on that shard. Thus
- // we have to spoof the replicationTracker and set the achieved rf to the number of active replicas.
+ // Implementing min_rf here was a bit tricky. When a request comes in for a delete by id to a
+ // replica that does _not_
+ // have any documents specified by those IDs, the request is not forwarded to any other replicas
+ // on that shard. Thus
+ // we have to spoof the replicationTracker and set the achieved rf to the number of active
+ // replicas.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/processor/DistributedUpdateProcessor.java
##########
@@ -1201,35 +1352,43 @@ private static String buildMsg(List<Error> errors) {
}
}
-
// Keeps track of the replication factor achieved for a distributed update request
- // originated in this distributed update processor. A RollupReplicationTracker is the only tracker that will
+ // originated in this distributed update processor. A RollupReplicationTracker is the only
+ // tracker that will
// persist across sub-requests.
//
- // Note that the replica that receives the original request has the only RollupReplicationTracker that exists for the
- // lifetime of the batch. The leader for each shard keeps track of its own achieved replication for its shard
- // and attaches that to the response to the originating node (i.e. the one with the RollupReplicationTracker).
- // Followers in general do not need a tracker of any sort with the sole exception of the RollupReplicationTracker
+ // Note that the replica that receives the original request has the only
+ // RollupReplicationTracker that exists for the
+ // lifetime of the batch. The leader for each shard keeps track of its own achieved replication
+ // for its shard
+ // and attaches that to the response to the originating node (i.e. the one with the
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/processor/DistributedUpdateProcessor.java
##########
@@ -877,7 +991,8 @@ protected void versionDeleteByQuery(DeleteUpdateCommand cmd) throws IOException
doLocalDeleteByQuery(cmd, versionOnUpdate, isReplayOrPeersync);
// since we don't know which documents were deleted, the easiest thing to do is to invalidate
- // all real-time caches (i.e. UpdateLog) which involves also getting a new version of the IndexReader
+ // all real-time caches (i.e. UpdateLog) which involves also getting a new version of the
+ // IndexReader
// (so cache misses will see up-to-date data)
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/processor/DistributedUpdateProcessor.java
##########
@@ -1019,13 +1156,21 @@ private boolean doVersionDelete(DeleteUpdateCommand cmd, long versionOnUpdate, l
if (signedVersionOnUpdate != 0) {
Long lastVersion = vinfo.lookupVersion(cmd.getIndexedId());
long foundVersion = lastVersion == null ? -1 : lastVersion;
- if ((signedVersionOnUpdate == foundVersion) || (signedVersionOnUpdate < 0 && foundVersion < 0)
+ if ((signedVersionOnUpdate == foundVersion)
+ || (signedVersionOnUpdate < 0 && foundVersion < 0)
|| (signedVersionOnUpdate == 1 && foundVersion > 0)) {
- // we're ok if versions match, or if both are negative (all missing docs are equal), or if cmd
+ // we're ok if versions match, or if both are negative (all missing docs are equal),
+ // or if cmd
// specified it must exist (versionOnUpdate==1) and it does.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/processor/DistributedUpdateProcessor.java
##########
@@ -769,28 +867,34 @@ protected void doDeleteById(DeleteUpdateCommand cmd) throws IOException {
if (returnVersions && rsp != null && cmd.getIndexedId() != null && idField != null) {
if (deleteResponse == null) {
deleteResponse = new NamedList<>(1);
- rsp.add("deletes",deleteResponse);
+ rsp.add("deletes", deleteResponse);
}
if (scratch == null) scratch = new CharsRefBuilder();
idField.getType().indexedToReadable(cmd.getIndexedId(), scratch);
- deleteResponse.add(scratch.toString(), cmd.getVersion()); // we're returning the version of the delete.. not the version of the doc we deleted.
+ deleteResponse.add(
+ scratch.toString(),
+ cmd.getVersion()); // we're returning the version of the delete.. not the version of the
+ // doc we deleted.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/processor/DistributedZkUpdateProcessor.java
##########
@@ -86,14 +85,18 @@
private boolean readOnlyCollection = false;
private boolean broadcastDeleteById = false;
- // The cached immutable clusterState for the update... usually refreshed for each individual update.
- // Different parts of this class used to request current clusterState views, which lead to subtle bugs and race conditions
- // such as SOLR-13815 (live split data loss.) Most likely, the only valid reasons for updating clusterState should be on
+ // The cached immutable clusterState for the update... usually refreshed for each individual
+ // update.
+ // Different parts of this class used to request current clusterState views, which lead to subtle
+ // bugs and race conditions
+ // such as SOLR-13815 (live split data loss.) Most likely, the only valid reasons for updating
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/processor/DistributedUpdateProcessor.java
##########
@@ -1244,25 +1403,29 @@ public void testAndSetAchievedRf(int rf) {
}
public String toString() {
- StringBuilder sb = new StringBuilder("RollupRequestReplicationTracker")
- .append(" achievedRf: ")
- .append(achievedRf);
+ StringBuilder sb =
+ new StringBuilder("RollupRequestReplicationTracker")
+ .append(" achievedRf: ")
+ .append(achievedRf);
return sb.toString();
}
}
-
- // Allocate a LeaderRequestReplicatinTracker if (and only if) we're a leader. If the request comes in to the leader
+ // Allocate a LeaderRequestReplicatinTracker if (and only if) we're a leader. If the request comes
+ // in to the leader
// at first, allocate both one of these and a RollupRequestReplicationTracker.
//
- // Since these are leader-only, all they really have to do is track the individual update request for this shard
- // and return it to be added to the rollup tracker. Which is kind of simple since we get an onSuccess method in
+ // Since these are leader-only, all they really have to do is track the individual update request
+ // for this shard
+ // and return it to be added to the rollup tracker. Which is kind of simple since we get an
+ // onSuccess method in
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/processor/DistributedUpdateProcessor.java
##########
@@ -1244,25 +1403,29 @@ public void testAndSetAchievedRf(int rf) {
}
public String toString() {
- StringBuilder sb = new StringBuilder("RollupRequestReplicationTracker")
- .append(" achievedRf: ")
- .append(achievedRf);
+ StringBuilder sb =
+ new StringBuilder("RollupRequestReplicationTracker")
+ .append(" achievedRf: ")
+ .append(achievedRf);
return sb.toString();
}
}
-
- // Allocate a LeaderRequestReplicatinTracker if (and only if) we're a leader. If the request comes in to the leader
+ // Allocate a LeaderRequestReplicatinTracker if (and only if) we're a leader. If the request comes
+ // in to the leader
// at first, allocate both one of these and a RollupRequestReplicationTracker.
//
- // Since these are leader-only, all they really have to do is track the individual update request for this shard
- // and return it to be added to the rollup tracker. Which is kind of simple since we get an onSuccess method in
+ // Since these are leader-only, all they really have to do is track the individual update request
+ // for this shard
+ // and return it to be added to the rollup tracker. Which is kind of simple since we get an
+ // onSuccess method in
// SolrCmdDistributor
public static class LeaderRequestReplicationTracker {
private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
- // Since we only allocate one of these on the leader and, by definition, the leader has been found and is running,
+ // Since we only allocate one of these on the leader and, by definition, the leader has been
+ // found and is running,
// we have a replication factor of one by default.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/processor/DistributedZkUpdateProcessor.java
##########
@@ -203,17 +217,20 @@ public void processCommit(CommitUpdateCommand cmd) throws IOException {
params.set(DISTRIB_UPDATE_PARAM, DistribPhase.TOLEADER.toString());
params.set(COMMIT_END_POINT, "leaders");
if (useNodes != null) {
- params.set(DISTRIB_FROM, ZkCoreNodeProps.getCoreUrl(
- zkController.getBaseUrl(), req.getCore().getName()));
+ params.set(
+ DISTRIB_FROM,
+ ZkCoreNodeProps.getCoreUrl(zkController.getBaseUrl(), req.getCore().getName()));
cmdDistrib.distribCommit(cmd, useNodes, params);
issuedDistribCommit = true;
}
}
if (isLeader) {
if (issuedDistribCommit) {
- // defensive copy of params, which was passed into distribCommit(...) above; will unconditionally replace
- // DISTRIB_UPDATE_PARAM, COMMIT_END_POINT, and DISTRIB_FROM if the new `params` val will actually be used
+ // defensive copy of params, which was passed into distribCommit(...) above; will
+ // unconditionally replace
+ // DISTRIB_UPDATE_PARAM, COMMIT_END_POINT, and DISTRIB_FROM if the new `params` val will
+ // actually be used
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/processor/DistributedZkUpdateProcessor.java
##########
@@ -223,22 +240,29 @@ public void processCommit(CommitUpdateCommand cmd) throws IOException {
useNodes = getReplicaNodesForLeader(cloudDesc.getShardId(), leaderReplica);
if (useNodes != null) {
- params.set(DISTRIB_FROM, ZkCoreNodeProps.getCoreUrl(
- zkController.getBaseUrl(), req.getCore().getName()));
+ params.set(
+ DISTRIB_FROM,
+ ZkCoreNodeProps.getCoreUrl(zkController.getBaseUrl(), req.getCore().getName()));
- // NOTE: distribCommit(...) internally calls `blockAndDoRetries()`, flushing any TOLEADER distrib commits
+ // NOTE: distribCommit(...) internally calls `blockAndDoRetries()`, flushing any TOLEADER
+ // distrib commits
cmdDistrib.distribCommit(cmd, useNodes, params);
issuedDistribCommit = true;
}
doLocalCommit(cmd);
}
if (issuedDistribCommit) {
- // TODO: according to discussion on SOLR-15045, this call (and all tracking of `issuedDistribCommit`) may
- // well be superfluous, and can probably simply be removed. It is left in place for now, intentionally
- // punting on the question of whether this internal `blockAndDoRetries()` is necessary. At worst, its
- // presence is misleading; but it should be harmless, and allows the change fixing SOLR-15045 to be as
- // tightly scoped as possible, leaving the behavior of the code otherwise functionally equivalent (for
+ // TODO: according to discussion on SOLR-15045, this call (and all tracking of
+ // `issuedDistribCommit`) may
+ // well be superfluous, and can probably simply be removed. It is left in place for now,
+ // intentionally
+ // punting on the question of whether this internal `blockAndDoRetries()` is necessary. At
+ // worst, its
+ // presence is misleading; but it should be harmless, and allows the change fixing
+ // SOLR-15045 to be as
+ // tightly scoped as possible, leaving the behavior of the code otherwise functionally
+ // equivalent (for
// better or worse!)
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/processor/DistributedZkUpdateProcessor.java
##########
@@ -257,7 +281,8 @@ public void processAdd(AddUpdateCommand cmd) throws IOException {
setupRequest(cmd);
- // check if client has requested minimum replication factor information. will set replicationTracker to null if
+ // check if client has requested minimum replication factor information. will set
+ // replicationTracker to null if
// we aren't the leader or subShardLeader
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/processor/DistributedZkUpdateProcessor.java
##########
@@ -425,37 +473,43 @@ protected void doDeleteByQuery(DeleteUpdateCommand cmd) throws IOException {
isLeader = false;
}
- // check if client has requested minimum replication factor information. will set replicationTracker to null if
+ // check if client has requested minimum replication factor information. will set
+ // replicationTracker to null if
// we aren't the leader or subShardLeader
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/processor/DistributedZkUpdateProcessor.java
##########
@@ -331,15 +366,20 @@ public void processDelete(DeleteUpdateCommand cmd) throws IOException {
protected void doDeleteById(DeleteUpdateCommand cmd) throws IOException {
setupRequest(cmd);
- if (broadcastDeleteById && DistribPhase.NONE == DistribPhase.parseParam(req.getParams().get(DISTRIB_UPDATE_PARAM))) {
+ if (broadcastDeleteById
+ && DistribPhase.NONE
+ == DistribPhase.parseParam(req.getParams().get(DISTRIB_UPDATE_PARAM))) {
DocCollection coll = clusterState.getCollection(collection);
if (log.isDebugEnabled()) {
- log.debug("The deleteById command for doc {} is missing the required route, broadcasting to leaders of other shards", cmd.getId());
+ log.debug(
+ "The deleteById command for doc {} is missing the required route, broadcasting to leaders of other shards",
+ cmd.getId());
}
forwardDelete(coll, cmd);
}
- // check if client has requested minimum replication factor information. will set replicationTracker to null if
+ // check if client has requested minimum replication factor information. will set
+ // replicationTracker to null if
// we aren't the leader or subShardLeader
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/processor/DistributedZkUpdateProcessor.java
##########
@@ -754,37 +880,41 @@ protected boolean shouldCloneCmdDoc() {
// Sets replicationTracker = null if we aren't the leader
// We have two possibilities here:
//
- // 1> we are a leader: Allocate a LeaderTracker and, if we're getting the original request, a RollupTracker
+ // 1> we are a leader: Allocate a LeaderTracker and, if we're getting the original request, a
+ // RollupTracker
// 2> we're a follower: allocat a RollupTracker
//
private void checkReplicationTracker(UpdateCommand cmd) {
SolrParams rp = cmd.getReq().getParams();
String distribUpdate = rp.get(DISTRIB_UPDATE_PARAM);
- // Ok,we're receiving the original request, we need a rollup tracker, but only one so we accumulate over the
+ // Ok,we're receiving the original request, we need a rollup tracker, but only one so we
+ // accumulate over the
// course of a batch.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/processor/DocBasedVersionConstraintsProcessor.java
##########
@@ -64,20 +64,22 @@
private final SolrCore core;
private final NamedList<Object> tombstoneConfig;
- private final DistributedUpdateProcessor distribProc; // the distributed update processor following us
+ private final DistributedUpdateProcessor
+ distribProc; // the distributed update processor following us
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/processor/DistributedZkUpdateProcessor.java
##########
@@ -655,7 +768,9 @@ void setupRequest(UpdateCommand cmd) {
assert TestInjection.injectFailReplicaRequests();
- isLeader = false; // we actually might be the leader, but we don't want leader-logic for these types of updates anyway.
+ isLeader =
+ false; // we actually might be the leader, but we don't want leader-logic for these
+ // types of updates anyway.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/processor/DistributedZkUpdateProcessor.java
##########
@@ -602,25 +702,34 @@ void setupRequest(UpdateCommand cmd) {
updateCommand = cmd;
zkCheck();
if (cmd instanceof AddUpdateCommand) {
- AddUpdateCommand acmd = (AddUpdateCommand)cmd;
+ AddUpdateCommand acmd = (AddUpdateCommand) cmd;
nodes = setupRequest(acmd.getIndexedIdStr(), acmd.getSolrInputDocument());
} else if (cmd instanceof DeleteUpdateCommand) {
- DeleteUpdateCommand dcmd = (DeleteUpdateCommand)cmd;
- nodes = setupRequest(dcmd.getId(), null, (null != dcmd.getRoute() ? dcmd.getRoute() : req.getParams().get(ShardParams._ROUTE_)) );
+ DeleteUpdateCommand dcmd = (DeleteUpdateCommand) cmd;
+ nodes =
+ setupRequest(
+ dcmd.getId(),
+ null,
+ (null != dcmd.getRoute()
+ ? dcmd.getRoute()
+ : req.getParams().get(ShardParams._ROUTE_)));
}
}
protected List<SolrCmdDistributor.Node> setupRequest(String id, SolrInputDocument doc) {
return setupRequest(id, doc, null);
}
- protected List<SolrCmdDistributor.Node> setupRequest(String id, SolrInputDocument doc, String route) {
+ protected List<SolrCmdDistributor.Node> setupRequest(
+ String id, SolrInputDocument doc, String route) {
// if we are in zk mode...
assert TestInjection.injectUpdateRandomPause();
if ((updateCommand.getFlags() & (UpdateCommand.REPLAY | UpdateCommand.PEER_SYNC)) != 0) {
- isLeader = false; // we actually might be the leader, but we don't want leader-logic for these types of updates anyway.
+ isLeader =
+ false; // we actually might be the leader, but we don't want leader-logic for these types
+ // of updates anyway.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/processor/DistributedZkUpdateProcessor.java
##########
@@ -576,24 +666,34 @@ protected void doDistribDeleteByQuery(DeleteUpdateCommand cmd, List<SolrCmdDistr
protected String getLeaderUrl(String id) {
// try get leader from req params, fallback to zk lookup if not found.
String distribFrom = req.getParams().get(DISTRIB_FROM);
- if(distribFrom != null) {
+ if (distribFrom != null) {
return distribFrom;
}
return getLeaderUrlZk(id);
}
private String getLeaderUrlZk(String id) {
- // An update we're dependent upon didn't arrive! This is unexpected. Perhaps likely our leader is
- // down or partitioned from us for some reason. Lets force refresh cluster state, and request the
+ // An update we're dependent upon didn't arrive! This is unexpected. Perhaps likely our leader
+ // is
+ // down or partitioned from us for some reason. Lets force refresh cluster state, and request
+ // the
// leader for the update.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/processor/DistributedZkUpdateProcessor.java
##########
@@ -754,37 +880,41 @@ protected boolean shouldCloneCmdDoc() {
// Sets replicationTracker = null if we aren't the leader
// We have two possibilities here:
//
- // 1> we are a leader: Allocate a LeaderTracker and, if we're getting the original request, a RollupTracker
+ // 1> we are a leader: Allocate a LeaderTracker and, if we're getting the original request, a
+ // RollupTracker
// 2> we're a follower: allocat a RollupTracker
//
private void checkReplicationTracker(UpdateCommand cmd) {
SolrParams rp = cmd.getReq().getParams();
String distribUpdate = rp.get(DISTRIB_UPDATE_PARAM);
- // Ok,we're receiving the original request, we need a rollup tracker, but only one so we accumulate over the
+ // Ok,we're receiving the original request, we need a rollup tracker, but only one so we
+ // accumulate over the
// course of a batch.
- if ((distribUpdate == null || DistribPhase.NONE.toString().equals(distribUpdate)) &&
- rollupReplicationTracker == null) {
+ if ((distribUpdate == null || DistribPhase.NONE.toString().equals(distribUpdate))
+ && rollupReplicationTracker == null) {
rollupReplicationTracker = new RollupRequestReplicationTracker();
}
- // If we're a leader, we need a leader replication tracker, so let's do that. If there are multiple docs in
+ // If we're a leader, we need a leader replication tracker, so let's do that. If there are
+ // multiple docs in
// a batch we need to use the _same_ leader replication tracker.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/processor/DistributedZkUpdateProcessor.java
##########
@@ -1125,23 +1302,31 @@ protected void doDistribFinish() {
// recover - the doc was already added locally, so it should have been
// legit
- DistribPhase phase = DistribPhase.parseParam(error.req.uReq.getParams().get(DISTRIB_UPDATE_PARAM));
+ DistribPhase phase =
+ DistribPhase.parseParam(error.req.uReq.getParams().get(DISTRIB_UPDATE_PARAM));
if (phase != DistribPhase.FROMLEADER)
continue; // don't have non-leaders try to recovery other nodes
- // commits are special -- they can run on any node irrespective of whether it is a leader or not
+ // commits are special -- they can run on any node irrespective of whether it is a leader or
+ // not
// we don't want to run recovery on a node which missed a commit command
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/processor/TolerantUpdateProcessor.java
##########
@@ -164,22 +154,26 @@ public void processAdd(AddUpdateCommand cmd) throws IOException {
@Override
public void processDelete(DeleteUpdateCommand cmd) throws IOException {
-
+
try {
super.processDelete(cmd);
} catch (Throwable t) {
firstErrTracker.caught(t);
- ToleratedUpdateError err = new ToleratedUpdateError(cmd.isDeleteById() ? CmdType.DELID : CmdType.DELQ,
- cmd.isDeleteById() ? cmd.id : cmd.query,
- t.getMessage());
+ ToleratedUpdateError err =
+ new ToleratedUpdateError(
+ cmd.isDeleteById() ? CmdType.DELID : CmdType.DELQ,
+ cmd.isDeleteById() ? cmd.id : cmd.query,
+ t.getMessage());
knownErrors.add(err);
// NOTE: we're not using this to dedup before adding to knownErrors.
- // if we're lucky enough to get an immediate local failure (ie: we're a leader, or some other processor
- // failed) then recording the multiple failures is a good thing -- helps us with an accurate fail
+ // if we're lucky enough to get an immediate local failure (ie: we're a leader, or some other
+ // processor
+ // failed) then recording the multiple failures is a good thing -- helps us with an accurate
+ // fail
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/processor/RoutedAliasUpdateProcessor.java
##########
@@ -72,7 +72,8 @@
private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
// make sure we don't request collection properties any more frequently than once a minute during
- // slow continuous indexing, and even less frequently during bulk indexing. (cache is updated by zk
+ // slow continuous indexing, and even less frequently during bulk indexing. (cache is updated by
+ // zk
// watch instead of re-requested until indexing has been stopped for the duration specified here)
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/processor/DocBasedVersionConstraintsProcessor.java
##########
@@ -397,22 +427,23 @@ public void processAdd(AddUpdateCommand cmd) throws IOException {
}
try {
- cmd.setVersion(oldSolrVersion); // use optimistic concurrency to ensure that the doc has not changed in the meantime
+ cmd.setVersion(
+ oldSolrVersion); // use optimistic concurrency to ensure that the doc has not changed in
+ // the meantime
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/processor/DocBasedVersionConstraintsProcessor.java
##########
@@ -464,42 +499,58 @@ public void processDelete(DeleteUpdateCommand cmd) throws IOException {
// drop the delete, and instead propagate an AddDoc that
// replaces the doc with a new "empty" one that records the deleted version
- SolrInputDocument newDoc = createTombstoneDocument(core.getLatestSchema(), cmd.getId(), versionFieldNames, deleteParamValues, this.tombstoneConfig);
+ SolrInputDocument newDoc =
+ createTombstoneDocument(
+ core.getLatestSchema(),
+ cmd.getId(),
+ versionFieldNames,
+ deleteParamValues,
+ this.tombstoneConfig);
AddUpdateCommand newCmd = new AddUpdateCommand(cmd.getReq());
newCmd.solrDoc = newDoc;
newCmd.commitWithin = cmd.commitWithin;
- newCmd.setVersion(oldSolrVersion); // use optimistic concurrency to ensure that the doc has not changed in the meantime
+ newCmd.setVersion(
+ oldSolrVersion); // use optimistic concurrency to ensure that the doc has not changed in
+ // the meantime
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/processor/TolerantUpdateProcessor.java
##########
@@ -231,23 +225,25 @@ public void processRollback(RollbackUpdateCommand cmd) throws IOException {
public void finish() throws IOException {
// even if processAdd threw an error, this.finish() is still called and we might have additional
- // errors from other remote leaders that we need to check for from the finish method of downstream processors
+ // errors from other remote leaders that we need to check for from the finish method of
+ // downstream processors
// (like DUP)
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/processor/TolerantUpdateProcessor.java
##########
@@ -292,19 +289,21 @@ public void finish() throws IOException {
// decide if we have hit a situation where we know an error needs to be thrown.
if ((DistribPhase.TOLEADER.equals(distribPhase) ? 0 : maxErrors) < knownErrors.size()) {
- // NOTE: even if maxErrors wasn't exceeded, we need to throw an error when we have any errors if we're
- // a leader that was forwarded to by another node so that the forwarding node knows we encountered some
+ // NOTE: even if maxErrors wasn't exceeded, we need to throw an error when we have any errors
+ // if we're
+ // a leader that was forwarded to by another node so that the forwarding node knows we
+ // encountered some
// problems and can aggregate the results
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/processor/TolerantUpdateProcessor.java
##########
@@ -231,23 +225,25 @@ public void processRollback(RollbackUpdateCommand cmd) throws IOException {
public void finish() throws IOException {
// even if processAdd threw an error, this.finish() is still called and we might have additional
- // errors from other remote leaders that we need to check for from the finish method of downstream processors
+ // errors from other remote leaders that we need to check for from the finish method of
+ // downstream processors
// (like DUP)
try {
super.finish();
} catch (DistributedUpdateProcessor.DistributedUpdatesAsyncException duae) {
firstErrTracker.caught(duae);
-
// adjust our stats based on each of the distributed errors
for (Error error : duae.errors) {
- // we can't trust the req info from the Error, because multiple original requests might have been
+ // we can't trust the req info from the Error, because multiple original requests might have
+ // been
// lumped together
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/util/ConcurrentLRUCache.java
##########
@@ -366,23 +416,26 @@ private void markAndSweepByIdleTime() {
}
/*
- Must be called after acquiring markAndSweepLock
- */
+ Must be called after acquiring markAndSweepLock
+ */
private void markAndSweepByRamSize() {
assert markAndSweepLock.isHeldByCurrentThread() : "markAndSweepLock held by another thread";
List<CacheEntry<K, V>> entriesInAccessOrder = new ArrayList<>(map.size());
- map.forEach((o, kvCacheEntry) -> {
- kvCacheEntry.lastAccessedCopy = kvCacheEntry.lastAccessed; // important because we want to avoid volatile read during comparisons
- entriesInAccessOrder.add(kvCacheEntry);
- });
+ map.forEach(
+ (o, kvCacheEntry) -> {
+ kvCacheEntry.lastAccessedCopy =
+ kvCacheEntry.lastAccessed; // important because we want to avoid volatile read during
+ // comparisons
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/util/FileUtils.java
##########
@@ -94,17 +87,20 @@ public static boolean fileExists(String filePathString) {
}
// Files.createDirectories has odd behavior if the path is a symlink and it already exists
- // _even if it's a symlink to a directory_.
- //
- // oddly, if the path to be created just contains a symlink in intermediate levels, Files.createDirectories
+ // _even if it's a symlink to a directory_.
+ //
+ // oddly, if the path to be created just contains a symlink in intermediate levels,
+ // Files.createDirectories
// works just fine.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/processor/DistributedZkUpdateProcessor.java
##########
@@ -630,22 +739,26 @@ void setupRequest(UpdateCommand cmd) {
Slice slice = coll.getRouter().getTargetSlice(id, doc, route, req.getParams(), coll);
if (slice == null) {
- // No slice found. Most strict routers will have already thrown an exception, so a null return is
+ // No slice found. Most strict routers will have already thrown an exception, so a null
+ // return is
// a signal to use the slice of this core.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/update/processor/TolerantUpdateProcessor.java
##########
@@ -381,12 +382,14 @@ public void annotate(List<ToleratedUpdateError> errors) {
firstErrMetadata = new NamedList<String>();
first.setMetadata(firstErrMetadata);
} else {
- // any existing metadata representing ToleratedUpdateErrors in this single exception needs removed
+ // any existing metadata representing ToleratedUpdateErrors in this single exception needs
+ // removed
// so we can add *all* of the known ToleratedUpdateErrors (from this and other exceptions)
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/util/LongSet.java
##########
@@ -69,7 +73,8 @@ public boolean add(long val) {
// and bit 52 for double precision.
// Many values will only have significant bits just to the right of that.
- // For now, lets just settle to get first 8 significant mantissa bits of double or float in the lowest bits of our hash
+ // For now, lets just settle to get first 8 significant mantissa bits of double or float in the
+ // lowest bits of our hash
// The upper bits of our hash will be irrelevant.
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/util/PackageTool.java
##########
@@ -234,92 +283,92 @@ protected void runImpl(CommandLine cli) throws Exception {
log.info("Finished: {}", cmd);
} catch (Exception ex) {
- ex.printStackTrace(); // We need to print this since SolrCLI drops the stack trace in favour of brevity. Package tool should surely print full stacktraces!
+ ex.printStackTrace(); // We need to print this since SolrCLI drops the stack trace in favour
+ // of brevity. Package tool should surely print full stacktraces!
Review comment:
Fix this
##########
File path: solr/core/src/java/org/apache/solr/util/SolrCLI.java
##########
@@ -3092,30 +3328,43 @@ protected void waitToSeeLiveNodes(int maxWaitSecs, String zkHost, int numNodes)
// don't display a huge path for solr home if it is relative to the cwd
if (!isWindows && cwdPath.length() > 1 && solrHome.startsWith(cwdPath))
- solrHome = solrHome.substring(cwdPath.length()+1);
+ solrHome = solrHome.substring(cwdPath.length() + 1);
String startCmd =
- String.format(Locale.ROOT, "\"%s\" start %s -p %d -s \"%s\" %s %s %s %s %s %s",
- callScript, cloudModeArg, port, solrHome, hostArg, zkHostArg, memArg, forceArg, extraArgs, addlOptsArg);
+ String.format(
+ Locale.ROOT,
+ "\"%s\" start %s -p %d -s \"%s\" %s %s %s %s %s %s",
+ callScript,
+ cloudModeArg,
+ port,
+ solrHome,
+ hostArg,
+ zkHostArg,
+ memArg,
+ forceArg,
+ extraArgs,
+ addlOptsArg);
startCmd = startCmd.replaceAll("\\s+", " ").trim(); // for pretty printing
echo("\nStarting up Solr on port " + port + " using command:");
echo(startCmd + "\n");
String solrUrl =
- String.format(Locale.ROOT, "%s://%s:%d/solr", urlScheme, (host != null ? host : "localhost"), port);
+ String.format(
+ Locale.ROOT, "%s://%s:%d/solr", urlScheme, (host != null ? host : "localhost"), port);
- Map<String,Object> nodeStatus = checkPortConflict(solrUrl, solrHomeDir, port, cli);
+ Map<String, Object> nodeStatus = checkPortConflict(solrUrl, solrHomeDir, port, cli);
if (nodeStatus != null)
return nodeStatus; // the server they are trying to start is already running
int code = 0;
if (isWindows) {
// On Windows, the execution doesn't return, so we have to execute async
- // and when calling the script, it seems to be inheriting the environment that launched this app
+ // and when calling the script, it seems to be inheriting the environment that launched this
+ // app
// so we have to prune out env vars that may cause issues
Review comment:
Fix this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] risdenk commented on a change in pull request #705: SOLR-14920: Spotless formatting for core - non-test only
Posted by GitBox <gi...@apache.org>.
risdenk commented on a change in pull request #705:
URL: https://github.com/apache/solr/pull/705#discussion_r819026369
##########
File path: solr/core/src/java/org/apache/solr/parser/QueryParser.java
##########
@@ -8,25 +8,25 @@
import java.util.HashSet;
import java.util.List;
import java.util.Set;
-
import org.apache.lucene.analysis.Analyzer;
+import org.apache.lucene.queryparser.charstream.CharStream;
+import org.apache.lucene.queryparser.charstream.FastCharStream;
import org.apache.lucene.search.BooleanClause;
import org.apache.lucene.search.Query;
-import org.apache.solr.search.SyntaxError;
import org.apache.solr.search.QParser;
-
-import org.apache.lucene.queryparser.charstream.CharStream;
-import org.apache.lucene.queryparser.charstream.FastCharStream;
+import org.apache.solr.search.SyntaxError;
public class QueryParser extends SolrQueryParserBase implements QueryParserConstants {
Review comment:
I don't really agree, but not worth arguing over. Fixed in fade0a75a06504e9d50ba21314fa2f35ff2a5532
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] dsmiley commented on a change in pull request #705: SOLR-14920: Spotless formatting for core - non-test only
Posted by GitBox <gi...@apache.org>.
dsmiley commented on a change in pull request #705:
URL: https://github.com/apache/solr/pull/705#discussion_r817349815
##########
File path: solr/core/src/java/org/apache/solr/parser/QueryParser.java
##########
@@ -8,25 +8,25 @@
import java.util.HashSet;
import java.util.List;
import java.util.Set;
-
import org.apache.lucene.analysis.Analyzer;
+import org.apache.lucene.queryparser.charstream.CharStream;
+import org.apache.lucene.queryparser.charstream.FastCharStream;
import org.apache.lucene.search.BooleanClause;
import org.apache.lucene.search.Query;
-import org.apache.solr.search.SyntaxError;
import org.apache.solr.search.QParser;
-
-import org.apache.lucene.queryparser.charstream.CharStream;
-import org.apache.lucene.queryparser.charstream.FastCharStream;
+import org.apache.solr.search.SyntaxError;
public class QueryParser extends SolrQueryParserBase implements QueryParserConstants {
Review comment:
This file is machine generated; do we still format it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org