You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@solr.apache.org by GitBox <gi...@apache.org> on 2021/08/05 02:05:25 UTC
[GitHub] [solr] sonatype-lift[bot] commented on a change in pull request #214: SOLR-15428: Integrate the OpenJDK JMH micro benchmark framework for m…
sonatype-lift[bot] commented on a change in pull request #214:
URL: https://github.com/apache/solr/pull/214#discussion_r683075217
##########
File path: solr/benchmark/src/java/org/apache/solr/bench/MiniClusterState.java
##########
@@ -0,0 +1,370 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.bench;
+
+import static org.apache.commons.io.file.PathUtils.deleteDirectory;
+
+import com.codahale.metrics.Meter;
+import java.io.IOException;
+import java.lang.management.ManagementFactory;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Random;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.ThreadLocalRandom;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import org.apache.commons.io.output.NullPrintStream;
+import org.apache.solr.client.solrj.SolrClient;
+import org.apache.solr.client.solrj.SolrQuery;
+import org.apache.solr.client.solrj.embedded.JettySolrRunner;
+import org.apache.solr.client.solrj.impl.Http2SolrClient;
+import org.apache.solr.client.solrj.request.CollectionAdminRequest;
+import org.apache.solr.client.solrj.request.QueryRequest;
+import org.apache.solr.client.solrj.request.UpdateRequest;
+import org.apache.solr.cloud.MiniSolrCloudCluster;
+import org.apache.solr.common.SolrInputDocument;
+import org.apache.solr.common.params.ModifiableSolrParams;
+import org.apache.solr.common.util.IOUtils;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SolrNamedThreadFactory;
+import org.apache.solr.common.util.SuppressForbidden;
+import org.openjdk.jmh.annotations.Level;
+import org.openjdk.jmh.annotations.Scope;
+import org.openjdk.jmh.annotations.Setup;
+import org.openjdk.jmh.annotations.State;
+import org.openjdk.jmh.annotations.TearDown;
+import org.openjdk.jmh.infra.BenchmarkParams;
+import org.openjdk.jmh.infra.Control;
+
+/** The base class for Solr JMH benchmarks that operate against a {@link MiniSolrCloudCluster}. */
+public class MiniClusterState {
+
+ public static final boolean DEBUG_OUTPUT = false;
+
+ public static final int PROC_COUNT =
+ ManagementFactory.getOperatingSystemMXBean().getAvailableProcessors();
+
+ private static boolean quietLog = Boolean.getBoolean("quietLog");
+
+ @SuppressForbidden(reason = "JMH uses std out for user output")
+ public static void log(String value) {
+ if (!quietLog) {
+ System.out.println((value.equals("") ? "" : "--> ") + value);
+ }
+ }
+
+ @State(Scope.Benchmark)
+ public static class MiniClusterBenchState {
+
+ boolean metricsEnabled = true;
+
+ public List<String> nodes;
+ MiniSolrCloudCluster cluster;
+ public SolrClient client;
+
+ int runCnt = 0;
+
+ boolean createCollectionAndIndex = true;
+
+ boolean deleteMiniCluster = true;
+
+ Path baseDir;
+ boolean allowClusterReuse = false;
+
+ ThreadPoolExecutor exec;
+
+ boolean isWarmup;
+
+ @TearDown(Level.Iteration)
+ public void tearDown(BenchmarkParams benchmarkParams) throws Exception {
+
+ // dump Solr metrics
+ Path metricsResults =
+ Paths.get(
Review comment:
*PATH_TRAVERSAL_IN:* This API (java/nio/file/Paths.get(Ljava/lang/String;[Ljava/lang/String;)Ljava/nio/file/Path;) reads a file whose location might be specified by user input [(details)](https://find-sec-bugs.github.io/bugs.htm#PATH_TRAVERSAL_IN)
(at-me [in a reply](https://help.sonatype.com/lift) with `help` or `ignore`)
##########
File path: solr/core/src/java/org/apache/solr/client/solrj/embedded/JettySolrRunner.java
##########
@@ -697,6 +716,52 @@ public void stop() throws Exception {
}
}
+ public void outputMetrics(File outputDirectory, String fileName) throws IOException {
+ if (getCoreContainer() != null) {
+
+ if (outputDirectory != null) {
+ Path outDir = outputDirectory.toPath();
+ if (!Files.exists(outDir)) {
+ Files.createDirectories(outDir);
+ }
+ }
+
+ SolrMetricManager metricsManager = getCoreContainer().getMetricManager();
+
+ Set<String> registryNames = metricsManager.registryNames();
+ for (String registryName : registryNames) {
+ MetricRegistry metricsRegisty = metricsManager.registry(registryName);
+ try (PrintStream ps = outputDirectory == null ? new NullPrintStream() : new PrintStream(new File(outputDirectory, registryName + "_" + fileName), StandardCharsets.UTF_8)) {
Review comment:
*PATH_TRAVERSAL_IN:* This API (java/io/File.<init>(Ljava/io/File;Ljava/lang/String;)V) reads a file whose location might be specified by user input [(details)](https://find-sec-bugs.github.io/bugs.htm#PATH_TRAVERSAL_IN)
(at-me [in a reply](https://help.sonatype.com/lift) with `help` or `ignore`)
##########
File path: solr/benchmark/src/java/org/apache/solr/bench/DocMaker.java
##########
@@ -0,0 +1,301 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.bench;
+
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Queue;
+import java.util.Random;
+import java.util.SplittableRandom;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ThreadLocalRandom;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import org.apache.commons.lang3.RandomStringUtils;
+import org.apache.commons.lang3.Validate;
+import org.apache.lucene.util.RamUsageEstimator;
+import org.apache.lucene.util.TestUtil;
+import org.apache.solr.common.SolrInputDocument;
+import org.apache.solr.common.util.SolrNamedThreadFactory;
+import org.apache.solr.common.util.SuppressForbidden;
+
+/**
+ * A tool to generate controlled random data for a benchmark. {@link SolrInputDocument}s are created
+ * based on supplied FieldDef definitions.
+ *
+ * <p>You can call getDocument to build and retrieve one {@link SolrInputDocument} at a time, or you
+ * can call {@link #preGenerateDocs} to generate the given number of documents in RAM, and then
+ * retrieve them via {@link #getGeneratedDocsIterator}.
+ */
+public class DocMaker {
+
+ private Queue<SolrInputDocument> docs = new ConcurrentLinkedQueue<>();
+
+ private final Map<String, FieldDef> fields = new HashMap<>();
+
+ private static final AtomicInteger ID = new AtomicInteger();
+
+ private ExecutorService executorService;
+
+ private SplittableRandom threadRandom;
+
+ public DocMaker() {
+ Long seed = Long.getLong("randomSeed");
+ if (seed != null) {
+ threadRandom = new SplittableRandom(seed);
+ } else {
+ threadRandom = new SplittableRandom();
+ }
+ }
+
+ @SuppressForbidden(reason = "This module does not need to deal with logging context")
+ public void preGenerateDocs(int numDocs) throws InterruptedException {
+ MiniClusterState.log("preGenerateDocs " + numDocs + " ...");
+
+ executorService =
+ Executors.newFixedThreadPool(
+ Runtime.getRuntime().availableProcessors() + 1,
+ new SolrNamedThreadFactory("SolrJMH DocMaker"));
+
+ for (int i = 0; i < numDocs; i++) {
+ executorService.submit(
+ () -> {
+ try {
+ SolrInputDocument doc = getDocument();
+ docs.add(doc);
+ } catch (Exception e) {
+ executorService.shutdownNow();
+ throw new RuntimeException(e);
+ }
+ });
+ }
+
+ executorService.shutdown();
+ boolean result = executorService.awaitTermination(10, TimeUnit.MINUTES);
+ if (!result) {
+ throw new RuntimeException("Timeout waiting for doc adds to finish");
+ }
+ MiniClusterState.log(
+ "done preGenerateDocs docs="
+ + docs.size()
+ + " ram="
+ + RamUsageEstimator.humanReadableUnits(RamUsageEstimator.sizeOfObject(docs)));
+
+ if (numDocs != docs.size()) {
+ throw new IllegalStateException("numDocs != " + docs.size());
+ }
+ }
+
+ public Iterator<SolrInputDocument> getGeneratedDocsIterator() {
+ return docs.iterator();
+ }
+
+ public SolrInputDocument getDocument() {
+ SolrInputDocument doc = new SolrInputDocument();
+
+ for (Map.Entry<String, FieldDef> entry : fields.entrySet()) {
+ doc.addField(entry.getKey(), getValue(entry.getValue()));
+ }
+
+ return doc;
+ }
+
+ public void addField(String name, FieldDef.FieldDefBuilder builder) {
+ fields.put(name, builder.build());
+ }
+
+ private Object getValue(FieldDef fieldDef) {
+ switch (fieldDef.getContent()) {
+ case UNIQUE_INT:
+ return ID.incrementAndGet();
+ case INTEGER:
+ if (fieldDef.getMaxCardinality() > 0) {
+ long start = fieldDef.getCardinalityStart();
+ long seed = nextLong(start, start + fieldDef.getMaxCardinality(), threadRandom.split());
+ return nextInt(0, Integer.MAX_VALUE, new SplittableRandom(seed));
+ }
+
+ return ThreadLocalRandom.current().nextInt(Integer.MAX_VALUE);
+ case ALPHEBETIC:
+ return getString(fieldDef, value -> getAlphabeticString(fieldDef));
+ case UNICODE:
+ return getString(fieldDef, value -> getUnicodeString(fieldDef));
+ default:
+ throw new UnsupportedOperationException(
+ "Unsupported content type type=" + fieldDef.getContent());
+ }
+ }
+
+ private String getString(FieldDef fieldDef, StringSupplier supplier) {
+ if (fieldDef.getNumTokens() > 1 || fieldDef.getMaxNumTokens() > 1) {
+ StringBuilder sb =
+ new StringBuilder(
+ fieldDef.getNumTokens()
+ * (Math.max(fieldDef.getLength(), fieldDef.getMaxLength()) + 1));
+ SplittableRandom random = threadRandom.split();
+ for (int i = 0;
+ i
+ < (fieldDef.getMaxNumTokens() > 1
+ ? random.nextInt(1, fieldDef.getMaxNumTokens())
+ : fieldDef.getNumTokens());
+ i++) {
+ if (i > 0) {
+ sb.append(' ');
+ }
+ sb.append(supplier.getString(fieldDef));
+ }
+ return sb.toString();
+ }
+ return supplier.getString(fieldDef);
+ }
+
+ private String getUnicodeString(FieldDef fieldDef) {
+ try {
+ if (fieldDef.getMaxCardinality() > 0) {
+ long start = fieldDef.getCardinalityStart();
+ long seed = nextLong(start, start + fieldDef.getMaxCardinality(), threadRandom.split());
+ if (fieldDef.getLength() > -1) {
+ return TestUtil.randomRealisticUnicodeString(
+ new Random(seed), fieldDef.getLength(), fieldDef.getLength());
+ } else {
+ return TestUtil.randomRealisticUnicodeString(
+ new Random(seed), 1, fieldDef.getMaxLength());
+ }
+ }
+
+ if (fieldDef.getLength() > -1) {
+ return TestUtil.randomRealisticUnicodeString(
+ ThreadLocalRandom.current(), fieldDef.getLength(), fieldDef.getLength());
+ } else {
+ return TestUtil.randomRealisticUnicodeString(
+ ThreadLocalRandom.current(), 1, fieldDef.getMaxLength());
+ }
+ } catch (Exception e) {
+ throw new RuntimeException("Failed getting UnicodeString with FieldDef=" + fieldDef, e);
+ }
+ }
+
+ private String getAlphabeticString(FieldDef fieldDef) {
+ try {
+ if (fieldDef.getMaxCardinality() > 0) {
+ long start = fieldDef.getCardinalityStart();
+ long seed = nextLong(start, start + fieldDef.getMaxCardinality(), threadRandom.split());
+ SplittableRandom random = new SplittableRandom(seed);
+ if (fieldDef.getLength() > -1) {
+ return RandomStringUtils.random(
+ nextInt(fieldDef.getLength(), fieldDef.getLength(), random),
Review comment:
*PREDICTABLE_RANDOM:* This random generator (java.util.Random) is predictable [(details)](https://find-sec-bugs.github.io/bugs.htm#PREDICTABLE_RANDOM)
(at-me [in a reply](https://help.sonatype.com/lift) with `help` or `ignore`)
##########
File path: solr/benchmark/src/java/org/apache/solr/bench/DocMaker.java
##########
@@ -0,0 +1,301 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.bench;
+
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Queue;
+import java.util.Random;
+import java.util.SplittableRandom;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ThreadLocalRandom;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import org.apache.commons.lang3.RandomStringUtils;
+import org.apache.commons.lang3.Validate;
+import org.apache.lucene.util.RamUsageEstimator;
+import org.apache.lucene.util.TestUtil;
+import org.apache.solr.common.SolrInputDocument;
+import org.apache.solr.common.util.SolrNamedThreadFactory;
+import org.apache.solr.common.util.SuppressForbidden;
+
+/**
+ * A tool to generate controlled random data for a benchmark. {@link SolrInputDocument}s are created
+ * based on supplied FieldDef definitions.
+ *
+ * <p>You can call getDocument to build and retrieve one {@link SolrInputDocument} at a time, or you
+ * can call {@link #preGenerateDocs} to generate the given number of documents in RAM, and then
+ * retrieve them via {@link #getGeneratedDocsIterator}.
+ */
+public class DocMaker {
+
+ private Queue<SolrInputDocument> docs = new ConcurrentLinkedQueue<>();
+
+ private final Map<String, FieldDef> fields = new HashMap<>();
+
+ private static final AtomicInteger ID = new AtomicInteger();
+
+ private ExecutorService executorService;
+
+ private SplittableRandom threadRandom;
+
+ public DocMaker() {
+ Long seed = Long.getLong("randomSeed");
+ if (seed != null) {
+ threadRandom = new SplittableRandom(seed);
+ } else {
+ threadRandom = new SplittableRandom();
+ }
+ }
+
+ @SuppressForbidden(reason = "This module does not need to deal with logging context")
+ public void preGenerateDocs(int numDocs) throws InterruptedException {
+ MiniClusterState.log("preGenerateDocs " + numDocs + " ...");
+
+ executorService =
+ Executors.newFixedThreadPool(
+ Runtime.getRuntime().availableProcessors() + 1,
+ new SolrNamedThreadFactory("SolrJMH DocMaker"));
+
+ for (int i = 0; i < numDocs; i++) {
+ executorService.submit(
+ () -> {
+ try {
+ SolrInputDocument doc = getDocument();
+ docs.add(doc);
+ } catch (Exception e) {
+ executorService.shutdownNow();
+ throw new RuntimeException(e);
+ }
+ });
+ }
+
+ executorService.shutdown();
+ boolean result = executorService.awaitTermination(10, TimeUnit.MINUTES);
+ if (!result) {
+ throw new RuntimeException("Timeout waiting for doc adds to finish");
+ }
+ MiniClusterState.log(
+ "done preGenerateDocs docs="
+ + docs.size()
+ + " ram="
+ + RamUsageEstimator.humanReadableUnits(RamUsageEstimator.sizeOfObject(docs)));
+
+ if (numDocs != docs.size()) {
+ throw new IllegalStateException("numDocs != " + docs.size());
+ }
+ }
+
+ public Iterator<SolrInputDocument> getGeneratedDocsIterator() {
+ return docs.iterator();
+ }
+
+ public SolrInputDocument getDocument() {
+ SolrInputDocument doc = new SolrInputDocument();
+
+ for (Map.Entry<String, FieldDef> entry : fields.entrySet()) {
+ doc.addField(entry.getKey(), getValue(entry.getValue()));
+ }
+
+ return doc;
+ }
+
+ public void addField(String name, FieldDef.FieldDefBuilder builder) {
+ fields.put(name, builder.build());
+ }
+
+ private Object getValue(FieldDef fieldDef) {
+ switch (fieldDef.getContent()) {
+ case UNIQUE_INT:
+ return ID.incrementAndGet();
+ case INTEGER:
+ if (fieldDef.getMaxCardinality() > 0) {
+ long start = fieldDef.getCardinalityStart();
+ long seed = nextLong(start, start + fieldDef.getMaxCardinality(), threadRandom.split());
+ return nextInt(0, Integer.MAX_VALUE, new SplittableRandom(seed));
+ }
+
+ return ThreadLocalRandom.current().nextInt(Integer.MAX_VALUE);
+ case ALPHEBETIC:
+ return getString(fieldDef, value -> getAlphabeticString(fieldDef));
+ case UNICODE:
+ return getString(fieldDef, value -> getUnicodeString(fieldDef));
+ default:
+ throw new UnsupportedOperationException(
+ "Unsupported content type type=" + fieldDef.getContent());
+ }
+ }
+
+ private String getString(FieldDef fieldDef, StringSupplier supplier) {
+ if (fieldDef.getNumTokens() > 1 || fieldDef.getMaxNumTokens() > 1) {
+ StringBuilder sb =
+ new StringBuilder(
+ fieldDef.getNumTokens()
+ * (Math.max(fieldDef.getLength(), fieldDef.getMaxLength()) + 1));
+ SplittableRandom random = threadRandom.split();
+ for (int i = 0;
+ i
+ < (fieldDef.getMaxNumTokens() > 1
+ ? random.nextInt(1, fieldDef.getMaxNumTokens())
+ : fieldDef.getNumTokens());
+ i++) {
+ if (i > 0) {
+ sb.append(' ');
+ }
+ sb.append(supplier.getString(fieldDef));
+ }
+ return sb.toString();
+ }
+ return supplier.getString(fieldDef);
+ }
+
+ private String getUnicodeString(FieldDef fieldDef) {
+ try {
+ if (fieldDef.getMaxCardinality() > 0) {
+ long start = fieldDef.getCardinalityStart();
+ long seed = nextLong(start, start + fieldDef.getMaxCardinality(), threadRandom.split());
+ if (fieldDef.getLength() > -1) {
+ return TestUtil.randomRealisticUnicodeString(
+ new Random(seed), fieldDef.getLength(), fieldDef.getLength());
+ } else {
+ return TestUtil.randomRealisticUnicodeString(
+ new Random(seed), 1, fieldDef.getMaxLength());
+ }
+ }
+
+ if (fieldDef.getLength() > -1) {
+ return TestUtil.randomRealisticUnicodeString(
+ ThreadLocalRandom.current(), fieldDef.getLength(), fieldDef.getLength());
+ } else {
+ return TestUtil.randomRealisticUnicodeString(
+ ThreadLocalRandom.current(), 1, fieldDef.getMaxLength());
Review comment:
*PREDICTABLE_RANDOM:* This random generator (java.util.concurrent.ThreadLocalRandom) is predictable [(details)](https://find-sec-bugs.github.io/bugs.htm#PREDICTABLE_RANDOM)
(at-me [in a reply](https://help.sonatype.com/lift) with `help` or `ignore`)
##########
File path: solr/benchmark/src/java/org/apache/solr/bench/FieldDef.java
##########
@@ -0,0 +1,171 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.bench;
+
+import java.util.concurrent.ThreadLocalRandom;
+import org.apache.solr.common.SolrInputDocument;
+
+/**
+ * Provides the definition for a randomly generated field in a {@link SolrInputDocument} created by
+ * a {@link DocMaker}.
+ */
+public class FieldDef {
+ public static final int DEFAULT_MAX_LENGTH = 64;
+
+ private DocMaker.Content content;
+ private int numTokens = 1;
+ private int maxNumTokens = -1;
+ private int maxCardinality = -1;
+ private int maxLength = -1;
+ private int length = -1;
+ private long cardinalityStart;
+
+ public int getNumTokens() {
+ return numTokens;
+ }
+
+ public int getMaxNumTokens() {
+ return maxNumTokens;
+ }
+
+ public int getMaxCardinality() {
+ return maxCardinality;
+ }
+
+ public long getCardinalityStart() {
+ return cardinalityStart;
+ }
+
+ public int getMaxLength() {
+ return maxLength;
+ }
+
+ public int getLength() {
+ return length;
+ }
+
+ public DocMaker.Content getContent() {
+ return content;
+ }
+
+ public static final class FieldDefBuilder {
+
+ private DocMaker.Content content;
+ private int numTokens = 1;
+ private int maxNumTokens = -1;
+ private int maxCardinality = -1;
+ private int maxLength = -1;
+ private int length = -1;
+ private long cardinalityStart;
+
+ private FieldDefBuilder() {}
+
+ public static FieldDefBuilder aFieldDef() {
+ return new FieldDefBuilder();
+ }
+
+ public FieldDefBuilder withContent(DocMaker.Content content) {
+ this.content = content;
+ return this;
+ }
+
+ public FieldDefBuilder withTokenCount(int numTokens) {
+ if (numTokens > 1 && content == DocMaker.Content.UNIQUE_INT) {
+ throw new UnsupportedOperationException(
+ "UNIQUE_INT content type cannot be used with token count > 1");
+ }
+ if (maxCardinality > 1) {
+ throw new UnsupportedOperationException(
+ "tokenCount cannot be used with maxCardinality > 0");
+ }
+ this.numTokens = numTokens;
+ return this;
+ }
+
+ public FieldDefBuilder withMaxTokenCount(int maxNumTokens) {
+ if (numTokens > 1 && content == DocMaker.Content.UNIQUE_INT) {
+ throw new UnsupportedOperationException(
+ "UNIQUE_INT content type cannot be used with token count > 1");
+ }
+ if (maxCardinality > 1) {
+ throw new UnsupportedOperationException(
+ "maxNumTokens cannot be used with maxCardinality > 0");
+ }
+ this.maxNumTokens = maxNumTokens;
+ return this;
+ }
+
+ public FieldDefBuilder withMaxCardinality(int maxCardinality) {
+ if (numTokens > 1) {
+ throw new UnsupportedOperationException(
+ "maxCardinality cannot be used with token count > 1");
+ }
+ this.maxCardinality = maxCardinality;
+ this.cardinalityStart =
+ ThreadLocalRandom.current().nextLong(0, Long.MAX_VALUE - maxCardinality);
Review comment:
*PREDICTABLE_RANDOM:* This random generator (java.util.concurrent.ThreadLocalRandom) is predictable [(details)](https://find-sec-bugs.github.io/bugs.htm#PREDICTABLE_RANDOM)
(at-me [in a reply](https://help.sonatype.com/lift) with `help` or `ignore`)
##########
File path: solr/benchmark/src/java/org/apache/solr/bench/MiniClusterState.java
##########
@@ -0,0 +1,370 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.bench;
+
+import static org.apache.commons.io.file.PathUtils.deleteDirectory;
+
+import com.codahale.metrics.Meter;
+import java.io.IOException;
+import java.lang.management.ManagementFactory;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Random;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.ThreadLocalRandom;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import org.apache.commons.io.output.NullPrintStream;
+import org.apache.solr.client.solrj.SolrClient;
+import org.apache.solr.client.solrj.SolrQuery;
+import org.apache.solr.client.solrj.embedded.JettySolrRunner;
+import org.apache.solr.client.solrj.impl.Http2SolrClient;
+import org.apache.solr.client.solrj.request.CollectionAdminRequest;
+import org.apache.solr.client.solrj.request.QueryRequest;
+import org.apache.solr.client.solrj.request.UpdateRequest;
+import org.apache.solr.cloud.MiniSolrCloudCluster;
+import org.apache.solr.common.SolrInputDocument;
+import org.apache.solr.common.params.ModifiableSolrParams;
+import org.apache.solr.common.util.IOUtils;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SolrNamedThreadFactory;
+import org.apache.solr.common.util.SuppressForbidden;
+import org.openjdk.jmh.annotations.Level;
+import org.openjdk.jmh.annotations.Scope;
+import org.openjdk.jmh.annotations.Setup;
+import org.openjdk.jmh.annotations.State;
+import org.openjdk.jmh.annotations.TearDown;
+import org.openjdk.jmh.infra.BenchmarkParams;
+import org.openjdk.jmh.infra.Control;
+
+/** The base class for Solr JMH benchmarks that operate against a {@link MiniSolrCloudCluster}. */
+public class MiniClusterState {
+
+ public static final boolean DEBUG_OUTPUT = false;
+
+ public static final int PROC_COUNT =
+ ManagementFactory.getOperatingSystemMXBean().getAvailableProcessors();
+
+ private static boolean quietLog = Boolean.getBoolean("quietLog");
+
+ @SuppressForbidden(reason = "JMH uses std out for user output")
+ public static void log(String value) {
+ if (!quietLog) {
+ System.out.println((value.equals("") ? "" : "--> ") + value);
+ }
+ }
+
+ @State(Scope.Benchmark)
+ public static class MiniClusterBenchState {
+
+ boolean metricsEnabled = true;
+
+ public List<String> nodes;
+ MiniSolrCloudCluster cluster;
+ public SolrClient client;
+
+ int runCnt = 0;
+
+ boolean createCollectionAndIndex = true;
+
+ boolean deleteMiniCluster = true;
+
+ Path baseDir;
+ boolean allowClusterReuse = false;
+
+ ThreadPoolExecutor exec;
+
+ boolean isWarmup;
+
+ @TearDown(Level.Iteration)
+ public void tearDown(BenchmarkParams benchmarkParams) throws Exception {
+
+ // dump Solr metrics
+ Path metricsResults =
+ Paths.get(
+ "work/metrics-results",
+ benchmarkParams.id(),
+ String.valueOf(runCnt++),
+ benchmarkParams.getBenchmark() + ".txt");
+ if (!Files.exists(metricsResults.getParent())) {
+ Files.createDirectories(metricsResults.getParent());
+ }
+
+ cluster.outputMetrics(
+ metricsResults.getParent().toFile(), metricsResults.getFileName().toString());
+ }
+
+ @Setup(Level.Iteration)
+ public void checkWarmUp(Control control) throws Exception {
+ isWarmup = control.stopMeasurement;
+ }
+
+ @TearDown(Level.Trial)
+ public void shutdownMiniCluster() throws Exception {
+ if (DEBUG_OUTPUT) log("closing client and shutting down minicluster");
+ IOUtils.closeQuietly(client);
+ cluster.shutdown();
+ }
+
+ @Setup(Level.Trial)
+ public void doSetup(BenchmarkParams benchmarkParams) throws Exception {
+
+ MiniClusterState.log("");
+ Path currentRelativePath = Paths.get("");
+ String s = currentRelativePath.toAbsolutePath().toString();
+ log("current relative path is: " + s);
+
+ Long seed = Long.getLong("solr.bench.seed");
+
+ if (seed == null) {
+ seed = ThreadLocalRandom.current().nextLong();
+ }
+
+ // set the seed used by ThreadLocalRandom
+ System.setProperty("randomSeed", Long.toString(new Random(seed).nextLong()));
Review comment:
*PREDICTABLE_RANDOM:* This random generator (java.util.Random) is predictable [(details)](https://find-sec-bugs.github.io/bugs.htm#PREDICTABLE_RANDOM)
(at-me [in a reply](https://help.sonatype.com/lift) with `help` or `ignore`)
##########
File path: solr/benchmark/src/java/org/apache/solr/bench/MiniClusterState.java
##########
@@ -0,0 +1,370 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.bench;
+
+import static org.apache.commons.io.file.PathUtils.deleteDirectory;
+
+import com.codahale.metrics.Meter;
+import java.io.IOException;
+import java.lang.management.ManagementFactory;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Random;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.ThreadLocalRandom;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import org.apache.commons.io.output.NullPrintStream;
+import org.apache.solr.client.solrj.SolrClient;
+import org.apache.solr.client.solrj.SolrQuery;
+import org.apache.solr.client.solrj.embedded.JettySolrRunner;
+import org.apache.solr.client.solrj.impl.Http2SolrClient;
+import org.apache.solr.client.solrj.request.CollectionAdminRequest;
+import org.apache.solr.client.solrj.request.QueryRequest;
+import org.apache.solr.client.solrj.request.UpdateRequest;
+import org.apache.solr.cloud.MiniSolrCloudCluster;
+import org.apache.solr.common.SolrInputDocument;
+import org.apache.solr.common.params.ModifiableSolrParams;
+import org.apache.solr.common.util.IOUtils;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SolrNamedThreadFactory;
+import org.apache.solr.common.util.SuppressForbidden;
+import org.openjdk.jmh.annotations.Level;
+import org.openjdk.jmh.annotations.Scope;
+import org.openjdk.jmh.annotations.Setup;
+import org.openjdk.jmh.annotations.State;
+import org.openjdk.jmh.annotations.TearDown;
+import org.openjdk.jmh.infra.BenchmarkParams;
+import org.openjdk.jmh.infra.Control;
+
+/** The base class for Solr JMH benchmarks that operate against a {@link MiniSolrCloudCluster}. */
+public class MiniClusterState {
+
+ public static final boolean DEBUG_OUTPUT = false;
+
+ public static final int PROC_COUNT =
+ ManagementFactory.getOperatingSystemMXBean().getAvailableProcessors();
+
+ private static boolean quietLog = Boolean.getBoolean("quietLog");
+
+ @SuppressForbidden(reason = "JMH uses std out for user output")
+ public static void log(String value) {
+ if (!quietLog) {
+ System.out.println((value.equals("") ? "" : "--> ") + value);
+ }
+ }
+
+ @State(Scope.Benchmark)
+ public static class MiniClusterBenchState {
+
+ boolean metricsEnabled = true;
+
+ public List<String> nodes;
+ MiniSolrCloudCluster cluster;
+ public SolrClient client;
+
+ int runCnt = 0;
+
+ boolean createCollectionAndIndex = true;
+
+ boolean deleteMiniCluster = true;
+
+ Path baseDir;
+ boolean allowClusterReuse = false;
+
+ ThreadPoolExecutor exec;
+
+ boolean isWarmup;
+
+ @TearDown(Level.Iteration)
+ public void tearDown(BenchmarkParams benchmarkParams) throws Exception {
+
+ // dump Solr metrics
+ Path metricsResults =
+ Paths.get(
+ "work/metrics-results",
+ benchmarkParams.id(),
+ String.valueOf(runCnt++),
+ benchmarkParams.getBenchmark() + ".txt");
+ if (!Files.exists(metricsResults.getParent())) {
+ Files.createDirectories(metricsResults.getParent());
+ }
+
+ cluster.outputMetrics(
+ metricsResults.getParent().toFile(), metricsResults.getFileName().toString());
+ }
+
+ @Setup(Level.Iteration)
+ public void checkWarmUp(Control control) throws Exception {
+ isWarmup = control.stopMeasurement;
+ }
+
+ @TearDown(Level.Trial)
+ public void shutdownMiniCluster() throws Exception {
+ if (DEBUG_OUTPUT) log("closing client and shutting down minicluster");
+ IOUtils.closeQuietly(client);
+ cluster.shutdown();
+ }
+
+ @Setup(Level.Trial)
+ public void doSetup(BenchmarkParams benchmarkParams) throws Exception {
+
+ MiniClusterState.log("");
+ Path currentRelativePath = Paths.get("");
+ String s = currentRelativePath.toAbsolutePath().toString();
+ log("current relative path is: " + s);
+
+ Long seed = Long.getLong("solr.bench.seed");
+
+ if (seed == null) {
+ seed = ThreadLocalRandom.current().nextLong();
+ }
+
+ // set the seed used by ThreadLocalRandom
+ System.setProperty("randomSeed", Long.toString(new Random(seed).nextLong()));
+
+ System.setProperty("pkiHandlerPrivateKeyPath", "");
+ System.setProperty("pkiHandlerPublicKeyPath", "");
+
+ System.setProperty("solr.log.name", benchmarkParams.id());
+
+ System.setProperty("solr.default.confdir", "../server/solr/configsets/_default");
+
+ // not currently usable, but would enable JettySolrRunner's ill-conceived jetty.testMode and
+ // allow using SSL
+
+ // System.getProperty("jetty.testMode", "true");
+ // SolrCloudTestCase.sslConfig = SolrTestCaseJ4.buildSSLConfig();
+
+ String baseDirSysProp = System.getProperty("miniClusterBaseDir");
+ if (baseDirSysProp != null) {
+ deleteMiniCluster = false;
+ baseDir = Paths.get(baseDirSysProp);
+ if (Files.exists(baseDir)) {
+ createCollectionAndIndex = false;
+ }
+ } else {
+ baseDir = Paths.get("work/mini-cluster");
+ }
+
+ System.setProperty("metricsEnabled", String.valueOf(metricsEnabled));
+ }
+
+ public void allowClusterReuse(boolean allowClusterReuse) {
+ this.allowClusterReuse = allowClusterReuse;
+ }
+
+ public void metricsEnabled(boolean metricsEnabled) {
+ this.metricsEnabled = metricsEnabled;
+ }
+
+ public void startMiniCluster(int nodeCount) {
+ log("starting mini cluster at base directory: " + baseDir.toAbsolutePath());
+
+ if (!allowClusterReuse && Files.exists(baseDir)) {
+ log(
+ "mini cluster base directory exists, removing according to allowClusterReuse="
+ + allowClusterReuse);
+ try {
+ deleteDirectory(baseDir);
+ } catch (IOException e) {
+ throw new RuntimeException(e);
+ }
+ createCollectionAndIndex = true;
+ } else if (Files.exists(baseDir)) {
+ createCollectionAndIndex = false;
+ deleteMiniCluster = false;
+ }
+
+ try {
+ cluster =
+ new MiniSolrCloudCluster.Builder(nodeCount, baseDir)
+ .formatZkServer(false)
+ .addConfig("conf", Paths.get("src/resources/configs/cloud-minimal/conf"))
+ .configure();
+ } catch (Exception e) {
+ if (Files.exists(baseDir)) {
+ try {
+ deleteDirectory(baseDir);
+ } catch (IOException ex) {
+ e.addSuppressed(ex);
+ }
+ }
+ throw new RuntimeException(e);
+ }
+
+ nodes = new ArrayList<>(nodeCount);
+ List<JettySolrRunner> jetties = cluster.getJettySolrRunners();
+ for (JettySolrRunner runner : jetties) {
+ nodes.add(runner.getBaseUrl().toString());
+ }
+
+ client = new Http2SolrClient.Builder().build();
+
+ log("done starting mini cluster");
+ log("");
+ }
+
+ public void createCollection(String collection, int numShards, int numReplicas)
+ throws Exception {
+ if (createCollectionAndIndex) {
+ try {
+
+ CollectionAdminRequest.Create request =
+ CollectionAdminRequest.createCollection(collection, "conf", numShards, numReplicas);
+ request.setBasePath(
+ nodes.get(ThreadLocalRandom.current().nextInt(cluster.getJettySolrRunners().size())));
+
+ client.request(request);
+
+ cluster.waitForActiveCollection(
+ collection, 15, TimeUnit.SECONDS, numShards, numShards * numReplicas);
+ } catch (Exception e) {
+ if (Files.exists(baseDir)) {
+ deleteDirectory(baseDir);
+ }
+ throw e;
+ }
+ }
+ }
+
+ @SuppressForbidden(reason = "This module does not need to deal with logging context")
+ public void index(String collection, DocMaker docMaker, int docCount) throws Exception {
+ if (createCollectionAndIndex) {
+
+ log("indexing data for benchmark...");
+ Meter meter = new Meter();
+ ExecutorService executorService =
+ Executors.newFixedThreadPool(
+ Runtime.getRuntime().availableProcessors(),
+ new SolrNamedThreadFactory("SolrJMH Indexer Progress"));
+ ScheduledExecutorService scheduledExecutor =
+ Executors.newSingleThreadScheduledExecutor(
+ new SolrNamedThreadFactory("SolrJMH Indexer"));
+ scheduledExecutor.scheduleAtFixedRate(
+ () -> {
+ if (meter.getCount() == docCount) {
+ scheduledExecutor.shutdown();
+ } else {
+ log(meter.getCount() + " docs at " + meter.getMeanRate() + " doc/s");
+ }
+ },
+ 10,
+ 10,
+ TimeUnit.SECONDS);
+ for (int i = 0; i < docCount; i++) {
+ executorService.submit(
+ () -> {
+ UpdateRequest updateRequest = new UpdateRequest();
+ updateRequest.setBasePath(
+ nodes.get(
+ ThreadLocalRandom.current().nextInt(cluster.getJettySolrRunners().size())));
+ SolrInputDocument doc = docMaker.getDocument();
+ // log("add doc " + doc);
+ updateRequest.add(doc);
+ meter.mark();
+
+ try {
+ client.request(updateRequest, collection);
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+ });
+ }
+
+ log("done adding docs, waiting for executor to terminate...");
+
+ executorService.shutdown();
+ boolean result = executorService.awaitTermination(600, TimeUnit.MINUTES);
+
+ scheduledExecutor.shutdown();
+
+ if (!result) {
+ throw new RuntimeException("Timeout waiting for doc adds to finish");
+ }
+ log("done indexing data for benchmark");
+
+ if (allowClusterReuse) {
+ docMaker.clear();
+ }
+
+ log("committing data ...");
+ UpdateRequest commitRequest = new UpdateRequest();
+ commitRequest.setBasePath(
+ nodes.get(ThreadLocalRandom.current().nextInt(cluster.getJettySolrRunners().size())));
+ commitRequest.setAction(UpdateRequest.ACTION.COMMIT, false, true);
+ commitRequest.process(client, collection);
+ log("done committing data");
+ } else {
+ cluster.waitForActiveCollection(collection, 15, TimeUnit.SECONDS);
+ }
+
+ QueryRequest queryRequest = new QueryRequest(new SolrQuery("q", "*:*", "rows", "1"));
+ queryRequest.setBasePath(
+ nodes.get(ThreadLocalRandom.current().nextInt(cluster.getJettySolrRunners().size())));
Review comment:
*PREDICTABLE_RANDOM:* This random generator (java.util.concurrent.ThreadLocalRandom) is predictable [(details)](https://find-sec-bugs.github.io/bugs.htm#PREDICTABLE_RANDOM)
(at-me [in a reply](https://help.sonatype.com/lift) with `help` or `ignore`)
##########
File path: solr/benchmark/src/java/org/apache/solr/bench/index/CloudIndexing.java
##########
@@ -0,0 +1,170 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.bench.index;
+
+import java.util.Iterator;
+import java.util.concurrent.ThreadLocalRandom;
+import java.util.concurrent.TimeUnit;
+import org.apache.solr.bench.DocMaker;
+import org.apache.solr.bench.FieldDef;
+import org.apache.solr.bench.MiniClusterState;
+import org.apache.solr.client.solrj.request.UpdateRequest;
+import org.apache.solr.common.SolrInputDocument;
+import org.openjdk.jmh.annotations.Benchmark;
+import org.openjdk.jmh.annotations.BenchmarkMode;
+import org.openjdk.jmh.annotations.Fork;
+import org.openjdk.jmh.annotations.Level;
+import org.openjdk.jmh.annotations.Measurement;
+import org.openjdk.jmh.annotations.Mode;
+import org.openjdk.jmh.annotations.OutputTimeUnit;
+import org.openjdk.jmh.annotations.Param;
+import org.openjdk.jmh.annotations.Scope;
+import org.openjdk.jmh.annotations.Setup;
+import org.openjdk.jmh.annotations.State;
+import org.openjdk.jmh.annotations.TearDown;
+import org.openjdk.jmh.annotations.Threads;
+import org.openjdk.jmh.annotations.Timeout;
+import org.openjdk.jmh.annotations.Warmup;
+
+@BenchmarkMode(Mode.Throughput)
+@OutputTimeUnit(TimeUnit.SECONDS)
+@Threads(6)
+@Warmup(time = 3, iterations = 5)
+@Measurement(time = 15, iterations = 5)
+@Fork(value = 1)
+@Timeout(time = 60)
+/** A benchmark to experiment with the performance of distributed indexing. */
+public class CloudIndexing {
+
+ @State(Scope.Benchmark)
+ public static class BenchState {
+
+ String collection = "testCollection";
+
+ int nodeCount = 4;
+ int numShards = 5;
+
+ @Param({"1", "3", "9"})
+ int numReplicas;
+
+ @Param({"50000"})
+ public int docCount;
+
+ @Setup(Level.Iteration)
+ public void doSetup(MiniClusterState.MiniClusterBenchState miniClusterState) throws Exception {
+
+ miniClusterState.startMiniCluster(nodeCount);
+
+ miniClusterState.createCollection(collection, numShards, numReplicas);
+ }
+
+ @TearDown(Level.Iteration)
+ public void doTearDown(MiniClusterState.MiniClusterBenchState miniClusterState)
+ throws Exception {
+ miniClusterState.shutdownMiniCluster();
+ }
+
+ @State(Scope.Thread)
+ public static class Docs {
+
+ private DocMaker largeDocMaker;
+ private Iterator<SolrInputDocument> largeDocIterator;
+
+ private DocMaker smallDocMaker;
+ private Iterator<SolrInputDocument> smallDocIterator;
+
+ @Setup(Level.Trial)
+ public void setupDoc(BenchState state) throws Exception {
+ largeDocMaker = new DocMaker();
+ largeDocMaker.addField(
+ "id", FieldDef.FieldDefBuilder.aFieldDef().withContent(DocMaker.Content.UNIQUE_INT));
+ largeDocMaker.addField(
+ "text",
+ FieldDef.FieldDefBuilder.aFieldDef()
+ .withContent(DocMaker.Content.ALPHEBETIC)
+ .withMaxLength(64)
+ .withTokenCount(ThreadLocalRandom.current().nextInt(512) + 1));
+
+ largeDocMaker.preGenerateDocs(state.docCount);
+
+ largeDocIterator = largeDocMaker.getGeneratedDocsIterator();
+
+ smallDocMaker = new DocMaker();
+ smallDocMaker.addField(
+ "id", FieldDef.FieldDefBuilder.aFieldDef().withContent(DocMaker.Content.UNIQUE_INT));
+ smallDocMaker.addField(
+ "text",
+ FieldDef.FieldDefBuilder.aFieldDef()
+ .withContent(DocMaker.Content.ALPHEBETIC)
+ .withMaxLength(32)
+ .withTokenCount(1));
+
+ smallDocMaker.preGenerateDocs(state.docCount);
+
+ smallDocIterator = smallDocMaker.getGeneratedDocsIterator();
+ }
+
+ public SolrInputDocument getLargeDoc() {
+ if (!largeDocIterator.hasNext()) {
+ largeDocIterator = largeDocMaker.getGeneratedDocsIterator();
+ }
+ return largeDocIterator.next();
+ }
+
+ public SolrInputDocument getSmallDoc() {
+ if (!smallDocIterator.hasNext()) {
+ smallDocIterator = smallDocMaker.getGeneratedDocsIterator();
+ }
+ return smallDocIterator.next();
+ }
+ }
+ }
+
+ @Benchmark
+ @Timeout(time = 300)
+ public Object indexLargeDoc(
+ MiniClusterState.MiniClusterBenchState miniClusterState,
+ BenchState state,
+ BenchState.Docs docState)
+ throws Exception {
+ UpdateRequest updateRequest = new UpdateRequest();
+ updateRequest.setBasePath(
+ miniClusterState.nodes.get(ThreadLocalRandom.current().nextInt(state.nodeCount)));
+ SolrInputDocument doc = docState.getLargeDoc();
+
+ updateRequest.add(doc);
+
+ return miniClusterState.client.request(updateRequest, state.collection);
+ }
+
+ @Benchmark
+ @Timeout(time = 300)
+ public Object indexSmallDoc(
+ MiniClusterState.MiniClusterBenchState miniClusterState,
+ BenchState state,
+ BenchState.Docs docState)
+ throws Exception {
+ UpdateRequest updateRequest = new UpdateRequest();
+ updateRequest.setBasePath(
+ miniClusterState.nodes.get(ThreadLocalRandom.current().nextInt(state.nodeCount)));
Review comment:
*PREDICTABLE_RANDOM:* This random generator (java.util.concurrent.ThreadLocalRandom) is predictable [(details)](https://find-sec-bugs.github.io/bugs.htm#PREDICTABLE_RANDOM)
(at-me [in a reply](https://help.sonatype.com/lift) with `help` or `ignore`)
##########
File path: solr/benchmark/src/java/org/apache/solr/bench/search/JsonFaceting.java
##########
@@ -0,0 +1,194 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.bench.search;
+
+import java.util.concurrent.ThreadLocalRandom;
+import java.util.concurrent.TimeUnit;
+import org.apache.solr.bench.DocMaker;
+import org.apache.solr.bench.FieldDef;
+import org.apache.solr.bench.MiniClusterState;
+import org.apache.solr.client.solrj.request.QueryRequest;
+import org.apache.solr.common.params.ModifiableSolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.openjdk.jmh.annotations.Benchmark;
+import org.openjdk.jmh.annotations.BenchmarkMode;
+import org.openjdk.jmh.annotations.Fork;
+import org.openjdk.jmh.annotations.Level;
+import org.openjdk.jmh.annotations.Measurement;
+import org.openjdk.jmh.annotations.Mode;
+import org.openjdk.jmh.annotations.OutputTimeUnit;
+import org.openjdk.jmh.annotations.Param;
+import org.openjdk.jmh.annotations.Scope;
+import org.openjdk.jmh.annotations.Setup;
+import org.openjdk.jmh.annotations.State;
+import org.openjdk.jmh.annotations.Threads;
+import org.openjdk.jmh.annotations.Timeout;
+import org.openjdk.jmh.annotations.Warmup;
+import org.openjdk.jmh.infra.BenchmarkParams;
+
+/** A benchmark to experiment with the performance of json faceting. */
+@BenchmarkMode(Mode.Throughput)
+@OutputTimeUnit(TimeUnit.SECONDS)
+@Threads(1)
+@Warmup(time = 5, iterations = 3)
+@Measurement(time = 15, iterations = 5)
+@Fork(value = 1)
+@Timeout(time = 60)
+public class JsonFaceting {
+
+ @State(Scope.Benchmark)
+ public static class BenchState {
+
+ public String collection = "testCollection";
+
+ @Param({"10000"})
+ public int docCount;
+
+ int nodeCount = 2;
+ int numReplicas = 1;
+
+ @Param("2")
+ int numShards;
+
+ // DV, // DocValues, collect into ordinal array
+ // UIF, // UnInvertedField, collect into ordinal array
+ // DVHASH, // DocValues, collect into hash
+ // ENUM, // TermsEnum then intersect DocSet (stream-able)
+ // STREAM, // presently equivalent to ENUM
+ // SMART,
+ // "dv"
+ // "uif"
+ // "dvhash">
+ // "enum"
+ // "stream"
+ // "smart"
+ @Param({"smart"})
+ String fm;
+
+ @Param({"80000"})
+ int facetCard;
+
+ private ModifiableSolrParams params;
+
+ @Setup(Level.Trial)
+ public void setup(
+ BenchmarkParams benchmarkParams, MiniClusterState.MiniClusterBenchState miniClusterState)
+ throws Exception {
+ System.setProperty("maxMergeAtOnce", "20");
+ System.setProperty("segmentsPerTier", "20");
+
+ miniClusterState.allowClusterReuse(true);
+
+ miniClusterState.startMiniCluster(nodeCount);
+
+ miniClusterState.createCollection(collection, numShards, numReplicas);
+
+ // Define random documents
+ DocMaker docMaker = new DocMaker();
+ docMaker.addField(
+ "id", FieldDef.FieldDefBuilder.aFieldDef().withContent(DocMaker.Content.UNIQUE_INT));
+ docMaker.addField(
+ "facet_s",
+ FieldDef.FieldDefBuilder.aFieldDef()
+ .withContent(DocMaker.Content.ALPHEBETIC)
+ .withMaxLength(64)
+ .withMaxCardinality(facetCard));
+ docMaker.addField(
+ "facet2_s",
+ FieldDef.FieldDefBuilder.aFieldDef()
+ .withContent(DocMaker.Content.ALPHEBETIC)
+ .withMaxLength(16)
+ .withMaxCardinality(facetCard));
+ docMaker.addField(
+ "facet3_s",
+ FieldDef.FieldDefBuilder.aFieldDef()
+ .withContent(DocMaker.Content.UNICODE)
+ .withMaxLength(128)
+ .withMaxCardinality(12000));
+ docMaker.addField(
+ "text",
+ FieldDef.FieldDefBuilder.aFieldDef()
+ .withContent(DocMaker.Content.ALPHEBETIC)
+ .withMaxLength(64)
+ .withTokenCount(ThreadLocalRandom.current().nextInt(512) + 1));
+ docMaker.addField(
+ "int_i", FieldDef.FieldDefBuilder.aFieldDef().withContent(DocMaker.Content.INTEGER));
+ docMaker.addField(
+ "int2_i",
+ FieldDef.FieldDefBuilder.aFieldDef()
+ .withContent(DocMaker.Content.INTEGER)
+ .withMaxCardinality(500));
+ docMaker.addField(
+ "int3_i",
+ FieldDef.FieldDefBuilder.aFieldDef()
+ .withContent(DocMaker.Content.INTEGER)
+ .withMaxCardinality(200000));
+ docMaker.addField(
+ "int4_i",
+ FieldDef.FieldDefBuilder.aFieldDef()
+ .withContent(DocMaker.Content.INTEGER)
+ .withMaxCardinality(5));
+
+ miniClusterState.index(collection, docMaker, docCount);
+ miniClusterState.waitForMerges(collection);
+
+ params = new ModifiableSolrParams();
+
+ MiniClusterState.params(
+ params,
+ "q",
+ "*:*",
+ "json.facet",
+ "{f1:{method:'"
+ + fm
+ + "', type:terms, field:'facet_s', sort:'x desc', facet:{x:'min(int3_i)'} }"
+ + " , f2:{method:'"
+ + fm
+ + "',, type:terms, field:'facet_s', sort:'x desc', facet:{x:'max(int3_i)'} } "
+ + " , f3:{method:'"
+ + fm
+ + "', type:terms, field:'facet_s', sort:'x desc', facet:{x:'unique(facet2_s)'} } "
+ + " , f4:{method:'"
+ + fm
+ + "', type:terms, field:'facet_s', sort:'x desc', facet:{x:'hll(facet2_s)'} } "
+ + " , f5:{method:'"
+ + fm
+ + "', type:terms, field:'facet_s', sort:'x desc', facet:{x:'variance(int3_i)'} } "
+ + " , f6:{type:terms, field:'int3_i', limit:1, sort:'x desc', facet:{x:'hll(int2_i)'} } "
+ + " , f7:{type:terms, field:'facet_s', limit:2, sort:'x desc', facet:{x:'missing(int4_i)'} } "
+ + " , f8:{type:terms, field:'facet_s', limit:2, sort:'x desc', facet:{x:'countvals(int4_i)'} } "
+ + "}");
+
+ // MiniClusterState.log("params: " + params + "\n");
+ }
+ }
+
+ @Benchmark
+ @Timeout(time = 500, timeUnit = TimeUnit.SECONDS)
+ public Object jsonFacet(MiniClusterState.MiniClusterBenchState miniClusterState, BenchState state)
+ throws Exception {
+ QueryRequest queryRequest = new QueryRequest(state.params);
+ queryRequest.setBasePath(
+ miniClusterState.nodes.get(ThreadLocalRandom.current().nextInt(state.nodeCount)));
Review comment:
*PREDICTABLE_RANDOM:* This random generator (java.util.concurrent.ThreadLocalRandom) is predictable [(details)](https://find-sec-bugs.github.io/bugs.htm#PREDICTABLE_RANDOM)
(at-me [in a reply](https://help.sonatype.com/lift) with `help` or `ignore`)
##########
File path: solr/core/src/java/org/apache/solr/client/solrj/embedded/JettySolrRunner.java
##########
@@ -697,6 +716,52 @@ public void stop() throws Exception {
}
}
+ public void outputMetrics(File outputDirectory, String fileName) throws IOException {
+ if (getCoreContainer() != null) {
+
+ if (outputDirectory != null) {
+ Path outDir = outputDirectory.toPath();
+ if (!Files.exists(outDir)) {
+ Files.createDirectories(outDir);
+ }
+ }
+
+ SolrMetricManager metricsManager = getCoreContainer().getMetricManager();
Review comment:
*NULL_DEREFERENCE:* object returned by `getCoreContainer()` could be null and is dereferenced at line 729.
(at-me [in a reply](https://help.sonatype.com/lift) with `help` or `ignore`)
##########
File path: solr/core/src/java/org/apache/solr/core/SolrCore.java
##########
@@ -487,6 +488,14 @@ public long getIndexSize() {
return size;
}
+ public int getSegmentCount() {
+ try {
+ return withSearcher( solrIndexSearcher -> solrIndexSearcher.getRawReader().getIndexCommit().getSegmentCount());
Review comment:
*THREAD_SAFETY_VIOLATION:* Read/Write race. Non-private method `SolrCore.getSegmentCount()` indirectly reads with synchronization from `this.indexReaderFactory`. Potentially races with unsynchronized write in method `SolrCore.initIndex(...)`.
Reporting because another access to the same memory occurs on a background thread, although this access may not.
(at-me [in a reply](https://help.sonatype.com/lift) with `help` or `ignore`)
##########
File path: solr/test-framework/src/java/org/apache/solr/cloud/MiniSolrCloudCluster.java
##########
@@ -212,7 +197,7 @@ public MiniSolrCloudCluster(int numServers, String hostContext, Path baseDir, St
* @throws Exception if there was an error starting the cluster
*/
public MiniSolrCloudCluster(int numServers, Path baseDir, String solrXml, JettyConfig jettyConfig) throws Exception {
- this(numServers, baseDir, solrXml, jettyConfig, null);
+ this(numServers, baseDir, solrXml, jettyConfig, null, false);
Review comment:
*THREAD_SAFETY_VIOLATION:* Unprotected write. Non-private method `MiniSolrCloudCluster(...)` indirectly mutates container `util.ObjectReleaseTracker.OBJECTS` via call to `Map.remove(...)` outside of synchronization.
Reporting because another access to the same memory occurs on a background thread, although this access may not.
(at-me [in a reply](https://help.sonatype.com/lift) with `help` or `ignore`)
##########
File path: solr/test-framework/src/java/org/apache/solr/cloud/MiniSolrCloudCluster.java
##########
@@ -227,8 +212,8 @@ public MiniSolrCloudCluster(int numServers, Path baseDir, String solrXml, JettyC
* @throws Exception if there was an error starting the cluster
*/
public MiniSolrCloudCluster(int numServers, Path baseDir, String solrXml, JettyConfig jettyConfig,
- ZkTestServer zkTestServer) throws Exception {
- this(numServers, baseDir, solrXml, jettyConfig, zkTestServer, Optional.empty());
+ ZkTestServer zkTestServer, boolean formatZkServer) throws Exception {
+ this(numServers, baseDir, solrXml, jettyConfig, zkTestServer, Optional.empty(), formatZkServer);
Review comment:
*THREAD_SAFETY_VIOLATION:* Unprotected write. Non-private method `MiniSolrCloudCluster(...)` indirectly mutates container `util.ObjectReleaseTracker.OBJECTS` via call to `Map.remove(...)` outside of synchronization.
Reporting because another access to the same memory occurs on a background thread, although this access may not.
(at-me [in a reply](https://help.sonatype.com/lift) with `help` or `ignore`)
##########
File path: solr/test-framework/src/java/org/apache/solr/cloud/MiniSolrCloudCluster.java
##########
@@ -295,12 +280,14 @@ public MiniSolrCloudCluster(int numServers, Path baseDir, String solrXml, JettyC
this.zkServer = zkTestServer;
try (SolrZkClient zkClient = new SolrZkClient(zkServer.getZkHost(), AbstractZkTestCase.TIMEOUT)) {
- zkClient.makePath("/solr/solr.xml", solrXml.getBytes(Charset.defaultCharset()), true);
- if (jettyConfig.sslConfig != null && jettyConfig.sslConfig.isSSLMode()) {
- zkClient.makePath("/solr" + ZkStateReader.CLUSTER_PROPS, "{'urlScheme':'https'}".getBytes(StandardCharsets.UTF_8), true);
- }
- if (securityJson.isPresent()) { // configure Solr security
- zkClient.makePath("/solr/security.json", securityJson.get().getBytes(Charset.defaultCharset()), true);
+ if (!zkClient.exists("/solr/solr.xml", true)) {
+ zkClient.makePath("/solr/solr.xml", solrXml.getBytes(Charset.defaultCharset()), true);
+ if (jettyConfig.sslConfig != null && jettyConfig.sslConfig.isSSLMode()) {
+ zkClient.makePath("/solr" + ZkStateReader.CLUSTER_PROPS, "{'urlScheme':'https'}".getBytes(StandardCharsets.UTF_8), true);
+ }
+ if (securityJson.isPresent()) { // configure Solr security
+ zkClient.makePath("/solr/security.json", securityJson.get().getBytes(Charset.defaultCharset()), true);
+ }
}
}
Review comment:
*THREAD_SAFETY_VIOLATION:* Unprotected write. Non-private method `MiniSolrCloudCluster(...)` indirectly mutates container `util.ObjectReleaseTracker.OBJECTS` via call to `Map.remove(...)` outside of synchronization.
Reporting because another access to the same memory occurs on a background thread, although this access may not.
(at-me [in a reply](https://help.sonatype.com/lift) with `help` or `ignore`)
##########
File path: solr/test-framework/src/java/org/apache/solr/cloud/MiniSolrCloudCluster.java
##########
@@ -788,6 +777,26 @@ public void waitForActiveCollection(String collection, long wait, TimeUnit unit,
public void waitForActiveCollection(String collection, int shards, int totalReplicas) {
waitForActiveCollection(collection, 30, TimeUnit.SECONDS, shards, totalReplicas);
}
+
+ public void waitForActiveCollection(String collection, long wait, TimeUnit unit) {
+ log.info("waitForActiveCollection: {}", collection);
+ CollectionStatePredicate predicate = expectedActive();
+
+ AtomicReference<DocCollection> state = new AtomicReference<>();
+ AtomicReference<Set<String>> liveNodesLastSeen = new AtomicReference<>();
+ try {
+ getSolrClient().waitForState(collection, wait, unit, (n, c) -> {
Review comment:
*THREAD_SAFETY_VIOLATION:* Unprotected write. Non-private method `MiniSolrCloudCluster.waitForActiveCollection(...)` indirectly writes to field `noggit.JSONParser.devNull.buf` outside of synchronization.
Reporting because another access to the same memory occurs on a background thread, although this access may not.
(at-me [in a reply](https://help.sonatype.com/lift) with `help` or `ignore`)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org