You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@solr.apache.org by GitBox <gi...@apache.org> on 2021/08/10 20:28:28 UTC
[GitHub] [solr] markrmiller opened a new pull request #254: Solr 15581
markrmiller opened a new pull request #254:
URL: https://github.com/apache/solr/pull/254
Includes an EnglishWOrdGenerator as an initial proof of concept.
First PR is incomplete, a variety of things to smooth out and straighten up and the EnglishWordGenerator is only single token ATM.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] madrob commented on a change in pull request #254: Solr 15581
Posted by GitBox <gi...@apache.org>.
madrob commented on a change in pull request #254:
URL: https://github.com/apache/solr/pull/254#discussion_r698717978
##########
File path: solr/benchmark/src/java/org/apache/solr/bench/generators/StringsDSL.java
##########
@@ -0,0 +1,305 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.bench.generators;
+
+import static org.apache.solr.bench.generators.SourceDSL.checkArguments;
+import static org.apache.solr.bench.generators.SourceDSL.integers;
+
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.util.List;
+import java.util.SplittableRandom;
+import org.eclipse.jetty.io.RuntimeIOException;
+import org.quicktheories.core.RandomnessSource;
+import org.quicktheories.generators.Generate;
+import org.quicktheories.impl.BenchmarkRandomSource;
+import org.quicktheories.impl.SplittableRandomSource;
+
+public class StringsDSL {
+
+ private static final int BASIC_LATIN_LAST_CODEPOINT = 0x007E;
+ private static final int BASIC_LATIN_FIRST_CODEPOINT = 0x0020;
+ private static final int ASCII_LAST_CODEPOINT = 0x007F;
+ private static final int LARGEST_DEFINED_BMP_CODEPOINT = 65533;
+
+ private static final List<String> words;
+
+ private static final int wordsSize;
+
+ static {
+ try {
+ words =
+ Files.readAllLines(
+ Paths.get(
+ "src",
+ "resources",
+ "words.txt")); // english word list via https://github.com/dwyl/english-words
+ } catch (IOException e) {
+ throw new RuntimeIOException(e);
+ }
+ wordsSize = words.size();
+ }
+
+ public WordListGeneratorBuilder wordList() {
+ return new WordListGeneratorBuilder(
+ new SolrGen(
Review comment:
rawtypes
##########
File path: solr/benchmark/src/java/org/apache/solr/bench/Docs.java
##########
@@ -0,0 +1,199 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.bench;
+
+import static org.apache.solr.bench.BaseBenchState.log;
+
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Queue;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import org.apache.commons.math3.random.RandomGenerator;
+import org.apache.lucene.util.RamUsageEstimator;
+import org.apache.solr.bench.generators.SolrGen;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrInputDocument;
+import org.apache.solr.common.util.SolrNamedThreadFactory;
+import org.apache.solr.common.util.SuppressForbidden;
+import org.quicktheories.core.Gen;
+import org.quicktheories.impl.BenchmarkRandomSource;
+
+/**
+ * A tool to generate controlled random data for a benchmark. {@link SolrInputDocument}s are created
+ * based on supplied FieldDef definitions.
+ *
+ * <p>You can call getDocument to build and retrieve one {@link SolrInputDocument} at a time, or you
+ * can call {@link #preGenerate} to generate the given number of documents in RAM, and then retrieve
+ * them via {@link #generatedDocsIterator}.
+ */
+public class Docs {
+
+ private final ThreadLocal<RandomGenerator> random;
+ private final RandomGenerator randomParent;
+ private Queue<SolrInputDocument> docs = new ConcurrentLinkedQueue<>();
+
+ private final Map<String, Gen<?>> fields = new HashMap<>();
+
+ private static final AtomicInteger ID = new AtomicInteger();
+
+ private ExecutorService executorService;
+ private int stringFields;
+ private int multiStringFields;
+ private int integerFields;
+ private int longFields;
+
+ public static Docs docs() {
+ return new Docs();
+ }
+
+ public static Docs docs(RandomGenerator random) {
+ return new Docs(random);
+ }
+
+ private Docs(RandomGenerator random) {
+ this.randomParent = random;
+ this.random =
+ ThreadLocal.withInitial(
+ () ->
+ new SplittableRandomGenerator(
+ random.nextLong())); // TODO: pluggable RandomGenerator
+ }
+
+ private Docs() {
+ this(new SplittableRandomGenerator(Long.getLong("randomSeed")));
+ }
+
+ @SuppressForbidden(reason = "This module does not need to deal with logging context")
+ public Iterator<SolrInputDocument> preGenerate(int numDocs) throws InterruptedException {
+ log("preGenerate docs " + numDocs + " ...");
+ docs.clear();
+ executorService =
+ Executors.newFixedThreadPool(
+ Runtime.getRuntime().availableProcessors() + 1,
+ new SolrNamedThreadFactory("SolrJMH DocMaker"));
+
+ for (int i = 0; i < numDocs; i++) {
+ executorService.submit(
+ () -> {
+ try {
+ SolrInputDocument doc = Docs.this.inputDocument();
+ docs.add(doc);
+ } catch (Exception e) {
+ executorService.shutdownNow();
+ throw new RuntimeException(e);
+ }
+ });
+ }
+
+ executorService.shutdown();
+ boolean result = executorService.awaitTermination(10, TimeUnit.MINUTES);
+ if (!result) {
+ throw new RuntimeException("Timeout waiting for doc adds to finish");
+ }
+ log(
+ "done preGenerateDocs docs="
+ + docs.size()
+ + " ram="
+ + RamUsageEstimator.humanReadableUnits(RamUsageEstimator.sizeOfObject(docs)));
+
+ if (numDocs != docs.size()) {
+ throw new IllegalStateException("numDocs != " + docs.size());
+ }
+
+ return docs.iterator();
+ }
+
+ public Iterator<SolrInputDocument> generatedDocsIterator() {
+ return docs.iterator();
+ }
+
+ public SolrInputDocument inputDocument() {
+ SolrInputDocument doc = new SolrInputDocument();
+
+ for (Map.Entry<String, Gen<?>> entry : fields.entrySet()) {
+ doc.addField(
+ entry.getKey(), entry.getValue().generate(new BenchmarkRandomSource(random.get())));
+ }
+
+ return doc;
+ }
+
+ public SolrDocument document() {
+ SolrDocument doc = new SolrDocument();
+
+ for (Map.Entry<String, Gen<?>> entry : fields.entrySet()) {
+ doc.addField(
+ entry.getKey(), entry.getValue().generate(new BenchmarkRandomSource(random.get())));
+ }
+
+ return doc;
+ }
+
+ public Docs field(String name, SolrGen<?> generator) {
Review comment:
```suggestion
public Docs field(String name, Gen<?> generator) {
```
##########
File path: solr/benchmark/src/java/org/apache/solr/bench/generators/ArraysDSL.java
##########
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.bench.generators;
+
+import static org.apache.solr.bench.generators.SourceDSL.checkArguments;
+
+import java.lang.reflect.Array;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import org.quicktheories.api.AsString;
+import org.quicktheories.core.Gen;
+import org.quicktheories.generators.Generate;
+
+/**
+ * A Class for creating Sources of type T[], that will generate and shrink array objects of either
+ * fixed or bounded size. If fixed size, an array will be shrunk by reducing the individual items in
+ * tandem. If the array is of bounded size, shrinking will begin by removing random items until the
+ * array is of minimum size, at which point it will continue to shrink as if an array of fixed
+ * length.
+ *
+ * <p>There is a possibility that the shrinker will get stuck at a local minima when one value in
+ * the array shrinks to it's "smallest" value without falsifying
+ */
+public class ArraysDSL {
+
+ /**
+ * Creates an ArrayGeneratorBuilder of Integers that can be used to create an array Source
+ *
+ * @param source a Source of type Integer
+ * @return an ArrayGeneratorBuilder of type Integer
+ */
+ public ArrayGeneratorBuilder<Integer> ofIntegers(Gen<Integer> source) {
+ return new ArrayGeneratorBuilder<>(source, Integer.class);
+ }
+
+ /**
+ * Creates an ArrayGeneratorBuilder of Characters that can be used to create an array Source
+ *
+ * @param source a Source of type Character
+ * @return an ArrayGeneratorBuilder of type Character
+ */
+ public ArrayGeneratorBuilder<Character> ofCharacters(Gen<Character> source) {
+ return new ArrayGeneratorBuilder<>(source, Character.class);
+ }
+
+ /**
+ * Creates an ArrayGeneratorBuilder of Strings that can be used to create an array Source
+ *
+ * @param source a Source of type String
+ * @return an ArrayGeneratorBuilder of type String
+ */
+ public ArrayGeneratorBuilder<String> ofStrings(Gen<String> source) {
+ return new ArrayGeneratorBuilder<>(source, String.class);
+ }
+
+ /**
+ * Creates an ArrayGeneratorBuilder of the given class that can be used to create an array Source
+ *
+ * @param <T> type of value to generate
+ * @param source a Source of type T
+ * @param c a Class of type T
+ * @return an ArrayGeneratorBuilder of type T
+ */
+ public <T> ArrayGeneratorBuilder<T> ofClass(Gen<T> source, Class<T> c) {
+ return new ArrayGeneratorBuilder<>(source, c);
+ }
+
+ public static class ArrayGeneratorBuilder<T> {
+
+ private final Gen<T> source;
+ private final Class<T> c;
+
+ ArrayGeneratorBuilder(Gen<T> source, Class<T> c) {
+ this.source = source;
+ this.c = c;
+ }
+
+ /**
+ * Generates arrays of specified type T of fixed length
+ *
+ * @param length - fixed length
+ * @return a Source of type T[]
+ */
+ public Gen<T[]> withLength(int length) {
+ return withLengthBetween(length, length);
+ }
+
+ public Gen<T[]> withLengths(Gen<Integer> lengths) {
+ return Lists.listsOf(source, Lists.arrayList(), lengths)
+ .map(
+ l -> l.toArray((T[]) Array.newInstance(c, 0)) // will generate
+ // correct size if
+ // zero is less than
+ // the length of the
+ // array
+ )
Review comment:
I think this is a weird thing that spotless apply did, can you put the comment all on one line?
##########
File path: solr/benchmark/src/java/org/apache/solr/bench/generators/SolrGen.java
##########
@@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.bench.generators;
+
+import org.quicktheories.core.Gen;
+import org.quicktheories.core.RandomnessSource;
+import org.quicktheories.impl.BenchmarkRandomSource;
+
+public class SolrGen<T> implements Gen<T> {
+
+ protected final Gen<T> child;
+ private final Type type;
+ private Distribution distribution = Distribution.Uniform;
+
+ SolrGen() {
+ this(Type.Unknown);
+ }
+
+ SolrGen(Gen<T> child, Type type) {
+ this.child = child;
+ if (child instanceof SolrGen) {
+ ((SolrGen<Object>) child).distribution = distribution;
+ }
+ this.type = type;
+ }
+
+ SolrGen(Type type) {
+ child = null;
+ this.type = type;
+ }
+
+ SolrGen(Gen<T> child) {
+ this(child, Type.Unknown);
+ }
+
+ @Override
+ public T generate(RandomnessSource in) {
+ // System.out.println("gen : " + toString() + " child: " + child.toString());
+ if (in instanceof BenchmarkRandomSource) {
+ return child.generate(((BenchmarkRandomSource) in).withDistribution(distribution));
+ }
+ return child.generate(in);
+ }
+
+ public Type type() {
+ return type;
+ }
+
+ public SolrGen<T> tracked(RandomDataHistogram.Counts collector) {
+ return new TrackingGenerator<>(this, collector);
+ }
+
+ public SolrGen<T> withDistribution(Distribution distribution) {
+ // System.out.println("set dist gen : " + toString() + " child: " + child.toString());
+ this.distribution = distribution;
+ if (this.child instanceof SolrGen) {
+ ((SolrGen<Object>) this.child).distribution = distribution;
+ }
+ return this;
+ }
+
+ @Override
+ public String toString() {
+ return "SolrGen{"
+ + "child="
+ + child
+ + ", type="
+ + type
+ + ", distribution="
+ + distribution
+ + '}';
+ }
+
+ protected Distribution getDistribution() {
+ return this.distribution;
+ }
+
+ public class TrackingGenerator<T> extends SolrGen<T> {
Review comment:
This class doesn't make sense to me, it both extends and wraps `SolrGen` when it feels like maybe we only need one or the other. Should probably be a static inner class, regardless.
##########
File path: solr/benchmark/src/java/org/apache/solr/bench/generators/SolrGen.java
##########
@@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.bench.generators;
+
+import org.quicktheories.core.Gen;
+import org.quicktheories.core.RandomnessSource;
+import org.quicktheories.impl.BenchmarkRandomSource;
+
+public class SolrGen<T> implements Gen<T> {
+
+ protected final Gen<T> child;
+ private final Type type;
+ private Distribution distribution = Distribution.Uniform;
+
+ SolrGen() {
+ this(Type.Unknown);
+ }
+
+ SolrGen(Gen<T> child, Type type) {
+ this.child = child;
+ if (child instanceof SolrGen) {
+ ((SolrGen<Object>) child).distribution = distribution;
+ }
+ this.type = type;
+ }
+
+ SolrGen(Type type) {
+ child = null;
+ this.type = type;
+ }
+
+ SolrGen(Gen<T> child) {
+ this(child, Type.Unknown);
+ }
+
+ @Override
+ public T generate(RandomnessSource in) {
+ // System.out.println("gen : " + toString() + " child: " + child.toString());
+ if (in instanceof BenchmarkRandomSource) {
+ return child.generate(((BenchmarkRandomSource) in).withDistribution(distribution));
+ }
+ return child.generate(in);
+ }
+
+ public Type type() {
+ return type;
+ }
+
+ public SolrGen<T> tracked(RandomDataHistogram.Counts collector) {
+ return new TrackingGenerator<>(this, collector);
+ }
+
+ public SolrGen<T> withDistribution(Distribution distribution) {
+ // System.out.println("set dist gen : " + toString() + " child: " + child.toString());
+ this.distribution = distribution;
+ if (this.child instanceof SolrGen) {
+ ((SolrGen<Object>) this.child).distribution = distribution;
Review comment:
```suggestion
((SolrGen<?>) this.child).distribution = distribution;
```
##########
File path: solr/benchmark/src/java/org/apache/solr/bench/generators/SolrGen.java
##########
@@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.bench.generators;
+
+import org.quicktheories.core.Gen;
+import org.quicktheories.core.RandomnessSource;
+import org.quicktheories.impl.BenchmarkRandomSource;
+
+public class SolrGen<T> implements Gen<T> {
+
+ protected final Gen<T> child;
+ private final Type type;
+ private Distribution distribution = Distribution.Uniform;
+
+ SolrGen() {
+ this(Type.Unknown);
+ }
+
+ SolrGen(Gen<T> child, Type type) {
+ this.child = child;
+ if (child instanceof SolrGen) {
+ ((SolrGen<Object>) child).distribution = distribution;
Review comment:
```suggestion
((SolrGen<?>) child).distribution = distribution;
```
##########
File path: solr/benchmark/src/java/org/apache/solr/bench/generators/ArraysDSL.java
##########
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.bench.generators;
+
+import static org.apache.solr.bench.generators.SourceDSL.checkArguments;
+
+import java.lang.reflect.Array;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import org.quicktheories.api.AsString;
+import org.quicktheories.core.Gen;
+import org.quicktheories.generators.Generate;
+
+/**
+ * A Class for creating Sources of type T[], that will generate and shrink array objects of either
+ * fixed or bounded size. If fixed size, an array will be shrunk by reducing the individual items in
+ * tandem. If the array is of bounded size, shrinking will begin by removing random items until the
+ * array is of minimum size, at which point it will continue to shrink as if an array of fixed
+ * length.
+ *
+ * <p>There is a possibility that the shrinker will get stuck at a local minima when one value in
+ * the array shrinks to it's "smallest" value without falsifying
+ */
+public class ArraysDSL {
+
+ /**
+ * Creates an ArrayGeneratorBuilder of Integers that can be used to create an array Source
+ *
+ * @param source a Source of type Integer
+ * @return an ArrayGeneratorBuilder of type Integer
+ */
+ public ArrayGeneratorBuilder<Integer> ofIntegers(Gen<Integer> source) {
+ return new ArrayGeneratorBuilder<>(source, Integer.class);
+ }
+
+ /**
+ * Creates an ArrayGeneratorBuilder of Characters that can be used to create an array Source
+ *
+ * @param source a Source of type Character
+ * @return an ArrayGeneratorBuilder of type Character
+ */
+ public ArrayGeneratorBuilder<Character> ofCharacters(Gen<Character> source) {
+ return new ArrayGeneratorBuilder<>(source, Character.class);
+ }
+
+ /**
+ * Creates an ArrayGeneratorBuilder of Strings that can be used to create an array Source
+ *
+ * @param source a Source of type String
+ * @return an ArrayGeneratorBuilder of type String
+ */
+ public ArrayGeneratorBuilder<String> ofStrings(Gen<String> source) {
+ return new ArrayGeneratorBuilder<>(source, String.class);
+ }
+
+ /**
+ * Creates an ArrayGeneratorBuilder of the given class that can be used to create an array Source
+ *
+ * @param <T> type of value to generate
+ * @param source a Source of type T
+ * @param c a Class of type T
+ * @return an ArrayGeneratorBuilder of type T
+ */
+ public <T> ArrayGeneratorBuilder<T> ofClass(Gen<T> source, Class<T> c) {
+ return new ArrayGeneratorBuilder<>(source, c);
+ }
+
+ public static class ArrayGeneratorBuilder<T> {
+
+ private final Gen<T> source;
+ private final Class<T> c;
+
+ ArrayGeneratorBuilder(Gen<T> source, Class<T> c) {
+ this.source = source;
+ this.c = c;
+ }
+
+ /**
+ * Generates arrays of specified type T of fixed length
+ *
+ * @param length - fixed length
+ * @return a Source of type T[]
+ */
+ public Gen<T[]> withLength(int length) {
+ return withLengthBetween(length, length);
+ }
+
+ public Gen<T[]> withLengths(Gen<Integer> lengths) {
+ return Lists.listsOf(source, Lists.arrayList(), lengths)
+ .map(
+ l -> l.toArray((T[]) Array.newInstance(c, 0)) // will generate
Review comment:
Suppress the unchecked warning here please.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] markrmiller commented on pull request #254: Solr 15581
Posted by GitBox <gi...@apache.org>.
markrmiller commented on pull request #254:
URL: https://github.com/apache/solr/pull/254#issuecomment-896324201
FYI, english words and license from https://github.com/dwyl/english-words/blob/master/LICENSE.md
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] markrmiller commented on pull request #254: Solr 15581
Posted by GitBox <gi...@apache.org>.
markrmiller commented on pull request #254:
URL: https://github.com/apache/solr/pull/254#issuecomment-909438956
@madrob updated pr, not merged up to master this time yet as the last time that bit me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] sonatype-lift[bot] commented on a change in pull request #254: Solr 15581
Posted by GitBox <gi...@apache.org>.
sonatype-lift[bot] commented on a change in pull request #254:
URL: https://github.com/apache/solr/pull/254#discussion_r699665335
##########
File path: solr/benchmark/src/java/org/apache/solr/bench/BaseBenchState.java
##########
@@ -0,0 +1,92 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.bench;
+
+import com.sun.management.HotSpotDiagnosticMXBean;
+import java.io.File;
+import java.io.IOException;
+import java.lang.management.ManagementFactory;
+import java.util.SplittableRandom;
+import javax.management.MBeanServer;
+import org.apache.commons.io.FileUtils;
+import org.apache.solr.common.util.SuppressForbidden;
+import org.openjdk.jmh.annotations.Level;
+import org.openjdk.jmh.annotations.Scope;
+import org.openjdk.jmh.annotations.Setup;
+import org.openjdk.jmh.annotations.State;
+import org.openjdk.jmh.annotations.TearDown;
+import org.openjdk.jmh.infra.BenchmarkParams;
+
+@State(Scope.Benchmark)
+public class BaseBenchState {
+
+ private static final long RANDOM_SEED = 6624420638116043983L;
+
+ private static SplittableRandom random = new SplittableRandom(getInitRandomeSeed());
+
+ public static Long getRandomSeed() {
+ return random.split().nextLong();
+ }
+
+ public static boolean quietLog = Boolean.getBoolean("quietLog");
+
+ @SuppressForbidden(reason = "JMH uses std out for user output")
+ public static void log(String value) {
+ if (!quietLog) {
+ System.out.println((value.isEmpty() ? "" : "--> ") + value);
+ }
+ }
+
+ @Setup(Level.Trial)
+ public void doSetup(BenchmarkParams benchmarkParams) throws Exception {
+ System.setProperty("solr.log.name", benchmarkParams.id());
+ }
+
+ @TearDown(Level.Trial)
+ public static void doTearDown(BenchmarkParams benchmarkParams) throws Exception {
+ String heapDump = System.getProperty("dumpheap");
+ if (heapDump != null) {
+ File file = new File(heapDump);
+ FileUtils.deleteDirectory(file);
+ file.mkdirs();
+ File dumpFile = new File(file, benchmarkParams.id() + ".hprof");
Review comment:
*PATH_TRAVERSAL_IN:* This API (java/io/File.<init>(Ljava/io/File;Ljava/lang/String;)V) reads a file whose location might be specified by user input [(details)](https://find-sec-bugs.github.io/bugs.htm#PATH_TRAVERSAL_IN)
(at-me [in a reply](https://help.sonatype.com/lift) with `help` or `ignore`)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] markrmiller merged pull request #254: Solr 15581
Posted by GitBox <gi...@apache.org>.
markrmiller merged pull request #254:
URL: https://github.com/apache/solr/pull/254
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] sonatype-lift[bot] commented on a change in pull request #254: Solr 15581
Posted by GitBox <gi...@apache.org>.
sonatype-lift[bot] commented on a change in pull request #254:
URL: https://github.com/apache/solr/pull/254#discussion_r699714106
##########
File path: solr/core/src/java/org/apache/solr/core/SolrCore.java
##########
@@ -828,6 +828,7 @@ void initIndex(boolean passOnPreviousState, boolean reload) throws IOException {
try {
writer = SolrIndexWriter.create(this, "SolrCore.initIndex", indexDir, getDirectoryFactory(), true,
Review comment:
*THREAD_SAFETY_VIOLATION:* Unprotected write. Non-private method `SolrCore.initIndex(...)` indirectly mutates container `core.SolrResourceLoader.classNameCache` via call to `Map.put(...)` outside of synchronization.
Reporting because this access may occur on a background thread.
(at-me [in a reply](https://help.sonatype.com/lift) with `help` or `ignore`)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] sonatype-lift[bot] commented on a change in pull request #254: Solr 15581
Posted by GitBox <gi...@apache.org>.
sonatype-lift[bot] commented on a change in pull request #254:
URL: https://github.com/apache/solr/pull/254#discussion_r697185496
##########
File path: solr/benchmark/src/java/org/apache/solr/bench/BaseBenchState.java
##########
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.bench;
+
+import java.util.Random;
+import java.util.SplittableRandom;
+import org.apache.solr.common.util.SuppressForbidden;
+import org.openjdk.jmh.annotations.Level;
+import org.openjdk.jmh.annotations.Scope;
+import org.openjdk.jmh.annotations.Setup;
+import org.openjdk.jmh.annotations.State;
+import org.openjdk.jmh.infra.BenchmarkParams;
+
+@State(Scope.Benchmark)
+public class BaseBenchState {
+
+ private static final long RANDOM_SEED = 6624420638116043983L;
+
+ private SplittableRandom random;
+
+ public static boolean quietLog = Boolean.getBoolean("quietLog");
+
+ @SuppressForbidden(reason = "JMH uses std out for user output")
+ public static void log(String value) {
+ if (!quietLog) {
+ System.out.println((value.equals("") ? "" : "--> ") + value);
+ }
+ }
+
+ @Setup(Level.Trial)
+ public void doSetup(BenchmarkParams benchmarkParams) throws Exception {
+ System.setProperty("solr.log.name", benchmarkParams.id());
+
+ Long seed = getRandomSeed();
+
+ this.random = new SplittableRandom(seed);
+ }
+
+ public static Long getRandomSeed() {
+ Long seed = Long.getLong("solr.bench.seed");
+
+ if (seed == null) {
+ seed = RANDOM_SEED;
+ }
+
+ log("benchmark random seed: " + seed);
+
+ // set the seed used by hard to reach places
+ System.setProperty("randomSeed", Long.toString(new Random(seed).nextLong()));
Review comment:
*PREDICTABLE_RANDOM:* This random generator (java.util.Random) is predictable [(details)](https://find-sec-bugs.github.io/bugs.htm#PREDICTABLE_RANDOM)
(at-me [in a reply](https://help.sonatype.com/lift) with `help` or `ignore`)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] madrob commented on a change in pull request #254: Solr 15581
Posted by GitBox <gi...@apache.org>.
madrob commented on a change in pull request #254:
URL: https://github.com/apache/solr/pull/254#discussion_r698717978
##########
File path: solr/benchmark/src/java/org/apache/solr/bench/generators/StringsDSL.java
##########
@@ -0,0 +1,305 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.bench.generators;
+
+import static org.apache.solr.bench.generators.SourceDSL.checkArguments;
+import static org.apache.solr.bench.generators.SourceDSL.integers;
+
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+import java.util.List;
+import java.util.SplittableRandom;
+import org.eclipse.jetty.io.RuntimeIOException;
+import org.quicktheories.core.RandomnessSource;
+import org.quicktheories.generators.Generate;
+import org.quicktheories.impl.BenchmarkRandomSource;
+import org.quicktheories.impl.SplittableRandomSource;
+
+public class StringsDSL {
+
+ private static final int BASIC_LATIN_LAST_CODEPOINT = 0x007E;
+ private static final int BASIC_LATIN_FIRST_CODEPOINT = 0x0020;
+ private static final int ASCII_LAST_CODEPOINT = 0x007F;
+ private static final int LARGEST_DEFINED_BMP_CODEPOINT = 65533;
+
+ private static final List<String> words;
+
+ private static final int wordsSize;
+
+ static {
+ try {
+ words =
+ Files.readAllLines(
+ Paths.get(
+ "src",
+ "resources",
+ "words.txt")); // english word list via https://github.com/dwyl/english-words
+ } catch (IOException e) {
+ throw new RuntimeIOException(e);
+ }
+ wordsSize = words.size();
+ }
+
+ public WordListGeneratorBuilder wordList() {
+ return new WordListGeneratorBuilder(
+ new SolrGen(
Review comment:
rawtypes
##########
File path: solr/benchmark/src/java/org/apache/solr/bench/Docs.java
##########
@@ -0,0 +1,199 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.bench;
+
+import static org.apache.solr.bench.BaseBenchState.log;
+
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Queue;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import org.apache.commons.math3.random.RandomGenerator;
+import org.apache.lucene.util.RamUsageEstimator;
+import org.apache.solr.bench.generators.SolrGen;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrInputDocument;
+import org.apache.solr.common.util.SolrNamedThreadFactory;
+import org.apache.solr.common.util.SuppressForbidden;
+import org.quicktheories.core.Gen;
+import org.quicktheories.impl.BenchmarkRandomSource;
+
+/**
+ * A tool to generate controlled random data for a benchmark. {@link SolrInputDocument}s are created
+ * based on supplied FieldDef definitions.
+ *
+ * <p>You can call getDocument to build and retrieve one {@link SolrInputDocument} at a time, or you
+ * can call {@link #preGenerate} to generate the given number of documents in RAM, and then retrieve
+ * them via {@link #generatedDocsIterator}.
+ */
+public class Docs {
+
+ private final ThreadLocal<RandomGenerator> random;
+ private final RandomGenerator randomParent;
+ private Queue<SolrInputDocument> docs = new ConcurrentLinkedQueue<>();
+
+ private final Map<String, Gen<?>> fields = new HashMap<>();
+
+ private static final AtomicInteger ID = new AtomicInteger();
+
+ private ExecutorService executorService;
+ private int stringFields;
+ private int multiStringFields;
+ private int integerFields;
+ private int longFields;
+
+ public static Docs docs() {
+ return new Docs();
+ }
+
+ public static Docs docs(RandomGenerator random) {
+ return new Docs(random);
+ }
+
+ private Docs(RandomGenerator random) {
+ this.randomParent = random;
+ this.random =
+ ThreadLocal.withInitial(
+ () ->
+ new SplittableRandomGenerator(
+ random.nextLong())); // TODO: pluggable RandomGenerator
+ }
+
+ private Docs() {
+ this(new SplittableRandomGenerator(Long.getLong("randomSeed")));
+ }
+
+ @SuppressForbidden(reason = "This module does not need to deal with logging context")
+ public Iterator<SolrInputDocument> preGenerate(int numDocs) throws InterruptedException {
+ log("preGenerate docs " + numDocs + " ...");
+ docs.clear();
+ executorService =
+ Executors.newFixedThreadPool(
+ Runtime.getRuntime().availableProcessors() + 1,
+ new SolrNamedThreadFactory("SolrJMH DocMaker"));
+
+ for (int i = 0; i < numDocs; i++) {
+ executorService.submit(
+ () -> {
+ try {
+ SolrInputDocument doc = Docs.this.inputDocument();
+ docs.add(doc);
+ } catch (Exception e) {
+ executorService.shutdownNow();
+ throw new RuntimeException(e);
+ }
+ });
+ }
+
+ executorService.shutdown();
+ boolean result = executorService.awaitTermination(10, TimeUnit.MINUTES);
+ if (!result) {
+ throw new RuntimeException("Timeout waiting for doc adds to finish");
+ }
+ log(
+ "done preGenerateDocs docs="
+ + docs.size()
+ + " ram="
+ + RamUsageEstimator.humanReadableUnits(RamUsageEstimator.sizeOfObject(docs)));
+
+ if (numDocs != docs.size()) {
+ throw new IllegalStateException("numDocs != " + docs.size());
+ }
+
+ return docs.iterator();
+ }
+
+ public Iterator<SolrInputDocument> generatedDocsIterator() {
+ return docs.iterator();
+ }
+
+ public SolrInputDocument inputDocument() {
+ SolrInputDocument doc = new SolrInputDocument();
+
+ for (Map.Entry<String, Gen<?>> entry : fields.entrySet()) {
+ doc.addField(
+ entry.getKey(), entry.getValue().generate(new BenchmarkRandomSource(random.get())));
+ }
+
+ return doc;
+ }
+
+ public SolrDocument document() {
+ SolrDocument doc = new SolrDocument();
+
+ for (Map.Entry<String, Gen<?>> entry : fields.entrySet()) {
+ doc.addField(
+ entry.getKey(), entry.getValue().generate(new BenchmarkRandomSource(random.get())));
+ }
+
+ return doc;
+ }
+
+ public Docs field(String name, SolrGen<?> generator) {
Review comment:
```suggestion
public Docs field(String name, Gen<?> generator) {
```
##########
File path: solr/benchmark/src/java/org/apache/solr/bench/generators/ArraysDSL.java
##########
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.bench.generators;
+
+import static org.apache.solr.bench.generators.SourceDSL.checkArguments;
+
+import java.lang.reflect.Array;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import org.quicktheories.api.AsString;
+import org.quicktheories.core.Gen;
+import org.quicktheories.generators.Generate;
+
+/**
+ * A Class for creating Sources of type T[], that will generate and shrink array objects of either
+ * fixed or bounded size. If fixed size, an array will be shrunk by reducing the individual items in
+ * tandem. If the array is of bounded size, shrinking will begin by removing random items until the
+ * array is of minimum size, at which point it will continue to shrink as if an array of fixed
+ * length.
+ *
+ * <p>There is a possibility that the shrinker will get stuck at a local minima when one value in
+ * the array shrinks to it's "smallest" value without falsifying
+ */
+public class ArraysDSL {
+
+ /**
+ * Creates an ArrayGeneratorBuilder of Integers that can be used to create an array Source
+ *
+ * @param source a Source of type Integer
+ * @return an ArrayGeneratorBuilder of type Integer
+ */
+ public ArrayGeneratorBuilder<Integer> ofIntegers(Gen<Integer> source) {
+ return new ArrayGeneratorBuilder<>(source, Integer.class);
+ }
+
+ /**
+ * Creates an ArrayGeneratorBuilder of Characters that can be used to create an array Source
+ *
+ * @param source a Source of type Character
+ * @return an ArrayGeneratorBuilder of type Character
+ */
+ public ArrayGeneratorBuilder<Character> ofCharacters(Gen<Character> source) {
+ return new ArrayGeneratorBuilder<>(source, Character.class);
+ }
+
+ /**
+ * Creates an ArrayGeneratorBuilder of Strings that can be used to create an array Source
+ *
+ * @param source a Source of type String
+ * @return an ArrayGeneratorBuilder of type String
+ */
+ public ArrayGeneratorBuilder<String> ofStrings(Gen<String> source) {
+ return new ArrayGeneratorBuilder<>(source, String.class);
+ }
+
+ /**
+ * Creates an ArrayGeneratorBuilder of the given class that can be used to create an array Source
+ *
+ * @param <T> type of value to generate
+ * @param source a Source of type T
+ * @param c a Class of type T
+ * @return an ArrayGeneratorBuilder of type T
+ */
+ public <T> ArrayGeneratorBuilder<T> ofClass(Gen<T> source, Class<T> c) {
+ return new ArrayGeneratorBuilder<>(source, c);
+ }
+
+ public static class ArrayGeneratorBuilder<T> {
+
+ private final Gen<T> source;
+ private final Class<T> c;
+
+ ArrayGeneratorBuilder(Gen<T> source, Class<T> c) {
+ this.source = source;
+ this.c = c;
+ }
+
+ /**
+ * Generates arrays of specified type T of fixed length
+ *
+ * @param length - fixed length
+ * @return a Source of type T[]
+ */
+ public Gen<T[]> withLength(int length) {
+ return withLengthBetween(length, length);
+ }
+
+ public Gen<T[]> withLengths(Gen<Integer> lengths) {
+ return Lists.listsOf(source, Lists.arrayList(), lengths)
+ .map(
+ l -> l.toArray((T[]) Array.newInstance(c, 0)) // will generate
+ // correct size if
+ // zero is less than
+ // the length of the
+ // array
+ )
Review comment:
I think this is a weird thing that spotless apply did, can you put the comment all on one line?
##########
File path: solr/benchmark/src/java/org/apache/solr/bench/generators/SolrGen.java
##########
@@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.bench.generators;
+
+import org.quicktheories.core.Gen;
+import org.quicktheories.core.RandomnessSource;
+import org.quicktheories.impl.BenchmarkRandomSource;
+
+public class SolrGen<T> implements Gen<T> {
+
+ protected final Gen<T> child;
+ private final Type type;
+ private Distribution distribution = Distribution.Uniform;
+
+ SolrGen() {
+ this(Type.Unknown);
+ }
+
+ SolrGen(Gen<T> child, Type type) {
+ this.child = child;
+ if (child instanceof SolrGen) {
+ ((SolrGen<Object>) child).distribution = distribution;
+ }
+ this.type = type;
+ }
+
+ SolrGen(Type type) {
+ child = null;
+ this.type = type;
+ }
+
+ SolrGen(Gen<T> child) {
+ this(child, Type.Unknown);
+ }
+
+ @Override
+ public T generate(RandomnessSource in) {
+ // System.out.println("gen : " + toString() + " child: " + child.toString());
+ if (in instanceof BenchmarkRandomSource) {
+ return child.generate(((BenchmarkRandomSource) in).withDistribution(distribution));
+ }
+ return child.generate(in);
+ }
+
+ public Type type() {
+ return type;
+ }
+
+ public SolrGen<T> tracked(RandomDataHistogram.Counts collector) {
+ return new TrackingGenerator<>(this, collector);
+ }
+
+ public SolrGen<T> withDistribution(Distribution distribution) {
+ // System.out.println("set dist gen : " + toString() + " child: " + child.toString());
+ this.distribution = distribution;
+ if (this.child instanceof SolrGen) {
+ ((SolrGen<Object>) this.child).distribution = distribution;
+ }
+ return this;
+ }
+
+ @Override
+ public String toString() {
+ return "SolrGen{"
+ + "child="
+ + child
+ + ", type="
+ + type
+ + ", distribution="
+ + distribution
+ + '}';
+ }
+
+ protected Distribution getDistribution() {
+ return this.distribution;
+ }
+
+ public class TrackingGenerator<T> extends SolrGen<T> {
Review comment:
This class doesn't make sense to me, it both extends and wraps `SolrGen` when it feels like maybe we only need one or the other. Should probably be a static inner class, regardless.
##########
File path: solr/benchmark/src/java/org/apache/solr/bench/generators/SolrGen.java
##########
@@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.bench.generators;
+
+import org.quicktheories.core.Gen;
+import org.quicktheories.core.RandomnessSource;
+import org.quicktheories.impl.BenchmarkRandomSource;
+
+public class SolrGen<T> implements Gen<T> {
+
+ protected final Gen<T> child;
+ private final Type type;
+ private Distribution distribution = Distribution.Uniform;
+
+ SolrGen() {
+ this(Type.Unknown);
+ }
+
+ SolrGen(Gen<T> child, Type type) {
+ this.child = child;
+ if (child instanceof SolrGen) {
+ ((SolrGen<Object>) child).distribution = distribution;
+ }
+ this.type = type;
+ }
+
+ SolrGen(Type type) {
+ child = null;
+ this.type = type;
+ }
+
+ SolrGen(Gen<T> child) {
+ this(child, Type.Unknown);
+ }
+
+ @Override
+ public T generate(RandomnessSource in) {
+ // System.out.println("gen : " + toString() + " child: " + child.toString());
+ if (in instanceof BenchmarkRandomSource) {
+ return child.generate(((BenchmarkRandomSource) in).withDistribution(distribution));
+ }
+ return child.generate(in);
+ }
+
+ public Type type() {
+ return type;
+ }
+
+ public SolrGen<T> tracked(RandomDataHistogram.Counts collector) {
+ return new TrackingGenerator<>(this, collector);
+ }
+
+ public SolrGen<T> withDistribution(Distribution distribution) {
+ // System.out.println("set dist gen : " + toString() + " child: " + child.toString());
+ this.distribution = distribution;
+ if (this.child instanceof SolrGen) {
+ ((SolrGen<Object>) this.child).distribution = distribution;
Review comment:
```suggestion
((SolrGen<?>) this.child).distribution = distribution;
```
##########
File path: solr/benchmark/src/java/org/apache/solr/bench/generators/SolrGen.java
##########
@@ -0,0 +1,135 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.bench.generators;
+
+import org.quicktheories.core.Gen;
+import org.quicktheories.core.RandomnessSource;
+import org.quicktheories.impl.BenchmarkRandomSource;
+
+public class SolrGen<T> implements Gen<T> {
+
+ protected final Gen<T> child;
+ private final Type type;
+ private Distribution distribution = Distribution.Uniform;
+
+ SolrGen() {
+ this(Type.Unknown);
+ }
+
+ SolrGen(Gen<T> child, Type type) {
+ this.child = child;
+ if (child instanceof SolrGen) {
+ ((SolrGen<Object>) child).distribution = distribution;
Review comment:
```suggestion
((SolrGen<?>) child).distribution = distribution;
```
##########
File path: solr/benchmark/src/java/org/apache/solr/bench/generators/ArraysDSL.java
##########
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.bench.generators;
+
+import static org.apache.solr.bench.generators.SourceDSL.checkArguments;
+
+import java.lang.reflect.Array;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+import org.quicktheories.api.AsString;
+import org.quicktheories.core.Gen;
+import org.quicktheories.generators.Generate;
+
+/**
+ * A Class for creating Sources of type T[], that will generate and shrink array objects of either
+ * fixed or bounded size. If fixed size, an array will be shrunk by reducing the individual items in
+ * tandem. If the array is of bounded size, shrinking will begin by removing random items until the
+ * array is of minimum size, at which point it will continue to shrink as if an array of fixed
+ * length.
+ *
+ * <p>There is a possibility that the shrinker will get stuck at a local minima when one value in
+ * the array shrinks to it's "smallest" value without falsifying
+ */
+public class ArraysDSL {
+
+ /**
+ * Creates an ArrayGeneratorBuilder of Integers that can be used to create an array Source
+ *
+ * @param source a Source of type Integer
+ * @return an ArrayGeneratorBuilder of type Integer
+ */
+ public ArrayGeneratorBuilder<Integer> ofIntegers(Gen<Integer> source) {
+ return new ArrayGeneratorBuilder<>(source, Integer.class);
+ }
+
+ /**
+ * Creates an ArrayGeneratorBuilder of Characters that can be used to create an array Source
+ *
+ * @param source a Source of type Character
+ * @return an ArrayGeneratorBuilder of type Character
+ */
+ public ArrayGeneratorBuilder<Character> ofCharacters(Gen<Character> source) {
+ return new ArrayGeneratorBuilder<>(source, Character.class);
+ }
+
+ /**
+ * Creates an ArrayGeneratorBuilder of Strings that can be used to create an array Source
+ *
+ * @param source a Source of type String
+ * @return an ArrayGeneratorBuilder of type String
+ */
+ public ArrayGeneratorBuilder<String> ofStrings(Gen<String> source) {
+ return new ArrayGeneratorBuilder<>(source, String.class);
+ }
+
+ /**
+ * Creates an ArrayGeneratorBuilder of the given class that can be used to create an array Source
+ *
+ * @param <T> type of value to generate
+ * @param source a Source of type T
+ * @param c a Class of type T
+ * @return an ArrayGeneratorBuilder of type T
+ */
+ public <T> ArrayGeneratorBuilder<T> ofClass(Gen<T> source, Class<T> c) {
+ return new ArrayGeneratorBuilder<>(source, c);
+ }
+
+ public static class ArrayGeneratorBuilder<T> {
+
+ private final Gen<T> source;
+ private final Class<T> c;
+
+ ArrayGeneratorBuilder(Gen<T> source, Class<T> c) {
+ this.source = source;
+ this.c = c;
+ }
+
+ /**
+ * Generates arrays of specified type T of fixed length
+ *
+ * @param length - fixed length
+ * @return a Source of type T[]
+ */
+ public Gen<T[]> withLength(int length) {
+ return withLengthBetween(length, length);
+ }
+
+ public Gen<T[]> withLengths(Gen<Integer> lengths) {
+ return Lists.listsOf(source, Lists.arrayList(), lengths)
+ .map(
+ l -> l.toArray((T[]) Array.newInstance(c, 0)) // will generate
Review comment:
Suppress the unchecked warning here please.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] markrmiller commented on pull request #254: Solr 15581
Posted by GitBox <gi...@apache.org>.
markrmiller commented on pull request #254:
URL: https://github.com/apache/solr/pull/254#issuecomment-909438956
@madrob updated pr, not merged up to master this time yet as the last time that bit me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] sonatype-lift[bot] commented on a change in pull request #254: Solr 15581
Posted by GitBox <gi...@apache.org>.
sonatype-lift[bot] commented on a change in pull request #254:
URL: https://github.com/apache/solr/pull/254#discussion_r699665335
##########
File path: solr/benchmark/src/java/org/apache/solr/bench/BaseBenchState.java
##########
@@ -0,0 +1,92 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.bench;
+
+import com.sun.management.HotSpotDiagnosticMXBean;
+import java.io.File;
+import java.io.IOException;
+import java.lang.management.ManagementFactory;
+import java.util.SplittableRandom;
+import javax.management.MBeanServer;
+import org.apache.commons.io.FileUtils;
+import org.apache.solr.common.util.SuppressForbidden;
+import org.openjdk.jmh.annotations.Level;
+import org.openjdk.jmh.annotations.Scope;
+import org.openjdk.jmh.annotations.Setup;
+import org.openjdk.jmh.annotations.State;
+import org.openjdk.jmh.annotations.TearDown;
+import org.openjdk.jmh.infra.BenchmarkParams;
+
+@State(Scope.Benchmark)
+public class BaseBenchState {
+
+ private static final long RANDOM_SEED = 6624420638116043983L;
+
+ private static SplittableRandom random = new SplittableRandom(getInitRandomeSeed());
+
+ public static Long getRandomSeed() {
+ return random.split().nextLong();
+ }
+
+ public static boolean quietLog = Boolean.getBoolean("quietLog");
+
+ @SuppressForbidden(reason = "JMH uses std out for user output")
+ public static void log(String value) {
+ if (!quietLog) {
+ System.out.println((value.isEmpty() ? "" : "--> ") + value);
+ }
+ }
+
+ @Setup(Level.Trial)
+ public void doSetup(BenchmarkParams benchmarkParams) throws Exception {
+ System.setProperty("solr.log.name", benchmarkParams.id());
+ }
+
+ @TearDown(Level.Trial)
+ public static void doTearDown(BenchmarkParams benchmarkParams) throws Exception {
+ String heapDump = System.getProperty("dumpheap");
+ if (heapDump != null) {
+ File file = new File(heapDump);
+ FileUtils.deleteDirectory(file);
+ file.mkdirs();
+ File dumpFile = new File(file, benchmarkParams.id() + ".hprof");
Review comment:
*PATH_TRAVERSAL_IN:* This API (java/io/File.<init>(Ljava/io/File;Ljava/lang/String;)V) reads a file whose location might be specified by user input [(details)](https://find-sec-bugs.github.io/bugs.htm#PATH_TRAVERSAL_IN)
(at-me [in a reply](https://help.sonatype.com/lift) with `help` or `ignore`)
##########
File path: solr/core/src/java/org/apache/solr/core/SolrCore.java
##########
@@ -828,6 +828,7 @@ void initIndex(boolean passOnPreviousState, boolean reload) throws IOException {
try {
writer = SolrIndexWriter.create(this, "SolrCore.initIndex", indexDir, getDirectoryFactory(), true,
Review comment:
*THREAD_SAFETY_VIOLATION:* Unprotected write. Non-private method `SolrCore.initIndex(...)` indirectly mutates container `core.SolrResourceLoader.classNameCache` via call to `Map.put(...)` outside of synchronization.
Reporting because this access may occur on a background thread.
(at-me [in a reply](https://help.sonatype.com/lift) with `help` or `ignore`)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
[GitHub] [solr] sonatype-lift[bot] commented on a change in pull request #254: Solr 15581
Posted by GitBox <gi...@apache.org>.
sonatype-lift[bot] commented on a change in pull request #254:
URL: https://github.com/apache/solr/pull/254#discussion_r686357228
##########
File path: solr/benchmark/src/java/org/apache/solr/bench/FieldDefValueGenerator.java
##########
@@ -0,0 +1,242 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.bench;
+
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Queue;
+import java.util.Random;
+import java.util.SplittableRandom;
+import java.util.concurrent.ConcurrentLinkedQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.atomic.AtomicInteger;
+import org.apache.commons.lang3.RandomStringUtils;
+import org.apache.commons.lang3.Validate;
+import org.apache.lucene.util.TestUtil;
+import org.apache.solr.common.SolrInputDocument;
+
+/** */
+public class FieldDefValueGenerator implements ValueGenerator {
+
+ private final FieldDef fieldDef;
+ private Queue<SolrInputDocument> docs = new ConcurrentLinkedQueue<>();
+
+ private final Map<String, FieldDef> fields = new HashMap<>();
+
+ private static final AtomicInteger ID = new AtomicInteger();
+
+ private ExecutorService executorService;
+ private volatile Object lastValue;
+
+ public FieldDefValueGenerator(FieldDef fieldDef) {
+ this.fieldDef = fieldDef;
+ }
+
+ @Override
+ public Object getNextValue(SplittableRandom random) {
+ Object value;
+ switch (fieldDef.getContent()) {
+ case UNIQUE_INT:
+ value = ID.incrementAndGet();
+ break;
+ case INTEGER:
+ if (fieldDef.getMaxCardinality() > 0) {
+ long start = fieldDef.getCardinalityStart();
+ long seed = nextLong(start, start + fieldDef.getMaxCardinality(), random.split());
+ value = Integer.toString(nextInt(0, Integer.MAX_VALUE, new SplittableRandom(seed)));
+ break;
+ }
+
+ value = Integer.toString(random.nextInt(Integer.MAX_VALUE));
+ break;
+ case ALPHEBETIC:
+ value = getString(fieldDef, val -> getAlphabeticString(fieldDef, random), random);
+ break;
+ case UNICODE:
+ value = getString(fieldDef, val -> getUnicodeString(fieldDef, random), random);
+ break;
+ default:
+ throw new UnsupportedOperationException(
+ "Unsupported content type type=" + fieldDef.getContent());
+ }
+
+ lastValue = value;
+ return value;
+ }
+
+ @Override
+ public Object getLastValue() {
+ return lastValue;
+ }
+
+ private String getString(
+ FieldDef fieldDef, StringSupplier supplier, SplittableRandom threadRandom) {
+ if (fieldDef.getNumTokens() > 1 || fieldDef.getMaxNumTokens() > 1) {
+ StringBuilder sb =
+ new StringBuilder(
+ fieldDef.getNumTokens()
+ * (Math.max(fieldDef.getLength(), fieldDef.getMaxLength()) + 1));
+ SplittableRandom random = threadRandom.split();
+ for (int i = 0;
+ i
+ < (fieldDef.getMaxNumTokens() > 1
+ ? random.nextInt(1, fieldDef.getMaxNumTokens())
+ : fieldDef.getNumTokens());
+ i++) {
+ if (i > 0) {
+ sb.append(' ');
+ }
+ sb.append(supplier.getString(fieldDef));
+ }
+ return sb.toString();
+ }
+ return supplier.getString(fieldDef);
+ }
+
+ private String getUnicodeString(FieldDef fieldDef, SplittableRandom threadRandom) {
+ try {
+ if (fieldDef.getMaxCardinality() > 0) {
+ long start = fieldDef.getCardinalityStart();
+ long seed = nextLong(start, start + fieldDef.getMaxCardinality(), threadRandom.split());
+ if (fieldDef.getLength() > -1) {
+ return TestUtil.randomRealisticUnicodeString(
+ new Random(seed), fieldDef.getLength(), fieldDef.getLength());
+ } else {
+ return TestUtil.randomRealisticUnicodeString(
+ new Random(seed), 1, fieldDef.getMaxLength());
+ }
+ }
+
+ if (fieldDef.getLength() > -1) {
+ return TestUtil.randomRealisticUnicodeString(
+ new Random(threadRandom.nextLong()), fieldDef.getLength(), fieldDef.getLength());
+ } else {
+ return TestUtil.randomRealisticUnicodeString(
+ new Random(threadRandom.nextLong()), 1, fieldDef.getMaxLength());
+ }
+ } catch (Exception e) {
+ throw new RuntimeException("Failed getting UnicodeString with FieldDef=" + fieldDef, e);
+ }
+ }
+
+ private String getAlphabeticString(FieldDef fieldDef, SplittableRandom threadRandom) {
+ try {
+ if (fieldDef.getMaxCardinality() > 0) {
+ long start = fieldDef.getCardinalityStart();
+ long seed = nextLong(start, start + fieldDef.getMaxCardinality(), threadRandom.split());
+ SplittableRandom random = new SplittableRandom(seed);
+ if (fieldDef.getLength() > -1) {
+ return RandomStringUtils.random(
+ nextInt(fieldDef.getLength(), fieldDef.getLength(), random),
Review comment:
*PREDICTABLE_RANDOM:* This random generator (java.util.Random) is predictable [(details)](https://find-sec-bugs.github.io/bugs.htm#PREDICTABLE_RANDOM)
(at-me [in a reply](https://help.sonatype.com/lift) with `help` or `ignore`)
##########
File path: solr/benchmark/src/java/org/apache/solr/bench/UniformLongGenerator.java
##########
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.bench;
+
+import java.util.SplittableRandom;
+import java.util.concurrent.ThreadLocalRandom;
+
+/** Generates longs randomly uniform from an interval. */
+public class UniformLongGenerator implements ValueGenerator {
+ private final long lb, ub, interval;
+ private long lastValue;
+
+ /**
+ * Creates a generator that will return longs uniformly randomly from the interval [lb,ub]
+ * inclusive (that is, lb and ub are possible values) (lb and ub are possible values).
+ *
+ * @param lb the lower bound (inclusive) of generated values
+ * @param ub the upper bound (inclusive) of generated values
+ */
+ public UniformLongGenerator(long lb, long ub) {
+ this.lb = lb;
+ this.ub = ub;
+ interval = this.ub - this.lb + 1;
+ }
+
+ @Override
+ public Long getNextValue(SplittableRandom random) {
+ long ret = Math.abs(ThreadLocalRandom.current().nextLong()) % interval + lb;
Review comment:
*PREDICTABLE_RANDOM:* This random generator (java.util.concurrent.ThreadLocalRandom) is predictable [(details)](https://find-sec-bugs.github.io/bugs.htm#PREDICTABLE_RANDOM)
(at-me [in a reply](https://help.sonatype.com/lift) with `help` or `ignore`)
##########
File path: solr/benchmark/src/java/org/apache/solr/bench/ZipfianGenerator.java
##########
@@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.bench;
+
+import java.util.SplittableRandom;
+
+/**
+ * A generator of a zipfian distribution. It produces a sequence of items, such that some items are
+ * more popular than others, according to a zipfian distribution. When you construct an instance of
+ * this class, you specify the number of items in the set to draw from, either by specifying an
+ * itemcount (so that the sequence is of items from 0 to itemcount-1) or by specifying a min and a
+ * max (so that the sequence is of items from min to max inclusive). After you construct the
+ * instance, you can change the number of items by calling nextInt(itemcount) or
+ * nextLong(itemcount).
+ *
+ * <p>Note that the popular items will be clustered together, e.g. item 0 is the most popular, item
+ * 1 the second most popular, and so on (or min is the most popular, min+1 the next most popular,
+ * etc.) If you don't want this clustering, and instead want the popular items scattered throughout
+ * the item space, then use ScrambledZipfianGenerator instead.
+ *
+ * <p>Be aware: initializing this generator may take a long time if there are lots of items to
+ * choose from (e.g. over a minute for 100 million objects). This is because certain mathematical
+ * values need to be computed to properly generate a zipfian skew, and one of those values (zeta) is
+ * a sum sequence from 1 to n, where n is the itemcount. Note that if you increase the number of
+ * items in the set, we can compute a new zeta incrementally, so it should be fast unless you have
+ * added millions of items. However, if you decrease the number of items, we recompute zeta from
+ * scratch, so this can take a long time.
+ *
+ * <p>The algorithm used here is from "Quickly Generating Billion-Record Synthetic Databases", Jim
+ * Gray et al, SIGMOD 1994.
+ */
+public class ZipfianGenerator implements ValueGenerator {
+ public static final double ZIPFIAN_CONSTANT = 0.99;
+
+ /** Number of items. */
+ private final long items;
+
+ /** Min item to generate. */
+ private final long base;
+
+ /** The zipfian constant to use. */
+ private final double zipfianconstant;
+
+ /** Computed parameters for generating the distribution. */
+ private double alpha, zetan, eta, theta, zeta2theta;
+
+ /** The number of items used to compute zetan the last time. */
+ private long countforzeta;
+
+ /**
+ * Flag to prevent problems. If you increase the number of items the zipfian generator is allowed
+ * to choose from, this code will incrementally compute a new zeta value for the larger itemcount.
+ * However, if you decrease the number of items, the code computes zeta from scratch; this is
+ * expensive for large itemsets. Usually this is not intentional; e.g. one thread thinks the
+ * number of items is 1001 and calls "nextLong()" with that item count; then another thread who
+ * thinks the number of items is 1000 calls nextLong() with itemcount=1000 triggering the
+ * expensive recomputation. (It is expensive for 100 million items, not really for 1000 items.)
+ * Why did the second thread think there were only 1000 items? maybe it read the item count before
+ * the first thread incremented it. So this flag allows you to say if you really do want that
+ * recomputation. If true, then the code will recompute zeta if the itemcount goes down. If false,
+ * the code will assume itemcount only goes up, and never recompute.
+ */
+ private boolean allowitemcountdecrease = false;
+
+ private long lastValue;
+
+ /******************************* Constructors **************************************/
+
+ /**
+ * Create a zipfian generator for the specified number of items.
+ *
+ * @param items The number of items in the distribution.
+ */
+ public ZipfianGenerator(long items, SplittableRandom random) {
+ this(0, items - 1, random);
+ }
+
+ /**
+ * Create a zipfian generator for items between min and max.
+ *
+ * @param min The smallest integer to generate in the sequence.
+ * @param max The largest integer to generate in the sequence.
+ */
+ public ZipfianGenerator(long min, long max, SplittableRandom random) {
+ this(min, max, ZIPFIAN_CONSTANT, random);
+ }
+
+ /**
+ * Create a zipfian generator for the specified number of items using the specified zipfian
+ * constant.
+ *
+ * @param items The number of items in the distribution.
+ * @param zipfianconstant The zipfian constant to use.
+ */
+ public ZipfianGenerator(long items, double zipfianconstant, SplittableRandom random) {
+ this(0, items - 1, zipfianconstant, random);
+ }
+
+ /**
+ * Create a zipfian generator for items between min and max (inclusive) for the specified zipfian
+ * constant.
+ *
+ * @param min The smallest integer to generate in the sequence.
+ * @param max The largest integer to generate in the sequence.
+ * @param zipfianconstant The zipfian constant to use.
+ */
+ public ZipfianGenerator(long min, long max, double zipfianconstant, SplittableRandom random) {
+ this(min, max, zipfianconstant, zetastatic(max - min + 1, zipfianconstant), random);
+ }
+
+ /**
+ * Create a zipfian generator for items between min and max (inclusive) for the specified zipfian
+ * constant, using the precomputed value of zeta.
+ *
+ * @param min The smallest integer to generate in the sequence.
+ * @param max The largest integer to generate in the sequence.
+ * @param zipfianconstant The zipfian constant to use.
+ * @param zetan The precomputed zeta constant.
+ */
+ public ZipfianGenerator(
+ long min, long max, double zipfianconstant, double zetan, SplittableRandom random) {
+
+ items = max - min + 1;
+ base = min;
+ this.zipfianconstant = zipfianconstant;
+
+ theta = this.zipfianconstant;
+
+ zeta2theta = zeta(2, theta);
+
+ alpha = 1.0 / (1.0 - theta);
+ this.zetan = zetan;
+ countforzeta = items;
+ eta = (1 - Math.pow(2.0 / items, 1 - theta)) / (1 - zeta2theta / this.zetan);
+
+ getNextValue(random);
+ }
+
+ /**************************************************************************/
+
+ /**
+ * Compute the zeta constant needed for the distribution. Do this from scratch for a distribution
+ * with n items, using the zipfian constant thetaVal. Remember the value of n, so if we change the
+ * itemcount, we can recompute zeta.
+ *
+ * @param n The number of items to compute zeta over.
+ * @param thetaVal The zipfian constant.
+ */
+ double zeta(long n, double thetaVal) {
+ countforzeta = n;
Review comment:
*THREAD_SAFETY_VIOLATION:* Unprotected write. Non-private method `ZipfianGenerator.zeta(...)` writes to field `this.countforzeta` outside of synchronization.
Reporting because another access to the same memory occurs on a background thread, although this access may not.
(at-me [in a reply](https://help.sonatype.com/lift) with `help` or `ignore`)
##########
File path: solr/benchmark/src/java/org/apache/solr/bench/ZipfianGenerator.java
##########
@@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.bench;
+
+import java.util.SplittableRandom;
+
+/**
+ * A generator of a zipfian distribution. It produces a sequence of items, such that some items are
+ * more popular than others, according to a zipfian distribution. When you construct an instance of
+ * this class, you specify the number of items in the set to draw from, either by specifying an
+ * itemcount (so that the sequence is of items from 0 to itemcount-1) or by specifying a min and a
+ * max (so that the sequence is of items from min to max inclusive). After you construct the
+ * instance, you can change the number of items by calling nextInt(itemcount) or
+ * nextLong(itemcount).
+ *
+ * <p>Note that the popular items will be clustered together, e.g. item 0 is the most popular, item
+ * 1 the second most popular, and so on (or min is the most popular, min+1 the next most popular,
+ * etc.) If you don't want this clustering, and instead want the popular items scattered throughout
+ * the item space, then use ScrambledZipfianGenerator instead.
+ *
+ * <p>Be aware: initializing this generator may take a long time if there are lots of items to
+ * choose from (e.g. over a minute for 100 million objects). This is because certain mathematical
+ * values need to be computed to properly generate a zipfian skew, and one of those values (zeta) is
+ * a sum sequence from 1 to n, where n is the itemcount. Note that if you increase the number of
+ * items in the set, we can compute a new zeta incrementally, so it should be fast unless you have
+ * added millions of items. However, if you decrease the number of items, we recompute zeta from
+ * scratch, so this can take a long time.
+ *
+ * <p>The algorithm used here is from "Quickly Generating Billion-Record Synthetic Databases", Jim
+ * Gray et al, SIGMOD 1994.
+ */
+public class ZipfianGenerator implements ValueGenerator {
+ public static final double ZIPFIAN_CONSTANT = 0.99;
+
+ /** Number of items. */
+ private final long items;
+
+ /** Min item to generate. */
+ private final long base;
+
+ /** The zipfian constant to use. */
+ private final double zipfianconstant;
+
+ /** Computed parameters for generating the distribution. */
+ private double alpha, zetan, eta, theta, zeta2theta;
+
+ /** The number of items used to compute zetan the last time. */
+ private long countforzeta;
+
+ /**
+ * Flag to prevent problems. If you increase the number of items the zipfian generator is allowed
+ * to choose from, this code will incrementally compute a new zeta value for the larger itemcount.
+ * However, if you decrease the number of items, the code computes zeta from scratch; this is
+ * expensive for large itemsets. Usually this is not intentional; e.g. one thread thinks the
+ * number of items is 1001 and calls "nextLong()" with that item count; then another thread who
+ * thinks the number of items is 1000 calls nextLong() with itemcount=1000 triggering the
+ * expensive recomputation. (It is expensive for 100 million items, not really for 1000 items.)
+ * Why did the second thread think there were only 1000 items? maybe it read the item count before
+ * the first thread incremented it. So this flag allows you to say if you really do want that
+ * recomputation. If true, then the code will recompute zeta if the itemcount goes down. If false,
+ * the code will assume itemcount only goes up, and never recompute.
+ */
+ private boolean allowitemcountdecrease = false;
+
+ private long lastValue;
+
+ /******************************* Constructors **************************************/
+
+ /**
+ * Create a zipfian generator for the specified number of items.
+ *
+ * @param items The number of items in the distribution.
+ */
+ public ZipfianGenerator(long items, SplittableRandom random) {
+ this(0, items - 1, random);
+ }
+
+ /**
+ * Create a zipfian generator for items between min and max.
+ *
+ * @param min The smallest integer to generate in the sequence.
+ * @param max The largest integer to generate in the sequence.
+ */
+ public ZipfianGenerator(long min, long max, SplittableRandom random) {
+ this(min, max, ZIPFIAN_CONSTANT, random);
+ }
+
+ /**
+ * Create a zipfian generator for the specified number of items using the specified zipfian
+ * constant.
+ *
+ * @param items The number of items in the distribution.
+ * @param zipfianconstant The zipfian constant to use.
+ */
+ public ZipfianGenerator(long items, double zipfianconstant, SplittableRandom random) {
+ this(0, items - 1, zipfianconstant, random);
+ }
+
+ /**
+ * Create a zipfian generator for items between min and max (inclusive) for the specified zipfian
+ * constant.
+ *
+ * @param min The smallest integer to generate in the sequence.
+ * @param max The largest integer to generate in the sequence.
+ * @param zipfianconstant The zipfian constant to use.
+ */
+ public ZipfianGenerator(long min, long max, double zipfianconstant, SplittableRandom random) {
+ this(min, max, zipfianconstant, zetastatic(max - min + 1, zipfianconstant), random);
+ }
+
+ /**
+ * Create a zipfian generator for items between min and max (inclusive) for the specified zipfian
+ * constant, using the precomputed value of zeta.
+ *
+ * @param min The smallest integer to generate in the sequence.
+ * @param max The largest integer to generate in the sequence.
+ * @param zipfianconstant The zipfian constant to use.
+ * @param zetan The precomputed zeta constant.
+ */
+ public ZipfianGenerator(
+ long min, long max, double zipfianconstant, double zetan, SplittableRandom random) {
+
+ items = max - min + 1;
+ base = min;
+ this.zipfianconstant = zipfianconstant;
+
+ theta = this.zipfianconstant;
+
+ zeta2theta = zeta(2, theta);
+
+ alpha = 1.0 / (1.0 - theta);
+ this.zetan = zetan;
+ countforzeta = items;
+ eta = (1 - Math.pow(2.0 / items, 1 - theta)) / (1 - zeta2theta / this.zetan);
+
+ getNextValue(random);
+ }
+
+ /**************************************************************************/
+
+ /**
+ * Compute the zeta constant needed for the distribution. Do this from scratch for a distribution
+ * with n items, using the zipfian constant thetaVal. Remember the value of n, so if we change the
+ * itemcount, we can recompute zeta.
+ *
+ * @param n The number of items to compute zeta over.
+ * @param thetaVal The zipfian constant.
+ */
+ double zeta(long n, double thetaVal) {
+ countforzeta = n;
+ return zetastatic(n, thetaVal);
+ }
+
+ /**
+ * Compute the zeta constant needed for the distribution. Do this from scratch for a distribution
+ * with n items, using the zipfian constant theta. This is a static version of the function which
+ * will not remember n.
+ *
+ * @param n The number of items to compute zeta over.
+ * @param theta The zipfian constant.
+ */
+ static double zetastatic(long n, double theta) {
+ return zetastatic(0, n, theta, 0);
+ }
+
+ /**
+ * Compute the zeta constant needed for the distribution. Do this incrementally for a distribution
+ * that has n items now but used to have st items. Use the zipfian constant thetaVal. Remember the
+ * new value of n so that if we change the itemcount, we'll know to recompute zeta.
+ *
+ * @param st The number of items used to compute the last initialsum
+ * @param n The number of items to compute zeta over.
+ * @param thetaVal The zipfian constant.
+ * @param initialsum The value of zeta we are computing incrementally from.
+ */
+ double zeta(long st, long n, double thetaVal, double initialsum) {
+ countforzeta = n;
Review comment:
*THREAD_SAFETY_VIOLATION:* Unprotected write. Non-private method `ZipfianGenerator.zeta(...)` writes to field `this.countforzeta` outside of synchronization.
Reporting because another access to the same memory occurs on a background thread, although this access may not.
(at-me [in a reply](https://help.sonatype.com/lift) with `help` or `ignore`)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org