You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@solr.apache.org by "gerlowskija (via GitHub)" <gi...@apache.org> on 2025/09/23 19:12:17 UTC
[PR] Add automation for running gatling nightly [solr-sandbox]
gerlowskija opened a new pull request, #124:
URL: https://github.com/apache/solr-sandbox/pull/124
Primary entrypoint is `run-benchmark-on-commits.sh`, which takes a
list of commits (or a "branch", "startCommit" and "endCommit")
and iterates over them, running Gatling benchmarks on each and storing
the results in a data directory.
Users who want to automate running benchmarks periodically (i.e.
nightly) may use the provided `periodic-benchmark.sh` script. This isn't
as nice as having a full Jenkins setup, that's probably the "right" way to
do this sort of recurring automation.
But providing a simple path for folks to get up and running that doesn't
involve installing and running Jenkins seems like a nice 80/20 tradeoff
that lowers the barrier-to-entry a good deal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Add automation for running gatling nightly [solr-sandbox]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on code in PR #124:
URL: https://github.com/apache/solr-sandbox/pull/124#discussion_r2398592741
##########
scripts/gatling/lib/solr.sh:
##########
@@ -0,0 +1,66 @@
+#!/bin/bash
+
+function solr_checkout_source_in() {
+ if [[ -z $1 ]]; then
+ >&2 echo "Required argument 'location' was not provided"
+ return 1
+ fi
+
+ local location=$1
+ if [[ -d $location/solr ]]; then
+ # The checkout already exists I guess?
+ return 0
+ fi
+
+ pushd $location
+ git clone git@github.com:apache/solr.git
+ popd
+}
+
+# Returns with an exit status indicating whether Solr is (or isn't)
+# running at the specified port. Callers provide the port as a required
+# argument. Assumes that the wording directory is positioned so that
+# 'bin/solr' can be invoked.
+# Usage: solr_is_running 8983
+function solr_is_running() {
+ if [[ -z ${1:-} ]]; then
+ >&2 echo "Required argument 'port' was not provided"
+ return 1
+ fi
+
+ local port=$1
+ bin/solr assert --started "http://localhost:$port/solr"
+ return $?
+}
+
+function solr_kill_all() {
+ ps -ef | grep solr | grep java | awk {'print $2'} | xargs kill -9
+}
+
+# Start Solr (with custom ops pulled from START_SOLR_OPTS)
+# Assumes cwd of the Solr package root
+function solr_start() {
+ bin/solr start ${START_SOLR_OPTS:-}
Review Comment:
we are going to want to add some backcompat in solr 10 so -c doesn't raise an error I bet!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Add automation for running gatling nightly [solr-sandbox]
Posted by "gerlowskija (via GitHub)" <gi...@apache.org>.
gerlowskija commented on code in PR #124:
URL: https://github.com/apache/solr-sandbox/pull/124#discussion_r2419903196
##########
scripts/gatling/run-benchmark-on-commits.sh:
##########
@@ -0,0 +1,102 @@
+#!/bin/bash -x
+
+set -eu
+
+SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
+LIB_DIR="$SCRIPT_DIR/lib"
+SANDBOX_CHECKOUT_ROOT="$SCRIPT_DIR/../../"
+source $LIB_DIR/env-state.sh
+source $LIB_DIR/git.sh
+source $LIB_DIR/solr.sh
+source $LIB_DIR/gatling.sh
+
+###########################
+# "Main" - begin arg parsing
+COMMIT_HASHES=""
+START_HASH=""
+END_HASH=""
+BRANCH="main"
+
+if [ $# -gt 0 ]; then
+ while [ $# -gt 0 ]; do
+ case "${1:-}" in
+ # Inclusive
+ -s|--start-hash)
+ START_HASH=${2}
+ shift 2
+ ;;
+ # Inclusive
+ -e|--end-hash)
+ END_HASH=${2}
+ shift 2
+ ;;
+ -b|--branch)
+ BRANCH=${2}
+ shift 2
+ ;;
+ -c|--commit-hashes)
+ COMMIT_HASHES=${2}
+ shift 2
+ ;;
+ *)
+ shift
+ ;;
+ esac
+ done
+fi
+
+###############################################
+# Identify the branch and the commits to run on
+env_state_bootstrap
+
+pushd $BENCH_SOLR_CHECKOUT_DIR
+ if [[ -n "${COMMIT_HASHES}" ]]; then
+ COMMIT_HASHES="$(echo "$COMMIT_HASHES" | sed 's/,/\n/g')"
+ else # [[ -z "${COMMIT_HASHES}" ]]; then
+ if [[ -z "${START_HASH}" ]]; then
+ >&2 echo "Either '-c' or '-s'/'-e' argument must be provided"
+ exit 1
+ else
+ git_update_checkout $BRANCH
+ COMMIT_HASHES="$(git_list_commits_since $START_HASH $END_HASH)"
+ fi
+ fi
+popd
+
+####################################################
+# Download any benchmark-data needed for simulations
+pushd $SANDBOX_CHECKOUT_ROOT
+ gatling_download_wiki_data
+popd
+
+######################################################
+# Iterate over commits, building and benchmarking each
+pushd $BENCH_SOLR_CHECKOUT_DIR
+ for commit in $(echo "$COMMIT_HASHES") ; do
+ echo "Processing commit: [$commit]"
+
+ # Build and start Solr
+ git_checkout "$commit"
+ solr_kill_all
+ solr_build_package
+ package_dir="$(solr_get_package_directory)"
+ pushd $package_dir
+ export START_SOLR_OPTS=" -m 4g "
+ solr_start
Review Comment:
Ah, that makes sense then.
The way `START_SOLR_OPTS` works in this scripts, it's used to provide additional CLI args to the `bin/solr start` command. So you'd need to use the system-property syntax:
```
export START_SOLR_OPTS="-Dsolr.jetty.host=0.0.0.0"
./run-benchmark-on-commits.sh <script-args>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Add automation for running gatling nightly [solr-sandbox]
Posted by "gerlowskija (via GitHub)" <gi...@apache.org>.
gerlowskija commented on PR #124:
URL: https://github.com/apache/solr-sandbox/pull/124#issuecomment-3373255165
Alright, I think I've addressed all your comments above except for the `SOLR_JETTY_HOST` bit @epugh . Happy to revise this more, but otherwise will aim to merge in the next day or two and we can iterate from there! Thanks for the review!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Add automation for running gatling nightly [solr-sandbox]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on code in PR #124:
URL: https://github.com/apache/solr-sandbox/pull/124#discussion_r2398518595
##########
scripts/gatling/run-benchmark-on-commits.sh:
##########
@@ -0,0 +1,102 @@
+#!/bin/bash -x
+
+set -eu
+
+SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
+LIB_DIR="$SCRIPT_DIR/lib"
+SANDBOX_CHECKOUT_ROOT="$SCRIPT_DIR/../../"
+source $LIB_DIR/env-state.sh
+source $LIB_DIR/git.sh
+source $LIB_DIR/solr.sh
+source $LIB_DIR/gatling.sh
+
+###########################
+# "Main" - begin arg parsing
+COMMIT_HASHES=""
+START_HASH=""
+END_HASH=""
+BRANCH="main"
+
+if [ $# -gt 0 ]; then
+ while [ $# -gt 0 ]; do
+ case "${1:-}" in
+ # Inclusive
+ -s|--start-hash)
+ START_HASH=${2}
+ shift 2
+ ;;
+ # Inclusive
+ -e|--end-hash)
+ END_HASH=${2}
+ shift 2
+ ;;
+ -b|--branch)
+ BRANCH=${2}
+ shift 2
+ ;;
+ -c|--commit-hashes)
+ COMMIT_HASHES=${2}
+ shift 2
+ ;;
+ *)
+ shift
+ ;;
+ esac
+ done
+fi
+
+###############################################
+# Identify the branch and the commits to run on
+env_state_bootstrap
+
+pushd $BENCH_SOLR_CHECKOUT_DIR
+ if [[ -n "${COMMIT_HASHES}" ]]; then
+ COMMIT_HASHES="$(echo "$COMMIT_HASHES" | sed 's/,/\n/g')"
+ else # [[ -z "${COMMIT_HASHES}" ]]; then
+ if [[ -z "${START_HASH}" ]]; then
+ >&2 echo "Either '-c' or '-s'/'-e' argument must be provided"
+ exit 1
+ else
+ git_update_checkout $BRANCH
+ COMMIT_HASHES="$(git_list_commits_since $START_HASH $END_HASH)"
+ fi
+ fi
+popd
+
+####################################################
+# Download any benchmark-data needed for simulations
+pushd $SANDBOX_CHECKOUT_ROOT
+ gatling_download_wiki_data
+popd
+
+######################################################
+# Iterate over commits, building and benchmarking each
+pushd $BENCH_SOLR_CHECKOUT_DIR
+ for commit in $(echo "$COMMIT_HASHES") ; do
+ echo "Processing commit: [$commit]"
+
+ # Build and start Solr
+ git_checkout "$commit"
+ solr_kill_all
+ solr_build_package
+ package_dir="$(solr_get_package_directory)"
+ pushd $package_dir
+ export START_SOLR_OPTS=" -m 4g "
Review Comment:
does this overwrite my own START_SOLR_OPTS ???
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Add automation for running gatling nightly [solr-sandbox]
Posted by "gerlowskija (via GitHub)" <gi...@apache.org>.
gerlowskija commented on code in PR #124:
URL: https://github.com/apache/solr-sandbox/pull/124#discussion_r2407660974
##########
scripts/gatling/run-benchmark-on-commits.sh:
##########
@@ -0,0 +1,102 @@
+#!/bin/bash -x
+
+set -eu
+
+SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
+LIB_DIR="$SCRIPT_DIR/lib"
+SANDBOX_CHECKOUT_ROOT="$SCRIPT_DIR/../../"
+source $LIB_DIR/env-state.sh
+source $LIB_DIR/git.sh
+source $LIB_DIR/solr.sh
+source $LIB_DIR/gatling.sh
+
+###########################
+# "Main" - begin arg parsing
+COMMIT_HASHES=""
+START_HASH=""
+END_HASH=""
+BRANCH="main"
+
+if [ $# -gt 0 ]; then
+ while [ $# -gt 0 ]; do
+ case "${1:-}" in
+ # Inclusive
+ -s|--start-hash)
+ START_HASH=${2}
+ shift 2
+ ;;
+ # Inclusive
+ -e|--end-hash)
+ END_HASH=${2}
+ shift 2
+ ;;
+ -b|--branch)
+ BRANCH=${2}
+ shift 2
+ ;;
+ -c|--commit-hashes)
+ COMMIT_HASHES=${2}
+ shift 2
+ ;;
+ *)
+ shift
+ ;;
+ esac
+ done
+fi
+
+###############################################
+# Identify the branch and the commits to run on
+env_state_bootstrap
+
+pushd $BENCH_SOLR_CHECKOUT_DIR
+ if [[ -n "${COMMIT_HASHES}" ]]; then
+ COMMIT_HASHES="$(echo "$COMMIT_HASHES" | sed 's/,/\n/g')"
+ else # [[ -z "${COMMIT_HASHES}" ]]; then
+ if [[ -z "${START_HASH}" ]]; then
+ >&2 echo "Either '-c' or '-s'/'-e' argument must be provided"
+ exit 1
+ else
+ git_update_checkout $BRANCH
+ COMMIT_HASHES="$(git_list_commits_since $START_HASH $END_HASH)"
+ fi
+ fi
+popd
+
+####################################################
+# Download any benchmark-data needed for simulations
+pushd $SANDBOX_CHECKOUT_ROOT
+ gatling_download_wiki_data
+popd
+
+######################################################
+# Iterate over commits, building and benchmarking each
+pushd $BENCH_SOLR_CHECKOUT_DIR
+ for commit in $(echo "$COMMIT_HASHES") ; do
+ echo "Processing commit: [$commit]"
+
+ # Build and start Solr
+ git_checkout "$commit"
+ solr_kill_all
+ solr_build_package
+ package_dir="$(solr_get_package_directory)"
+ pushd $package_dir
+ export START_SOLR_OPTS=" -m 4g "
Review Comment:
Fixed; good catch!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Add automation for running gatling nightly [solr-sandbox]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on PR #124:
URL: https://github.com/apache/solr-sandbox/pull/124#issuecomment-3374688000
awesome.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Add automation for running gatling nightly [solr-sandbox]
Posted by "gerlowskija (via GitHub)" <gi...@apache.org>.
gerlowskija commented on code in PR #124:
URL: https://github.com/apache/solr-sandbox/pull/124#discussion_r2407656835
##########
scripts/gatling/lib/solr.sh:
##########
@@ -0,0 +1,66 @@
+#!/bin/bash
+
+function solr_checkout_source_in() {
+ if [[ -z $1 ]]; then
+ >&2 echo "Required argument 'location' was not provided"
+ return 1
+ fi
+
+ local location=$1
+ if [[ -d $location/solr ]]; then
+ # The checkout already exists I guess?
+ return 0
+ fi
+
+ pushd $location
+ git clone git@github.com:apache/solr.git
+ popd
+}
+
+# Returns with an exit status indicating whether Solr is (or isn't)
+# running at the specified port. Callers provide the port as a required
+# argument. Assumes that the wording directory is positioned so that
+# 'bin/solr' can be invoked.
+# Usage: solr_is_running 8983
+function solr_is_running() {
+ if [[ -z ${1:-} ]]; then
+ >&2 echo "Required argument 'port' was not provided"
+ return 1
+ fi
+
+ local port=$1
+ bin/solr assert --started "http://localhost:$port/solr"
+ return $?
+}
+
+function solr_kill_all() {
+ ps -ef | grep solr | grep java | awk {'print $2'} | xargs kill -9
+}
+
+# Start Solr (with custom ops pulled from START_SOLR_OPTS)
+# Assumes cwd of the Solr package root
+function solr_start() {
+ bin/solr start ${START_SOLR_OPTS:-}
Review Comment:
Yeah, that's why the current version omits the '-c'; I've mostly been testing on `main`.
For now I've tweaked the command so that we try both with and without the `-c` before erroring out. We'll want a better solution to these types of Solr major version syntax differences at some point, but I'm going to punt on figuring out that better solution now. Hopefully as we see more of these types of things we'll get a better sense of what that should look like.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Add automation for running gatling nightly [solr-sandbox]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on code in PR #124:
URL: https://github.com/apache/solr-sandbox/pull/124#discussion_r2396375861
##########
scripts/gatling/lib/solr.sh:
##########
@@ -0,0 +1,66 @@
+#!/bin/bash
+
+function solr_checkout_source_in() {
+ if [[ -z $1 ]]; then
+ >&2 echo "Required argument 'location' was not provided"
+ return 1
+ fi
+
+ local location=$1
+ if [[ -d $location/solr ]]; then
+ # The checkout already exists I guess?
+ return 0
+ fi
+
+ pushd $location
+ git clone git@github.com:apache/solr.git
Review Comment:
git clone https://github.com/apache/solr.git so you don't need keys to be set up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Add automation for running gatling nightly [solr-sandbox]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on code in PR #124:
URL: https://github.com/apache/solr-sandbox/pull/124#discussion_r2396376428
##########
scripts/gatling/lib/solr.sh:
##########
@@ -0,0 +1,66 @@
+#!/bin/bash
+
+function solr_checkout_source_in() {
+ if [[ -z $1 ]]; then
+ >&2 echo "Required argument 'location' was not provided"
+ return 1
+ fi
+
+ local location=$1
+ if [[ -d $location/solr ]]; then
+ # The checkout already exists I guess?
+ return 0
+ fi
+
+ pushd $location
+ git clone git@github.com:apache/solr.git
Review Comment:
```suggestion
git clone https://github.com/apache/solr.git
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Add automation for running gatling nightly [solr-sandbox]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on code in PR #124:
URL: https://github.com/apache/solr-sandbox/pull/124#discussion_r2398591259
##########
scripts/gatling/lib/solr.sh:
##########
@@ -0,0 +1,66 @@
+#!/bin/bash
+
+function solr_checkout_source_in() {
+ if [[ -z $1 ]]; then
+ >&2 echo "Required argument 'location' was not provided"
+ return 1
+ fi
+
+ local location=$1
+ if [[ -d $location/solr ]]; then
+ # The checkout already exists I guess?
+ return 0
+ fi
+
+ pushd $location
+ git clone git@github.com:apache/solr.git
+ popd
+}
+
+# Returns with an exit status indicating whether Solr is (or isn't)
+# running at the specified port. Callers provide the port as a required
+# argument. Assumes that the wording directory is positioned so that
+# 'bin/solr' can be invoked.
+# Usage: solr_is_running 8983
+function solr_is_running() {
+ if [[ -z ${1:-} ]]; then
+ >&2 echo "Required argument 'port' was not provided"
+ return 1
+ fi
+
+ local port=$1
+ bin/solr assert --started "http://localhost:$port/solr"
+ return $?
+}
+
+function solr_kill_all() {
+ ps -ef | grep solr | grep java | awk {'print $2'} | xargs kill -9
+}
+
+# Start Solr (with custom ops pulled from START_SOLR_OPTS)
+# Assumes cwd of the Solr package root
+function solr_start() {
+ bin/solr start ${START_SOLR_OPTS:-}
Review Comment:
```suggestion
bin/solr start -c ${START_SOLR_OPTS:-}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Add automation for running gatling nightly [solr-sandbox]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on PR #124:
URL: https://github.com/apache/solr-sandbox/pull/124#issuecomment-3358686529
to get solr to start, i had to add `export SOLR_JETTY_HOST=0.0.0.0` or it never saw that solr started...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Add automation for running gatling nightly [solr-sandbox]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on code in PR #124:
URL: https://github.com/apache/solr-sandbox/pull/124#discussion_r2398503510
##########
scripts/gatling/lib/solr.sh:
##########
@@ -0,0 +1,66 @@
+#!/bin/bash
+
+function solr_checkout_source_in() {
+ if [[ -z $1 ]]; then
+ >&2 echo "Required argument 'location' was not provided"
+ return 1
+ fi
+
+ local location=$1
+ if [[ -d $location/solr ]]; then
+ # The checkout already exists I guess?
+ return 0
+ fi
+
+ pushd $location
+ git clone git@github.com:apache/solr.git
+ popd
+}
+
+# Returns with an exit status indicating whether Solr is (or isn't)
+# running at the specified port. Callers provide the port as a required
+# argument. Assumes that the wording directory is positioned so that
+# 'bin/solr' can be invoked.
+# Usage: solr_is_running 8983
+function solr_is_running() {
+ if [[ -z ${1:-} ]]; then
+ >&2 echo "Required argument 'port' was not provided"
+ return 1
+ fi
+
+ local port=$1
+ bin/solr assert --started "http://localhost:$port/solr"
+ return $?
+}
+
+function solr_kill_all() {
+ ps -ef | grep solr | grep java | awk {'print $2'} | xargs kill -9
Review Comment:
```suggestion
ps -ef | grep solr | grep java | awk {'print $2'} | xargs -r kill -9
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Add automation for running gatling nightly [solr-sandbox]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on code in PR #124:
URL: https://github.com/apache/solr-sandbox/pull/124#discussion_r2407808469
##########
scripts/gatling/run-benchmark-on-commits.sh:
##########
@@ -0,0 +1,102 @@
+#!/bin/bash -x
+
+set -eu
+
+SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
+LIB_DIR="$SCRIPT_DIR/lib"
+SANDBOX_CHECKOUT_ROOT="$SCRIPT_DIR/../../"
+source $LIB_DIR/env-state.sh
+source $LIB_DIR/git.sh
+source $LIB_DIR/solr.sh
+source $LIB_DIR/gatling.sh
+
+###########################
+# "Main" - begin arg parsing
+COMMIT_HASHES=""
+START_HASH=""
+END_HASH=""
+BRANCH="main"
+
+if [ $# -gt 0 ]; then
+ while [ $# -gt 0 ]; do
+ case "${1:-}" in
+ # Inclusive
+ -s|--start-hash)
+ START_HASH=${2}
+ shift 2
+ ;;
+ # Inclusive
+ -e|--end-hash)
+ END_HASH=${2}
+ shift 2
+ ;;
+ -b|--branch)
+ BRANCH=${2}
+ shift 2
+ ;;
+ -c|--commit-hashes)
+ COMMIT_HASHES=${2}
+ shift 2
+ ;;
+ *)
+ shift
+ ;;
+ esac
+ done
+fi
+
+###############################################
+# Identify the branch and the commits to run on
+env_state_bootstrap
+
+pushd $BENCH_SOLR_CHECKOUT_DIR
+ if [[ -n "${COMMIT_HASHES}" ]]; then
+ COMMIT_HASHES="$(echo "$COMMIT_HASHES" | sed 's/,/\n/g')"
+ else # [[ -z "${COMMIT_HASHES}" ]]; then
+ if [[ -z "${START_HASH}" ]]; then
+ >&2 echo "Either '-c' or '-s'/'-e' argument must be provided"
+ exit 1
+ else
+ git_update_checkout $BRANCH
+ COMMIT_HASHES="$(git_list_commits_since $START_HASH $END_HASH)"
+ fi
+ fi
+popd
+
+####################################################
+# Download any benchmark-data needed for simulations
+pushd $SANDBOX_CHECKOUT_ROOT
+ gatling_download_wiki_data
+popd
+
+######################################################
+# Iterate over commits, building and benchmarking each
+pushd $BENCH_SOLR_CHECKOUT_DIR
+ for commit in $(echo "$COMMIT_HASHES") ; do
+ echo "Processing commit: [$commit]"
+
+ # Build and start Solr
+ git_checkout "$commit"
+ solr_kill_all
+ solr_build_package
+ package_dir="$(solr_get_package_directory)"
+ pushd $package_dir
+ export START_SOLR_OPTS=" -m 4g "
+ solr_start
Review Comment:
I suppose then, it should be passing in `SOLR_JETTY_HOST` as a`START_SOLR_OPTS` option?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Add automation for running gatling nightly [solr-sandbox]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on PR #124:
URL: https://github.com/apache/solr-sandbox/pull/124#issuecomment-3358686862
also had to add `--force` to start command.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Add automation for running gatling nightly [solr-sandbox]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on code in PR #124:
URL: https://github.com/apache/solr-sandbox/pull/124#discussion_r2407799563
##########
scripts/gatling/run-benchmark-on-commits.sh:
##########
@@ -0,0 +1,102 @@
+#!/bin/bash -x
+
+set -eu
+
+SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
+LIB_DIR="$SCRIPT_DIR/lib"
+SANDBOX_CHECKOUT_ROOT="$SCRIPT_DIR/../../"
+source $LIB_DIR/env-state.sh
+source $LIB_DIR/git.sh
+source $LIB_DIR/solr.sh
+source $LIB_DIR/gatling.sh
+
+###########################
+# "Main" - begin arg parsing
+COMMIT_HASHES=""
+START_HASH=""
+END_HASH=""
+BRANCH="main"
+
+if [ $# -gt 0 ]; then
+ while [ $# -gt 0 ]; do
+ case "${1:-}" in
+ # Inclusive
+ -s|--start-hash)
+ START_HASH=${2}
+ shift 2
+ ;;
+ # Inclusive
+ -e|--end-hash)
+ END_HASH=${2}
+ shift 2
+ ;;
+ -b|--branch)
+ BRANCH=${2}
+ shift 2
+ ;;
+ -c|--commit-hashes)
+ COMMIT_HASHES=${2}
+ shift 2
+ ;;
+ *)
+ shift
+ ;;
+ esac
+ done
+fi
+
+###############################################
+# Identify the branch and the commits to run on
+env_state_bootstrap
+
+pushd $BENCH_SOLR_CHECKOUT_DIR
+ if [[ -n "${COMMIT_HASHES}" ]]; then
+ COMMIT_HASHES="$(echo "$COMMIT_HASHES" | sed 's/,/\n/g')"
+ else # [[ -z "${COMMIT_HASHES}" ]]; then
+ if [[ -z "${START_HASH}" ]]; then
+ >&2 echo "Either '-c' or '-s'/'-e' argument must be provided"
+ exit 1
+ else
+ git_update_checkout $BRANCH
+ COMMIT_HASHES="$(git_list_commits_since $START_HASH $END_HASH)"
+ fi
+ fi
+popd
+
+####################################################
+# Download any benchmark-data needed for simulations
+pushd $SANDBOX_CHECKOUT_ROOT
+ gatling_download_wiki_data
+popd
+
+######################################################
+# Iterate over commits, building and benchmarking each
+pushd $BENCH_SOLR_CHECKOUT_DIR
+ for commit in $(echo "$COMMIT_HASHES") ; do
+ echo "Processing commit: [$commit]"
+
+ # Build and start Solr
+ git_checkout "$commit"
+ solr_kill_all
+ solr_build_package
+ package_dir="$(solr_get_package_directory)"
+ pushd $package_dir
+ export START_SOLR_OPTS=" -m 4g "
+ solr_start
Review Comment:
so, yeah, I wanted to acces solr from externally. I was running solr on a Digital Ocean droplet, and wanted to log into solr and see how it was doing...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Add automation for running gatling nightly [solr-sandbox]
Posted by "gerlowskija (via GitHub)" <gi...@apache.org>.
gerlowskija commented on PR #124:
URL: https://github.com/apache/solr-sandbox/pull/124#issuecomment-3373124302
> I don't love the hidden .solr-benchmarks. I needed to pull the reports down for more processing and it was hard to find it.
Fair - I was kindof following the model of tools like Gradle, and of our own releaseWizard which creates a similar directory for holding state related to different releases. I didn't like the thought of polluting the checkout dir, etc.
Open to suggestions though, if you have a particular (non-checkout) place you think makes sense for storing that state? In the meantime I'll aim to make it configurable to folks can specify their own if they want.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Add automation for running gatling nightly [solr-sandbox]
Posted by "gerlowskija (via GitHub)" <gi...@apache.org>.
gerlowskija merged PR #124:
URL: https://github.com/apache/solr-sandbox/pull/124
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Add automation for running gatling nightly [solr-sandbox]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on code in PR #124:
URL: https://github.com/apache/solr-sandbox/pull/124#discussion_r2398536922
##########
scripts/gatling/run-benchmark-on-commits.sh:
##########
@@ -0,0 +1,102 @@
+#!/bin/bash -x
+
+set -eu
+
+SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
+LIB_DIR="$SCRIPT_DIR/lib"
+SANDBOX_CHECKOUT_ROOT="$SCRIPT_DIR/../../"
+source $LIB_DIR/env-state.sh
+source $LIB_DIR/git.sh
+source $LIB_DIR/solr.sh
+source $LIB_DIR/gatling.sh
+
+###########################
+# "Main" - begin arg parsing
+COMMIT_HASHES=""
+START_HASH=""
+END_HASH=""
+BRANCH="main"
+
+if [ $# -gt 0 ]; then
+ while [ $# -gt 0 ]; do
+ case "${1:-}" in
+ # Inclusive
+ -s|--start-hash)
+ START_HASH=${2}
+ shift 2
+ ;;
+ # Inclusive
+ -e|--end-hash)
+ END_HASH=${2}
+ shift 2
+ ;;
+ -b|--branch)
+ BRANCH=${2}
+ shift 2
+ ;;
+ -c|--commit-hashes)
+ COMMIT_HASHES=${2}
+ shift 2
+ ;;
+ *)
+ shift
+ ;;
+ esac
+ done
+fi
+
+###############################################
+# Identify the branch and the commits to run on
+env_state_bootstrap
+
+pushd $BENCH_SOLR_CHECKOUT_DIR
+ if [[ -n "${COMMIT_HASHES}" ]]; then
+ COMMIT_HASHES="$(echo "$COMMIT_HASHES" | sed 's/,/\n/g')"
+ else # [[ -z "${COMMIT_HASHES}" ]]; then
+ if [[ -z "${START_HASH}" ]]; then
+ >&2 echo "Either '-c' or '-s'/'-e' argument must be provided"
+ exit 1
+ else
+ git_update_checkout $BRANCH
+ COMMIT_HASHES="$(git_list_commits_since $START_HASH $END_HASH)"
+ fi
+ fi
+popd
+
+####################################################
+# Download any benchmark-data needed for simulations
+pushd $SANDBOX_CHECKOUT_ROOT
+ gatling_download_wiki_data
+popd
+
+######################################################
+# Iterate over commits, building and benchmarking each
+pushd $BENCH_SOLR_CHECKOUT_DIR
+ for commit in $(echo "$COMMIT_HASHES") ; do
+ echo "Processing commit: [$commit]"
+
+ # Build and start Solr
+ git_checkout "$commit"
+ solr_kill_all
+ solr_build_package
+ package_dir="$(solr_get_package_directory)"
+ pushd $package_dir
+ export START_SOLR_OPTS=" -m 4g "
+ solr_start
Review Comment:
```suggestion
export SOLR_JETTY_HOST=0.0.0.0
solr_start
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Add automation for running gatling nightly [solr-sandbox]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on code in PR #124:
URL: https://github.com/apache/solr-sandbox/pull/124#discussion_r2398504819
##########
scripts/gatling/lib/solr.sh:
##########
@@ -0,0 +1,66 @@
+#!/bin/bash
+
+function solr_checkout_source_in() {
+ if [[ -z $1 ]]; then
+ >&2 echo "Required argument 'location' was not provided"
+ return 1
+ fi
+
+ local location=$1
+ if [[ -d $location/solr ]]; then
+ # The checkout already exists I guess?
+ return 0
+ fi
+
+ pushd $location
+ git clone git@github.com:apache/solr.git
+ popd
+}
+
+# Returns with an exit status indicating whether Solr is (or isn't)
+# running at the specified port. Callers provide the port as a required
+# argument. Assumes that the wording directory is positioned so that
+# 'bin/solr' can be invoked.
+# Usage: solr_is_running 8983
+function solr_is_running() {
+ if [[ -z ${1:-} ]]; then
+ >&2 echo "Required argument 'port' was not provided"
+ return 1
+ fi
+
+ local port=$1
+ bin/solr assert --started "http://localhost:$port/solr"
+ return $?
+}
+
+function solr_kill_all() {
+ ps -ef | grep solr | grep java | awk {'print $2'} | xargs kill -9
Review Comment:
if solr isn't running the then kill gets no args, and errors... so add `-r`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Add automation for running gatling nightly [solr-sandbox]
Posted by "epugh (via GitHub)" <gi...@apache.org>.
epugh commented on PR #124:
URL: https://github.com/apache/solr-sandbox/pull/124#issuecomment-3362004403
I don't love the hidden `.solr-benchmarks`. I needed to pull the reports down for more processing and it was hard to find it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org
Re: [PR] Add automation for running gatling nightly [solr-sandbox]
Posted by "gerlowskija (via GitHub)" <gi...@apache.org>.
gerlowskija commented on code in PR #124:
URL: https://github.com/apache/solr-sandbox/pull/124#discussion_r2407689858
##########
scripts/gatling/run-benchmark-on-commits.sh:
##########
@@ -0,0 +1,102 @@
+#!/bin/bash -x
+
+set -eu
+
+SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
+LIB_DIR="$SCRIPT_DIR/lib"
+SANDBOX_CHECKOUT_ROOT="$SCRIPT_DIR/../../"
+source $LIB_DIR/env-state.sh
+source $LIB_DIR/git.sh
+source $LIB_DIR/solr.sh
+source $LIB_DIR/gatling.sh
+
+###########################
+# "Main" - begin arg parsing
+COMMIT_HASHES=""
+START_HASH=""
+END_HASH=""
+BRANCH="main"
+
+if [ $# -gt 0 ]; then
+ while [ $# -gt 0 ]; do
+ case "${1:-}" in
+ # Inclusive
+ -s|--start-hash)
+ START_HASH=${2}
+ shift 2
+ ;;
+ # Inclusive
+ -e|--end-hash)
+ END_HASH=${2}
+ shift 2
+ ;;
+ -b|--branch)
+ BRANCH=${2}
+ shift 2
+ ;;
+ -c|--commit-hashes)
+ COMMIT_HASHES=${2}
+ shift 2
+ ;;
+ *)
+ shift
+ ;;
+ esac
+ done
+fi
+
+###############################################
+# Identify the branch and the commits to run on
+env_state_bootstrap
+
+pushd $BENCH_SOLR_CHECKOUT_DIR
+ if [[ -n "${COMMIT_HASHES}" ]]; then
+ COMMIT_HASHES="$(echo "$COMMIT_HASHES" | sed 's/,/\n/g')"
+ else # [[ -z "${COMMIT_HASHES}" ]]; then
+ if [[ -z "${START_HASH}" ]]; then
+ >&2 echo "Either '-c' or '-s'/'-e' argument must be provided"
+ exit 1
+ else
+ git_update_checkout $BRANCH
+ COMMIT_HASHES="$(git_list_commits_since $START_HASH $END_HASH)"
+ fi
+ fi
+popd
+
+####################################################
+# Download any benchmark-data needed for simulations
+pushd $SANDBOX_CHECKOUT_ROOT
+ gatling_download_wiki_data
+popd
+
+######################################################
+# Iterate over commits, building and benchmarking each
+pushd $BENCH_SOLR_CHECKOUT_DIR
+ for commit in $(echo "$COMMIT_HASHES") ; do
+ echo "Processing commit: [$commit]"
+
+ # Build and start Solr
+ git_checkout "$commit"
+ solr_kill_all
+ solr_build_package
+ package_dir="$(solr_get_package_directory)"
+ pushd $package_dir
+ export START_SOLR_OPTS=" -m 4g "
+ solr_start
Review Comment:
This is a bit of a headscratcher IMO. On one hand I believe you when you say that you were unable to reach Solr without this environment variable, but on the other - I can't reproduce it and it really shouldn't be necessary.
`SOLR_JETTY_HOST` is a setting used to control which network interface Solr listens on. The default is `127.0.0.1`, meaning that Solr's Jetty should only accept connections coming from its own box...but that should be fine since that's exactly how this script is set up: Solr and Gatling are running on the same machine.
Any chance you have something unique in your `/etc/hosts` or other networking configuration that might've required this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org