You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@couchdb.apache.org by pg...@apache.org on 2023/12/05 11:48:26 UTC
(couchdb) branch main updated (9fa56e087 -> 4ace5f139)
This is an automated email from the ASF dual-hosted git repository.
pgj pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/couchdb.git
from 9fa56e087 Bump 3.3.3 version numbers
new a8b4bb2f7 Make the Clouseau development flow more flexible
new 4ace5f139 mango: increase timeout for flaky `text` search test with floats
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
Makefile | 15 ++--
Makefile.win | 15 ++--
README-DEV.rst | 25 ++++++
configure | 125 +++++++++++++++++++++---------
configure.ps1 | 86 +++++++++++----------
dev/run | 145 +++++++++++++++++++++++++++++------
src/mango/test/06-basic-text-test.py | 3 +-
7 files changed, 297 insertions(+), 117 deletions(-)
(couchdb) 01/02: Make the Clouseau development flow more flexible
Posted by pg...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
pgj pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/couchdb.git
commit a8b4bb2f7659ce09a51ff1346070e75838ac064c
Author: Gabor Pali <ga...@ibm.com>
AuthorDate: Thu Nov 30 02:36:56 2023 +0100
Make the Clouseau development flow more flexible
This change features a set of extensions to the integration of
Clouseau in the developer setup, added in 2d8a33b9:
- Introduce the `--clouseau-dir` option for the `./dev/run` script
to allow for ad-hoc integration of Clouseau with CouchDB, even
without being configured previously. This facilities
experimentation with various alternatives and lets the user switch
between them with ease.
- The directory specified by `--clouseau-dir` could either hold an
unpacked set of JAR files or `git` clone of the Clouseau source
code. The script can automagically determine which version is
being used and invoke Apache Maven [1] to compile the sources and
launch the nodes with the proper classpath. In this latter case,
check for JDK 1.7 because that is the edition that could be used
for compiling the sources currently.
- Modify the `./configure` script to support deploying locally a
specific branch of Clouseau's `git` clone instead of a tagged
release.
- Adjust the `Makefile`s to set `--clouseau-dir` by the
`CLOUSEAU_DIR` variable.
[1] https://maven.apache.org/
---
Makefile | 15 +++---
Makefile.win | 15 +++---
README-DEV.rst | 25 ++++++++++
configure | 125 ++++++++++++++++++++++++++++++++++---------------
configure.ps1 | 86 +++++++++++++++++-----------------
dev/run | 145 +++++++++++++++++++++++++++++++++++++++++++++++----------
6 files changed, 295 insertions(+), 116 deletions(-)
diff --git a/Makefile b/Makefile
index 35e0252d8..22c4217ff 100644
--- a/Makefile
+++ b/Makefile
@@ -262,13 +262,20 @@ elixir: elixir-init devclean
--erlang-config rel/files/eunit.config \
--no-eval 'mix test --trace --include test/elixir/test/config/suite.elixir --exclude test/elixir/test/config/skip.elixir $(EXUNIT_OPTS)'
+ifneq ($(CLOUSEAU_DIR),)
+_WITH_CLOUSEAU="--with-clouseau --clouseau-dir=$(CLOUSEAU_DIR)"
+else ifeq ($(with_clouseau), 1)
+_WITH_CLOUSEAU="--with-clouseau"
+endif
+
.PHONY: elixir-search
# target: elixir-search - Run search tests, requires a configured Clouseau instance
elixir-search: export MIX_ENV=integration
elixir-search: elixir-init devclean
-ifeq ($(with_clouseau), 1)
+ifneq ($(_WITH_CLOUSEAU), )
@dev/run -n 1 -q -a adm:pass \
- --with-clouseau \
+ "$(_WITH_CLOUSEAU)" \
+ "$(TEST_OPTS)" \
--locald-config test/config/test-config.ini \
--no-eval 'mix test --trace --include test/elixir/test/config/search.elixir'
else
@@ -318,10 +325,6 @@ list-eunit-suites:
build-test:
@test/build/test-configure.sh
-ifeq ($(with_clouseau), 1)
-_WITH_CLOUSEAU="--with-clouseau"
-endif
-
.PHONY: mango-test
# target: mango-test - Run Mango tests
mango-test: devclean all
diff --git a/Makefile.win b/Makefile.win
index 0354fe29d..4d7a2e9d8 100644
--- a/Makefile.win
+++ b/Makefile.win
@@ -241,13 +241,20 @@ elixir: elixir-init devclean
--erlang-config rel/files/eunit.config \
--no-eval 'mix test --trace --include test\elixir\test\config\suite.elixir --exclude test\elixir\test\config\skip.elixir $(EXUNIT_OPTS)'
+ifneq ($(CLOUSEAU_DIR),)
+_WITH_CLOUSEAU="--with-clouseau --clouseau-dir=$(CLOUSEAU_DIR)"
+else ifeq ($(with_clouseau), 1)
+_WITH_CLOUSEAU="--with-clouseau"
+endif
+
.PHONY: elixir-search
# target: elixir-search - Run search tests, requires a configured Clouseau instance
elixir-search: export MIX_ENV=integration
elixir-search: elixir-init devclean
-ifeq ($(with_clouseau), 1)
+ifneq ($(_WITH_CLOUSEAU), )
@dev\run -n 1 -q -a adm:pass \
- --with-clouseau \
+ "$(_WITH_CLOUSEAU)" \
+ "$(TEST_OPTS)" \
--locald-config test/config/test-config.ini \
--no-eval 'mix test --trace --include test/elixir/test/config/search.elixir'
else
@@ -286,10 +293,6 @@ list-eunit-apps:
list-eunit-suites:
@powershell -Command 'Get-ChildItem -Path src -Recurse -Filter "*_tests?.erl" | ForEach-Object { "{0}" -f $$_.BaseName } | Sort'
-ifeq ($(with_clouseau), 1)
-_WITH_CLOUSEAU="--with-clouseau"
-endif
-
.PHONY: mango-test
# target: mango-test - Run Mango tests
mango-test: export COUCHDB_TEST_ADMIN_PARTY_OVERRIDE=1
diff --git a/README-DEV.rst b/README-DEV.rst
index 3ab5c6bf6..0988780aa 100644
--- a/README-DEV.rst
+++ b/README-DEV.rst
@@ -317,6 +317,31 @@ used to configure Clouseau to work with that::
dev/run --with-clouseau --erlang-cookie=brumbrum
+It is possible to override Clouseau's location per invocation of
+``./dev/run`` in case some other version needs to be exercised for the
+moment. This can be done with the help of the ``--clouseau-dir``
+flag. The specified location could be either an unpacked bundle of
+JAR files or a git clone of the Clouseau source code repository::
+
+ dev/run --with-clouseau --clouseau-dir $HOME/git/clouseau.wip
+
+Through the ``CLOUSEAU_DIR`` variable the same could be forwarded to the
+respective test targets, e.g. ``mango-test``::
+
+ make mango-test CLOUSEAU_DIR=$HOME/git/clouseau.wip
+
+This can even be done if there was no local Clouseau deployment
+configured previously. Mind that this will require building Clouseau
+from source, which causes the nodes start up somewhat slower. It also
+requires JDK 1.7 and Apache Maven 3.8 to be present, that is why it is
+important the set the ``CLOUSEAU_JAVA_HOME`` and the
+``CLOUSEAU_M2_HOME`` environment variables accordingly, for instance::
+
+ asdf install java zulu-7.56.0.11
+ asdf plugin add maven
+ asdf install maven 3.8.8
+ export CLOUSEAU_JAVA_HOME=$(asdf where java zulu-7.56.0.11)
+ export CLOUSEAU_M2_HOME=$(asdf where maven 3.8.8)
Static Code Analysis
~~~~~~~~~~~~~~~~~~~~
diff --git a/configure b/configure
index a6032da68..9a99df0e5 100755
--- a/configure
+++ b/configure
@@ -39,6 +39,8 @@ run_erlang() {
COUCHDB_USER="$(whoami 2>/dev/null || echo couchdb)"
SM_VSN=${SM_VSN:-"91"}
+CLOUSEAU_MTH=${CLOUSEAU_MTH:-"dist"}
+CLOUSEAU_URI=${CLOUSEAU_URI:-"https://github.com/cloudant-labs/clouseau/releases/download/%s/clouseau-%s-dist.zip"}
CLOUSEAU_VSN=${CLOUSEAU_VSN:-"2.22.0"}
CLOUSEAU_DIR="$(pwd)"/clouseau
ARCH="$(uname -m)"
@@ -66,7 +68,9 @@ Options:
--erlang-md5 use erlang for md5 hash operations
--dev alias for --disable-docs --disable-fauxton
--spidermonkey-version VSN specify the version of SpiderMonkey to use (defaults to $SM_VSN)
- --clouseau-version VSN specify the version of Clouseau to use (defaults to $CLOUSEAU_VSN)
+ --clouseau-method MTH specify the method for Clouseau to deploy: git or dist (defaults to $CLOUSEAU_MTH)
+ --clouseau-uri URI specify the location for retrieving Clouseau (defaults $(printf "$CLOUSEAU_URI" "$CLOUSEAU_VSN" "$CLOUSEAU_VSN"))
+ --clouseau-version VSN specify the version (tag/branch for git) of Clouseau to use (defaults to $CLOUSEAU_VSN)
--skip-deps do not update erlang dependencies
--rebar=PATH use rebar by specified path (version >=2.6.0 && <3.0 required)
--rebar3=PATH use rebar3 by specified path
@@ -220,7 +224,7 @@ parse_opts() {
--clouseau-version)
if [ -n "$2" ]; then
- eval CLOUSEAU_SVN=$2
+ eval CLOUSEAU_VSN=$2
shift 2
continue
else
@@ -236,6 +240,42 @@ parse_opts() {
exit 1
;;
+ --clouseau-method)
+ if [ -n "$2" ]; then
+ eval CLOUSEAU_MTH=$2
+ shift 2
+ continue
+ else
+ printf 'ERROR: "--clouseau-method" requires a non-empty argument.\n' >&2
+ exit 1
+ fi
+ ;;
+ --clouseau-method=?*)
+ eval CLOUSEAU_MTH=${1#*=}
+ ;;
+ --clouseau-method=)
+ printf 'ERROR: "--clouseau-method" requires a non-empty argument.\n' >&2
+ exit 1
+ ;;
+
+ --clouseau-uri)
+ if [ -n "$2" ]; then
+ eval CLOUSEAU_URI=$2
+ shift 2
+ continue
+ else
+ printf 'ERROR: "--clouseau-uri" requires a non-empty argument.\n' >&2
+ exit 1
+ fi
+ ;;
+ --clouseau-uri=?*)
+ eval CLOUSEAU_URI=${1#*=}
+ ;;
+ --clouseau-uri=)
+ printf 'ERROR: "--clouseau-uri" requires a non-empty argument.\n' >&2
+ exit 1
+ ;;
+
--) # End of options
shift
break
@@ -380,43 +420,56 @@ install_local_erlfmt() {
fi
}
-install_local_clouseau() {
- _DIST_URL=https://github.com/cloudant-labs/clouseau/releases/download/"$CLOUSEAU_VSN"/clouseau-"$CLOUSEAU_VSN"-dist.zip
- _MAVEN_BASE_URI=https://repo1.maven.org/maven2
-
- _SLF4J_SIMPLE_VSN=${SLF4J_SIMPLE_VERSION:-1.7.36}
- _SLF4J_SIMPLE_JAR=slf4j-simple-"$_SLF4J_SIMPLE_VSN".jar
- _SLF4J_SIMPLE_URL="$_MAVEN_BASE_URI"/org/slf4j/slf4j-simple/"$_SLF4J_SIMPLE_VSN"/"$_SLF4J_SIMPLE_JAR"
-
- rm -rf "$CLOUSEAU_DIR"
- mkdir -p "$CLOUSEAU_DIR"
-
- if ! curl -sSL --max-redirs 1 -o clouseau.zip "$_DIST_URL"; then
- printf "ERROR: %s could not be downloaded.\n" "$_DIST_URL" >&2
- exit 1
- fi
-
- if ! unzip -q -j clouseau.zip -d "$CLOUSEAU_DIR"; then
- printf "ERROR: Clouseau distribution package (clouseau.zip) could not be extracted.\n" >&2
- exit 1
- fi
-
- rm clouseau.zip
+check_local_clouseau_dir() {
+ _dir="$1"
- if ! curl -sSL --max-redirs 1 -o "$CLOUSEAU_DIR"/"$_SLF4J_SIMPLE_JAR" "$_SLF4J_SIMPLE_URL"; then
- printf "ERROR: %s could not be downloaded.\n" "$_SLF4J_SIMPLE_URL" >&2
+ if [ -e "$_dir" ]; then
+ printf "ERROR: \"%s\" already exists. Please remove or move it away first.\n" "$_dir" >&2
exit 1
fi
+}
- cat <<EOF > "$CLOUSEAU_DIR"/clouseau.ini
-[clouseau]
-EOF
- cat <<EOF > "$CLOUSEAU_DIR"/log4j.properties
-log4j.rootLogger=debug, CONSOLE
-log4j.appender.CONSOLE=org.apache.log4j.ConsoleAppender
-log4j.appender.CONSOLE.layout=org.apache.log4j.PatternLayout
-log4j.appender.CONSOLE.layout.ConversionPattern=%d{ISO8601} %c [%p] %m%n
-EOF
+install_local_clouseau() {
+ case "$CLOUSEAU_MTH" in
+ dist)
+ _DIST_URL=$(printf "$CLOUSEAU_URI" "$CLOUSEAU_VSN" "$CLOUSEAU_VSN")
+ _MAVEN_BASE_URI=https://repo1.maven.org/maven2
+
+ _SLF4J_SIMPLE_VSN=${SLF4J_SIMPLE_VERSION:-1.7.36}
+ _SLF4J_SIMPLE_JAR=slf4j-simple-"$_SLF4J_SIMPLE_VSN".jar
+ _SLF4J_SIMPLE_URL="$_MAVEN_BASE_URI"/org/slf4j/slf4j-simple/"$_SLF4J_SIMPLE_VSN"/"$_SLF4J_SIMPLE_JAR"
+
+ check_local_clouseau_dir "$CLOUSEAU_DIR"
+ mkdir -p "$CLOUSEAU_DIR"
+
+ echo "Fetching Clouseau from $_DIST_URL..."
+ if ! curl -sSL --max-redirs 1 -o clouseau.zip "$_DIST_URL"; then
+ printf "ERROR: %s could not be downloaded.\n" "$_DIST_URL" >&2
+ exit 1
+ fi
+
+ if ! unzip -q -j clouseau.zip -d "$CLOUSEAU_DIR"; then
+ printf "ERROR: Clouseau distribution package (clouseau.zip) could not be extracted.\n" >&2
+ exit 1
+ fi
+
+ rm clouseau.zip
+
+ if ! curl -sSL --max-redirs 1 -o "$CLOUSEAU_DIR"/"$_SLF4J_SIMPLE_JAR" "$_SLF4J_SIMPLE_URL"; then
+ printf "ERROR: %s could not be downloaded.\n" "$_SLF4J_SIMPLE_URL" >&2
+ exit 1
+ fi
+ ;;
+
+ git)
+ echo "Cloning Clouseau from $CLOUSEAU_URI ($CLOUSEAU_VSN)..."
+ check_local_clouseau_dir "$CLOUSEAU_DIR"
+ git clone --depth 1 --branch "$CLOUSEAU_VSN" "$CLOUSEAU_URI" "$CLOUSEAU_DIR"
+ ;;
+
+ *) printf "ERROR: Invalid deployment method for Clouseau. Please use either \`dist\` or \`git\` verbatim.\n" >&2
+ exit 1
+ esac
}
if [ -z "${REBAR}" ]; then
@@ -436,8 +489,6 @@ fi
if [ $WITH_CLOUSEAU -ne 0 ]; then
install_local_clouseau
-else
- rm -rf "$CLOUSEAU_DIR"
fi
# only update dependencies, when we are not in a release tarball
diff --git a/configure.ps1 b/configure.ps1
index 7d13b5505..6c4a61e6f 100644
--- a/configure.ps1
+++ b/configure.ps1
@@ -13,6 +13,8 @@
-CouchDBUser USER set the username to run as (defaults to current user)
-SpiderMonkeyVersion VSN select the version of SpiderMonkey to use (default 91)
-ClouseauVersion VSN select the version of Clouseau to use (default 2.22.0)
+ -ClouseauMethod MTH method for Clouseau to deploy: git or dist (default dist)
+ -ClouseauUri URI location for retrieving Clouseau (default https://github.com/cloudant-labs/clouseau/releases/download/2.22.0/clouseau-2.22.0-dist.zip)
Installation directories:
-Prefix PREFIX install architecture-independent files in PREFIX
@@ -57,8 +59,12 @@ Param(
[ValidateNotNullOrEmpty()]
[string]$SpiderMonkeyVersion = "91", # select the version of SpiderMonkey to use (default 91)
[ValidateNotNullOrEmpty()]
+ [string]$ClouseauMethod = "dist", # method for Clouseau to deploy: git or dist (default dist)
+ [ValidateNotNullOrEmpty()]
[string]$ClouseauVersion = "2.22.0", # select the version of Clouseau to use (default 2.22.0)
[ValidateNotNullOrEmpty()]
+ [string]$ClouseauUri = "https://github.com/cloudant-labs/clouseau/releases/download/{0}/clouseau-{0}-dist.zip", # location for retrieving Clouseau (default https://github.com/cloudant-labs/clouseau/releases/download/2.22.0/clouseau-2.22.0-dist.zip)
+ [ValidateNotNullOrEmpty()]
[string]$Prefix = "C:\Program Files\Apache\CouchDB", # install architecture-independent file location (default C:\Program Files\Apache\CouchDB)
[ValidateNotNullOrEmpty()]
[string]$ExecPrefix = $Prefix, # install architecture-dependent file location (default C:\Program Files\Apache\CouchDB)
@@ -278,56 +284,52 @@ $ClouseauDir = "$rootdir\clouseau"
if ($EnableClouseau)
{
- Write-Verbose "===> downloading Clouseau distribution..."
if (Test-Path $ClouseauDir) {
- Remove-Item -Recurse -Force $ClouseauDir
- }
- New-Item -Path $ClouseauDir -ItemType Directory | Out-File Null
-
- $Slf4jVersion = "1.7.36"
- $ClouseauDistUrl = "https://github.com/cloudant-labs/clouseau/releases/download/$ClouseauVersion/clouseau-$ClouseauVersion-dist.zip"
- $Slf4jSimpleJar = "slf4j-simple-$Slf4jVersion.jar"
- $Slf4jSimpleUrl = "https://repo1.maven.org/maven2/org/slf4j/slf4j-simple/$Slf4jVersion/$Slf4jSimpleJar"
-
- Set-Variable ProgressPreference SilentlyContinue
- Invoke-WebRequest -MaximumRedirection 1 -OutFile clouseau.zip $ClouseauDistUrl
- If ($LASTEXITCODE -ne 0) {
- Write-Output "ERROR: $ClouseauDistUrl could not be downloaded."
+ Write-Output "ERROR: ""$ClouseauDir"" already exists. Please remove or move it away first."
exit 1
}
- Expand-Archive clouseau.zip -DestinationPath $ClouseauDir -Force
- If ($LASTEXITCODE -ne 0) {
- Write-Output "ERROR: Clouseau distribution package (clouseau.zip) could not be extracted."
- exit 1
+ if ($ClouseauMethod -eq "dist") {
+ Write-Verbose "===> fetching Clouseau from $ClouseauDistUrl..."
+
+ New-Item -Path $ClouseauDir -ItemType Directory | Out-File Null
+
+ $Slf4jVersion = "1.7.36"
+ $ClouseauDistUrl = $ClouseauUri -f $ClouseauVersion
+ $Slf4jSimpleJar = "slf4j-simple-$Slf4jVersion.jar"
+ $Slf4jSimpleUrl = "https://repo1.maven.org/maven2/org/slf4j/slf4j-simple/$Slf4jVersion/$Slf4jSimpleJar"
+
+ Set-Variable ProgressPreference SilentlyContinue
+ Invoke-WebRequest -MaximumRedirection 1 -OutFile clouseau.zip $ClouseauDistUrl
+ If ($LASTEXITCODE -ne 0) {
+ Write-Output "ERROR: $ClouseauDistUrl could not be downloaded."
+ exit 1
+ }
+
+ Expand-Archive clouseau.zip -DestinationPath $ClouseauDir -Force
+ If ($LASTEXITCODE -ne 0) {
+ Write-Output "ERROR: Clouseau distribution package (clouseau.zip) could not be extracted."
+ exit 1
+ }
+ mv "$ClouseauDir\*\*.jar" "$ClouseauDir"
+ rm "$ClouseauDir\clouseau-$ClouseauVersion"
+ rm clouseau.zip
+
+ Invoke-WebRequest -MaximumRedirection 1 -OutFile "$ClouseauDir\$Slf4jSimpleJar" $Slf4jSimpleUrl
+ If ($LASTEXITCODE -ne 0) {
+ Write-Output "ERROR: $Slf4jSimpleJarUrl could not be downloaded."
+ exit 1
+ }
}
- mv "$ClouseauDir\*\*.jar" "$ClouseauDir"
- rm "$ClouseauDir\clouseau-$ClouseauVersion"
- rm clouseau.zip
+ elseif ($ClouseauMethod -eq "git") {
+ Write-Verbose "===> cloning Clouseau from $ClouseauDistUrl ($ClouseauVersion)..."
- Invoke-WebRequest -MaximumRedirection 1 -OutFile "$ClouseauDir\$Slf4jSimpleJar" $Slf4jSimpleUrl
- If ($LASTEXITCODE -ne 0) {
- Write-Output "ERROR: $Slf4jSimpleJarUrl could not be downloaded."
- exit 1
+ git clone --depth 1 --branch $ClouseauVersion $ClouseauUri $ClouseauDir
}
-
- $ClouseauIni = @"
-[clouseau]
-"@
- $ClouseauIni | Out-File "$ClouseauDir\clouseau.ini" -encoding ascii
-
- $Log4JProperties = @"
-log4j.rootLogger=debug, CONSOLE
-log4j.appender.CONSOLE=org.apache.log4j.ConsoleAppender
-log4j.appender.CONSOLE.layout=org.apache.log4j.PatternLayout
-log4j.appender.CONSOLE.layout.ConversionPattern=%d{ISO8601} %c [%p] %m%n
-"@
- $Log4JProperties | Out-File "$ClouseauDir\log4j.properties"
-}
-else {
- if (Test-Path $ClouseauDir) {
- Remove-Item -Recurse -Force $ClouseauDir
+ else {
+ Write-Output "ERROR: Invalid deployment method for Clouseau. Please use either `dist` or `git`."
+ exit 1
}
}
diff --git a/dev/run b/dev/run
index 62dc528d6..dbdfecf03 100755
--- a/dev/run
+++ b/dev/run
@@ -26,6 +26,7 @@ import os
import platform
import posixpath
import re
+import shutil
import signal
import socket
import subprocess as sp
@@ -253,6 +254,12 @@ def get_args_parser():
default=None,
help="Erlang cookie string",
)
+ parser.add_option(
+ "--clouseau-dir",
+ dest="clouseau_dir",
+ default=None,
+ help="Use a specific directory for finding Clouseau files",
+ ),
parser.add_option(
"-t",
"--enable-tls",
@@ -270,6 +277,9 @@ def get_args_parser():
return parser
+CLOUSEAU_DIR = "clouseau"
+
+
def setup_context(opts, args):
fpath = os.path.abspath(__file__)
return {
@@ -297,6 +307,7 @@ def setup_context(opts, args):
"locald_configs": opts.locald_configs,
"with_nouveau": opts.with_nouveau,
"with_clouseau": opts.with_clouseau,
+ "clouseau_dir": opts.clouseau_dir or CLOUSEAU_DIR,
"erlang_cookie": opts.erlang_cookie,
"enable_tls": opts.enable_tls,
"no_tls": opts.no_tls,
@@ -525,7 +536,6 @@ def boot_nouveau(ctx):
)
-CLOUSEAU_DIR = "clouseau"
JAVA_VERSION_RE = re.compile(r'"(\d+\.\d+).*"')
@@ -547,6 +557,27 @@ class StartupError(Exception):
super().__init__(self.message)
+def generate_default_clouseau_ini(path):
+ print("(generate default clouseau.ini)")
+ with open(path, "w") as handle:
+ handle.write(
+ """[clouseau]
+"""
+ )
+
+
+def generate_default_log4j_properties(path):
+ print("(generate default log4j.properties)")
+ with open(path, "w") as handle:
+ handle.write(
+ """log4j.rootLogger=debug, CONSOLE
+log4j.appender.CONSOLE=org.apache.log4j.ConsoleAppender
+log4j.appender.CONSOLE.layout=org.apache.log4j.PatternLayout
+log4j.appender.CONSOLE.layout.ConversionPattern=%d{ISO8601} %c [%p] %m%n
+"""
+ )
+
+
@log("Start Clouseau node clouseau{idx}")
def boot_clouseau(ctx, idx):
configure_cmd = (
@@ -554,47 +585,115 @@ def boot_clouseau(ctx, idx):
if platform.system() == "Windows"
else "./configure --enable-clouseau"
)
+ clouseau_dir = ctx["clouseau_dir"]
- if not os.path.isdir(CLOUSEAU_DIR):
+ if not os.path.isdir(clouseau_dir):
raise StartupError(
- "Clouseau deployment cannot be found, please run `{}`".format(configure_cmd)
+ (
+ "Clouseau files cannot be found. "
+ "Please run `{}` or make `--clouseau-dir` point to a valid directory"
+ ).format(configure_cmd)
)
- clouseau_jdk_home = os.environ.get("CLOUSEAU_JAVA_HOME") or os.environ.get(
+ clouseau_java_home = os.environ.get("CLOUSEAU_JAVA_HOME") or os.environ.get(
"JAVA_HOME"
)
- java = clouseau_jdk_home + "/bin/java" if clouseau_jdk_home else "java"
+ java = (
+ clouseau_java_home + "/bin/java" if clouseau_java_home else shutil.which("java")
+ )
java_version = get_java_version(java)
+ pom_file = os.path.join(clouseau_dir, "pom.xml")
+ method = "src" if os.path.isfile(pom_file) else "dist"
+
if not java_version:
print(
"Warning: Java version could not be determined, Clouseau may not be able to run"
)
else:
- if java_version < 1.7 or java_version > 1.8:
+ if method == "src" and (java_version < 1.7 or 1.7 < java_version):
+ raise StartupError(
+ "Java is not suitable to run Clouseau. Please use JDK 1.7 and configure its (root) path in `CLOUSEAU_JAVA_HOME`"
+ )
+ elif java_version < 1.7 or 1.8 < java_version:
raise StartupError(
"Java is not suitable to run Clouseau. Please use JRE 1.7 or 1.8 and configure its (root) path in `CLOUSEAU_JAVA_HOME`"
)
- clouseau_jars = [
- "{}/{}".format(CLOUSEAU_DIR, fname)
- for fname in os.listdir(CLOUSEAU_DIR)
- if fname.endswith(".jar")
- ]
+ logfname = os.path.join(ctx["devdir"], "logs", "clouseau{}.log".format(idx))
+ log = open(logfname, "w")
+ separator = ";" if platform.system() == "Windows" else ":"
- if not clouseau_jars:
- raise StartupError("Clouseau has no JAR files")
+ if method == "src":
+ target_dir = os.path.join(clouseau_dir, "target")
+ cp_file = os.path.join(ctx["devdir"], "lib", "clouseau.classpath")
- clouseau_ini = "{}/clouseau.ini".format(CLOUSEAU_DIR)
- if not os.path.isfile(clouseau_ini):
- raise StartupError("Clouseau has no ini file")
+ # attempt building the sources only once
+ if idx == 1:
+ env = os.environ.copy()
+ if clouseau_java_home:
+ env["JAVA_HOME"] = clouseau_java_home
- log4j_properties = "{}/log4j.properties".format(CLOUSEAU_DIR)
- if not os.path.isfile(log4j_properties):
- raise StartupError("Clouseau has no Log4J configuration")
+ clouseau_mvn_home = os.environ.get("CLOUSEAU_M2_HOME") or os.environ.get(
+ "M2_HOME"
+ )
+ mvn = (
+ clouseau_mvn_home + "/bin/mvn"
+ if clouseau_mvn_home
+ else shutil.which("mvn")
+ )
+
+ if not mvn:
+ raise StartupError(
+ "Maven could not be found. Please install Maven 3.8 and configure its (root) path in `CLOUSEAU_M2_HOME`"
+ )
+
+ try:
+ sp.check_call(
+ [
+ mvn,
+ "-B",
+ "compile",
+ "dependency:build-classpath",
+ "-Dmdep.outputFile={}".format(cp_file),
+ ],
+ cwd=clouseau_dir,
+ env=env,
+ stdin=sp.PIPE,
+ stdout=log,
+ stderr=log,
+ )
+ except Exception as exc:
+ log.write(str(exc))
+ raise StartupError("Could not build Clouseau from sources.")
+
+ with open(cp_file, "r") as handle:
+ classpath_deps = handle.read().rstrip()
+
+ targets = os.path.join(clouseau_dir, "target", "classes")
+ classpath = separator.join([classpath_deps, targets])
+ else:
+ clouseau_jars = [
+ os.path.join(clouseau_dir, fname)
+ for fname in os.listdir(clouseau_dir)
+ if fname.endswith(".jar")
+ ]
+
+ if not clouseau_jars:
+ raise StartupError("Clouseau has no JAR files")
+
+ classpath = separator.join(clouseau_jars)
+
+ clouseau_ini = os.path.join(clouseau_dir, "clouseau.ini")
+ if not os.path.exists(clouseau_ini):
+ generate_default_clouseau_ini(clouseau_ini)
+
+ log4j_properties = os.path.join(clouseau_dir, "log4j.properties")
+ if not os.path.exists(log4j_properties):
+ generate_default_log4j_properties(log4j_properties)
clouseau_name = "clouseau{}@127.0.0.1".format(idx)
- clouseau_indexes_dir = "{}/clouseau{}/data".format(ctx["devdir"], idx)
+ clouseau_indexes_dir = os.path.join(ctx["devdir"], "clouseau{}".format(idx), "data")
if ctx["erlang_cookie"]:
clouseau_cookie = ["-Dclouseau.cookie={}".format(ctx["erlang_cookie"])]
@@ -608,8 +707,6 @@ def boot_clouseau(ctx, idx):
if alternativeHome:
os.environ["HOME"] = alternativeHome
- separator = ";" if platform.system() == "Windows" else ":"
-
cmd = (
[
java,
@@ -621,7 +718,7 @@ def boot_clouseau(ctx, idx):
"-XX:+UseConcMarkSweepGC",
"-XX:+CMSParallelRemarkEnabled",
"-cp",
- separator.join(clouseau_jars),
+ classpath,
"-Dlog4j.configuration=file:{}".format(log4j_properties),
"-Dclouseau.name={}".format(clouseau_name),
"-Dclouseau.dir={}".format(clouseau_indexes_dir),
@@ -632,8 +729,6 @@ def boot_clouseau(ctx, idx):
clouseau_ini,
]
)
- logfname = os.path.join(ctx["devdir"], "logs", "clouseau{}.log".format(idx))
- log = open(logfname, "w")
try:
return sp.Popen(
(couchdb) 02/02: mango: increase timeout for flaky `text` search test with floats
Posted by pg...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
pgj pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/couchdb.git
commit 4ace5f13971aba8d3f858a69ea5f4404b5355951
Author: Gabor Pali <ga...@ibm.com>
AuthorDate: Thu Nov 30 14:36:46 2023 +0100
mango: increase timeout for flaky `text` search test with floats
The following exception was observed on the CI (excerpt):
```
ERROR: test_floating_point_val (06-basic-text-test.NumStringTests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/jenkins/workspace/jenkins-cm1_PullRequests_PR-4871/24.3.4.14/build/apache-couchdb-3.3.2-e464422/src/mango/test/06-basic-text-test.py", line 583, in test_floating_point_val
@example("NaN")
File "/home/jenkins/workspace/jenkins-cm1_PullRequests_PR-4871/24.3.4.14/build/apache-couchdb-3.3.2-e464422/src/mango/.venv/lib/python3.9/site-packages/hypothesis/core.py", line 1190, in wrapped_test
raise the_error_hypothesis_found
File "/home/jenkins/workspace/jenkins-cm1_PullRequests_PR-4871/24.3.4.14/build/apache-couchdb-3.3.2-e464422/src/mango/.venv/lib/python3.9/site-packages/hypothesis/core.py", line 886, in __flaky
raise Flaky(message)
hypothesis.errors.Flaky: Hypothesis test_floating_point_val(self=<06-basic-text-test.NumStringTests testMethod=test_floating_point_val>, f='-0x1.70cc4f85b0b06p-938') produces unreliable results: Falsified on the first call but did not on a subsequent one
[..]
Falsifying example: test_floating_point_val(
f='-0x1.70cc4f85b0b06p-938',
self=<06-basic-text-test.NumStringTests testMethod=test_floating_point_val>,
)
Unreliable test timings! On an initial run, this test took 255.29ms, which exceeded the deadline of 200.00ms, but on a subsequent run it took 49.44 ms, which did not. If you expect this sort of variability in your test timings, consider turning deadlines off for this test by setting deadline=None.
You can reproduce this example by temporarily adding @reproduce_failure('6.31.6', b'AAFGBwQA+pYNDaHyMw4B') as a decorator on your test case
```
As the note suggests in the report, make an attempt to fix the
issue by bumping the value of the associated default deadline.
---
src/mango/test/06-basic-text-test.py | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/src/mango/test/06-basic-text-test.py b/src/mango/test/06-basic-text-test.py
index 03891a840..da3eb6ff5 100644
--- a/src/mango/test/06-basic-text-test.py
+++ b/src/mango/test/06-basic-text-test.py
@@ -15,7 +15,7 @@ import mango
import unittest
import user_docs
import math
-from hypothesis import given, assume, example
+from hypothesis import given, assume, example, settings
import hypothesis.strategies as st
@@ -580,6 +580,7 @@ class NumStringTests(mango.DbPerClass):
not (math.isinf(num) or math.isnan(num))
@given(f=st.floats().filter(isFinite).map(str) | st.floats().map(lambda f: f.hex()))
+ @settings(deadline=1000)
@example("NaN")
@example("Infinity")
def test_floating_point_val(self, f):