You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@couchdb.apache.org by "pgj (via GitHub)" <gi...@apache.org> on 2023/11/09 17:29:46 UTC
[PR] feat(`dev`): add Clouseau to the developer setup [couchdb]
pgj opened a new pull request, #4835:
URL: https://github.com/apache/couchdb/pull/4835
Provide a way to the developers to deploy and launch the Clouseau search module more easily, therefore making it more accessible. This can help with running the Search-based Elixir tests as well as the Mango `text` search tests. This could allow us to catch more bugs ahead of time and might even inspire more improvements in the area.
The extension is designed in a way to make it simple to integrate with the CI — only the Java environment of the proper version needs to be deployed, everything else could be managed from this repository. Such as bumping the version of Clouseau or fine-tuning the configuration parameters.
This is an opt-in feature on two levels. First, one has to tell `./configure --enable-clouseau` to instantiate Clouseau locally. This will create the `clouseau` sub-directory that holds the contents of the Clouseau distribution of the specified version, which currently defaults to 2.22.0 (the latest). If an older version is needed, use the `--clouseau-version` flag.
Then the `mango-test` and `elixir-search` targets will try to use Clouseau automatically just to make it easy to use in the old way. However, the `dev/run` script will not do the same. It requires the `--with-clouseau` flag to be passed for that. That is because the developer may not necessarily want to launch Clouseau ad hoc, even if it is available.
With this, note that `elixir-search` is added to the `check` target. In lack of a configured Clouseau instance, it will become a no-op and a warning is emitted.
## Testing recommendations
There are many ways to try this change. The conventional approach is to have some flavor of JDK 8 installed, via [`asdf`](https://asdf-vm.com/) for example, configure CouchDB to deploy Clouseau and run the corresponding test suites.
```console
asdf plugin add java
asdf install java zulu-8.74.0.17
export CLOUSEAU_JAVA_HOME=$(asdf where java zulu-8.74.0.17)
./configure --enable-clouseau <other-configure-flag-of-yours>
make elixir-search
make mango-test
```
The `elixir-search` target should not emit a warning (but run the tests) and `mango-test` should have only a single skipped test.
The `dev/run` script could be run individually where Search could be talked to, e.g.:
```console
dev/run --with-clouseau -n 1 -a adm:pass
curl -sS http://127.0.0.1:15984/ | jq -rc '.features'
```
The last command should show something similar to this:
```console
["search","access-ready","partitioned","pluggable-storage-engines","reshard","scheduler"]
```
Conversely, if no Clouseau is configured, there shall be no `clouseau` sub-directory, no Elixir Search tests or Mango `text` tests are run (more skipped cases), and no `search` is reported as a feature. When trying to use `dev/run --with-clouseau` without Clouseau being added, there shall be errors.
## Related Issues or Pull Requests
- [`apache/couchdb-ci` #62](https://github.com/apache/couchdb-ci/pull/62)
## Checklist
- [ ] Code is written and works correctly
- [ ] Add Windows support
- [ ] Add `Jenkinsfile` changes (maybe as a follow-up PR once the Docker images are updated)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(`dev`): add Clouseau to the developer setup [couchdb]
Posted by "jiahuili430 (via GitHub)" <gi...@apache.org>.
jiahuili430 commented on code in PR #4835:
URL: https://github.com/apache/couchdb/pull/4835#discussion_r1390458823
##########
dev/run:
##########
@@ -504,6 +512,119 @@ def boot_nouveau(ctx):
)
+CLOUSEAU_DIR = "clouseau"
+JAVA_VERSION_RE = re.compile('"(\d+\.\d+).*"')
+
+
+def get_java_version(java):
+ output = sp.check_output([java, "-version"], stderr=sp.STDOUT).decode("utf-8")
+ if output:
+ matches = JAVA_VERSION_RE.search(output)
+ if matches:
+ return float(matches.groups()[0])
+
+
+@log("Start Clouseau node clouseau{idx}")
+def boot_clouseau(ctx, idx):
+ def warn(reason):
+ print("Warning: " + reason)
Review Comment:
Instead of multiple `return` in the following if statements, shall we add `sys.exit()` after this line?
- If we use `return` the log looks like this and never stops until we terminate it manually.
```bash
[ * ] Setup environment ... ok
[ * ] Ensure CouchDB is built ... ok
[ * ] Ensure Erlang boot script exists ... ok
[ * ] Prepare configuration files ... ok
[ * ] Start node node1 ... ok
[ * ] Start Clouseau node clouseau1 ... Warning: Clouseau deployment cannot be found, please run `./configure with --enable-clouseau`
ok
[ * ] Check node at http://127.0.0.1:15984/ ... ok
[ * ] Check Clouseau node clouseau1 ... failed: Clouseau is not working
[ * ] Check node at http://127.0.0.1:15984/ ... ok
[ * ] Check Clouseau node clouseau1 ... failed: Clouseau is not working
[ * ] Check node at http://127.0.0.1:15984/ ... ok
...
```
- If we use `sys.exit()` it will stop immediately after the warning.
```bash
[ * ] Setup environment ... ok
[ * ] Ensure CouchDB is built ... ok
[ * ] Ensure Erlang boot script exists ... ok
[ * ] Prepare configuration files ... ok
[ * ] Start node node1 ... ok
[ * ] Start Clouseau node clouseau1 ...
Warning: Clouseau deployment cannot be found, please run `./configure --enable-clouseau`
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(`dev`): add Clouseau to the developer setup [couchdb]
Posted by "pgj (via GitHub)" <gi...@apache.org>.
pgj commented on code in PR #4835:
URL: https://github.com/apache/couchdb/pull/4835#discussion_r1394474361
##########
dev/run:
##########
@@ -505,6 +517,119 @@ def boot_nouveau(ctx):
)
+CLOUSEAU_DIR = "clouseau"
+JAVA_VERSION_RE = re.compile('"(\d+\.\d+).*"')
+
+
+def get_java_version(java):
+ output = sp.check_output([java, "-version"], stderr=sp.STDOUT).decode("utf-8")
+ if output:
+ matches = JAVA_VERSION_RE.search(output)
+ if matches:
+ return float(matches.groups()[0])
+
+
+class ClouseauError(Exception):
+ def __init__(self, message):
+ self.message = message
+ super().__init__(self.message)
+
+
+@log("Start Clouseau node clouseau{idx}")
+def boot_clouseau(ctx, idx):
+ if not os.path.isdir(CLOUSEAU_DIR):
+ raise ClouseauError(
+ "Clouseau deployment cannot be found, please run `./configure with --enable-clouseau`"
+ )
+
+ clouseau_jdk_home = os.environ.get("CLOUSEAU_JAVA_HOME") or os.environ.get(
+ "JAVA_HOME"
+ )
+ java = clouseau_jdk_home + "/bin/java" if clouseau_jdk_home else "java"
+ java_version = get_java_version(java)
+
+ if not java_version:
+ print(
+ "Warning: Java version could not be determined, Clouseau may not be able to run"
+ )
Review Comment:
This is a not a real error: it still might be possible to run Clouseau, but there are no guarantees for that. However, the `TypeError` above you included should not be there, thanks for reporting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(`dev`): add Clouseau to the developer setup [couchdb]
Posted by "big-r81 (via GitHub)" <gi...@apache.org>.
big-r81 commented on code in PR #4835:
URL: https://github.com/apache/couchdb/pull/4835#discussion_r1389552537
##########
README-DEV.rst:
##########
@@ -272,6 +266,50 @@ to keep those databases around for further investigation::
MANGO_TESTS_KEEP_DBS=please \
make mango-test MANGO_TEST_OPTS='03-operator-test'
+Running Clouseau
+~~~~~~~~~~~~~~~~
+
+When configured with the ``./configure`` script, the ``dev/run``
Review Comment:
Do we want to use `./dev/run`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(`dev`): add Clouseau to the developer setup [couchdb]
Posted by "pgj (via GitHub)" <gi...@apache.org>.
pgj commented on code in PR #4835:
URL: https://github.com/apache/couchdb/pull/4835#discussion_r1389490610
##########
dev/run:
##########
@@ -504,6 +512,119 @@ def boot_nouveau(ctx):
)
+CLOUSEAU_DIR = "clouseau"
+JAVA_VERSION_RE = re.compile('"(\d+\.\d+).*"')
+
+
+def get_java_version(java):
+ output = sp.check_output([java, "-version"], stderr=sp.STDOUT).decode("utf-8")
+ if output:
+ matches = JAVA_VERSION_RE.search(output)
+ if matches:
+ return float(matches.groups()[0])
+
+
+@log("Start Clouseau node clouseau{idx}")
+def boot_clouseau(ctx, idx):
+ def warn(reason):
+ print("Warning: " + reason)
+
+ if not os.path.isdir(CLOUSEAU_DIR):
+ warn(
+ "Clouseau deployment cannot be found, please run ./configure with --enable-clouseau"
Review Comment:
Sure. Good idea.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(`dev`): add Clouseau to the developer setup [couchdb]
Posted by "jiahuili430 (via GitHub)" <gi...@apache.org>.
jiahuili430 commented on code in PR #4835:
URL: https://github.com/apache/couchdb/pull/4835#discussion_r1394383054
##########
dev/run:
##########
@@ -505,6 +517,119 @@ def boot_nouveau(ctx):
)
+CLOUSEAU_DIR = "clouseau"
+JAVA_VERSION_RE = re.compile('"(\d+\.\d+).*"')
+
+
+def get_java_version(java):
+ output = sp.check_output([java, "-version"], stderr=sp.STDOUT).decode("utf-8")
+ if output:
+ matches = JAVA_VERSION_RE.search(output)
+ if matches:
+ return float(matches.groups()[0])
+
+
+class ClouseauError(Exception):
+ def __init__(self, message):
+ self.message = message
+ super().__init__(self.message)
+
+
+@log("Start Clouseau node clouseau{idx}")
+def boot_clouseau(ctx, idx):
+ if not os.path.isdir(CLOUSEAU_DIR):
+ raise ClouseauError(
+ "Clouseau deployment cannot be found, please run `./configure with --enable-clouseau`"
+ )
+
+ clouseau_jdk_home = os.environ.get("CLOUSEAU_JAVA_HOME") or os.environ.get(
+ "JAVA_HOME"
+ )
+ java = clouseau_jdk_home + "/bin/java" if clouseau_jdk_home else "java"
+ java_version = get_java_version(java)
+
+ if not java_version:
+ print(
+ "Warning: Java version could not be determined, Clouseau may not be able to run"
+ )
Review Comment:
Shall we use `raise ClouseauError()` here as well?
Tried with Java openjdk-17, get the following error
```bash
[ * ] Start Clouseau node clouseau1 ... Warning: Java version could not be determined, Clouseau may not be able to run
failed: '<' not supported between instances of 'NoneType' and 'float'
Traceback (most recent call last):
File "/Users/jiahuili/src/pgj/./dev/run", line 1064, in <module>
main()
File "/Users/jiahuili/src/pgj/./dev/run", line 94, in main
startup(ctx)
File "/Users/jiahuili/src/pgj/./dev/run", line 711, in startup
boot_nodes(ctx)
File "/Users/jiahuili/src/pgj/./dev/run", line 757, in boot_nodes
clouseau_proc = maybe_boot_clouseau(ctx, idx)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/jiahuili/src/pgj/./dev/run", line 611, in maybe_boot_clouseau
return boot_clouseau(ctx, idx)
^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/jiahuili/src/pgj/./dev/run", line 73, in wrapper
res = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/Users/jiahuili/src/pgj/./dev/run", line 556, in boot_clouseau
if java_version < 1.7 or java_version > 1.8:
^^^^^^^^^^^^^^^^^^
TypeError: '<' not supported between instances of 'NoneType' and 'float'
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(`dev`): add Clouseau to the developer setup [couchdb]
Posted by "jiahuili430 (via GitHub)" <gi...@apache.org>.
jiahuili430 commented on code in PR #4835:
URL: https://github.com/apache/couchdb/pull/4835#discussion_r1389437909
##########
dev/run:
##########
@@ -504,6 +512,119 @@ def boot_nouveau(ctx):
)
+CLOUSEAU_DIR = "clouseau"
+JAVA_VERSION_RE = re.compile('"(\d+\.\d+).*"')
+
+
+def get_java_version(java):
+ output = sp.check_output([java, "-version"], stderr=sp.STDOUT).decode("utf-8")
+ if output:
+ matches = JAVA_VERSION_RE.search(output)
+ if matches:
+ return float(matches.groups()[0])
+
+
+@log("Start Clouseau node clouseau{idx}")
+def boot_clouseau(ctx, idx):
+ def warn(reason):
+ print("Warning: " + reason)
+
+ if not os.path.isdir(CLOUSEAU_DIR):
+ warn(
+ "Clouseau deployment cannot be found, please run ./configure with --enable-clouseau"
Review Comment:
Shall we highlight the command?
"Clouseau deployment cannot be found, please run `./configure --enable-clouseau`"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(`dev`): add Clouseau to the developer setup [couchdb]
Posted by "jiahuili430 (via GitHub)" <gi...@apache.org>.
jiahuili430 commented on PR #4835:
URL: https://github.com/apache/couchdb/pull/4835#issuecomment-1805790263
The code is very clean and makes Clouseau easier to use. Thank you, Gábor.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(`dev`): add Clouseau to the developer setup [couchdb]
Posted by "pgj (via GitHub)" <gi...@apache.org>.
pgj commented on code in PR #4835:
URL: https://github.com/apache/couchdb/pull/4835#discussion_r1389623988
##########
README-DEV.rst:
##########
@@ -272,6 +266,50 @@ to keep those databases around for further investigation::
MANGO_TESTS_KEEP_DBS=please \
make mango-test MANGO_TEST_OPTS='03-operator-test'
+Running Clouseau
+~~~~~~~~~~~~~~~~
+
+When configured with the ``./configure`` script, the ``dev/run``
+script is capable of launching Clouseau instances alongside the
+CouchDB nodes and hooking them up. This is what the ``mango-test``
+and ``elixir-search`` targets also use to run their respective test
+suites and let Clouseau automatically managed for them.
+
+Although the ``./configure`` and the ``dev/run`` scripts try to take
Review Comment:
Right.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(`dev`): add Clouseau to the developer setup [couchdb]
Posted by "pgj (via GitHub)" <gi...@apache.org>.
pgj commented on code in PR #4835:
URL: https://github.com/apache/couchdb/pull/4835#discussion_r1391453720
##########
dev/run:
##########
@@ -504,6 +512,119 @@ def boot_nouveau(ctx):
)
+CLOUSEAU_DIR = "clouseau"
+JAVA_VERSION_RE = re.compile('"(\d+\.\d+).*"')
+
+
+def get_java_version(java):
+ output = sp.check_output([java, "-version"], stderr=sp.STDOUT).decode("utf-8")
+ if output:
+ matches = JAVA_VERSION_RE.search(output)
+ if matches:
+ return float(matches.groups()[0])
+
+
+@log("Start Clouseau node clouseau{idx}")
+def boot_clouseau(ctx, idx):
+ def warn(reason):
+ print("Warning: " + reason)
Review Comment:
The "check node" action is not repeated indefinitely but for a fixed amount (10) of times. See the `ensure_all_nodes_alive()` function for the logic. I admit that it could be annoying but the script will eventually give up on trying.
Perhaps it may make sense to differentiate between warnings and errors. In case of errors, the script could stop earlier ("fail fast"). I am not convinced that `sys.exit()` is the best to express that though. I consider it an aggressive break in the control flow and I usually keep it as the last resort. I will look into options on how to implement this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(`dev`): add Clouseau to the developer setup [couchdb]
Posted by "big-r81 (via GitHub)" <gi...@apache.org>.
big-r81 commented on code in PR #4835:
URL: https://github.com/apache/couchdb/pull/4835#discussion_r1396052971
##########
Makefile:
##########
@@ -313,19 +319,24 @@ list-eunit-suites:
build-test:
@test/build/test-configure.sh
+ifeq ($(with_clouseau), 1)
+_WITH_CLOUSEAU="--with-clouseau"
+endif
.PHONY: mango-test
# target: mango-test - Run Mango tests
mango-test: export COUCHDB_TEST_ADMIN_PARTY_OVERRIDE=1
mango-test: devclean all
- @cd src/mango && \
- python3 -m venv .venv && \
- .venv/bin/python3 -m pip install -r requirements.txt
- @cd src/mango && \
- ../../dev/run "$(TEST_OPTS)" \
+ @python3 -m venv src/mango/.venv && \
Review Comment:
Only a small nit. I don't know if we used it consequently, but should we use the defined env var `PYTHON` here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(`dev`): add Clouseau to the developer setup [couchdb]
Posted by "pgj (via GitHub)" <gi...@apache.org>.
pgj merged PR #4835:
URL: https://github.com/apache/couchdb/pull/4835
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(`dev`): add Clouseau to the developer setup [couchdb]
Posted by "big-r81 (via GitHub)" <gi...@apache.org>.
big-r81 commented on code in PR #4835:
URL: https://github.com/apache/couchdb/pull/4835#discussion_r1396174749
##########
Makefile:
##########
@@ -313,19 +319,24 @@ list-eunit-suites:
build-test:
@test/build/test-configure.sh
+ifeq ($(with_clouseau), 1)
+_WITH_CLOUSEAU="--with-clouseau"
+endif
.PHONY: mango-test
# target: mango-test - Run Mango tests
mango-test: export COUCHDB_TEST_ADMIN_PARTY_OVERRIDE=1
mango-test: devclean all
- @cd src/mango && \
- python3 -m venv .venv && \
- .venv/bin/python3 -m pip install -r requirements.txt
- @cd src/mango && \
- ../../dev/run "$(TEST_OPTS)" \
+ @python3 -m venv src/mango/.venv && \
Review Comment:
Yes, I mean `Makefile.win`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(`dev`): add Clouseau to the developer setup [couchdb]
Posted by "jiahuili430 (via GitHub)" <gi...@apache.org>.
jiahuili430 commented on code in PR #4835:
URL: https://github.com/apache/couchdb/pull/4835#discussion_r1396039018
##########
dev/run:
##########
@@ -505,6 +517,119 @@ def boot_nouveau(ctx):
)
+CLOUSEAU_DIR = "clouseau"
+JAVA_VERSION_RE = re.compile('"(\d+\.\d+).*"')
+
+
+def get_java_version(java):
+ output = sp.check_output([java, "-version"], stderr=sp.STDOUT).decode("utf-8")
+ if output:
+ matches = JAVA_VERSION_RE.search(output)
+ if matches:
+ return float(matches.groups()[0])
+
+
+class ClouseauError(Exception):
+ def __init__(self, message):
+ self.message = message
+ super().__init__(self.message)
+
+
+@log("Start Clouseau node clouseau{idx}")
+def boot_clouseau(ctx, idx):
+ if not os.path.isdir(CLOUSEAU_DIR):
+ raise ClouseauError(
+ "Clouseau deployment cannot be found, please run `./configure with --enable-clouseau`"
+ )
+
+ clouseau_jdk_home = os.environ.get("CLOUSEAU_JAVA_HOME") or os.environ.get(
+ "JAVA_HOME"
+ )
+ java = clouseau_jdk_home + "/bin/java" if clouseau_jdk_home else "java"
+ java_version = get_java_version(java)
+
+ if not java_version:
+ print(
+ "Warning: Java version could not be determined, Clouseau may not be able to run"
+ )
Review Comment:
No `TypeError` on my end, +1, and thanks for the improvement!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(`dev`): add Clouseau to the developer setup [couchdb]
Posted by "pgj (via GitHub)" <gi...@apache.org>.
pgj commented on code in PR #4835:
URL: https://github.com/apache/couchdb/pull/4835#discussion_r1389623756
##########
README-DEV.rst:
##########
@@ -272,6 +266,50 @@ to keep those databases around for further investigation::
MANGO_TESTS_KEEP_DBS=please \
make mango-test MANGO_TEST_OPTS='03-operator-test'
+Running Clouseau
+~~~~~~~~~~~~~~~~
+
+When configured with the ``./configure`` script, the ``dev/run``
Review Comment:
Yes, that would make the text consistent with other instances.
##########
README-DEV.rst:
##########
@@ -272,6 +266,50 @@ to keep those databases around for further investigation::
MANGO_TESTS_KEEP_DBS=please \
make mango-test MANGO_TEST_OPTS='03-operator-test'
+Running Clouseau
+~~~~~~~~~~~~~~~~
+
+When configured with the ``./configure`` script, the ``dev/run``
+script is capable of launching Clouseau instances alongside the
+CouchDB nodes and hooking them up. This is what the ``mango-test``
+and ``elixir-search`` targets also use to run their respective test
+suites and let Clouseau automatically managed for them.
+
+Although the ``./configure`` and the ``dev/run`` scripts try to take
+care of the details of the Clouseau deployment, it is still the
+responsibility of the user to provide a suitable Java environment for
+running. Clouseau can run with JDK 1.7 and 1.8 only. Also, when
+Nouveau is in use, which uses a more recent Java environment, the old
+JDK has to be installed separately and the ``CLOUSEAU_JAVA_HOME``
+environment variable has to be set to point its location.
+
+Fortunately, the ```asdf`` tool <https://asdf-vm.com/>` provides a
+convenient way to install old versions of JDK through its ```java``
+plugin <https://github.com/halcyon/asdf-java>`::
+
+ asdf plugin add java
+
+Then use ``asdf`` to install it::
+
+ asdf install java zulu-8.74.0.17
+
+Finally, use ``asdf`` to set the ``CLOUSEAU_JAVA_HOME`` environment
+variable::
+
+ export CLOUSEAU_JAVA_HOME=$(asdf where java zulu-8.74.0.17)
+
+If the use of ``asdf`` is not an option, `the Zulu site
+<https://cdn.azul.com/zulu/bin/>` could be used directly to get the
+distribution package for the appropriate JDK version. But this is
+just one of the possibilities to access installers for old Java
+environments.
+
+Once both Clouseau and the corresponding Java environment are set,
+they are not put in use automatically. In order to do so, the
+``dev/run`` script needs to be run with Clouseau enabled as follows::
Review Comment:
Right.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(`dev`): add Clouseau to the developer setup [couchdb]
Posted by "jiahuili430 (via GitHub)" <gi...@apache.org>.
jiahuili430 commented on code in PR #4835:
URL: https://github.com/apache/couchdb/pull/4835#discussion_r1396039018
##########
dev/run:
##########
@@ -505,6 +517,119 @@ def boot_nouveau(ctx):
)
+CLOUSEAU_DIR = "clouseau"
+JAVA_VERSION_RE = re.compile('"(\d+\.\d+).*"')
+
+
+def get_java_version(java):
+ output = sp.check_output([java, "-version"], stderr=sp.STDOUT).decode("utf-8")
+ if output:
+ matches = JAVA_VERSION_RE.search(output)
+ if matches:
+ return float(matches.groups()[0])
+
+
+class ClouseauError(Exception):
+ def __init__(self, message):
+ self.message = message
+ super().__init__(self.message)
+
+
+@log("Start Clouseau node clouseau{idx}")
+def boot_clouseau(ctx, idx):
+ if not os.path.isdir(CLOUSEAU_DIR):
+ raise ClouseauError(
+ "Clouseau deployment cannot be found, please run `./configure with --enable-clouseau`"
+ )
+
+ clouseau_jdk_home = os.environ.get("CLOUSEAU_JAVA_HOME") or os.environ.get(
+ "JAVA_HOME"
+ )
+ java = clouseau_jdk_home + "/bin/java" if clouseau_jdk_home else "java"
+ java_version = get_java_version(java)
+
+ if not java_version:
+ print(
+ "Warning: Java version could not be determined, Clouseau may not be able to run"
+ )
Review Comment:
I don't get `TypeError` anymore, +1, and thanks for the improvement!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(`dev`): add Clouseau to the developer setup [couchdb]
Posted by "pgj (via GitHub)" <gi...@apache.org>.
pgj commented on code in PR #4835:
URL: https://github.com/apache/couchdb/pull/4835#discussion_r1396154669
##########
Makefile:
##########
@@ -313,19 +319,24 @@ list-eunit-suites:
build-test:
@test/build/test-configure.sh
+ifeq ($(with_clouseau), 1)
+_WITH_CLOUSEAU="--with-clouseau"
+endif
.PHONY: mango-test
# target: mango-test - Run Mango tests
mango-test: export COUCHDB_TEST_ADMIN_PARTY_OVERRIDE=1
mango-test: devclean all
- @cd src/mango && \
- python3 -m venv .venv && \
- .venv/bin/python3 -m pip install -r requirements.txt
- @cd src/mango && \
- ../../dev/run "$(TEST_OPTS)" \
+ @python3 -m venv src/mango/.venv && \
Review Comment:
Are you talking about `Makefile.win`? `PYTHON` does not seem to be defined in `Makefile`...?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(`dev`): add Clouseau to the developer setup [couchdb]
Posted by "jaydoane (via GitHub)" <gi...@apache.org>.
jaydoane commented on code in PR #4835:
URL: https://github.com/apache/couchdb/pull/4835#discussion_r1390273239
##########
README-DEV.rst:
##########
@@ -272,6 +266,51 @@ to keep those databases around for further investigation::
MANGO_TESTS_KEEP_DBS=please \
make mango-test MANGO_TEST_OPTS='03-operator-test'
+Running Clouseau
+~~~~~~~~~~~~~~~~
+
+When configured with the ``./configure`` script, the ``./dev/run``
+script is capable of launching Clouseau instances alongside the
+CouchDB nodes and hooking them up. This is what the ``mango-test``
+and ``elixir-search`` targets also use to run their respective test
+suites and let Clouseau automatically managed for them.
Review Comment:
Maybe insert a comma after "suites", change "managed" to "manage", and delete "for".
##########
configure:
##########
@@ -38,6 +39,8 @@ run_erlang() {
COUCHDB_USER="$(whoami 2>/dev/null || echo couchdb)"
SM_VSN=${SM_VSN:-"91"}
+CLOUSEAU_VSN=${CLOUSEAU_VSN:-"2.22.0"}
+CLOUSEAU_DIR="$(pwd)"/clouseau
Review Comment:
I was pondering whether it made sense to stick this into `src` like all the other (Erlang) deps, but then noticed that `nouveau` was also at the top level, so guessing that was the precedent you used for deciding where this should go?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(`dev`): add Clouseau to the developer setup [couchdb]
Posted by "big-r81 (via GitHub)" <gi...@apache.org>.
big-r81 commented on code in PR #4835:
URL: https://github.com/apache/couchdb/pull/4835#discussion_r1389553098
##########
README-DEV.rst:
##########
@@ -272,6 +266,50 @@ to keep those databases around for further investigation::
MANGO_TESTS_KEEP_DBS=please \
make mango-test MANGO_TEST_OPTS='03-operator-test'
+Running Clouseau
+~~~~~~~~~~~~~~~~
+
+When configured with the ``./configure`` script, the ``dev/run``
+script is capable of launching Clouseau instances alongside the
+CouchDB nodes and hooking them up. This is what the ``mango-test``
+and ``elixir-search`` targets also use to run their respective test
+suites and let Clouseau automatically managed for them.
+
+Although the ``./configure`` and the ``dev/run`` scripts try to take
+care of the details of the Clouseau deployment, it is still the
+responsibility of the user to provide a suitable Java environment for
+running. Clouseau can run with JDK 1.7 and 1.8 only. Also, when
+Nouveau is in use, which uses a more recent Java environment, the old
+JDK has to be installed separately and the ``CLOUSEAU_JAVA_HOME``
+environment variable has to be set to point its location.
+
+Fortunately, the ```asdf`` tool <https://asdf-vm.com/>` provides a
+convenient way to install old versions of JDK through its ```java``
+plugin <https://github.com/halcyon/asdf-java>`::
+
+ asdf plugin add java
+
+Then use ``asdf`` to install it::
+
+ asdf install java zulu-8.74.0.17
+
+Finally, use ``asdf`` to set the ``CLOUSEAU_JAVA_HOME`` environment
+variable::
+
+ export CLOUSEAU_JAVA_HOME=$(asdf where java zulu-8.74.0.17)
+
+If the use of ``asdf`` is not an option, `the Zulu site
+<https://cdn.azul.com/zulu/bin/>` could be used directly to get the
+distribution package for the appropriate JDK version. But this is
+just one of the possibilities to access installers for old Java
+environments.
+
+Once both Clouseau and the corresponding Java environment are set,
+they are not put in use automatically. In order to do so, the
+``dev/run`` script needs to be run with Clouseau enabled as follows::
Review Comment:
Do we want to use `./dev/run`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(`dev`): add Clouseau to the developer setup [couchdb]
Posted by "big-r81 (via GitHub)" <gi...@apache.org>.
big-r81 commented on code in PR #4835:
URL: https://github.com/apache/couchdb/pull/4835#discussion_r1389554272
##########
README-DEV.rst:
##########
@@ -272,6 +266,50 @@ to keep those databases around for further investigation::
MANGO_TESTS_KEEP_DBS=please \
make mango-test MANGO_TEST_OPTS='03-operator-test'
+Running Clouseau
+~~~~~~~~~~~~~~~~
+
+When configured with the ``./configure`` script, the ``dev/run``
+script is capable of launching Clouseau instances alongside the
+CouchDB nodes and hooking them up. This is what the ``mango-test``
+and ``elixir-search`` targets also use to run their respective test
+suites and let Clouseau automatically managed for them.
+
+Although the ``./configure`` and the ``dev/run`` scripts try to take
Review Comment:
same here, do we want to use `./dev/run`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(`dev`): add Clouseau to the developer setup [couchdb]
Posted by "pgj (via GitHub)" <gi...@apache.org>.
pgj commented on code in PR #4835:
URL: https://github.com/apache/couchdb/pull/4835#discussion_r1391440212
##########
configure:
##########
@@ -38,6 +39,8 @@ run_erlang() {
COUCHDB_USER="$(whoami 2>/dev/null || echo couchdb)"
SM_VSN=${SM_VSN:-"91"}
+CLOUSEAU_VSN=${CLOUSEAU_VSN:-"2.22.0"}
+CLOUSEAU_DIR="$(pwd)"/clouseau
Review Comment:
Yes, the top-level `nouveau` directory inspired me. Also, I wanted to make Clouseau, as its sibling, look similar therefore easier for the developers to understand how it works. Mind that `clouseau` will not hold any source files but "executable" files (JARs) and configuration files. Thus it would not make a good addition to `src`, I believe.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(`dev`): add Clouseau to the developer setup [couchdb]
Posted by "pgj (via GitHub)" <gi...@apache.org>.
pgj commented on code in PR #4835:
URL: https://github.com/apache/couchdb/pull/4835#discussion_r1396303802
##########
Makefile:
##########
@@ -313,19 +319,24 @@ list-eunit-suites:
build-test:
@test/build/test-configure.sh
+ifeq ($(with_clouseau), 1)
+_WITH_CLOUSEAU="--with-clouseau"
+endif
.PHONY: mango-test
# target: mango-test - Run Mango tests
mango-test: export COUCHDB_TEST_ADMIN_PARTY_OVERRIDE=1
mango-test: devclean all
- @cd src/mango && \
- python3 -m venv .venv && \
- .venv/bin/python3 -m pip install -r requirements.txt
- @cd src/mango && \
- ../../dev/run "$(TEST_OPTS)" \
+ @python3 -m venv src/mango/.venv && \
Review Comment:
All right, done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@couchdb.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org