[GitHub] [druid] abhagraw commented on a diff in pull request #13669: Generate the IT docker-compose.yaml files

[GitBox <gi...@apache.org>]

abhagraw commented on code in PR #13669:
URL: https://github.com/apache/druid/pull/13669#discussion_r1080856667


##########
integration-tests-ex/docs/compose.md:
##########
@@ -236,3 +236,56 @@ To define a test cluster, do the following:
 * If you need multiple instances of the same service, extend that service
   twice, and define distinct names and port numbers.
 * Add any test-specific environment configuration required.
+
+## Generating `docker-compose.yaml` Files
+
+Each test has somewhat different needs for its test cluster. Yet, there is a
+great amount of consistency across test clusters and across services. The result,
+if we create files by hand, is a great amount of copy/paste redundancy, with all
+the problems that copy/paste implies.
+
+As an alternative, the framework provides a simple-minded way to generate the
+`docker-compose.yaml` file using a simple Bash-based template mechanism. To use
+this:
+
+* Create your test cluster directory: `cluster/<category>`.
+* Create a file in that directory called `docker-compose.sh`.
+* The minimal file appears below:
+
+```bash
+set -e
+
+TEMPLATE=$0
+export MODULE_DIR=$(cd $(dirname $0) && pwd)
+CATEGORY=$(basename $MODULE_DIR)
+
+. $MODULE_DIR/../Common/gen-docker.sh
+
+# Custom functions go here
+
+gen_compose_file $CATEGORY
+```
+
+The above will generate a "generic" cluster: one of each kind of service, with
+either a Middle Manager or Indexer depending on the `USE_INDEXER`
+env var.
+
+You customize your specific cluster by "overriding" (really, just replacing) the
+various Bash functions that do the generation. See any of the existing files for
+examples.
+
+For example, you can:
+
+* Add test-specific environment config to one, some or all services.
+* Add or remove services.
+* Create multiples of selected services.
+
+The advantage is that, as Druid evolves and we change the basics, those changes
+are automatically propagated to all test clusters.
+
+Once you've created your file, the test framework will re-generate the
+`docker-compose.yaml` file on each run to reflect any per-run customization.

Review Comment:
   On the second run of the test, how does the framework know if the existing `docker-compose.yaml` file is auto-generated or static?



##########
integration-tests-ex/cases/cluster.sh:
##########
@@ -137,17 +142,34 @@ function build_shared_dir {
 	chmod -R a+rwx $SHARED_DIR
 }
 
-# Each test must have a default docker-compose.yaml file which corresponds to using
+# Either generate the docker-compose file, or use "static" versions.
+function docker_file {
+
+	# If a template exists, generate the docker-compose.yaml file. Copy over the Common
+	# folder.
+	if [ -f "$CLUSTER_DIR/docker-compose.sh" ]; then

Review Comment:
   Should be base this on some env var instead of file presence?
   
   We can have the default to be used if the variable is not set, else use the compose file path set in that variable. This gives the freedom to easy use any compose file easily.
   
   In the current scenario, we will need to delete the `docker-compose.sh` file and then create/edit the `docker-compose.yaml` - if we want to try a different docker compose file.



##########
integration-tests-ex/cases/cluster.sh:
##########
@@ -171,33 +206,41 @@ case $CMD in
 		;;
 	"help" )
 		usage
-		docker-compose help
+		docker compose help
+		;;
+	"prepare" )
+		category $*
+		build_shared_dir
+		docker_file
 		;;
 	"up" )
 		category $*
 		echo "Starting cluster $DRUID_INTEGRATION_TEST_GROUP"
 		build_shared_dir
-	    cd $CLUSTER_DIR
-		docker-compose `docker_file` up -d
+		docker_file
+	    cd $COMPOSE_DIR

Review Comment:
   nit: indentation is not correct here and at some lines below this for `cd $COMPOSE_DIR.`



##########
integration-tests-ex/cases/cluster/Common/gen-docker.sh:
##########
@@ -0,0 +1,396 @@
+#! /bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#--------------------------------------------------------------------
+
+# Generates docker-compose.yaml files for a test. Avoids the need for
+# copy/paste to create test configs. Also automatically switches from
+# MiddleManager to Indexer based on the setting of the
+# USE_INDEXER env var.
+#
+# This script acts like a simple template engine. Since the generated
+# files are simple, using shell commands for the template is not quite
+# as crazy as it sounds. If needes get more complex, we can switch to
+# the Freemarker template engine used to generate the Calcite parser.
+#
+# To customize, find the function that generates the bit to be changed.
+# Copy that function to your per-test template. This master template
+# includes the per-test template after defining all functions, so that
+# any in your template replace (not override) those defined here.
+#
+# Any function with "custom" in the name is intended to help with

Review Comment:
   I think instead of `"custom"` - it should be `"gen"`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org