You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2023/01/12 22:56:10 UTC
[GitHub] [druid] paul-rogers opened a new pull request, #13669: Generate the IT docker-compose.yaml files
paul-rogers opened a new pull request, #13669:
URL: https://github.com/apache/druid/pull/13669
The "new ITs" use Docker compose. As we add tests, we find we're creating a large amount of redundancy: each test "category" needs basically the same cluster, with a few variations. Some tests need both a Middle Manager and an Indexer version of the cluster. While Docker Compose provides some tools to reduce the resulting redundancy, we still find that we must maintain a large amount of copy/paste near-duplicates, with the maintenance cost which copy/paste implies.
This PR uses a very simple Bash script to generate the `docker-compose.yaml` files using "templates." A "base" template provides all the boilerplate. A category-specific template adds test-specific customization. The `cluster.sh` script which launches the cluster checks if a test has a template file. If so, it generates the required `docker-compose.yaml` file on the fly when starting (or stopping) the cluster.
The result is to vastly reduce redundancy. Templates are only as complex as needed to express how some specific cluster differs from the "standard" Druid cluster.
This PR converts three of the test categories as a proof-of-concept. The `cluster.sh` script works as previously if a category uses a "static" `docker-compose.yaml` file. It is transparent to the users of the test.
This PR builds on another. Before commit, we'll merge this one onto the latest master so that it contains only the template code. The files unique to this PR are in `integration-tests-ex/cases`:
* `cluster.sh`
* `cluster/BatchIndex/docker-compose.sh`
* `cluster/Common/gen-docker.sh`
* `cluster/HighAvailability/docker-compose.sh`
* `cluster/S3DeepStorage/docker-compose.sh`
To see the code gen in action:
```bash
cd integration-tests-ex/cases
cluster/S3DeepStorage/docker-compose.sh
more target/cluster/S3DeepStorage/docker-compose.yaml
```
The code generator was validated by comparing the generated file against the existing hand-written file. Once validated, the hand-written file was deleted. If you want to do the same validation, just re-add the hand-written file:
```
git checkout master integration-tests-ex/cases/cluster/S3DeepStorage/docker-compose.yaml
```
<hr>
This PR has:
- [X] been self-reviewed.
- [X] added documentation for new or modified features or behaviors.
- [X] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] paul-rogers commented on a diff in pull request #13669: Generate the IT docker-compose.yaml files
Posted by "paul-rogers (via GitHub)" <gi...@apache.org>.
paul-rogers commented on code in PR #13669:
URL: https://github.com/apache/druid/pull/13669#discussion_r1084411970
##########
integration-tests-ex/cases/cluster.sh:
##########
@@ -137,17 +142,34 @@ function build_shared_dir {
chmod -R a+rwx $SHARED_DIR
}
-# Each test must have a default docker-compose.yaml file which corresponds to using
+# Either generate the docker-compose file, or use "static" versions.
+function docker_file {
+
+ # If a template exists, generate the docker-compose.yaml file. Copy over the Common
+ # folder.
+ if [ -f "$CLUSTER_DIR/docker-compose.sh" ]; then
Review Comment:
The idea here is that we already have a directory structure like this:
```text
integration-test-ex/cases
- cluster
- SomeTest
- docker-compose.yaml
- AnotherTest
- docker-compose.yaml
```
Now, suppose we convert just `SomeTest` to use the generator. Then we'll have:
```text
integration-test-ex/cases
- template
- SomeTest.py
- AnotherTest.py
```
The `cluster/<category>` directory is no longer needed. The logic in the script looks at this file structure to decide if there is a hand-created `docker-compose.yaml` file, or if we should generate one.
If we used an environment variable, then every developer and GHA test script would have to set that variable to match the files on disk. Since there is only one right answer (or the test will fail), we might as well infer that answer from the file structure rather than forcing a human to replicate that information.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] abhishekagarwal87 commented on a diff in pull request #13669: Generate the IT docker-compose.yaml files
Posted by "abhishekagarwal87 (via GitHub)" <gi...@apache.org>.
abhishekagarwal87 commented on code in PR #13669:
URL: https://github.com/apache/druid/pull/13669#discussion_r1106905070
##########
integration-tests-ex/cases/cluster.sh:
##########
@@ -54,114 +62,159 @@ shift
# All commands need env vars
ENV_FILE=$MODULE_DIR/../image/target/env.sh
if [ ! -f $ENV_FILE ]; then
- echo "Please build the Docker test image before testing" 1>&2
- exit 1
+ echo "Please build the Docker test image before testing" 1>&2
+ exit 1
fi
source $ENV_FILE
function category {
- if [ $# -eq 0 ]; then
- usage 1>&2
- exit 1
- fi
- export CATEGORY=$1
- # The untranslated category is used for the local name of the
- # shared folder.
-
- # DRUID_INTEGRATION_TEST_GROUP is used in
- # docker-compose files and here. Despite the name, it is the
- # name of the cluster configuration we want to run, not the
- # test category. Multiple categories can map to the same cluster
- # definition.
-
- # Map from category name to shared cluster definition name.
- # Add an entry here if you create a new category that shares
- # a definition.
- case $CATEGORY in
- "InputSource")
- export DRUID_INTEGRATION_TEST_GROUP=BatchIndex
- ;;
- "InputFormat")
- export DRUID_INTEGRATION_TEST_GROUP=BatchIndex
- ;;
- "Catalog")
- export DRUID_INTEGRATION_TEST_GROUP=BatchIndex
- ;;
- *)
- export DRUID_INTEGRATION_TEST_GROUP=$CATEGORY
- ;;
- esac
-
- export CLUSTER_DIR=$MODULE_DIR/cluster/$DRUID_INTEGRATION_TEST_GROUP
- if [ ! -d $CLUSTER_DIR ]; then
- echo "Cluster directory $CLUSTER_DIR does not exist." 1>&2
- echo "$USAGE" 1>&2
- exit 1
- fi
-
- export TARGET_DIR=$MODULE_DIR/target
- export SHARED_DIR=$TARGET_DIR/$CATEGORY
- export ENV_FILE="$TARGET_DIR/${CATEGORY}.env"
+ if [ $# -eq 0 ]; then
+ usage 1>&2
+ exit 1
+ fi
+ export CATEGORY=$1
+ # The untranslated category is used for the local name of the
+ # shared folder.
+
+ # DRUID_INTEGRATION_TEST_GROUP is used in
+ # docker-compose files and here. Despite the name, it is the
+ # name of the cluster configuration we want to run, not the
+ # test category. Multiple categories can map to the same cluster
+ # definition.
+
+ # Map from category name to shared cluster definition name.
+ # Add an entry here if you create a new category that shares
+ # a definition.
+ case $CATEGORY in
+ "InputSource")
+ export DRUID_INTEGRATION_TEST_GROUP=BatchIndex
+ ;;
+ "InputFormat")
+ export DRUID_INTEGRATION_TEST_GROUP=BatchIndex
+ ;;
+ "Catalog")
+ export DRUID_INTEGRATION_TEST_GROUP=BatchIndex
+ ;;
+ *)
+ export DRUID_INTEGRATION_TEST_GROUP=$CATEGORY
+ ;;
+ esac
+
+ export CLUSTER_DIR=$MODULE_DIR/cluster/$DRUID_INTEGRATION_TEST_GROUP
+ export TARGET_DIR=$MODULE_DIR/target
+ export SHARED_DIR=$TARGET_DIR/$CATEGORY
+ export ENV_FILE="$TARGET_DIR/${CATEGORY}.env"
}
# Dump lots of information to debug Docker failures when run inside
# of a build environment where we can't inspect Docker directly.
function show_status {
- echo "===================================="
- ls -l target/shared
- echo "docker ps -a"
- docker ps -a
- # Was: --filter status=exited
- for id in $(docker ps -a --format "{{.ID}}"); do
- echo "===================================="
- echo "Logs for Container ID $id"
- docker logs $id | tail -n 20
- done
- echo "===================================="
+ echo "===================================="
+ ls -l target/shared
+ echo "docker ps -a"
+ docker ps -a
+ # Was: --filter status=exited
+ for id in $(docker ps -a --format "{{.ID}}"); do
+ echo "===================================="
+ echo "Logs for Container ID $id"
+ docker logs $id | tail -n 20
+ done
+ echo "===================================="
}
function build_shared_dir {
- mkdir -p $SHARED_DIR
- # Must start with an empty DB to keep MySQL happy
- rm -rf $SHARED_DIR/db
- mkdir -p $SHARED_DIR/logs
- mkdir -p $SHARED_DIR/tasklogs
- mkdir -p $SHARED_DIR/db
- mkdir -p $SHARED_DIR/kafka
- mkdir -p $SHARED_DIR/resources
- cp $MODULE_DIR/assets/log4j2.xml $SHARED_DIR/resources
- # Permissions in some build setups are screwed up. See above. The user
- # which runs Docker does not have permission to write into the /shared
- # directory. Force ownership to allow writing.
- chmod -R a+rwx $SHARED_DIR
+ mkdir -p $SHARED_DIR
+ # Must start with an empty DB to keep MySQL happy
+ rm -rf $SHARED_DIR/db
+ mkdir -p $SHARED_DIR/logs
+ mkdir -p $SHARED_DIR/tasklogs
+ mkdir -p $SHARED_DIR/db
+ mkdir -p $SHARED_DIR/kafka
+ mkdir -p $SHARED_DIR/resources
+ cp $MODULE_DIR/assets/log4j2.xml $SHARED_DIR/resources
+ # Permissions in some build setups are screwed up. See above. The user
+ # which runs Docker does not have permission to write into the /shared
+ # directory. Force ownership to allow writing.
+ chmod -R a+rwx $SHARED_DIR
+}
+
+# Either generate the docker-compose file, or use "static" versions.
+function docker_file {
+
+ # If a template exists, generate the docker-compose.yaml file. Copy over the Common
+ # folder.
+ TEMPLATE_DIR=$MODULE_DIR/templates
+ TEMPLATE_FILE=${DRUID_INTEGRATION_TEST_GROUP}.py
Review Comment:
the name `template_file` seems confusing as it is not really a template.
##########
integration-tests-ex/cases/templates/template.py:
##########
@@ -0,0 +1,319 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+'''
+Generates a docker-compose.yaml file from a test-specific template. Each
+test template either uses the base template directly, or extends the template
+to customize bits of the cluster.
+
+Since the cluster is defined as YAML, the cluster definition is build up
+internally as a Python data structure made up of maps, arrays and scalars.
+PyYaml does the grunt work of converting the data structure to the YAML file.
+'''
+
+import yaml, os, os.path
+from pathlib import Path
+
+DRUID_NETWORK = 'druid-it-net'
+DRUID_SUBNET = '172.172.172'
+ZOO_KEEPER = 'zookeeper'
+METADATA = 'metadata'
+COORDINATOR = 'coordinator'
+OVERLORD = 'overlord'
+ROUTER = 'router'
+BROKER = 'broker'
+HISTORICAL = 'historical'
+INDEXER = 'indexer'
+MIDDLE_MANAGER = 'middlemanager'
+
+def generate(template_path, template):
+ #template_path = Path(sys.modules[template.__class__.__module__].__file__)
+ template_path = Path(template_path)
+ #print("template_path", template_path)
+ cluster = template_path.stem
+ #print("Cluster", cluster)
+ module_dir = Path(__file__).parent.parent
+ target_dir = module_dir.joinpath("target")
+ target_file = target_dir.joinpath('cluster', cluster, 'docker-compose.yaml')
+ #print("target_file", target_file)
+ with target_file.open("w") as f:
+ template.generate_file(f, cluster)
+ f.close()
Review Comment:
can you please remove the commented code?
##########
integration-tests-ex/cases/templates/template.py:
##########
@@ -0,0 +1,319 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+'''
+Generates a docker-compose.yaml file from a test-specific template. Each
+test template either uses the base template directly, or extends the template
+to customize bits of the cluster.
+
+Since the cluster is defined as YAML, the cluster definition is build up
+internally as a Python data structure made up of maps, arrays and scalars.
+PyYaml does the grunt work of converting the data structure to the YAML file.
+'''
+
+import yaml, os, os.path
+from pathlib import Path
+
+DRUID_NETWORK = 'druid-it-net'
+DRUID_SUBNET = '172.172.172'
+ZOO_KEEPER = 'zookeeper'
+METADATA = 'metadata'
+COORDINATOR = 'coordinator'
+OVERLORD = 'overlord'
+ROUTER = 'router'
+BROKER = 'broker'
+HISTORICAL = 'historical'
+INDEXER = 'indexer'
+MIDDLE_MANAGER = 'middlemanager'
+
+def generate(template_path, template):
+ #template_path = Path(sys.modules[template.__class__.__module__].__file__)
+ template_path = Path(template_path)
+ #print("template_path", template_path)
+ cluster = template_path.stem
+ #print("Cluster", cluster)
+ module_dir = Path(__file__).parent.parent
+ target_dir = module_dir.joinpath("target")
+ target_file = target_dir.joinpath('cluster', cluster, 'docker-compose.yaml')
+ #print("target_file", target_file)
+ with target_file.open("w") as f:
+ template.generate_file(f, cluster)
+ f.close()
+
+class BaseTemplate:
+
+ def __init__(self):
+ self.cluster = {}
+
+ def generate_file(self, out_file, cluster):
+ self.cluster_name = cluster
+ self.define_cluster()
+ self.out_file = out_file
+ self.generate()
+
+ def define_cluster(self):
+ self.define_network()
+ self.define_support_services()
+ self.define_druid_services()
+ self.define_custom_services()
+
+ def define_support_services(self):
+ self.define_zk()
+ self.define_metadata()
+
+ def define_druid_services(self):
+ self.define_coordinator()
+ self.define_overlord()
+ self.define_broker()
+ self.define_router()
+ self.define_historical()
+ self.define_indexer()
+
+ def define_custom_services(self):
+ '''
+ Override to define additional services for the cluster.
+ '''
+ pass
+
+ def generate(self):
+ self.gen_header()
+ self.gen_header_comment()
+ self.gen_body()
+
+ def emit(self, text):
+ # Chop off the newline that occurs when ''' is on a separate line
+ if len(text) > 0 and text[0] == '\n':
+ text = text[1:]
+ self.out_file.write(text)
+
+ def gen_header(self):
+ self.emit('''
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#--------------------------------------------------------------------
Review Comment:
Is this header needed if we are not checking in the generated files?
##########
integration-tests-ex/cases/templates/template.py:
##########
@@ -0,0 +1,319 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+'''
+Generates a docker-compose.yaml file from a test-specific template. Each
+test template either uses the base template directly, or extends the template
+to customize bits of the cluster.
+
+Since the cluster is defined as YAML, the cluster definition is build up
+internally as a Python data structure made up of maps, arrays and scalars.
+PyYaml does the grunt work of converting the data structure to the YAML file.
+'''
+
+import yaml, os, os.path
+from pathlib import Path
+
+DRUID_NETWORK = 'druid-it-net'
+DRUID_SUBNET = '172.172.172'
+ZOO_KEEPER = 'zookeeper'
+METADATA = 'metadata'
+COORDINATOR = 'coordinator'
+OVERLORD = 'overlord'
+ROUTER = 'router'
+BROKER = 'broker'
+HISTORICAL = 'historical'
+INDEXER = 'indexer'
+MIDDLE_MANAGER = 'middlemanager'
+
+def generate(template_path, template):
+ #template_path = Path(sys.modules[template.__class__.__module__].__file__)
+ template_path = Path(template_path)
+ #print("template_path", template_path)
+ cluster = template_path.stem
+ #print("Cluster", cluster)
+ module_dir = Path(__file__).parent.parent
+ target_dir = module_dir.joinpath("target")
+ target_file = target_dir.joinpath('cluster', cluster, 'docker-compose.yaml')
+ #print("target_file", target_file)
+ with target_file.open("w") as f:
+ template.generate_file(f, cluster)
+ f.close()
Review Comment:
you can though add examples in the comments for target_dir. Not necessary, but will be nice to have.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] github-code-scanning[bot] commented on a diff in pull request #13669: Generate the IT docker-compose.yaml files
Posted by "github-code-scanning[bot] (via GitHub)" <gi...@apache.org>.
github-code-scanning[bot] commented on code in PR #13669:
URL: https://github.com/apache/druid/pull/13669#discussion_r1096631151
##########
integration-tests-ex/cases/templates/template.py:
##########
@@ -0,0 +1,314 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+'''
+Generates a docker-compose.yaml file from a test-specific template. Each
+test template either uses the base template directly, or extends the template
+to customize bits of the cluster.
+
+Since the cluster is defined as YAML, the cluster definition is build up
+internally as a Python data structure made up of maps, arrays and scalars.
+PyYaml does the grunt work of converting the data structure to the YAML file.
+'''
+
+import yaml, os, sys, os.path
+from pathlib import Path
+
+DRUID_NETWORK = 'druid-it-net'
+DRUID_SUBNET = '172.172.172'
+ZOO_KEEPER = 'zookeeper'
+METADATA = 'metadata'
+COORDINATOR = 'coordinator'
+OVERLORD = 'overlord'
+ROUTER = 'router'
+BROKER = 'broker'
+HISTORICAL = 'historical'
+INDEXER = 'indexer'
+MIDDLE_MANAGER = 'middlemanager'
+
+def generate(template_path, template):
+ #template_path = Path(sys.modules[template.__class__.__module__].__file__)
+ template_path = Path(template_path)
+ #print("template_path", template_path)
+ cluster = template_path.stem
+ #print("Cluster", cluster)
+ module_dir = Path(__file__).parent.parent
+ target_dir = module_dir.joinpath("target")
+ target_file = target_dir.joinpath('cluster', cluster, 'docker-compose.yaml')
+ #print("target_file", target_file)
+ with target_file.open("w") as f:
+ template.generate_file(f, cluster)
+ f.close()
+
+class BaseTemplate:
+
+ def __init__(self):
+ self.cluster = {}
+
+ def generate_file(self, out_file, cluster):
+ self.cluster_name = cluster
+ self.define_cluster()
+ self.out_file = out_file
+ self.generate()
+
+ def define_cluster(self):
+ self.define_network()
+ self.define_support_services()
+ self.define_druid_services()
+ self.define_custom_services()
+
+ def define_support_services(self):
+ self.define_zk()
+ self.define_metadata()
+
+ def define_druid_services(self):
+ self.define_coordinator()
+ self.define_overlord()
+ self.define_broker()
+ self.define_router()
+ self.define_historical()
+ self.define_indexer()
+
+ def define_custom_services(self):
+ '''
+ Override to define additional services for the cluster.
+ '''
+ pass
+
+ def generate(self):
+ self.gen_header()
+ self.gen_header_comment()
+ self.gen_body()
+
+ def emit(self, text):
+ # Chop off the newline that occurs when ''' is on a separate line
+ if len(text) > 0 and text[0] == '\n':
+ text = text[1:]
+ self.out_file.write(text)
+
+ def gen_header(self):
+ self.emit('''
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#--------------------------------------------------------------------
+
+# THIS FILE IS GENERATED -- DO NOT EDIT!
+#
+# Instead, edit the template from which this file was generated.
+# Template: templates/{}.py
+
+'''.format(self.cluster_name))
+
+ def gen_header_comment(self):
+ '''
+ Override to generate a custom header comment after the standard header.
+ '''
+ pass
+
+ def gen_body(self):
+ yaml.dump(self.cluster, self.out_file, sort_keys=False)
+
+ def define_network(self):
+ self.cluster['networks'] = {
+ 'druid-it-net': {
+ 'name': DRUID_NETWORK,
+ 'ipam': {
+ 'config': [
+ {'subnet': DRUID_SUBNET + '.0/24'}
+ ]
+ }
+ }
+ }
+
+ def add_service(self, name, service):
+ services = self.cluster.setdefault('services', {})
+ services[name] = service
+
+ def add_volume(self, service, local, container):
+ '''
+ Add a volume to a service.
+ '''
+ volumes = service.setdefault('volumes', [])
+ volumes.append(local + ':' + container)
+
+ def add_env(self, service, var, value):
+ '''
+ Add an environment variable to a service.
+ '''
+ vars = service.setdefault('environment', [])
+ vars.append(var + '=' + value)
+
+ def add_property(self, service, prop, value):
+ '''
+ Sets a property for a service. The property is of the same form as the
+ .properties file: druid.some.property.
+ This method converts the property to the env var form so you don't have to.
+ '''
+ var = prop.replace('.', '_')
+ self.add_env(service, var, value)
+
+ def add_env_file(self, service, env_file):
+ '''
+ Add an environment file to a service.
+ '''
+ env_files = service.setdefault('env_file', [])
+ env_files.append(env_file)
+
+ def add_env_config(self, service, base_name):
+ '''
+ Add to a service one of the standard environment config files in
+ the Common/environment-configs directory
+ '''
+ self.add_env_file(service, '../Common/environment-configs/' + base_name + '.env')
+
+ def add_port(self, service, local, container):
+ '''
+ Add a port mapping to the service
+ '''
+ ports = service.setdefault('ports', [])
+ ports.append(local + ':' + container)
+
+ def define_external_service(self, name) -> dict:
+ service = {'extends': {
+ 'file': '../Common/dependencies.yaml',
+ 'service': name
+ }}
+ self.add_service(name, service)
+ return service
+
+ def define_zk(self):
+ self.define_external_service(ZOO_KEEPER)
+
+ def define_metadata(self):
+ self.define_external_service(METADATA)
+
+ def define_druid_service(self, name, base):
+ service = {}
+ if base is not None:
+ service['extends'] = {
+ 'file': '../Common/druid.yaml',
+ 'service': base
+ }
+ self.extend_druid_service(service)
+ self.add_service(name, service)
+ return service
+
+ def extend_druid_service(self, service):
+ '''
+ Override this to add options to all Druid services.
+ '''
+ pass
+
+ def add_depends(self, service, items):
+ if items is None or len(items) == 0:
+ return
+ depends = service.setdefault('depends_on', [])
+ depends += items
+ return depends
+
+ def define_master_service(self, name, base):
+ service = self.define_druid_service(name, base)
+ self.add_depends(service, [ZOO_KEEPER, METADATA])
+ return service
+
+ def define_std_master_service(self, name):
+ return self.define_master_service(name, name)
+
+ def define_coordinator(self):
+ return self.define_std_master_service(COORDINATOR)
+ return service
Review Comment:
## Unreachable code
This statement is unreachable.
[Show more details](https://github.com/apache/druid/security/code-scanning/4232)
##########
integration-tests-ex/cases/templates/template.py:
##########
@@ -0,0 +1,314 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+'''
+Generates a docker-compose.yaml file from a test-specific template. Each
+test template either uses the base template directly, or extends the template
+to customize bits of the cluster.
+
+Since the cluster is defined as YAML, the cluster definition is build up
+internally as a Python data structure made up of maps, arrays and scalars.
+PyYaml does the grunt work of converting the data structure to the YAML file.
+'''
+
+import yaml, os, sys, os.path
Review Comment:
## Unused import
Import of 'sys' is not used.
[Show more details](https://github.com/apache/druid/security/code-scanning/4231)
##########
integration-tests-ex/cases/templates/template.py:
##########
@@ -0,0 +1,314 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+'''
+Generates a docker-compose.yaml file from a test-specific template. Each
+test template either uses the base template directly, or extends the template
+to customize bits of the cluster.
+
+Since the cluster is defined as YAML, the cluster definition is build up
+internally as a Python data structure made up of maps, arrays and scalars.
+PyYaml does the grunt work of converting the data structure to the YAML file.
+'''
+
+import yaml, os, sys, os.path
+from pathlib import Path
+
+DRUID_NETWORK = 'druid-it-net'
+DRUID_SUBNET = '172.172.172'
+ZOO_KEEPER = 'zookeeper'
+METADATA = 'metadata'
+COORDINATOR = 'coordinator'
+OVERLORD = 'overlord'
+ROUTER = 'router'
+BROKER = 'broker'
+HISTORICAL = 'historical'
+INDEXER = 'indexer'
+MIDDLE_MANAGER = 'middlemanager'
+
+def generate(template_path, template):
+ #template_path = Path(sys.modules[template.__class__.__module__].__file__)
+ template_path = Path(template_path)
+ #print("template_path", template_path)
+ cluster = template_path.stem
+ #print("Cluster", cluster)
+ module_dir = Path(__file__).parent.parent
+ target_dir = module_dir.joinpath("target")
+ target_file = target_dir.joinpath('cluster', cluster, 'docker-compose.yaml')
+ #print("target_file", target_file)
+ with target_file.open("w") as f:
+ template.generate_file(f, cluster)
+ f.close()
+
+class BaseTemplate:
+
+ def __init__(self):
+ self.cluster = {}
+
+ def generate_file(self, out_file, cluster):
+ self.cluster_name = cluster
+ self.define_cluster()
+ self.out_file = out_file
+ self.generate()
+
+ def define_cluster(self):
+ self.define_network()
+ self.define_support_services()
+ self.define_druid_services()
+ self.define_custom_services()
+
+ def define_support_services(self):
+ self.define_zk()
+ self.define_metadata()
+
+ def define_druid_services(self):
+ self.define_coordinator()
+ self.define_overlord()
+ self.define_broker()
+ self.define_router()
+ self.define_historical()
+ self.define_indexer()
+
+ def define_custom_services(self):
+ '''
+ Override to define additional services for the cluster.
+ '''
+ pass
+
+ def generate(self):
+ self.gen_header()
+ self.gen_header_comment()
+ self.gen_body()
+
+ def emit(self, text):
+ # Chop off the newline that occurs when ''' is on a separate line
+ if len(text) > 0 and text[0] == '\n':
+ text = text[1:]
+ self.out_file.write(text)
+
+ def gen_header(self):
+ self.emit('''
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#--------------------------------------------------------------------
+
+# THIS FILE IS GENERATED -- DO NOT EDIT!
+#
+# Instead, edit the template from which this file was generated.
+# Template: templates/{}.py
+
+'''.format(self.cluster_name))
+
+ def gen_header_comment(self):
+ '''
+ Override to generate a custom header comment after the standard header.
+ '''
+ pass
+
+ def gen_body(self):
+ yaml.dump(self.cluster, self.out_file, sort_keys=False)
+
+ def define_network(self):
+ self.cluster['networks'] = {
+ 'druid-it-net': {
+ 'name': DRUID_NETWORK,
+ 'ipam': {
+ 'config': [
+ {'subnet': DRUID_SUBNET + '.0/24'}
+ ]
+ }
+ }
+ }
+
+ def add_service(self, name, service):
+ services = self.cluster.setdefault('services', {})
+ services[name] = service
+
+ def add_volume(self, service, local, container):
+ '''
+ Add a volume to a service.
+ '''
+ volumes = service.setdefault('volumes', [])
+ volumes.append(local + ':' + container)
+
+ def add_env(self, service, var, value):
+ '''
+ Add an environment variable to a service.
+ '''
+ vars = service.setdefault('environment', [])
+ vars.append(var + '=' + value)
+
+ def add_property(self, service, prop, value):
+ '''
+ Sets a property for a service. The property is of the same form as the
+ .properties file: druid.some.property.
+ This method converts the property to the env var form so you don't have to.
+ '''
+ var = prop.replace('.', '_')
+ self.add_env(service, var, value)
+
+ def add_env_file(self, service, env_file):
+ '''
+ Add an environment file to a service.
+ '''
+ env_files = service.setdefault('env_file', [])
+ env_files.append(env_file)
+
+ def add_env_config(self, service, base_name):
+ '''
+ Add to a service one of the standard environment config files in
+ the Common/environment-configs directory
+ '''
+ self.add_env_file(service, '../Common/environment-configs/' + base_name + '.env')
+
+ def add_port(self, service, local, container):
+ '''
+ Add a port mapping to the service
+ '''
+ ports = service.setdefault('ports', [])
+ ports.append(local + ':' + container)
+
+ def define_external_service(self, name) -> dict:
+ service = {'extends': {
+ 'file': '../Common/dependencies.yaml',
+ 'service': name
+ }}
+ self.add_service(name, service)
+ return service
+
+ def define_zk(self):
+ self.define_external_service(ZOO_KEEPER)
+
+ def define_metadata(self):
+ self.define_external_service(METADATA)
+
+ def define_druid_service(self, name, base):
+ service = {}
+ if base is not None:
+ service['extends'] = {
+ 'file': '../Common/druid.yaml',
+ 'service': base
+ }
+ self.extend_druid_service(service)
+ self.add_service(name, service)
+ return service
+
+ def extend_druid_service(self, service):
+ '''
+ Override this to add options to all Druid services.
+ '''
+ pass
+
+ def add_depends(self, service, items):
Review Comment:
## Explicit returns mixed with implicit (fall through) returns
Mixing implicit and explicit returns may indicate an error as implicit returns always return None.
[Show more details](https://github.com/apache/druid/security/code-scanning/4230)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] paul-rogers merged pull request #13669: Generate the IT docker-compose.yaml files
Posted by "paul-rogers (via GitHub)" <gi...@apache.org>.
paul-rogers merged PR #13669:
URL: https://github.com/apache/druid/pull/13669
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] abhagraw commented on a diff in pull request #13669: Generate the IT docker-compose.yaml files
Posted by GitBox <gi...@apache.org>.
abhagraw commented on code in PR #13669:
URL: https://github.com/apache/druid/pull/13669#discussion_r1080856667
##########
integration-tests-ex/docs/compose.md:
##########
@@ -236,3 +236,56 @@ To define a test cluster, do the following:
* If you need multiple instances of the same service, extend that service
twice, and define distinct names and port numbers.
* Add any test-specific environment configuration required.
+
+## Generating `docker-compose.yaml` Files
+
+Each test has somewhat different needs for its test cluster. Yet, there is a
+great amount of consistency across test clusters and across services. The result,
+if we create files by hand, is a great amount of copy/paste redundancy, with all
+the problems that copy/paste implies.
+
+As an alternative, the framework provides a simple-minded way to generate the
+`docker-compose.yaml` file using a simple Bash-based template mechanism. To use
+this:
+
+* Create your test cluster directory: `cluster/<category>`.
+* Create a file in that directory called `docker-compose.sh`.
+* The minimal file appears below:
+
+```bash
+set -e
+
+TEMPLATE=$0
+export MODULE_DIR=$(cd $(dirname $0) && pwd)
+CATEGORY=$(basename $MODULE_DIR)
+
+. $MODULE_DIR/../Common/gen-docker.sh
+
+# Custom functions go here
+
+gen_compose_file $CATEGORY
+```
+
+The above will generate a "generic" cluster: one of each kind of service, with
+either a Middle Manager or Indexer depending on the `USE_INDEXER`
+env var.
+
+You customize your specific cluster by "overriding" (really, just replacing) the
+various Bash functions that do the generation. See any of the existing files for
+examples.
+
+For example, you can:
+
+* Add test-specific environment config to one, some or all services.
+* Add or remove services.
+* Create multiples of selected services.
+
+The advantage is that, as Druid evolves and we change the basics, those changes
+are automatically propagated to all test clusters.
+
+Once you've created your file, the test framework will re-generate the
+`docker-compose.yaml` file on each run to reflect any per-run customization.
Review Comment:
On the second run of the test, how does the framework know if the existing `docker-compose.yaml` file is auto-generated or static?
If the `docker-compose.sh` file is present - there can be no static compose file?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] abhagraw commented on a diff in pull request #13669: Generate the IT docker-compose.yaml files
Posted by GitBox <gi...@apache.org>.
abhagraw commented on code in PR #13669:
URL: https://github.com/apache/druid/pull/13669#discussion_r1080827405
##########
integration-tests-ex/cases/cluster.sh:
##########
@@ -137,17 +142,34 @@ function build_shared_dir {
chmod -R a+rwx $SHARED_DIR
}
-# Each test must have a default docker-compose.yaml file which corresponds to using
+# Either generate the docker-compose file, or use "static" versions.
+function docker_file {
+
+ # If a template exists, generate the docker-compose.yaml file. Copy over the Common
+ # folder.
+ if [ -f "$CLUSTER_DIR/docker-compose.sh" ]; then
Review Comment:
Should be base this on some env var instead of file presence?
We can have the default to be used if the variable is not set, else use the compose file path set in that variable. This gives the freedom to use any compose file easily.
In the current scenario, we will need to delete the `docker-compose.sh` file or create/edit the `docker-compose.yaml` - if we want to try a different docker compose file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] paul-rogers commented on a diff in pull request #13669: Generate the IT docker-compose.yaml files
Posted by "paul-rogers (via GitHub)" <gi...@apache.org>.
paul-rogers commented on code in PR #13669:
URL: https://github.com/apache/druid/pull/13669#discussion_r1084411970
##########
integration-tests-ex/cases/cluster.sh:
##########
@@ -137,17 +142,34 @@ function build_shared_dir {
chmod -R a+rwx $SHARED_DIR
}
-# Each test must have a default docker-compose.yaml file which corresponds to using
+# Either generate the docker-compose file, or use "static" versions.
+function docker_file {
+
+ # If a template exists, generate the docker-compose.yaml file. Copy over the Common
+ # folder.
+ if [ -f "$CLUSTER_DIR/docker-compose.sh" ]; then
Review Comment:
The idea here is that we already have a directory structure like this:
```text
integration-test-ex/cases
- cluster
- SomeTest
- docker-compose.yaml
- AnotherTest
- docker-compose.yaml
```
Now, suppose we convert just `SomeTest` to use the generator. Then we'll have:
```text
integration-test-ex/cases
- cluster
- SomeTest
- docker-compose.sh
- AnotherTest
- docker-compose.yaml
```
Notice that the `.yaml` file is replaced with a `.sh` file. The logic in the script looks at this file structure to decide if there is a hand-created `docker-compose.yaml` file, or if we should generate one.
If we used an environment variable, then every developer and GHA test script would have to set that variable to match the files on disk. Since there is only one right answer (or the test will fail), we might as well infer that answer from the file structure rather than forcing a human to replicate that information.
##########
integration-tests-ex/docs/compose.md:
##########
@@ -236,3 +236,56 @@ To define a test cluster, do the following:
* If you need multiple instances of the same service, extend that service
twice, and define distinct names and port numbers.
* Add any test-specific environment configuration required.
+
+## Generating `docker-compose.yaml` Files
+
+Each test has somewhat different needs for its test cluster. Yet, there is a
+great amount of consistency across test clusters and across services. The result,
+if we create files by hand, is a great amount of copy/paste redundancy, with all
+the problems that copy/paste implies.
+
+As an alternative, the framework provides a simple-minded way to generate the
+`docker-compose.yaml` file using a simple Bash-based template mechanism. To use
+this:
+
+* Create your test cluster directory: `cluster/<category>`.
+* Create a file in that directory called `docker-compose.sh`.
+* The minimal file appears below:
+
+```bash
+set -e
+
+TEMPLATE=$0
+export MODULE_DIR=$(cd $(dirname $0) && pwd)
+CATEGORY=$(basename $MODULE_DIR)
+
+. $MODULE_DIR/../Common/gen-docker.sh
+
+# Custom functions go here
+
+gen_compose_file $CATEGORY
+```
+
+The above will generate a "generic" cluster: one of each kind of service, with
+either a Middle Manager or Indexer depending on the `USE_INDEXER`
+env var.
+
+You customize your specific cluster by "overriding" (really, just replacing) the
+various Bash functions that do the generation. See any of the existing files for
+examples.
+
+For example, you can:
+
+* Add test-specific environment config to one, some or all services.
+* Add or remove services.
+* Create multiples of selected services.
+
+The advantage is that, as Druid evolves and we change the basics, those changes
+are automatically propagated to all test clusters.
+
+Once you've created your file, the test framework will re-generate the
+`docker-compose.yaml` file on each run to reflect any per-run customization.
Review Comment:
Generated files go into the `target` directory, not the source tree. The script picks the proper directory depending on whether the file was hand-coded or generated. The script copies over the `Common` files if the file is generated.
Generation actually occurs only on `up`. `down` reuses the existing generated file so that we take down the cluster be brought up, even if you change `USE_INDEXER` in between.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] abhagraw commented on a diff in pull request #13669: Generate the IT docker-compose.yaml files
Posted by GitBox <gi...@apache.org>.
abhagraw commented on code in PR #13669:
URL: https://github.com/apache/druid/pull/13669#discussion_r1080856667
##########
integration-tests-ex/docs/compose.md:
##########
@@ -236,3 +236,56 @@ To define a test cluster, do the following:
* If you need multiple instances of the same service, extend that service
twice, and define distinct names and port numbers.
* Add any test-specific environment configuration required.
+
+## Generating `docker-compose.yaml` Files
+
+Each test has somewhat different needs for its test cluster. Yet, there is a
+great amount of consistency across test clusters and across services. The result,
+if we create files by hand, is a great amount of copy/paste redundancy, with all
+the problems that copy/paste implies.
+
+As an alternative, the framework provides a simple-minded way to generate the
+`docker-compose.yaml` file using a simple Bash-based template mechanism. To use
+this:
+
+* Create your test cluster directory: `cluster/<category>`.
+* Create a file in that directory called `docker-compose.sh`.
+* The minimal file appears below:
+
+```bash
+set -e
+
+TEMPLATE=$0
+export MODULE_DIR=$(cd $(dirname $0) && pwd)
+CATEGORY=$(basename $MODULE_DIR)
+
+. $MODULE_DIR/../Common/gen-docker.sh
+
+# Custom functions go here
+
+gen_compose_file $CATEGORY
+```
+
+The above will generate a "generic" cluster: one of each kind of service, with
+either a Middle Manager or Indexer depending on the `USE_INDEXER`
+env var.
+
+You customize your specific cluster by "overriding" (really, just replacing) the
+various Bash functions that do the generation. See any of the existing files for
+examples.
+
+For example, you can:
+
+* Add test-specific environment config to one, some or all services.
+* Add or remove services.
+* Create multiples of selected services.
+
+The advantage is that, as Druid evolves and we change the basics, those changes
+are automatically propagated to all test clusters.
+
+Once you've created your file, the test framework will re-generate the
+`docker-compose.yaml` file on each run to reflect any per-run customization.
Review Comment:
On the second run of the test, how does the framework know if the existing `docker-compose.yaml` file is auto-generated or static?
##########
integration-tests-ex/cases/cluster.sh:
##########
@@ -137,17 +142,34 @@ function build_shared_dir {
chmod -R a+rwx $SHARED_DIR
}
-# Each test must have a default docker-compose.yaml file which corresponds to using
+# Either generate the docker-compose file, or use "static" versions.
+function docker_file {
+
+ # If a template exists, generate the docker-compose.yaml file. Copy over the Common
+ # folder.
+ if [ -f "$CLUSTER_DIR/docker-compose.sh" ]; then
Review Comment:
Should be base this on some env var instead of file presence?
We can have the default to be used if the variable is not set, else use the compose file path set in that variable. This gives the freedom to easy use any compose file easily.
In the current scenario, we will need to delete the `docker-compose.sh` file and then create/edit the `docker-compose.yaml` - if we want to try a different docker compose file.
##########
integration-tests-ex/cases/cluster.sh:
##########
@@ -171,33 +206,41 @@ case $CMD in
;;
"help" )
usage
- docker-compose help
+ docker compose help
+ ;;
+ "prepare" )
+ category $*
+ build_shared_dir
+ docker_file
;;
"up" )
category $*
echo "Starting cluster $DRUID_INTEGRATION_TEST_GROUP"
build_shared_dir
- cd $CLUSTER_DIR
- docker-compose `docker_file` up -d
+ docker_file
+ cd $COMPOSE_DIR
Review Comment:
nit: indentation is not correct here and at some lines below this for `cd $COMPOSE_DIR.`
##########
integration-tests-ex/cases/cluster/Common/gen-docker.sh:
##########
@@ -0,0 +1,396 @@
+#! /bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#--------------------------------------------------------------------
+
+# Generates docker-compose.yaml files for a test. Avoids the need for
+# copy/paste to create test configs. Also automatically switches from
+# MiddleManager to Indexer based on the setting of the
+# USE_INDEXER env var.
+#
+# This script acts like a simple template engine. Since the generated
+# files are simple, using shell commands for the template is not quite
+# as crazy as it sounds. If needes get more complex, we can switch to
+# the Freemarker template engine used to generate the Calcite parser.
+#
+# To customize, find the function that generates the bit to be changed.
+# Copy that function to your per-test template. This master template
+# includes the per-test template after defining all functions, so that
+# any in your template replace (not override) those defined here.
+#
+# Any function with "custom" in the name is intended to help with
Review Comment:
I think instead of `"custom"` - it should be `"gen"`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] abhishekagarwal87 commented on a diff in pull request #13669: Generate the IT docker-compose.yaml files
Posted by "abhishekagarwal87 (via GitHub)" <gi...@apache.org>.
abhishekagarwal87 commented on code in PR #13669:
URL: https://github.com/apache/druid/pull/13669#discussion_r1106919054
##########
integration-tests-ex/cases/cluster.sh:
##########
@@ -54,114 +62,159 @@ shift
# All commands need env vars
ENV_FILE=$MODULE_DIR/../image/target/env.sh
if [ ! -f $ENV_FILE ]; then
- echo "Please build the Docker test image before testing" 1>&2
- exit 1
+ echo "Please build the Docker test image before testing" 1>&2
+ exit 1
fi
source $ENV_FILE
function category {
- if [ $# -eq 0 ]; then
- usage 1>&2
- exit 1
- fi
- export CATEGORY=$1
- # The untranslated category is used for the local name of the
- # shared folder.
-
- # DRUID_INTEGRATION_TEST_GROUP is used in
- # docker-compose files and here. Despite the name, it is the
- # name of the cluster configuration we want to run, not the
- # test category. Multiple categories can map to the same cluster
- # definition.
-
- # Map from category name to shared cluster definition name.
- # Add an entry here if you create a new category that shares
- # a definition.
- case $CATEGORY in
- "InputSource")
- export DRUID_INTEGRATION_TEST_GROUP=BatchIndex
- ;;
- "InputFormat")
- export DRUID_INTEGRATION_TEST_GROUP=BatchIndex
- ;;
- "Catalog")
- export DRUID_INTEGRATION_TEST_GROUP=BatchIndex
- ;;
- *)
- export DRUID_INTEGRATION_TEST_GROUP=$CATEGORY
- ;;
- esac
-
- export CLUSTER_DIR=$MODULE_DIR/cluster/$DRUID_INTEGRATION_TEST_GROUP
- if [ ! -d $CLUSTER_DIR ]; then
- echo "Cluster directory $CLUSTER_DIR does not exist." 1>&2
- echo "$USAGE" 1>&2
- exit 1
- fi
-
- export TARGET_DIR=$MODULE_DIR/target
- export SHARED_DIR=$TARGET_DIR/$CATEGORY
- export ENV_FILE="$TARGET_DIR/${CATEGORY}.env"
+ if [ $# -eq 0 ]; then
+ usage 1>&2
+ exit 1
+ fi
+ export CATEGORY=$1
+ # The untranslated category is used for the local name of the
+ # shared folder.
+
+ # DRUID_INTEGRATION_TEST_GROUP is used in
+ # docker-compose files and here. Despite the name, it is the
+ # name of the cluster configuration we want to run, not the
+ # test category. Multiple categories can map to the same cluster
+ # definition.
+
+ # Map from category name to shared cluster definition name.
+ # Add an entry here if you create a new category that shares
+ # a definition.
+ case $CATEGORY in
+ "InputSource")
+ export DRUID_INTEGRATION_TEST_GROUP=BatchIndex
+ ;;
+ "InputFormat")
+ export DRUID_INTEGRATION_TEST_GROUP=BatchIndex
+ ;;
+ "Catalog")
+ export DRUID_INTEGRATION_TEST_GROUP=BatchIndex
+ ;;
+ *)
+ export DRUID_INTEGRATION_TEST_GROUP=$CATEGORY
+ ;;
+ esac
+
+ export CLUSTER_DIR=$MODULE_DIR/cluster/$DRUID_INTEGRATION_TEST_GROUP
+ export TARGET_DIR=$MODULE_DIR/target
+ export SHARED_DIR=$TARGET_DIR/$CATEGORY
+ export ENV_FILE="$TARGET_DIR/${CATEGORY}.env"
}
# Dump lots of information to debug Docker failures when run inside
# of a build environment where we can't inspect Docker directly.
function show_status {
- echo "===================================="
- ls -l target/shared
- echo "docker ps -a"
- docker ps -a
- # Was: --filter status=exited
- for id in $(docker ps -a --format "{{.ID}}"); do
- echo "===================================="
- echo "Logs for Container ID $id"
- docker logs $id | tail -n 20
- done
- echo "===================================="
+ echo "===================================="
+ ls -l target/shared
+ echo "docker ps -a"
+ docker ps -a
+ # Was: --filter status=exited
+ for id in $(docker ps -a --format "{{.ID}}"); do
+ echo "===================================="
+ echo "Logs for Container ID $id"
+ docker logs $id | tail -n 20
+ done
+ echo "===================================="
}
function build_shared_dir {
- mkdir -p $SHARED_DIR
- # Must start with an empty DB to keep MySQL happy
- rm -rf $SHARED_DIR/db
- mkdir -p $SHARED_DIR/logs
- mkdir -p $SHARED_DIR/tasklogs
- mkdir -p $SHARED_DIR/db
- mkdir -p $SHARED_DIR/kafka
- mkdir -p $SHARED_DIR/resources
- cp $MODULE_DIR/assets/log4j2.xml $SHARED_DIR/resources
- # Permissions in some build setups are screwed up. See above. The user
- # which runs Docker does not have permission to write into the /shared
- # directory. Force ownership to allow writing.
- chmod -R a+rwx $SHARED_DIR
+ mkdir -p $SHARED_DIR
+ # Must start with an empty DB to keep MySQL happy
+ rm -rf $SHARED_DIR/db
+ mkdir -p $SHARED_DIR/logs
+ mkdir -p $SHARED_DIR/tasklogs
+ mkdir -p $SHARED_DIR/db
+ mkdir -p $SHARED_DIR/kafka
+ mkdir -p $SHARED_DIR/resources
+ cp $MODULE_DIR/assets/log4j2.xml $SHARED_DIR/resources
+ # Permissions in some build setups are screwed up. See above. The user
+ # which runs Docker does not have permission to write into the /shared
+ # directory. Force ownership to allow writing.
+ chmod -R a+rwx $SHARED_DIR
+}
+
+# Either generate the docker-compose file, or use "static" versions.
+function docker_file {
+
+ # If a template exists, generate the docker-compose.yaml file. Copy over the Common
+ # folder.
+ TEMPLATE_DIR=$MODULE_DIR/templates
+ TEMPLATE_FILE=${DRUID_INTEGRATION_TEST_GROUP}.py
Review Comment:
I looked at the rest of the code and it has a template as well the generation logic. I will leave it to you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] abhishekagarwal87 commented on a diff in pull request #13669: Generate the IT docker-compose.yaml files
Posted by "abhishekagarwal87 (via GitHub)" <gi...@apache.org>.
abhishekagarwal87 commented on code in PR #13669:
URL: https://github.com/apache/druid/pull/13669#discussion_r1106905070
##########
integration-tests-ex/cases/cluster.sh:
##########
@@ -54,114 +62,159 @@ shift
# All commands need env vars
ENV_FILE=$MODULE_DIR/../image/target/env.sh
if [ ! -f $ENV_FILE ]; then
- echo "Please build the Docker test image before testing" 1>&2
- exit 1
+ echo "Please build the Docker test image before testing" 1>&2
+ exit 1
fi
source $ENV_FILE
function category {
- if [ $# -eq 0 ]; then
- usage 1>&2
- exit 1
- fi
- export CATEGORY=$1
- # The untranslated category is used for the local name of the
- # shared folder.
-
- # DRUID_INTEGRATION_TEST_GROUP is used in
- # docker-compose files and here. Despite the name, it is the
- # name of the cluster configuration we want to run, not the
- # test category. Multiple categories can map to the same cluster
- # definition.
-
- # Map from category name to shared cluster definition name.
- # Add an entry here if you create a new category that shares
- # a definition.
- case $CATEGORY in
- "InputSource")
- export DRUID_INTEGRATION_TEST_GROUP=BatchIndex
- ;;
- "InputFormat")
- export DRUID_INTEGRATION_TEST_GROUP=BatchIndex
- ;;
- "Catalog")
- export DRUID_INTEGRATION_TEST_GROUP=BatchIndex
- ;;
- *)
- export DRUID_INTEGRATION_TEST_GROUP=$CATEGORY
- ;;
- esac
-
- export CLUSTER_DIR=$MODULE_DIR/cluster/$DRUID_INTEGRATION_TEST_GROUP
- if [ ! -d $CLUSTER_DIR ]; then
- echo "Cluster directory $CLUSTER_DIR does not exist." 1>&2
- echo "$USAGE" 1>&2
- exit 1
- fi
-
- export TARGET_DIR=$MODULE_DIR/target
- export SHARED_DIR=$TARGET_DIR/$CATEGORY
- export ENV_FILE="$TARGET_DIR/${CATEGORY}.env"
+ if [ $# -eq 0 ]; then
+ usage 1>&2
+ exit 1
+ fi
+ export CATEGORY=$1
+ # The untranslated category is used for the local name of the
+ # shared folder.
+
+ # DRUID_INTEGRATION_TEST_GROUP is used in
+ # docker-compose files and here. Despite the name, it is the
+ # name of the cluster configuration we want to run, not the
+ # test category. Multiple categories can map to the same cluster
+ # definition.
+
+ # Map from category name to shared cluster definition name.
+ # Add an entry here if you create a new category that shares
+ # a definition.
+ case $CATEGORY in
+ "InputSource")
+ export DRUID_INTEGRATION_TEST_GROUP=BatchIndex
+ ;;
+ "InputFormat")
+ export DRUID_INTEGRATION_TEST_GROUP=BatchIndex
+ ;;
+ "Catalog")
+ export DRUID_INTEGRATION_TEST_GROUP=BatchIndex
+ ;;
+ *)
+ export DRUID_INTEGRATION_TEST_GROUP=$CATEGORY
+ ;;
+ esac
+
+ export CLUSTER_DIR=$MODULE_DIR/cluster/$DRUID_INTEGRATION_TEST_GROUP
+ export TARGET_DIR=$MODULE_DIR/target
+ export SHARED_DIR=$TARGET_DIR/$CATEGORY
+ export ENV_FILE="$TARGET_DIR/${CATEGORY}.env"
}
# Dump lots of information to debug Docker failures when run inside
# of a build environment where we can't inspect Docker directly.
function show_status {
- echo "===================================="
- ls -l target/shared
- echo "docker ps -a"
- docker ps -a
- # Was: --filter status=exited
- for id in $(docker ps -a --format "{{.ID}}"); do
- echo "===================================="
- echo "Logs for Container ID $id"
- docker logs $id | tail -n 20
- done
- echo "===================================="
+ echo "===================================="
+ ls -l target/shared
+ echo "docker ps -a"
+ docker ps -a
+ # Was: --filter status=exited
+ for id in $(docker ps -a --format "{{.ID}}"); do
+ echo "===================================="
+ echo "Logs for Container ID $id"
+ docker logs $id | tail -n 20
+ done
+ echo "===================================="
}
function build_shared_dir {
- mkdir -p $SHARED_DIR
- # Must start with an empty DB to keep MySQL happy
- rm -rf $SHARED_DIR/db
- mkdir -p $SHARED_DIR/logs
- mkdir -p $SHARED_DIR/tasklogs
- mkdir -p $SHARED_DIR/db
- mkdir -p $SHARED_DIR/kafka
- mkdir -p $SHARED_DIR/resources
- cp $MODULE_DIR/assets/log4j2.xml $SHARED_DIR/resources
- # Permissions in some build setups are screwed up. See above. The user
- # which runs Docker does not have permission to write into the /shared
- # directory. Force ownership to allow writing.
- chmod -R a+rwx $SHARED_DIR
+ mkdir -p $SHARED_DIR
+ # Must start with an empty DB to keep MySQL happy
+ rm -rf $SHARED_DIR/db
+ mkdir -p $SHARED_DIR/logs
+ mkdir -p $SHARED_DIR/tasklogs
+ mkdir -p $SHARED_DIR/db
+ mkdir -p $SHARED_DIR/kafka
+ mkdir -p $SHARED_DIR/resources
+ cp $MODULE_DIR/assets/log4j2.xml $SHARED_DIR/resources
+ # Permissions in some build setups are screwed up. See above. The user
+ # which runs Docker does not have permission to write into the /shared
+ # directory. Force ownership to allow writing.
+ chmod -R a+rwx $SHARED_DIR
+}
+
+# Either generate the docker-compose file, or use "static" versions.
+function docker_file {
+
+ # If a template exists, generate the docker-compose.yaml file. Copy over the Common
+ # folder.
+ TEMPLATE_DIR=$MODULE_DIR/templates
+ TEMPLATE_FILE=${DRUID_INTEGRATION_TEST_GROUP}.py
Review Comment:
the name `template_file` seems confusing as it is not really a template. can this be called TEMPLATE_SCRIPT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org