[GitHub] csantanapr closed pull request #3105: auto detect docker-for-mac

[GitBox <gi...@apache.org>]

csantanapr closed pull request #3105: auto detect docker-for-mac
URL: https://github.com/apache/incubator-openwhisk/pull/3105
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/.gitignore b/.gitignore
index f65f024838..bfb7f48f58 100644
--- a/.gitignore
+++ b/.gitignore
@@ -54,6 +54,7 @@ out/
 
 # Ansible
 ansible/environments/docker-machine/hosts
+ansible/environments/local/hosts
 ansible/db_local.ini*
 ansible/tmp/*
 ansible/roles/nginx/files/openwhisk-client*
diff --git a/ansible/README.md b/ansible/README.md
index 8bb9596608..95bbcf9b9b 100644
--- a/ansible/README.md
+++ b/ansible/README.md
@@ -73,20 +73,32 @@ In all instructions, replace `<openwhisk_home>` with the base directory of your
 
 #### Setup
 
-This step needs to be done only once per development environment. It will generate configuration files based on your local settings. Notice that for the following playbook you don't need to specify a target environment as it will run only local actions.
-After the playbook is done you should see a file called `db_local.ini` in your `ansible` directory. It will by default contain settings for a local ephemeral CouchDB setup. Afterwards, you can change the values directly in `db_local.ini`.
-
-#####  Ephemeral CouchDB
-
-If you want to use the ephemeral CouchDB, run this command
+The following step must be executed once per development environment.
+It will generate the `hosts` configuration file based on your environment settings.
 
 ```
 ansible-playbook -i environments/<environment> setup.yml
 ```
 
-#####  Persistent CouchDB
+The default configuration does not run multiple instances of core components (e.g., controller, invoker, kafka).
+You may elect to enable high-availability (HA) mode by passing tne ansible option `-e mode=HA` when executing this playbook.
+This will configure your deployment with multiple instances (e.g., two kafka instancess, and two invokers).
+
+In addition to the host file generation, you need to configure the database for your deployment. This is done
+by creating a file `ansible/db_local.ini` to provide the following properties.
+
+```bash
+[db_creds]
+db_provider=
+db_username=
+db_password=
+db_protocol=
+db_host=
+db_port=
+```
 
-If you want to use the persistent CouchDB instead, you can use env variables that are read by the playbook:
+This file is generated automatically if you are using an ephermeral CouchDB instance. Otherwise, you must create it explicitly.
+For convenience, you can use shell environment variables that are read by the playbook to generate the required `db_local.ini` file as shown below.
 
 ```
 export OW_DB=CouchDB
@@ -96,14 +108,10 @@ export OW_DB_PROTOCOL=<your couchdb protocol>
 export OW_DB_HOST=<your couchdb host>
 export OW_DB_PORT=<your couchdb port>
 
-ansible-playbook -i environments/<environment> setup.yml
+ansible-playbook -i environments/<environment> couchdb.yml --tags ini
 ```
 
-If you deploy CouchDB manually (i.e., without using the deploy CouchDB playbook), you must set the `reduce_limit` property on views to `false`. This may be done via the REST API, as in: `curl -X PUT ${OW_DB_PROTOCOL}://${OW_DB_HOST}:${OW_DB_PORT}/_config/query_server_config/reduce_limit -d '"false"' -u ${OW_DB_USERNAME}:${OW_DB_PASSWORD}`.
-
-##### Cloudant
-
-If you want to use Cloudant instead, you can use env variables that are read by the playbook:
+Alternatively, if you want to use Cloudant as your datastore:
 
 ```
 export OW_DB=Cloudant
@@ -113,7 +121,7 @@ export OW_DB_PROTOCOL=https
 export OW_DB_HOST=<your cloudant user>.cloudant.com
 export OW_DB_PORT=443
 
-ansible-playbook -i environments/<environment> setup.yml
+ansible-playbook -i environments/<environment> couchdb.yml --tags ini
 ```
 
 #### Install Prerequisites
@@ -129,6 +137,8 @@ ansible-playbook -i environments/<environment> prereq.yml
 
 ### Deploying Using CouchDB
 - Make sure your `db_local.ini` file is set up for CouchDB. See [Setup](#setup)
+- If you deploy CouchDB manually (i.e., without using the deploy CouchDB playbook), you must set the `reduce_limit` property on views to `false`.
+This may be done via the REST API, as in: `curl -X PUT ${OW_DB_PROTOCOL}://${OW_DB_HOST}:${OW_DB_PORT}/_config/query_server_config/reduce_limit -d '"false"' -u ${OW_DB_USERNAME}:${OW_DB_PASSWORD}`.
 - Then execute
 
 ```
@@ -143,7 +153,7 @@ ansible-playbook -i environments/<environment> openwhisk.yml
 ansible-playbook -i environments/<environment> postdeploy.yml
 ```
 
-You need to run `initdb.yml` on couchdb **every time** you do a fresh deploy CouchDB to initialize the subjects database.
+You need to run `initdb.yml` **every time** you do a fresh deploy CouchDB to initialize the subjects database.
 The playbooks `wipe.yml` and `postdeploy.yml` should be run on a fresh deployment only, otherwise all transient
 data that include actions and activations are lost.
 
diff --git a/ansible/couchdb.yml b/ansible/couchdb.yml
index 70faba99db..b75cd099cd 100644
--- a/ansible/couchdb.yml
+++ b/ansible/couchdb.yml
@@ -1,6 +1,18 @@
 ---
 # This playbook deploys a CouchDB for Openwhisk.  
 
+- hosts: localhost
+  tasks:
+  - name: check if db_local.ini exists?
+    tags: ini
+    stat: path="{{ playbook_dir }}/db_local.ini"
+    register: db
+
+  - name: prepare db_local.ini
+    tags: ini
+    local_action: template src="db_local.ini.j2" dest="{{ playbook_dir }}/db_local.ini"
+    when: not db.stat.exists
+
 - hosts: db
   roles:
   - couchdb
\ No newline at end of file
diff --git a/ansible/environments/docker-machine/group_vars/all b/ansible/environments/docker-machine/group_vars/all
index 391fa1e1e3..abffd2fe3c 100644
--- a/ansible/environments/docker-machine/group_vars/all
+++ b/ansible/environments/docker-machine/group_vars/all
@@ -30,3 +30,12 @@ controller_arguments: '-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxre
 invoker_arguments: "{{ controller_arguments }}"
 
 invoker_allow_multiple_instances: true
+
+# Set kafka configuration
+kafka_heap: '512m'
+kafka_topics_completed_retentionBytes: 104857600
+kafka_topics_completed_retentionMS: 300000
+kafka_topics_health_retentionBytes: 104857600
+kafka_topics_health_retentionMS: 300000
+kafka_topics_invoker_retentionBytes: 104857600
+kafka_topics_invoker_retentionMS: 300000
diff --git a/ansible/environments/docker-machine/hosts.j2.ini b/ansible/environments/docker-machine/hosts.j2.ini
index a4990ae359..1a49698ced 100644
--- a/ansible/environments/docker-machine/hosts.j2.ini
+++ b/ansible/environments/docker-machine/hosts.j2.ini
@@ -8,17 +8,24 @@ ansible ansible_connection=local
 
 [controllers]
 controller0                 ansible_host={{ docker_machine_ip }}
-controller1                 ansible_host={{ docker_machine_ip }}
+;{% if mode is defined and 'HA' in mode %}
+;controller1                 ansible_host={{ docker_machine_ip }}
+;{% endif %}
 
 [kafkas]
-{{ docker_machine_ip }}     ansible_host={{ docker_machine_ip }}
+kafka0                      ansible_host={{ docker_machine_ip }}
+{% if mode is defined and 'HA' in mode %}
+kafka1                      ansible_host={{ docker_machine_ip }}
+{% endif %}
 
 [zookeepers:children]
 kafkas
 
 [invokers]
 invoker0                    ansible_host={{ docker_machine_ip }}
+{% if mode is defined and 'HA' in mode %}
 invoker1                    ansible_host={{ docker_machine_ip }}
+{% endif %}
 
 [db]
 {{ docker_machine_ip }}     ansible_host={{ docker_machine_ip }}
diff --git a/ansible/environments/local/hosts b/ansible/environments/local/hosts.j2.ini
similarity index 79%
rename from ansible/environments/local/hosts
rename to ansible/environments/local/hosts.j2.ini
index 4cf54aa811..a687adf413 100644
--- a/ansible/environments/local/hosts
+++ b/ansible/environments/local/hosts.j2.ini
@@ -8,17 +8,24 @@ ansible ansible_connection=local
 
 [controllers]
 controller0         ansible_host=172.17.0.1 ansible_connection=local
+;{% if mode is defined and 'HA' in mode %}
+;controller1         ansible_host=172.17.0.1 ansible_connection=local
+;{% endif %}
 
 [kafkas]
 kafka0              ansible_host=172.17.0.1 ansible_connection=local
+{% if mode is defined and 'HA' in mode %}
 kafka1              ansible_host=172.17.0.1 ansible_connection=local
+{% endif %}
 
 [zookeepers:children]
 kafkas
 
 [invokers]
 invoker0            ansible_host=172.17.0.1 ansible_connection=local
+{% if mode is defined and 'HA' in mode %}
 invoker1            ansible_host=172.17.0.1 ansible_connection=local
+{% endif %}
 
 ; db group is only used if db_provider is CouchDB
 [db]
diff --git a/ansible/setup.yml b/ansible/setup.yml
index 28d02c50bc..86904920e9 100644
--- a/ansible/setup.yml
+++ b/ansible/setup.yml
@@ -3,6 +3,10 @@
 
 - hosts: localhost
   tasks:
+  - name: gen hosts if 'local' env is used
+    local_action: template src="{{playbook_dir}}/environments/local/hosts.j2.ini" dest="{{ playbook_dir }}/environments/local/hosts"
+    when: "'environments/local' in inventory_dir"
+
   - name: find the ip of docker-machine
     local_action: shell "docker-machine" "ip" "{{docker_machine_name | default('whisk')}}"
     register: result
@@ -17,19 +21,6 @@
     local_action: template src="{{playbook_dir}}/environments/docker-machine/hosts.j2.ini" dest="{{ playbook_dir }}/environments/docker-machine/hosts"
     when: "'environments/docker-machine' in inventory_dir"
 
-  # this step is needed to generate db_local.ini which is required by later steps
-  - name: add new db host on-the-fly
-    add_host: name={{ docker_machine_ip }} groups=db
-    when: "'environments/docker-machine' in inventory_dir"
-
-  - name: check if db_local.ini exists?
-    stat: path="{{ playbook_dir }}/db_local.ini"
-    register: db
-
-  - name: prepare db_local.ini
-    local_action: template src="db_local.ini.j2" dest="{{ playbook_dir }}/db_local.ini"
-    when: not db.stat.exists
-
   - name: gen untrusted server certificate for host
     local_action: shell "{{ playbook_dir }}/roles/nginx/files/genssl.sh" "*.{{ whisk_api_localhost_name | default(whisk_api_host_name) | default(whisk_api_localhost_name_default) }}" "server"
     when: nginx.ssl.cert == "openwhisk-server-cert.pem"
diff --git a/tools/build/redo b/tools/build/redo
index 8b877b14aa..b152c24e34 100755
--- a/tools/build/redo
+++ b/tools/build/redo
@@ -75,7 +75,12 @@ def getArgs():
         if osname == 'Linux':
             return 'local'
         elif osname == 'Darwin':
-            return 'docker-machine'
+            if os.getenv('DOCKER_HOST', None) is not None:
+                # docker-machine typically has docker host set in the environment
+                return 'docker-machine'
+            else:
+                # otherwise assume docker-for-mac
+                return 'local'
         else:
             return None
 
@@ -83,7 +88,7 @@ def getArgs():
     parser.add_argument('-b', '--build', help='build component', action='store_const', const=True, default=False)
     parser.add_argument('-x', '--teardown', help='teardown component', action='store_const', const=True, default=False)
     parser.add_argument('-d', '--deploy', help='deploy component', action='store_const', const=True, default=False)
-    parser.add_argument('-t', '--target', help='deploy target (one of [mac, docker-machine, local])', default=detectDeployTarget())
+    parser.add_argument('-t', '--target', help='deploy target (one of [docker-machine, local])', default=detectDeployTarget())
     parser.add_argument('-y', '--yaml', help='deploy target using inferred YAML file if component is not one of known targets', action='store_const', const=True, default=False)
     parser.add_argument('-g', '--gradle', help='use target using inferred gradle file if component is not one of known targets', action='store_const', const=True, default=False)
     parser.add_argument('-n', '--just-print', help='prints the component configuration but does not run any targets', action='store_const', const=True, default=False, dest='skiprun')
diff --git a/tools/travis/build.sh b/tools/travis/build.sh
index 043c1cc4d6..4870acb72a 100755
--- a/tools/travis/build.sh
+++ b/tools/travis/build.sh
@@ -23,7 +23,7 @@ cd $ROOTDIR/ansible
 ANSIBLE_CMD="ansible-playbook -i environments/local -e docker_image_prefix=testing"
 GRADLE_PROJS_SKIP="-x :actionRuntimes:pythonAction:distDocker  -x :actionRuntimes:python2Action:distDocker -x actionRuntimes:swift3.1.1Action:distDocker -x :actionRuntimes:javaAction:distDocker"
 
-$ANSIBLE_CMD setup.yml
+$ANSIBLE_CMD setup.yml -e mode=HA
 $ANSIBLE_CMD prereq.yml
 $ANSIBLE_CMD couchdb.yml
 $ANSIBLE_CMD initdb.yml


 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services