You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@senssoft.apache.org by le...@apache.org on 2018/03/12 15:58:45 UTC
[31/40] incubator-senssoft git commit: Pushed huge update on SensSoft
docker deployments. Upgraded to docker-compose version 3.
Pushed huge update on SensSoft docker deployments. Upgraded to docker-compose version 3.
Project: http://git-wip-us.apache.org/repos/asf/incubator-senssoft/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-senssoft/commit/c3aa251d
Tree: http://git-wip-us.apache.org/repos/asf/incubator-senssoft/tree/c3aa251d
Diff: http://git-wip-us.apache.org/repos/asf/incubator-senssoft/diff/c3aa251d
Branch: refs/heads/asf-site
Commit: c3aa251dfbee2753d0750de109fdf2284b1c8470
Parents: 87d7e09
Author: msbeard <ms...@apache.org>
Authored: Wed Nov 1 17:47:27 2017 -0400
Committer: msbeard <ms...@apache.org>
Committed: Wed Nov 1 17:47:27 2017 -0400
----------------------------------------------------------------------
docker/README.md | 155 ++++++++++++++++----
docker/docker-compose.yml | 157 +++++++++++----------
docker/elasticsearch/Dockerfile | 27 ++++
docker/elasticsearch/config/elasticsearch.yml | 48 +++++++
docker/elasticsearch/config/log4j2.properties | 24 ++++
docker/es/elasticsearch.yml | 114 ---------------
docker/images/configure_index.png | Bin 0 -> 116678 bytes
docker/images/confirmation.png | Bin 0 -> 158876 bytes
docker/images/dashboard.png | Bin 0 -> 172635 bytes
docker/images/management.png | Bin 0 -> 60954 bytes
docker/kibana/Dockerfile | 23 +++
docker/kibana/config/kibana.yml | 18 +++
docker/kibana/dashboard.json | 21 +--
docker/kibana/visualizations.json | 58 ++++++++
docker/logstash/Dockerfile | 9 +-
docker/logstash/config/logstash-userale.conf | 2 +-
docker/requirements.txt | 1 +
docker/startup/entrypoint.sh | 3 +-
18 files changed, 426 insertions(+), 234 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-senssoft/blob/c3aa251d/docker/README.md
----------------------------------------------------------------------
diff --git a/docker/README.md b/docker/README.md
index 2669bdb..f9d395a 100644
--- a/docker/README.md
+++ b/docker/README.md
@@ -1,35 +1,140 @@
-How to Build Docker Containers
-------------------------------
+How to Build SensSoft Docker Containers
+---------------------------------------
1. Install [``Docker``](http://docker.com) on your machine.
-2. Install ``docker-compose`` in an virtual environment. Full instructions can be found [``here``](https://docs.docker.com/compose/install/).
- ```
- $ python3 -m venv env
- $ source env/bin/activate
- $ pip install docker-compose
- ```
-3. Before launching the Docker containers, ensure your ``vm_max_map_count`` kernel setting is set to at least 262144.
- Visit [``Running Elasticsearch in Production mode``](https://www.elastic.co/guide/en/elasticsearch/reference/5.5/docker.html#docker-cli-run-prod-mode) for OS specific instructions.
+
+1. Install ``docker-compose`` in an virtual environment.
+ Full instructions can be found [``here``](https://docs.docker.com/compose/install/).
+
+ ```bash
+ $ python3 -m venv env
+ $ source env/bin/activate
+ $ pip install -r requirements.txt
```
- # For Linux systems
+
+1. Before launching the Docker containers, ensure your ``vm_max_map_count``
+ kernel setting is set to at least 262144.
+ Visit [``Running Elasticsearch in Production mode``](https://www.elastic.co/guide/en/elasticsearch/reference/5.5/docker.html#docker-cli-run-prod-mode) for OS specific instructions.
+
+ ```bash
+ # Example for Linux systems
$ sysctl -w vm.max_map_count=262144
```
-4. To build and run all ``Docker`` containers.
- ```
- $ docker-compose up -d
- ```
-5. To run a specific ``Docker`` container.
- ```
- $ docker-compose up -d site
- # Note: site container is instrumented w/ userale; all userale logs will be sent to the
- # elasticsearch docker container.
- ```
-6. Verify the deployment by navigating to ``Kibana`` in your favorite browser.
- ```sh
- http://localhost:5601
+
+1. Start Elasticsearch cluster:
+
+ ```bash
+ $ docker-compose up -d elasticsearch loadbalancer
```
+
+ The loadbalancer node exposes port 9200 on localhost and is the only node
+ that has HTTP enabled. Services such as Kibana and Logstash connect to the
+ loadbalancer node directly. Loadbalancer accepts requests from Kibana and Logstash
+ and balances them across the elasticsearch worker nodes. The elasticsearch
+ worker nodes communicate to each other and the loadbalancer via TCP on port 9300.
+
+1. Once Elasticsearch cluster is fully started, scale out elasticsearch service:
+ ```bash
+ $ docker-compose up --scale elasticsearch=3
+ ```
+
+1. Confirm cluster state:
+ ```bash
+ $ curl -XGET http://localhost:9200/_cluster/health?pretty
+ {
+ "cluster_name" : "SensSoft",
+ "status" : "green",
+ "timed_out" : false,
+ "number_of_nodes" : 4,
+ "number_of_data_nodes" : 3,
+ "active_primary_shards" : 0,
+ "active_shards" : 0,
+ "relocating_shards" : 0,
+ "initializing_shards" : 0,
+ "unassigned_shards" : 0,
+ "delayed_unassigned_shards" : 0,
+ "number_of_pending_tasks" : 0,
+ "number_of_in_flight_fetch" : 0,
+ "task_max_waiting_in_queue_millis" : 0,
+ "active_shards_percent_as_number" : 100.0
+ }
+ ```
+ Confirm that the `number_of_nodes` is 4 and `number_of_data_nodes` is 3.
+
+1. Launch logging server. Give Logstash about 2 minutes to start before confirming
+ its state.
+
+ ```bash
+ $ docker-compose up -d logstash
+
+ $ curl -XGET http://localhost:8100
+ ok
+ ```
+
+1. Before Kibana can be used, we will need to generate some data. We have already
+ provided an example instrumented website to assist.
+
+ ```bash
+ $ docker-compose up -d site
+ ```
+
+ Visit `http://localhost:8080` and you will see Apache SensSoft's home page.
+
+1. Launch Kibana. Give Kibana about 2-5 minutes to start before accessing
+ `http://localhost:5601`.
+
+ ```bash
+ $ docker-compose up -d kibana
+ ```
+
+1. Register an index in Kibana to see the logs:
+
+ Goto: Management -> Index Patterns and enter `userale` in the Index pattern box.
+ Choose `clientTime` in the drop down `Time Filter field name` field.
+
+ ![alt text][configure_index]
+
+1. Load example Dashboard and Visualizations under docker/kibana/.
+
+ Goto: Management -> Saved Objects and select the `Import` button. Import the
+ `visualizations.json` and `dashboard.json` file.
+
+ ![alt text][management]
-7. Stop all the containers.
+ Confirm index conflicts if message appears.
+
+ ![alt text][confirmation]
+
+ Once that is complete, navigate to the `Dashboard` view in Kibana and click the
+ `Apache SensSoft Dashboard` object.
+
+ ![alt text][dashboard]
+
+1. To stop all containers.
```sh
$ docker-compose stop
```
+
+Having Issues?
+--------------
+1. Check out the docker-compose logs for the service(s) that are having issues.
+
+ ```bash
+ $ docker-compose ps
+ ```
+1. Make sure to send us the docker-compose logs to help diagnose your issues please!
+
+ ```bash
+ $ docker-compose logs > err.dump
+ ```
+
+Todo
+----
+- [ ] TAP docker deployment instructions.
+- [ ] Distill docker deployment instructions.
+- [ ] Apache SensSoft Docker + Kubernetes instructions.
+
+[configure_index]: ./images/configure_index.png "Configure Kibana index"
+[confirmation]: ./images/confirmation.png "Confirm index pattern conflicts"
+[dashboard]: ./images/dashboard.png "Apache Senssoft Dashboard"
+[management]: ./images/management.png "Kibana management console"
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-senssoft/blob/c3aa251d/docker/docker-compose.yml
----------------------------------------------------------------------
diff --git a/docker/docker-compose.yml b/docker/docker-compose.yml
index fc14c1c..17b1106 100644
--- a/docker/docker-compose.yml
+++ b/docker/docker-compose.yml
@@ -15,131 +15,140 @@
# Apache SensSoft Docker Build
-version: "2"
+version: "3"
services:
# Startup
startup:
build: ./startup
+ image: senssoft.incubator.apache.org/startup:latest
container_name: senssoft-start
depends_on:
- - "elasticsearch1"
- - "elasticsearch2"
+ - "loadbalancer"
command: bash /opt/entrypoint.sh
# Site
site:
build: ./site
+ image: senssoft.incubator.apache.org/website:latest
container_name: senssoft-app
ports:
- 8080:8080
depends_on:
- - "userale-forwarder"
+ - "logstash"
command: python -m SimpleHTTPServer 8080
- # networks:
- # - esnet
- # ELK Stack
-
- # The node elasticsearch1 listens on localhost:9200 while elasticsearch2
- # talks to elasticsearch1 over a Docker network.
- elasticsearch1:
- image: docker.elastic.co/elasticsearch/elasticsearch:5.5.0
- container_name: senssoft-elastic-1
+ # Elasticsearch Workers
+ elasticsearch:
+ build: ./elasticsearch
+ image: senssoft.incubator.apache.org/elasticsearch:5.6.3
+ command:
+ - /bin/sh
+ - -c
+ - |
+ elasticsearch-plugin remove --silent x-pack
+ elasticsearch
environment:
- - cluster.name=SensSoft
- - node.name=soft-01
- - bootstrap.memory_lock=true
- - xpack.security.enabled=false
+ - NODE_MASTER=true
+ - NUMBER_OF_MASTERS=2
+ - NODE_COORDINATE=true
+ - "DISCOVERY_SERVICE=elasticsearch"
+ - NODE_DATA=true
+ - NODE_INGEST=true
+ - MAX_LOCAL_STORAGE_NODES=1
+ - NETWORK_HOST=0.0.0.0
+ - HTTP_ENABLE=false
+ - HTTP_CORS_ENABLE=false
+ - HTTP_CORS_ALLOW_ORIGIN=*
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
- ulimits:
- memlock:
- soft: -1
- hard: -1
- mem_limit: 1g
- volumes:
- - esdata1:/usr/share/elasticsearch/data
- ports:
- - 9200:9200
- # networks:
- # - esnet
+ networks:
+ - esnet
- elasticsearch2:
- image: docker.elastic.co/elasticsearch/elasticsearch:5.5.0
- container_name: senssoft-elastic-2
+ # Elasticsearch Loadbalancer
+ loadbalancer:
+ build: ./elasticsearch
+ image: senssoft.incubator.apache.org/elasticsearch:5.6.3
+ container_name: senssoft-loadbalancer
+ command:
+ - /bin/sh
+ - -c
+ - |
+ elasticsearch-plugin remove --silent x-pack
+ elasticsearch
environment:
- - cluster.name=SensSoft
- - node.name=soft-02
- - bootstrap.memory_lock=true
- - xpack.security.enabled=false
+ - NODE_MASTER=false
+ - NUMBER_OF_MASTERS=1
+ - NODE_COORDINATE=true
+ - "DISCOVERY_SERVICE=elasticsearch"
+ - NODE_DATA=false
+ - NODE_INGEST=false
+ - MAX_LOCAL_STORAGE_NODES=1
+ - NETWORK_HOST=0.0.0.0
+ - HTTP_ENABLE=true
+ - HTTP_CORS_ENABLE=true
+ - HTTP_CORS_ALLOW_ORIGIN=*
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
- - "discovery.zen.ping.unicast.hosts=elasticsearch1"
- ulimits:
- memlock:
- soft: -1
- hard: -1
- mem_limit: 1g
- volumes:
- - esdata2:/usr/share/elasticsearch/data
- # networks:
- # - esnet
+ ports:
+ - "9200:9200"
+ - "9300"
+ networks:
+ - esnet
# Kibana
kibana:
- image: docker.elastic.co/kibana/kibana:5.5.0
+ build: ./kibana
+ image: senssoft.incubator.apache.org/kibana:5.6.3
container_name: senssoft-kibana
+ environment:
+ - "ES_JAVA_OPTS=-Xms2g -Xmx2g"
ports:
- - 5601:5601
+ - "5601:5601"
depends_on:
- - "elasticsearch1"
- - "elasticsearch2"
- - "startup"
- environment:
- XPACK_MONITORING_ENABLED: "false"
- ELASTICSEARCH_URL: http://elasticsearch1:9200
- # networks:
- # - esnet
+ - "loadbalancer"
+ networks:
+ - esnet
# Logstash
- userale-forwarder:
+ logstash:
build: ./logstash
- container_name: senssoft-userale-forwarder
+ image: senssoft.incubator.apache.org/logstash:5.6.3
+ container_name: senssoft-logstash
command: -f /etc/logstash/conf.d
ports:
- 8100:8100
- volumes:
+ volumes:
- ./logstash/config/logstash-userale.conf:/etc/logstash/conf.d/logstash-userale.conf
- ./logstash/templates/userale.json:/usr/share/logstash/templates/userale.json
depends_on:
- - "elasticsearch1"
- - "elasticsearch2"
- environment:
- XPACK_MONITORING_ENABLED: "false"
- # networks:
- # - esnet
+ - "loadbalancer"
+ networks:
+ - esnet
# Distill
distill:
build: ./distill
+ image: senssoft.incubator.apache.org/distill:0.1.0
container_name: senssoft-distill
ports:
- 8090:8090
depends_on:
- - "elasticsearch1"
- # networks:
- # - esnet
+ - "loadbalancer"
+ networks:
+ - esnet
# TAP's Database
db:
- container_name: senssoft-db
build: ./db
+ image: senssoft.incubator.apache.org/postgres:latest
+ container_name: senssoft-postgres
ports:
- "5432:5432"
# TAP
tap:
- container_name: senssoft-tap
build: ./tap
+ image: senssoft.incubator.apache.org/tap:0.1.0
+ container_name: senssoft-tap
# command: ["bash", "./wait-for-postgres.sh"]
command: bash -c "/usr/src/app/wait-for-postgres.sh"
ports:
@@ -149,11 +158,7 @@ services:
links:
- db:db
-volumes:
- esdata1:
- driver: local
- esdata2:
- driver: local
+networks:
-# networks:
-# esnet:
+ esnet:
+ driver: bridge
http://git-wip-us.apache.org/repos/asf/incubator-senssoft/blob/c3aa251d/docker/elasticsearch/Dockerfile
----------------------------------------------------------------------
diff --git a/docker/elasticsearch/Dockerfile b/docker/elasticsearch/Dockerfile
new file mode 100644
index 0000000..7cdbae7
--- /dev/null
+++ b/docker/elasticsearch/Dockerfile
@@ -0,0 +1,27 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+FROM docker.elastic.co/elasticsearch/elasticsearch:5.6.3
+
+MAINTAINER Michelle Beard <ms...@apache.org>
+
+# Override config
+COPY config /usr/share/elasticsearch/config
+
+# Set environment
+ENV CLUSTER_NAME SensSoft
+
+# Kubernetes requires swap is turned off
+ENV MEMORY_LOCK false
http://git-wip-us.apache.org/repos/asf/incubator-senssoft/blob/c3aa251d/docker/elasticsearch/config/elasticsearch.yml
----------------------------------------------------------------------
diff --git a/docker/elasticsearch/config/elasticsearch.yml b/docker/elasticsearch/config/elasticsearch.yml
new file mode 100644
index 0000000..ca31f63
--- /dev/null
+++ b/docker/elasticsearch/config/elasticsearch.yml
@@ -0,0 +1,48 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+cluster:
+ name: ${CLUSTER_NAME}
+
+node:
+ master: ${NODE_MASTER}
+ data: ${NODE_DATA}
+ #name: ${NODE_NAME}
+ ingest: ${NODE_INGEST}
+ max_local_storage_nodes: ${MAX_LOCAL_STORAGE_NODES}
+
+network.host: ${NETWORK_HOST}
+
+search.remote.connect: ${NODE_COORDINATE}
+
+path:
+ data: /usr/share/elasticsearch/data
+ logs: /usr/share/elasticsearch/log
+
+bootstrap:
+ memory_lock: ${MEMORY_LOCK}
+
+http:
+ enabled: ${HTTP_ENABLE}
+ compression: true
+ cors:
+ enabled: ${HTTP_CORS_ENABLE}
+ allow-origin: ${HTTP_CORS_ALLOW_ORIGIN}
+
+# Prevent the "split brain" by configuring the majority of nodes (total number of nodes / 2 + 1):
+discovery:
+ zen:
+ ping.unicast.hosts: ${DISCOVERY_SERVICE}
+ minimum_master_nodes: ${NUMBER_OF_MASTERS}
http://git-wip-us.apache.org/repos/asf/incubator-senssoft/blob/c3aa251d/docker/elasticsearch/config/log4j2.properties
----------------------------------------------------------------------
diff --git a/docker/elasticsearch/config/log4j2.properties b/docker/elasticsearch/config/log4j2.properties
new file mode 100644
index 0000000..7f7817d
--- /dev/null
+++ b/docker/elasticsearch/config/log4j2.properties
@@ -0,0 +1,24 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+status = error
+
+appender.console.type = Console
+appender.console.name = console
+appender.console.layout.type = PatternLayout
+appender.console.layout.pattern = [%d{ISO8601}][%-5p][%-25c{1.}] %marker%m%n
+
+rootLogger.level = info
+rootLogger.appenderRef.console.ref = console
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-senssoft/blob/c3aa251d/docker/es/elasticsearch.yml
----------------------------------------------------------------------
diff --git a/docker/es/elasticsearch.yml b/docker/es/elasticsearch.yml
deleted file mode 100644
index 5e0c014..0000000
--- a/docker/es/elasticsearch.yml
+++ /dev/null
@@ -1,114 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-# ======================== Elasticsearch Configuration =========================
-#
-# NOTE: Elasticsearch comes with reasonable defaults for most settings.
-# Before you set out to tweak and tune the configuration, make sure you
-# understand what are you trying to accomplish and the consequences.
-#
-# The primary way of configuring a node is via this file. This template lists
-# the most important settings you may want to configure for a production cluster.
-#
-# Please see the documentation for further information on configuration options:
-# <http://www.elastic.co/guide/en/elasticsearch/reference/current/setup-configuration.html>
-#
-# ---------------------------------- Cluster -----------------------------------
-#
-# Use a descriptive name for your cluster:
-#
-cluster.name: SensSoft
-#
-# ------------------------------------ Node ------------------------------------
-#
-# Use a descriptive name for the node:
-#
-node.name: soft-01
-#
-# Add custom attributes to the node:
-#
-# node.rack: r1
-#
-# ----------------------------------- Paths ------------------------------------
-#
-# Path to directory where to store the data (separate multiple locations by comma):
-#
-path.data: /usr/share/elasticsearch/data
-#
-# Path to log files:
-#
-path.logs: /usr/share/elasticsearch/logs
-#
-#
-# ----------------------------------- Memory -----------------------------------
-#
-# Lock the memory on startup:
-#
-# bootstrap.mlockall: true
-#
-# Make sure that the `ES_HEAP_SIZE` environment variable is set to about half the memory
-# available on the system and that the owner of the process is allowed to use this limit.
-#
-# Elasticsearch performs poorly when the system is swapping the memory.
-#
-# ---------------------------------- Network -----------------------------------
-#
-# Set the bind address to a specific IP (IPv4 or IPv6):
-#
-network.host: 0.0.0.0
-#
-# Set a custom port for HTTP:
-#
-# http.port: 9200
-#
-# For more information, see the documentation at:
-# <http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-network.html>
-#
-# --------------------------------- Discovery ----------------------------------
-#
-# Pass an initial list of hosts to perform discovery when new node is started:
-# The default list of hosts is ["127.0.0.1", "[::1]"]
-#
-# discovery.zen.ping.unicast.hosts: ["host1", "host2"]
-#
-# Prevent the "split brain" by configuring the majority of nodes (total number of nodes / 2 + 1):
-#
-discovery.zen.minimum_master_nodes: 1
-#
-# For more information, see the documentation at:
-# <http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-discovery.html>
-#
-# ---------------------------------- Gateway -----------------------------------
-#
-# Block initial recovery after a full cluster restart until N nodes are started:
-#
-# gateway.recover_after_nodes: 3
-#
-# For more information, see the documentation at:
-# <http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-gateway.html>
-#
-# ---------------------------------- Various -----------------------------------
-#
-# Disable starting multiple nodes on a single system:
-#
-# node.max_local_storage_nodes: 1
-#
-# Require explicit names when deleting indices:
-#
-# action.destructive_requires_name: true
-
-http.cors.enabled: true
-http.cors.allow-origin: "*"
-xpack.security.enabled: false
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-senssoft/blob/c3aa251d/docker/images/configure_index.png
----------------------------------------------------------------------
diff --git a/docker/images/configure_index.png b/docker/images/configure_index.png
new file mode 100644
index 0000000..31ebe02
Binary files /dev/null and b/docker/images/configure_index.png differ
http://git-wip-us.apache.org/repos/asf/incubator-senssoft/blob/c3aa251d/docker/images/confirmation.png
----------------------------------------------------------------------
diff --git a/docker/images/confirmation.png b/docker/images/confirmation.png
new file mode 100644
index 0000000..467b4d4
Binary files /dev/null and b/docker/images/confirmation.png differ
http://git-wip-us.apache.org/repos/asf/incubator-senssoft/blob/c3aa251d/docker/images/dashboard.png
----------------------------------------------------------------------
diff --git a/docker/images/dashboard.png b/docker/images/dashboard.png
new file mode 100644
index 0000000..7123c9e
Binary files /dev/null and b/docker/images/dashboard.png differ
http://git-wip-us.apache.org/repos/asf/incubator-senssoft/blob/c3aa251d/docker/images/management.png
----------------------------------------------------------------------
diff --git a/docker/images/management.png b/docker/images/management.png
new file mode 100644
index 0000000..90a26ae
Binary files /dev/null and b/docker/images/management.png differ
http://git-wip-us.apache.org/repos/asf/incubator-senssoft/blob/c3aa251d/docker/kibana/Dockerfile
----------------------------------------------------------------------
diff --git a/docker/kibana/Dockerfile b/docker/kibana/Dockerfile
new file mode 100644
index 0000000..894fd53
--- /dev/null
+++ b/docker/kibana/Dockerfile
@@ -0,0 +1,23 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+FROM docker.elastic.co/kibana/kibana:5.6.3
+
+MAINTAINER Michelle Beard <ms...@apache.org>
+
+# Override config
+COPY config /usr/share/kibana/config
+
+RUN kibana-plugin remove x-pack
http://git-wip-us.apache.org/repos/asf/incubator-senssoft/blob/c3aa251d/docker/kibana/config/kibana.yml
----------------------------------------------------------------------
diff --git a/docker/kibana/config/kibana.yml b/docker/kibana/config/kibana.yml
new file mode 100644
index 0000000..20f2999
--- /dev/null
+++ b/docker/kibana/config/kibana.yml
@@ -0,0 +1,18 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+server.name: kibana
+server.host: "0"
+elasticsearch.url: http://loadbalancer:9200
http://git-wip-us.apache.org/repos/asf/incubator-senssoft/blob/c3aa251d/docker/kibana/dashboard.json
----------------------------------------------------------------------
diff --git a/docker/kibana/dashboard.json b/docker/kibana/dashboard.json
index d1922d9..8a179a6 100644
--- a/docker/kibana/dashboard.json
+++ b/docker/kibana/dashboard.json
@@ -1,25 +1,18 @@
[
{
- "_id": "6e6b6380-3673-11e7-b2f7-0b10e220d9ac",
+ "_id": "513c8680-be7d-11e7-8adf-fd0a12531eca",
"_type": "dashboard",
"_source": {
- "title": "Apache UserALE Monitor",
+ "title": "Apache SensSoft Dashboard",
"hits": 0,
- "description": "Example Kibana Dashboard",
- "panelsJSON": "[\n {\n \"size_x\": 9,\n \"size_y\": 4,\n \"panelIndex\": 1,\n \"type\": \"visualization\",\n \"id\": \"49fa7860-3673-11e7-b2f7-0b10e220d9ac\",\n \"col\": 4,\n \"row\": 4\n },\n {\n \"size_x\": 3,\n \"size_y\": 3,\n \"panelIndex\": 2,\n \"type\": \"visualization\",\n \"id\": \"6e55ca90-3671-11e7-b2f7-0b10e220d9ac\",\n \"col\": 1,\n \"row\": 4\n },\n {\n \"size_x\": 5,\n \"size_y\": 3,\n \"panelIndex\": 3,\n \"type\": \"visualization\",\n \"id\": \"ef62c4e0-3684-11e7-9c77-0135be018b90\",\n \"col\": 1,\n \"row\": 1\n },\n {\n \"size_x\": 7,\n \"size_y\": 3,\n \"panelIndex\": 4,\n \"type\": \"visualization\",\n \"id\": \"7ee8abd0-3684-11e7-9c77-0135be018b90\",\n \"col\": 6,\n \"row\": 1\n }\n]",
- "optionsJSON": "{\n \"darkTheme\": false\n}",
+ "description": "",
+ "panelsJSON": "[{\"size_x\":6,\"size_y\":5,\"panelIndex\":1,\"type\":\"visualization\",\"id\":\"2bf40010-be7d-11e7-8adf-fd0a12531eca\",\"col\":1,\"row\":4},{\"size_x\":6,\"size_y\":3,\"panelIndex\":2,\"type\":\"visualization\",\"id\":\"f7ca4560-be7c-11e7-8adf-fd0a12531eca\",\"col\":7,\"row\":1},{\"size_x\":6,\"size_y\":3,\"panelIndex\":3,\"type\":\"visualization\",\"id\":\"bcd7c950-be7c-11e7-8adf-fd0a12531eca\",\"col\":7,\"row\":4},{\"size_x\":6,\"size_y\":3,\"panelIndex\":4,\"type\":\"visualization\",\"id\":\"e36b1f40-be7c-11e7-8adf-fd0a12531eca\",\"col\":1,\"row\":1}]",
+ "optionsJSON": "{\"darkTheme\":true}",
"uiStateJSON": "{}",
"version": 1,
- "timeRestore": true,
- "timeTo": "now",
- "timeFrom": "now-15m",
- "refreshInterval": {
- "display": "Off",
- "pause": false,
- "value": 0
- },
+ "timeRestore": false,
"kibanaSavedObjectMeta": {
- "searchSourceJSON": "{\n \"filter\": [\n {\n \"query\": {\n \"query_string\": {\n \"query\": \"*\",\n \"analyze_wildcard\": true\n }\n }\n }\n ],\n \"highlightAll\": true,\n \"version\": true\n}"
+ "searchSourceJSON": "{\"filter\":[{\"query\":{\"query_string\":{\"query\":\"*\",\"analyze_wildcard\":true}}}],\"highlightAll\":true,\"version\":true}"
}
}
}
http://git-wip-us.apache.org/repos/asf/incubator-senssoft/blob/c3aa251d/docker/kibana/visualizations.json
----------------------------------------------------------------------
diff --git a/docker/kibana/visualizations.json b/docker/kibana/visualizations.json
new file mode 100644
index 0000000..7052384
--- /dev/null
+++ b/docker/kibana/visualizations.json
@@ -0,0 +1,58 @@
+[
+ {
+ "_id": "bcd7c950-be7c-11e7-8adf-fd0a12531eca",
+ "_type": "visualization",
+ "_source": {
+ "title": "Top Event Types",
+ "visState": "{\"title\":\"Top Event Types\",\"type\":\"pie\",\"params\":{\"addTooltip\":true,\"addLegend\":true,\"legendPosition\":\"right\",\"isDonut\":false},\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"count\",\"schema\":\"metric\",\"params\":{\"customLabel\":\"Top Event Types\"}},{\"id\":\"2\",\"enabled\":true,\"type\":\"terms\",\"schema\":\"segment\",\"params\":{\"field\":\"type\",\"size\":100,\"order\":\"desc\",\"orderBy\":\"1\",\"customLabel\":\"Event types\"}}],\"listeners\":{}}",
+ "uiStateJSON": "{}",
+ "description": "",
+ "version": 1,
+ "kibanaSavedObjectMeta": {
+ "searchSourceJSON": "{\"index\":\"userale\",\"query\":{\"query_string\":{\"query\":\"*\",\"analyze_wildcard\":true}},\"filter\":[]}"
+ }
+ }
+ },
+ {
+ "_id": "e36b1f40-be7c-11e7-8adf-fd0a12531eca",
+ "_type": "visualization",
+ "_source": {
+ "title": "Total Number of Logs",
+ "visState": "{\"title\":\"Total Number of Logs\",\"type\":\"metric\",\"params\":{\"handleNoResults\":true,\"fontSize\":60},\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"count\",\"schema\":\"metric\",\"params\":{\"customLabel\":\"Total number of logs\"}}],\"listeners\":{}}",
+ "uiStateJSON": "{}",
+ "description": "",
+ "version": 1,
+ "kibanaSavedObjectMeta": {
+ "searchSourceJSON": "{\"index\":\"userale\",\"query\":{\"query_string\":{\"query\":\"*\",\"analyze_wildcard\":true}},\"filter\":[]}"
+ }
+ }
+ },
+ {
+ "_id": "f7ca4560-be7c-11e7-8adf-fd0a12531eca",
+ "_type": "visualization",
+ "_source": {
+ "title": "Time Series",
+ "visState": "{\"title\":\"Time Series\",\"type\":\"timelion\",\"params\":{\"expression\":\".es(*)\",\"interval\":\"auto\"},\"aggs\":[],\"listeners\":{}}",
+ "uiStateJSON": "{}",
+ "description": "",
+ "version": 1,
+ "kibanaSavedObjectMeta": {
+ "searchSourceJSON": "{\"query\":{\"query_string\":{\"query\":\"*\"}},\"filter\":[]}"
+ }
+ }
+ },
+ {
+ "_id": "2bf40010-be7d-11e7-8adf-fd0a12531eca",
+ "_type": "visualization",
+ "_source": {
+ "title": "Events by Targets",
+ "visState": "{\"title\":\"Events by Targets\",\"type\":\"histogram\",\"params\":{\"addLegend\":true,\"addTimeMarker\":false,\"addTooltip\":true,\"categoryAxes\":[{\"id\":\"CategoryAxis-1\",\"labels\":{\"show\":true,\"truncate\":100},\"position\":\"bottom\",\"scale\":{\"type\":\"linear\"},\"show\":true,\"style\":{},\"title\":{\"text\":\"HTML Tags\"},\"type\":\"category\"}],\"defaultYExtents\":false,\"drawLinesBetweenPoints\":true,\"grid\":{\"categoryLines\":false,\"style\":{\"color\":\"#eee\"}},\"interpolate\":\"linear\",\"legendPosition\":\"right\",\"radiusRatio\":9,\"scale\":\"linear\",\"seriesParams\":[{\"data\":{\"id\":\"1\",\"label\":\"Count\"},\"drawLinesBetweenPoints\":true,\"mode\":\"stacked\",\"show\":\"true\",\"showCircles\":true,\"type\":\"histogram\",\"valueAxis\":\"ValueAxis-1\"}],\"setYExtents\":false,\"showCircles\":true,\"times\":[],\"valueAxes\":[{\"id\":\"ValueAxis-1\",\"labels\":{\"filter\":false,\"rotate\":0,\"show\":true,\"truncate\":100},\"name\":\"LeftAxi
s-1\",\"position\":\"left\",\"scale\":{\"mode\":\"normal\",\"type\":\"linear\"},\"show\":true,\"style\":{},\"title\":{},\"type\":\"value\"}]},\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"count\",\"schema\":\"metric\",\"params\":{}},{\"id\":\"2\",\"enabled\":true,\"type\":\"terms\",\"schema\":\"segment\",\"params\":{\"field\":\"target\",\"size\":100,\"order\":\"desc\",\"orderBy\":\"1\",\"customLabel\":\"HTML Tags\"}},{\"id\":\"3\",\"enabled\":true,\"type\":\"terms\",\"schema\":\"group\",\"params\":{\"field\":\"type\",\"size\":20,\"order\":\"desc\",\"orderBy\":\"1\",\"customLabel\":\"JS Events\"}}],\"listeners\":{}}",
+ "uiStateJSON": "{}",
+ "description": "",
+ "version": 1,
+ "kibanaSavedObjectMeta": {
+ "searchSourceJSON": "{\"index\":\"userale\",\"query\":{\"query_string\":{\"query\":\"*\",\"analyze_wildcard\":true}},\"filter\":[]}"
+ }
+ }
+ }
+]
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-senssoft/blob/c3aa251d/docker/logstash/Dockerfile
----------------------------------------------------------------------
diff --git a/docker/logstash/Dockerfile b/docker/logstash/Dockerfile
index 776f935..6aebb9e 100644
--- a/docker/logstash/Dockerfile
+++ b/docker/logstash/Dockerfile
@@ -13,7 +13,12 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-FROM docker.elastic.co/logstash/logstash:5.5.0
+FROM docker.elastic.co/logstash/logstash:5.6.3
MAINTAINER Michelle Beard <ms...@apache.org>
-RUN mkdir /usr/share/logstash/templates
\ No newline at end of file
+RUN mkdir /usr/share/logstash/templates
+
+RUN logstash-plugin remove x-pack
+
+# Known issue: https://github.com/elastic/logstash-docker/issues/45
+RUN sed -i '/xpack/d' /usr/share/logstash/config/logstash.yml
http://git-wip-us.apache.org/repos/asf/incubator-senssoft/blob/c3aa251d/docker/logstash/config/logstash-userale.conf
----------------------------------------------------------------------
diff --git a/docker/logstash/config/logstash-userale.conf b/docker/logstash/config/logstash-userale.conf
index a16c434..cbe8b8c 100644
--- a/docker/logstash/config/logstash-userale.conf
+++ b/docker/logstash/config/logstash-userale.conf
@@ -39,7 +39,7 @@ filter {
output {
# Output data to Elasticsearch instance
elasticsearch {
- hosts => "elasticsearch1:9200"
+ hosts => "loadbalancer:9200"
index => "userale"
document_type => "logs"
manage_template => true
http://git-wip-us.apache.org/repos/asf/incubator-senssoft/blob/c3aa251d/docker/requirements.txt
----------------------------------------------------------------------
diff --git a/docker/requirements.txt b/docker/requirements.txt
new file mode 100644
index 0000000..6094def
--- /dev/null
+++ b/docker/requirements.txt
@@ -0,0 +1 @@
+docker-compose==1.16.1
http://git-wip-us.apache.org/repos/asf/incubator-senssoft/blob/c3aa251d/docker/startup/entrypoint.sh
----------------------------------------------------------------------
diff --git a/docker/startup/entrypoint.sh b/docker/startup/entrypoint.sh
index 00953e6..c33a63b 100644
--- a/docker/startup/entrypoint.sh
+++ b/docker/startup/entrypoint.sh
@@ -17,5 +17,4 @@
# Wait for the Elasticsearch container to be ready before starting Kibana.
echo "Stalling for Elasticsearch"
-dockerize -wait http://elasticsearch1:9200
-dockerize -wait http://elasticsearch2:9200
\ No newline at end of file
+dockerize -wait http://loadbalancer:9200