You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@senssoft.apache.org by ms...@apache.org on 2018/06/04 22:15:48 UTC
[1/2] incubator-senssoft git commit: Setup system and container
service monitoring for SensSoft stack.
Repository: incubator-senssoft
Updated Branches:
refs/heads/master 460766793 -> c4b63e90b
Setup system and container service monitoring for SensSoft stack.
Project: http://git-wip-us.apache.org/repos/asf/incubator-senssoft/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-senssoft/commit/ae3ebdfd
Tree: http://git-wip-us.apache.org/repos/asf/incubator-senssoft/tree/ae3ebdfd
Diff: http://git-wip-us.apache.org/repos/asf/incubator-senssoft/diff/ae3ebdfd
Branch: refs/heads/master
Commit: ae3ebdfd2381be1f710d960bc8ef59e877e78245
Parents: 4607667
Author: msbeard <ms...@apache.org>
Authored: Fri May 11 12:24:24 2018 -0400
Committer: msbeard <ms...@apache.org>
Committed: Fri May 11 12:24:24 2018 -0400
----------------------------------------------------------------------
docker/README.md | 44 ++++++--
docker/beats/metricbeat/metricbeat.yml | 24 +++++
docker/beats/metricbeat/modules.d/docker.yml | 6 ++
.../metricbeat/modules.d/elasticsearch.yml | 6 ++
docker/beats/metricbeat/modules.d/kibana.yml | 6 ++
.../beats/metricbeat/modules.d/kubernetes.yml | 29 ++++++
docker/beats/metricbeat/modules.d/logstash.yml | 6 ++
.../beats/metricbeat/modules.d/postgresql.yml | 25 +++++
docker/beats/metricbeat/modules.d/system.yml | 24 +++++
docker/docker-compose.cluster.yml | 85 +++++++++++-----
docker/docker-compose.single-5.6.3.yml | 28 ++---
docker/docker-compose.yml | 102 ++++++++++---------
12 files changed, 278 insertions(+), 107 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-senssoft/blob/ae3ebdfd/docker/README.md
----------------------------------------------------------------------
diff --git a/docker/README.md b/docker/README.md
index 42d9856..3b7f51d 100644
--- a/docker/README.md
+++ b/docker/README.md
@@ -8,13 +8,25 @@ Prerequisites
1. Install docker-compose. Full instructions can be found [``here``](https://docs.docker.com/compose/install/).
+1. Create docker-machine instance
+ ```bash
+ docker-machine create --virtualbox-memory 3072 --virtualbox-cpu-count 2 senssoft
+ ```
+
1. Before launching the Docker containers, ensure your ``vm_max_map_count``
kernel setting is set to at least 262144.
Visit [``Running Elasticsearch in Production mode``](https://www.elastic.co/guide/en/elasticsearch/reference/5.5/docker.html#docker-cli-run-prod-mode) for OS specific instructions.
```bash
# Example for Linux systems
- $ sysctl -w vm.max_map_count=262144
+ $ docker-machine ssh senssoft sudo sysctl -w vm.max_map_count=262144
+ ```
+
+1. Create externel docker network to enable system monitoring. Only enable if running
+ the Elasticsearch 6.2.2 configuration (single and cluster mode)
+
+ ```bash
+ $ docker network create esnet
```
Single Node Deployment
@@ -24,16 +36,16 @@ Single Node Deployment
its state.
```bash
- $ docker-compose -f docker-compose.single-5.6.3.yml up -d loadbalancer
+ $ docker-compose -f docker-compose.single-5.6.3.yml up -d elasticsearch
or
- $ docker-compose up -d loadbalancer
+ $ docker-compose up -d elasticsearch
```
1. Confirm state:
```bash
- $ curl -XGET http://localhost:9200/_cluster/health?pretty
+ $ docker-machine ssh senssoft curl -XGET http://localhost:9200/_cluster/health?pretty
{
"cluster_name" : "SensSoft",
"status" : "yellow",
@@ -58,7 +70,7 @@ Single Node Deployment
```bash
$ docker-compose up -d logstash
- $ curl -XGET http://localhost:8100
+ $ docker-machine ssh senssoft curl -XGET http://localhost:8100
ok
```
@@ -67,6 +79,7 @@ Single Node Deployment
```bash
$ docker-compose up -d site
+ $ ssh docker@$(docker-machine ip senssoft) -L 8080:localhost:8080
```
Visit `http://localhost:8080` and you will see Apache SensSoft's home page.
@@ -76,6 +89,7 @@ Single Node Deployment
```bash
$ docker-compose up -d kibana
+ $ ssh docker@$(docker-machine ip senssoft) -L 5601:localhost:5601
```
1. Register an index in Kibana to see the logs:
@@ -118,20 +132,32 @@ Multi-Node Deployment on a Single Machine
1. Install ``docker-compose`` in an virtual environment. Full instructions can be found [``here``](https://docs.docker.com/compose/install/).
+1. Create docker-machine instance
+ ```bash
+ docker-machine create --virtualbox-memory 2048 --virtualbox-cpu-count 2 senssoft
+ ```
+
1. Before launching the Docker containers, ensure your ``vm_max_map_count``
kernel setting is set to at least 262144.
Visit [``Running Elasticsearch in Production mode``](https://www.elastic.co/guide/en/elasticsearch/reference/5.5/docker.html#docker-cli-run-prod-mode) for OS specific instructions.
```bash
# Example for Linux systems
- $ sysctl -w vm.max_map_count=262144
+ $ docker-machine ssh senssoft sudo sysctl -w vm.max_map_count=262144
+ ```
+
+1. Create externel docker network to enable system monitoring. Only enable if running
+ the Elasticsearch 6.2.2 configuration (single and cluster mode)
+
+ ```bash
+ $ docker network create esnet
```
1. Start Elasticsearch cluster:
```bash
- $ docker-compose -f docker-compose.cluster.yaml up -d --scale elasticsearch=3 elasticsearch
- $ docker-compose -f docker-compose.cluster.yaml up -d loadbalancer
+ $ docker-compose -f docker-compose.cluster.yml up -d --scale elasticsearch=3 elasticsearch
+ $ docker-compose -f docker-compose.cluster.yml up -d loadbalancer
```
The loadbalancer node exposes port 9200 on localhost and is the only node
@@ -148,7 +174,7 @@ Multi-Node Deployment on a Single Machine
1. Confirm cluster state:
```bash
- $ curl -XGET http://localhost:9200/_cluster/health?pretty
+ $ docker-machine ssh senssoft curl -XGET http://localhost:9200/_cluster/health\?pretty
{
"cluster_name" : "SensSoft",
"status" : "green",
http://git-wip-us.apache.org/repos/asf/incubator-senssoft/blob/ae3ebdfd/docker/beats/metricbeat/metricbeat.yml
----------------------------------------------------------------------
diff --git a/docker/beats/metricbeat/metricbeat.yml b/docker/beats/metricbeat/metricbeat.yml
new file mode 100644
index 0000000..3c61d77
--- /dev/null
+++ b/docker/beats/metricbeat/metricbeat.yml
@@ -0,0 +1,24 @@
+metricbeat.config.modules:
+
+ # Glob pattern for configuration reloading
+ path: ${path.config}/modules.d/*.yml
+
+ # Period on which files under path should be checked for changes
+ reload.period: 10s
+
+ # Set to true to enable config reloading
+ reload.enabled: true
+
+# Maximum amount of time to randomly delay the start of a metricset. Use 0 to
+# disable startup delay.
+metricbeat.max_start_delay: 10s
+
+output.elasticsearch:
+ hosts: ["${ELASTICSEARCH_URL}"]
+
+setup.dashboards.enabled: true
+
+setup.kibana:
+ host: "${KIBANA_URL}"
+
+logging.to_files: false
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-senssoft/blob/ae3ebdfd/docker/beats/metricbeat/modules.d/docker.yml
----------------------------------------------------------------------
diff --git a/docker/beats/metricbeat/modules.d/docker.yml b/docker/beats/metricbeat/modules.d/docker.yml
new file mode 100644
index 0000000..87a18f0
--- /dev/null
+++ b/docker/beats/metricbeat/modules.d/docker.yml
@@ -0,0 +1,6 @@
+#------------------------------- Docker Module -------------------------------
+- module: docker
+ metricsets: ["container", "cpu", "diskio", "healthcheck", "info", "memory", "network"]
+ hosts: ["unix:///var/run/docker.sock"]
+ enabled: true
+ period: 10s
http://git-wip-us.apache.org/repos/asf/incubator-senssoft/blob/ae3ebdfd/docker/beats/metricbeat/modules.d/elasticsearch.yml
----------------------------------------------------------------------
diff --git a/docker/beats/metricbeat/modules.d/elasticsearch.yml b/docker/beats/metricbeat/modules.d/elasticsearch.yml
new file mode 100644
index 0000000..62f7dcb
--- /dev/null
+++ b/docker/beats/metricbeat/modules.d/elasticsearch.yml
@@ -0,0 +1,6 @@
+#---------------------------- Elasticsearch Module ---------------------------
+- module: elasticsearch
+ metricsets: ["node", "node_stats"]
+ period: 10s
+ enabled: true
+ hosts: ["${ELASTICSEARCH_URL}"]
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-senssoft/blob/ae3ebdfd/docker/beats/metricbeat/modules.d/kibana.yml
----------------------------------------------------------------------
diff --git a/docker/beats/metricbeat/modules.d/kibana.yml b/docker/beats/metricbeat/modules.d/kibana.yml
new file mode 100644
index 0000000..2134a62
--- /dev/null
+++ b/docker/beats/metricbeat/modules.d/kibana.yml
@@ -0,0 +1,6 @@
+#------------------------------- Kibana Module -------------------------------
+- module: kibana
+ metricsets: ["status"]
+ period: 10s
+ enabled: true
+ hosts: ["${KIBANA_URL}"]
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-senssoft/blob/ae3ebdfd/docker/beats/metricbeat/modules.d/kubernetes.yml
----------------------------------------------------------------------
diff --git a/docker/beats/metricbeat/modules.d/kubernetes.yml b/docker/beats/metricbeat/modules.d/kubernetes.yml
new file mode 100644
index 0000000..b3e018d
--- /dev/null
+++ b/docker/beats/metricbeat/modules.d/kubernetes.yml
@@ -0,0 +1,29 @@
+#----------------------------- Kubernetes Module -----------------------------
+# Node metrics, from kubelet:
+#- module: kubernetes
+# metricsets:
+# - node
+# - system
+# - pod
+# - container
+# - volume
+# period: 10s
+# hosts: ["localhost:10255"]
+
+# State metrics from kube-state-metrics service:
+#- module: kubernetes
+# enabled: false
+# metricsets:
+# - state_node
+# - state_deployment
+# - state_replicaset
+# - state_pod
+# - state_container
+# period: 10s
+# hosts: ["kube-state-metrics:8080"]
+
+# Kubernetes events
+#- module: kubernetes
+# enabled: false
+# metricsets:
+# - event
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-senssoft/blob/ae3ebdfd/docker/beats/metricbeat/modules.d/logstash.yml
----------------------------------------------------------------------
diff --git a/docker/beats/metricbeat/modules.d/logstash.yml b/docker/beats/metricbeat/modules.d/logstash.yml
new file mode 100644
index 0000000..55a67ff
--- /dev/null
+++ b/docker/beats/metricbeat/modules.d/logstash.yml
@@ -0,0 +1,6 @@
+#------------------------------ Logstash Module ------------------------------
+- module: logstash
+ metricsets: ["node", "node_stats"]
+ enabled: true
+ period: 10s
+ hosts: ["${LOGSTASH_URL}"]
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-senssoft/blob/ae3ebdfd/docker/beats/metricbeat/modules.d/postgresql.yml
----------------------------------------------------------------------
diff --git a/docker/beats/metricbeat/modules.d/postgresql.yml b/docker/beats/metricbeat/modules.d/postgresql.yml
new file mode 100644
index 0000000..2b02360
--- /dev/null
+++ b/docker/beats/metricbeat/modules.d/postgresql.yml
@@ -0,0 +1,25 @@
+#----------------------------- PostgreSQL Module -----------------------------
+#- module: postgresql
+# metricsets:
+# # Stats about every PostgreSQL database
+# - database
+#
+# # Stats about the background writer process's activity
+# - bgwriter
+#
+# # Stats about every PostgreSQL process
+# - activity
+#
+# period: 10s
+
+ # The host must be passed as PostgreSQL URL. Example:
+ # postgres://localhost:5432?sslmode=disable
+ # The available parameters are documented here:
+ # https://godoc.org/github.com/lib/pq#hdr-Connection_String_Parameters
+# hosts: ["postgres://localhost:5432"]
+
+ # Username to use when connecting to PostgreSQL. Empty by default.
+ #username: user
+
+ # Password to use when connecting to PostgreSQL. Empty by default.
+ #password: pass
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-senssoft/blob/ae3ebdfd/docker/beats/metricbeat/modules.d/system.yml
----------------------------------------------------------------------
diff --git a/docker/beats/metricbeat/modules.d/system.yml b/docker/beats/metricbeat/modules.d/system.yml
new file mode 100644
index 0000000..f2d6bb2
--- /dev/null
+++ b/docker/beats/metricbeat/modules.d/system.yml
@@ -0,0 +1,24 @@
+#------------------------------- System Module -------------------------------
+- module: system
+ metricsets:
+ - core
+ - cpu
+ - diskio
+ - filesystem
+ - fsstat
+ - load
+ - memory
+ - network
+ - process
+ - process_summary
+ - socket
+ - uptime
+ enabled: true
+ period: 10s
+ processes: ['.*']
+ process.include_top_n:
+ by_cpu: 5 # include top 5 processes by CPU
+ by_memory: 5 # incude top 5 processes by memory
+ core.metrics: [percentages, ticks]
+ cpu.metrics: [percentages, normalized_percentages, ticks]
+ process.cgroups.enabled: true
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-senssoft/blob/ae3ebdfd/docker/docker-compose.cluster.yml
----------------------------------------------------------------------
diff --git a/docker/docker-compose.cluster.yml b/docker/docker-compose.cluster.yml
index 5835716..96f6cc4 100644
--- a/docker/docker-compose.cluster.yml
+++ b/docker/docker-compose.cluster.yml
@@ -21,8 +21,6 @@ services:
# Startup
startup:
build: ./startup
- image: senssoft/startup:latest
- container_name: senssoft-start
depends_on:
- "loadbalancer"
command: bash /opt/entrypoint.sh
@@ -30,8 +28,6 @@ services:
# # Site
site:
build: ./site
- image: senssoft/website:latest
- container_name: senssoft-app
ports:
- 8080:8080
depends_on:
@@ -40,8 +36,7 @@ services:
# Elasticsearch Swarm Workers
elasticsearch:
- build: ./elasticsearch
- image: senssoft/elasticsearch:6.2.2
+ build: ./elasticsearch/6.2.2
environment:
- NODE_MASTER=true
- NUMBER_OF_MASTERS=3
@@ -54,15 +49,19 @@ services:
- HTTP_ENABLE=false
- HTTP_CORS_ENABLE=false
- HTTP_CORS_ALLOW_ORIGIN=*
- - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
+ - "ES_JAVA_OPTS=-Xms256m -Xmx256m"
+ ulimits:
+ memlock:
+ soft: -1
+ hard: -1
+ healthcheck:
+ test: ["CMD", "curl","-s" ,"-f", "-u", "http://localhost:9200/_cluster/health"]
networks:
- esnet
- # Elasticsearch Loadbalancer/Entry
+ # Elasticsearch Loadbalancer
loadbalancer:
- build: ./elasticsearch
- image: senssoft/elasticsearch:6.2.2
- container_name: senssoft-loadbalancer
+ build: ./elasticsearch/6.2.2
environment:
- NODE_MASTER=false
- NUMBER_OF_MASTERS=1
@@ -75,7 +74,13 @@ services:
- HTTP_ENABLE=true
- HTTP_CORS_ENABLE=true
- HTTP_CORS_ALLOW_ORIGIN=*
- - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
+ - "ES_JAVA_OPTS=-Xms256m -Xmx256m"
+ ulimits:
+ memlock:
+ soft: -1
+ hard: -1
+ healthcheck:
+ test: ["CMD", "curl","-s" ,"-f", "-u", "http://localhost:9200/_cluster/health"]
ports:
- "9200:9200"
- "9300"
@@ -84,11 +89,9 @@ services:
# Kibana
kibana:
- build: ./kibana
- image: senssoft/kibana:6.2.2
- container_name: senssoft-kibana
+ build: ./kibana/6.2.2
environment:
- - "ES_JAVA_OPTS=-Xms2g -Xmx2g"
+ - "ES_JAVA_OPTS=-Xms256m -Xmx256m"
- "ELASTICSEARCH_URL=http://loadbalancer:9200"
ports:
- "5601:5601"
@@ -99,16 +102,14 @@ services:
# Logstash
logstash:
- build: ./logstash
- image: senssoft/logstash:6.2.2
- container_name: senssoft-logstash
+ build: ./logstash/6.2.2
environment:
- "ELASTICSEARCH_URL=loadbalancer:9200"
ports:
- 8100:8100
volumes:
- - ./logstash/pipeline/logstash-userale.conf:/usr/share/logstash/pipeline/logstash-userale.conf
- - ./logstash/templates/userale.json:/usr/share/logstash/templates/userale.json
+ - ./logstash/6.2.2/pipeline/logstash-userale.conf:/usr/share/logstash/pipeline/logstash-userale.conf
+ - ./logstash/6.2.2/templates/userale.json:/usr/share/logstash/templates/userale.json
depends_on:
- "loadbalancer"
networks:
@@ -117,8 +118,6 @@ services:
# Distill
distill:
build: ./distill
- image: senssoft/distill:0.1.0
- container_name: senssoft-distill
ports:
- 8090:8090
depends_on:
@@ -130,16 +129,12 @@ services:
db:
build: ./db
image: senssoft/postgres:latest
- container_name: senssoft-postgres
ports:
- "5432:5432"
# TAP
tap:
build: ./tap
- image: senssoft/tap:0.1.0
- container_name: senssoft-tap
- # command: ["bash", "./wait-for-postgres.sh"]
command: bash -c "/usr/src/app/wait-for-postgres.sh"
ports:
- "8010:8010"
@@ -148,6 +143,40 @@ services:
links:
- db:db
+ # Specific container and configuration to monitor the HOST (filesystem, memory, processes,...)
+ metricbeat:
+ image: docker.elastic.co/beats/metricbeat:6.2.4
+ restart: unless-stopped
+ user: root
+ pid: host
+ volumes:
+ #The commented sections below enable Metricbeat to monitor the Docker host
+ #rather than the Metricbeat container. These are used by the system module.
+ - /proc:/hostfs/proc:ro
+ - /sys/fs/cgroup:/hostfs/sys/fs/cgroup:ro
+ #We mount the host filesystem so we can report on disk usage with the system module
+ - /:/hostfs:ro
+ #Mount the metricbeat configuration so users can make edit
+ - ./beats/metricbeat/metricbeat.yml:/usr/share/metricbeat/metricbeat.yml
+ #Mount the modules.d directory into the container. This allows user to
+ #potentially make changes to the modules and they will be dynamically loaded.
+ - ./beats/metricbeat/modules.d/:/usr/share/metricbeat/modules.d/
+ #Allows us to report on docker from the hosts information
+ - /var/run/docker.sock:/var/run/docker.sock
+ depends_on:
+ - loadbalancer
+ environment:
+ - ELASTICSEARCH_URL=http://loadbalancer:9200
+ - KIBANA_URL=http://kibana:5601
+ - LOGSTASH_URL=http://logstash:9600
+ extra_hosts:
+ - "loadbalancer:172.17.0.1" # The IP of docker0 interface to access host from container
+ - "kibana:172.17.0.1" # The IP of docker0 interface to access host from container
+ - "logstash:172.17.0.1"
+ command: metricbeat -e -system.hostfs=/hostfs -strict.perms=false
+ network_mode: host # Mandatory to monitor HOST filesystem, memory, processes,...
+
networks:
esnet:
- driver: bridge
+ external:
+ name: esnet
http://git-wip-us.apache.org/repos/asf/incubator-senssoft/blob/ae3ebdfd/docker/docker-compose.single-5.6.3.yml
----------------------------------------------------------------------
diff --git a/docker/docker-compose.single-5.6.3.yml b/docker/docker-compose.single-5.6.3.yml
index e8e333f..0198644 100644
--- a/docker/docker-compose.single-5.6.3.yml
+++ b/docker/docker-compose.single-5.6.3.yml
@@ -21,8 +21,6 @@ services:
# Startup
startup:
build: ./startup
- image: senssoft/startup:latest
- container_name: senssoft-start
depends_on:
- "loadbalancer"
command: bash /opt/entrypoint.sh
@@ -30,24 +28,20 @@ services:
# Site
site:
build: ./site
- image: senssoft/website:latest
- container_name: senssoft-app
ports:
- 8080:8080
depends_on:
- "logstash"
command: python -m SimpleHTTPServer 8080
- # Elasticsearch Loadbalancer/Entry
+ # Elasticsearch
loadbalancer:
build: ./elasticsearch/5.6.3
- image: senssoft/elasticsearch:5.6.3
- container_name: senssoft-loadbalancers
environment:
- NODE_MASTER=true
- NUMBER_OF_MASTERS=1
- NODE_COORDINATE=true
- - "DISCOVERY_SERVICE=elasticsearch"
+ - "DISCOVERY_SERVICE=0.0.0.0"
- NODE_DATA=true
- NODE_INGEST=true
- MAX_LOCAL_STORAGE_NODES=1
@@ -64,37 +58,31 @@ services:
# Kibana
kibana:
build: ./kibana/5.6.3
-# image: senssoft/kibana:5.6.3
- container_name: senssoft-kibana
environment:
- "ES_JAVA_OPTS=-Xms2g -Xmx2g"
- - "ELASTICSEARCH_URL=http://loadbalancer:9200"
+ - "ELASTICSEARCH_URL=http://elasticsearch:9200"
ports:
- "5601:5601"
depends_on:
- - "loadbalancer"
+ - "elasticsearch"
networks:
- esnet
# Logstash
logstash:
build: ./logstash/5.6.3
- image: senssoft/logstash:5.6.3
- container_name: senssoft-logstash
environment:
- - "ELASTICSEARCH_URL=loadbalancer:9200"
+ - "ELASTICSEARCH_URL=elasticsearch:9200"
ports:
- 8100:8100
depends_on:
- - "loadbalancer"
+ - "elasticsearch"
networks:
- esnet
# Distill
distill:
build: ./distill
- image: senssoft/distill:0.1.0
- container_name: senssoft-distill
ports:
- 8090:8090
depends_on:
@@ -106,15 +94,12 @@ services:
db:
build: ./db
image: senssoft/postgres:latest
- container_name: senssoft-postgres
ports:
- "5432:5432"
# TAP
tap:
build: ./tap
- image: senssoft/tap:0.1.0
- container_name: senssoft-tap
command: bash -c "/usr/src/app/wait-for-postgres.sh"
ports:
- "8010:8010"
@@ -123,6 +108,7 @@ services:
links:
- db:db
+
networks:
esnet:
driver: bridge
http://git-wip-us.apache.org/repos/asf/incubator-senssoft/blob/ae3ebdfd/docker/docker-compose.yml
----------------------------------------------------------------------
diff --git a/docker/docker-compose.yml b/docker/docker-compose.yml
index a675f7b..30507c6 100644
--- a/docker/docker-compose.yml
+++ b/docker/docker-compose.yml
@@ -21,54 +21,27 @@ services:
# Startup
startup:
build: ./startup
- image: senssoft/startup:latest
- container_name: senssoft-start
depends_on:
- - "loadbalancer"
+ - "elasticsearch"
command: bash /opt/entrypoint.sh
# Site
site:
build: ./site
- image: senssoft/website:latest
- container_name: senssoft-app
ports:
- 8080:8080
depends_on:
- "logstash"
command: python -m SimpleHTTPServer 8080
+ # Elasticsearch
elasticsearch:
build: ./elasticsearch/6.2.2
- image: senssoft/elasticsearch:6.2.2
environment:
- NODE_MASTER=true
- NUMBER_OF_MASTERS=1
- - NODE_COORDINATE=false
- - "DISCOVERY_SERVICE=elasticsearch"
- - NODE_DATA=false
- - NODE_INGEST=false
- - MAX_LOCAL_STORAGE_NODES=1
- - NETWORK_HOST=0.0.0.0
- - HTTP_ENABLE=false
- - HTTP_CORS_ENABLE=false
- - HTTP_CORS_ALLOW_ORIGIN=*
- - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
- ports:
- - "9300"
- networks:
- - esnet
-
- # Elasticsearch Loadbalancer/Entry
- loadbalancer:
- build: ./elasticsearch/6.2.2
- image: senssoft/elasticsearch:6.2.2
- container_name: senssoft-loadbalancer
- environment:
- - NODE_MASTER=false
- - NUMBER_OF_MASTERS=1
- NODE_COORDINATE=true
- - "DISCOVERY_SERVICE=elasticsearch"
+ - "DISCOVERY_SERVICE=0.0.0.0"
- NODE_DATA=true
- NODE_INGEST=true
- MAX_LOCAL_STORAGE_NODES=1
@@ -77,54 +50,55 @@ services:
- HTTP_CORS_ENABLE=true
- HTTP_CORS_ALLOW_ORIGIN=*
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
+ ulimits:
+ memlock:
+ soft: -1
+ hard: -1
+ healthcheck:
+ test: ["CMD", "curl","-s" ,"-f", "-u", "http://localhost:9200/_cluster/health"]
+ # Mount volumes to backup elasticsearch data
+# volumes:
+# - esdata:/usr/share/elasticsaerch/data
ports:
- "9200:9200"
networks:
- esnet
- depends_on:
- - elasticsearch
# Kibana
kibana:
build: ./kibana/6.2.2
- image: senssoft/kibana:6.2.2
- container_name: senssoft-kibana
environment:
- - "ES_JAVA_OPTS=-Xms2g -Xmx2g"
- - "ELASTICSEARCH_URL=http://loadbalancer:9200"
+ - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
+ - "ELASTICSEARCH_URL=http://elasticsearch:9200"
ports:
- "5601:5601"
depends_on:
- - "loadbalancer"
+ - "elasticsearch"
networks:
- esnet
# Logstash
logstash:
build: ./logstash/6.2.2
- image: senssoft/logstash:6.2.2
- container_name: senssoft-logstash
environment:
- - "ELASTICSEARCH_URL=loadbalancer:9200"
+ - "ELASTICSEARCH_URL=elasticsearch:9200"
ports:
- 8100:8100
volumes:
- ./logstash/6.2.2/pipeline/logstash-userale.conf:/usr/share/logstash/pipeline/logstash-userale.conf
- ./logstash/6.2.2/templates/userale.json:/usr/share/logstash/templates/userale.json
depends_on:
- - "loadbalancer"
+ - "elasticsearch"
networks:
- esnet
# Distill
distill:
build: ./distill
- image: senssoft/distill:0.1.0
- container_name: senssoft-distill
ports:
- 8090:8090
depends_on:
- - "loadbalancer"
+ - "elasticsearch"
networks:
- esnet
@@ -132,16 +106,12 @@ services:
db:
build: ./db
image: senssoft/postgres:latest
- container_name: senssoft-postgres
ports:
- "5432:5432"
# TAP
tap:
build: ./tap
- image: senssoft/tap:0.1.0
- container_name: senssoft-tap
- # command: ["bash", "./wait-for-postgres.sh"]
command: bash -c "/usr/src/app/wait-for-postgres.sh"
ports:
- "8010:8010"
@@ -150,6 +120,40 @@ services:
links:
- db:db
+ # Specific container and configuration to monitor the HOST (filesystem, memory, processes,...)
+ metricbeat:
+ image: docker.elastic.co/beats/metricbeat:6.2.4
+ restart: unless-stopped
+ user: root
+ pid: host
+ volumes:
+ #The commented sections below enable Metricbeat to monitor the Docker host
+ #rather than the Metricbeat container. These are used by the system module.
+ - /proc:/hostfs/proc:ro
+ - /sys/fs/cgroup:/hostfs/sys/fs/cgroup:ro
+ #We mount the host filesystem so we can report on disk usage with the system module
+ - /:/hostfs:ro
+ #Mount the metricbeat configuration so users can make edit
+ - ./beats/metricbeat/metricbeat.yml:/usr/share/metricbeat/metricbeat.yml
+ #Mount the modules.d directory into the container. This allows user to
+ #potentially make changes to the modules and they will be dynamically loaded.
+ - ./beats/metricbeat/modules.d/:/usr/share/metricbeat/modules.d/
+ #Allows us to report on docker from the hosts information
+ - /var/run/docker.sock:/var/run/docker.sock
+ depends_on:
+ - elasticsearch
+ environment:
+ - ELASTICSEARCH_URL=http://elasticsearch:9200
+ - KIBANA_URL=http://kibana:5601
+ - LOGSTASH_URL=http://logstash:9600
+ extra_hosts:
+ - "elasticsearch:172.17.0.1" # The IP of docker0 interface to access host from container
+ - "kibana:172.17.0.1" # The IP of docker0 interface to access host from container
+ - "logstash:172.17.0.1"
+ command: metricbeat -e -system.hostfs=/hostfs -strict.perms=false
+ network_mode: host # Mandatory to monitor HOST filesystem, memory, processes,...
+
networks:
esnet:
- driver: bridge
+ external:
+ name: esnet
[2/2] incubator-senssoft git commit: Added support for monitoring
kafka, zookeeper, and updated config for postgresql to monitor TAPs database.
Posted by ms...@apache.org.
Added support for monitoring kafka, zookeeper, and updated config for postgresql to monitor TAPs database.
Project: http://git-wip-us.apache.org/repos/asf/incubator-senssoft/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-senssoft/commit/c4b63e90
Tree: http://git-wip-us.apache.org/repos/asf/incubator-senssoft/tree/c4b63e90
Diff: http://git-wip-us.apache.org/repos/asf/incubator-senssoft/diff/c4b63e90
Branch: refs/heads/master
Commit: c4b63e90b1bda7807490d33b0cde19726831f786
Parents: ae3ebdf
Author: msbeard <ms...@apache.org>
Authored: Mon Jun 4 18:10:20 2018 -0400
Committer: msbeard <ms...@apache.org>
Committed: Mon Jun 4 18:10:20 2018 -0400
----------------------------------------------------------------------
.gitignore | 2 +
docker/beats/metricbeat/metricbeat.yml | 2 +-
docker/beats/metricbeat/modules.d/kafka.yml | 9 ++++
docker/beats/metricbeat/modules.d/kibana.yml | 2 +-
.../beats/metricbeat/modules.d/kubernetes.yml | 47 ++++++++++----------
.../beats/metricbeat/modules.d/postgresql.yml | 22 +++------
docker/beats/metricbeat/modules.d/zookeeper.yml | 6 +++
docker/docker-compose.yml | 8 +++-
8 files changed, 56 insertions(+), 42 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-senssoft/blob/c4b63e90/.gitignore
----------------------------------------------------------------------
diff --git a/.gitignore b/.gitignore
index ba433d6..c8facd7 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1,3 +1,5 @@
+env
+venv
.idea
release/incubator-*
release/apache-*
http://git-wip-us.apache.org/repos/asf/incubator-senssoft/blob/c4b63e90/docker/beats/metricbeat/metricbeat.yml
----------------------------------------------------------------------
diff --git a/docker/beats/metricbeat/metricbeat.yml b/docker/beats/metricbeat/metricbeat.yml
index 3c61d77..a987a43 100644
--- a/docker/beats/metricbeat/metricbeat.yml
+++ b/docker/beats/metricbeat/metricbeat.yml
@@ -4,7 +4,7 @@ metricbeat.config.modules:
path: ${path.config}/modules.d/*.yml
# Period on which files under path should be checked for changes
- reload.period: 10s
+ reload.period: 30s
# Set to true to enable config reloading
reload.enabled: true
http://git-wip-us.apache.org/repos/asf/incubator-senssoft/blob/c4b63e90/docker/beats/metricbeat/modules.d/kafka.yml
----------------------------------------------------------------------
diff --git a/docker/beats/metricbeat/modules.d/kafka.yml b/docker/beats/metricbeat/modules.d/kafka.yml
new file mode 100644
index 0000000..b1954cc
--- /dev/null
+++ b/docker/beats/metricbeat/modules.d/kafka.yml
@@ -0,0 +1,9 @@
+#---------------------------- Kafka Module ---------------------------
+- module: kafka
+ metricsets: ["consumergroup", "partition"]
+ period: 10s
+ hosts: ["${KAFKA_URL}"]
+ enabled: false
+
+ # List of Topics to query metadata for. If empty, all topics will be queried.
+ #topics: []
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-senssoft/blob/c4b63e90/docker/beats/metricbeat/modules.d/kibana.yml
----------------------------------------------------------------------
diff --git a/docker/beats/metricbeat/modules.d/kibana.yml b/docker/beats/metricbeat/modules.d/kibana.yml
index 2134a62..0d83b05 100644
--- a/docker/beats/metricbeat/modules.d/kibana.yml
+++ b/docker/beats/metricbeat/modules.d/kibana.yml
@@ -1,6 +1,6 @@
#------------------------------- Kibana Module -------------------------------
- module: kibana
metricsets: ["status"]
- period: 10s
+ period: 30s
enabled: true
hosts: ["${KIBANA_URL}"]
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-senssoft/blob/c4b63e90/docker/beats/metricbeat/modules.d/kubernetes.yml
----------------------------------------------------------------------
diff --git a/docker/beats/metricbeat/modules.d/kubernetes.yml b/docker/beats/metricbeat/modules.d/kubernetes.yml
index b3e018d..f58ea52 100644
--- a/docker/beats/metricbeat/modules.d/kubernetes.yml
+++ b/docker/beats/metricbeat/modules.d/kubernetes.yml
@@ -1,29 +1,30 @@
#----------------------------- Kubernetes Module -----------------------------
# Node metrics, from kubelet:
-#- module: kubernetes
-# metricsets:
-# - node
-# - system
-# - pod
-# - container
-# - volume
-# period: 10s
-# hosts: ["localhost:10255"]
+- module: kubernetes
+ enabled: false
+ metricsets:
+ - node
+ - system
+ - pod
+ - container
+ - volume
+ period: 10s
+ hosts: ["localhost:10255"]
# State metrics from kube-state-metrics service:
-#- module: kubernetes
-# enabled: false
-# metricsets:
-# - state_node
-# - state_deployment
-# - state_replicaset
-# - state_pod
-# - state_container
-# period: 10s
-# hosts: ["kube-state-metrics:8080"]
+- module: kubernetes
+ enabled: false
+ metricsets:
+ - state_node
+ - state_deployment
+ - state_replicaset
+ - state_pod
+ - state_container
+ period: 10s
+ hosts: ["kube-state-metrics:8080"]
# Kubernetes events
-#- module: kubernetes
-# enabled: false
-# metricsets:
-# - event
\ No newline at end of file
+- module: kubernetes
+ enabled: false
+ metricsets:
+ - event
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-senssoft/blob/c4b63e90/docker/beats/metricbeat/modules.d/postgresql.yml
----------------------------------------------------------------------
diff --git a/docker/beats/metricbeat/modules.d/postgresql.yml b/docker/beats/metricbeat/modules.d/postgresql.yml
index 2b02360..363fe4b 100644
--- a/docker/beats/metricbeat/modules.d/postgresql.yml
+++ b/docker/beats/metricbeat/modules.d/postgresql.yml
@@ -1,25 +1,17 @@
#----------------------------- PostgreSQL Module -----------------------------
-#- module: postgresql
-# metricsets:
-# # Stats about every PostgreSQL database
-# - database
-#
-# # Stats about the background writer process's activity
-# - bgwriter
-#
-# # Stats about every PostgreSQL process
-# - activity
-#
-# period: 10s
+- module: postgresql
+ metricsets: ["database", "activity"]
+ period: 10s
+ enabled: true
# The host must be passed as PostgreSQL URL. Example:
# postgres://localhost:5432?sslmode=disable
# The available parameters are documented here:
# https://godoc.org/github.com/lib/pq#hdr-Connection_String_Parameters
-# hosts: ["postgres://localhost:5432"]
+ hosts: ["${POSTGRESQL_URL}"]
# Username to use when connecting to PostgreSQL. Empty by default.
- #username: user
+ username: "${POSTGRESQL_USERNAME}"
# Password to use when connecting to PostgreSQL. Empty by default.
- #password: pass
\ No newline at end of file
+ password: "${POSTGRESQL_PASSWORD}"
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-senssoft/blob/c4b63e90/docker/beats/metricbeat/modules.d/zookeeper.yml
----------------------------------------------------------------------
diff --git a/docker/beats/metricbeat/modules.d/zookeeper.yml b/docker/beats/metricbeat/modules.d/zookeeper.yml
new file mode 100644
index 0000000..ca67993
--- /dev/null
+++ b/docker/beats/metricbeat/modules.d/zookeeper.yml
@@ -0,0 +1,6 @@
+#---------------------------- ZooKeeper Module ---------------------------
+- module: zookeeper
+ enabled: false
+ metricsets: ["mntr"]
+ period: 10s
+ hosts: ["${ZOOKEEPER_URL}"]
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-senssoft/blob/c4b63e90/docker/docker-compose.yml
----------------------------------------------------------------------
diff --git a/docker/docker-compose.yml b/docker/docker-compose.yml
index 30507c6..316b7f5 100644
--- a/docker/docker-compose.yml
+++ b/docker/docker-compose.yml
@@ -123,7 +123,7 @@ services:
# Specific container and configuration to monitor the HOST (filesystem, memory, processes,...)
metricbeat:
image: docker.elastic.co/beats/metricbeat:6.2.4
- restart: unless-stopped
+# restart: unless-stopped
user: root
pid: host
volumes:
@@ -133,7 +133,7 @@ services:
- /sys/fs/cgroup:/hostfs/sys/fs/cgroup:ro
#We mount the host filesystem so we can report on disk usage with the system module
- /:/hostfs:ro
- #Mount the metricbeat configuration so users can make edit
+ #Mount the metricbeat configuration so users can make edits
- ./beats/metricbeat/metricbeat.yml:/usr/share/metricbeat/metricbeat.yml
#Mount the modules.d directory into the container. This allows user to
#potentially make changes to the modules and they will be dynamically loaded.
@@ -146,10 +146,14 @@ services:
- ELASTICSEARCH_URL=http://elasticsearch:9200
- KIBANA_URL=http://kibana:5601
- LOGSTASH_URL=http://logstash:9600
+ - POSTGRESQL_URL="postgresql://db:5432/tapdb?sslmode=disable"
+ - POSTGRESQL_USERNAME=tapuser
+ - POSTGRESQL_PASSWORD="Dr@p3rUs3r"
extra_hosts:
- "elasticsearch:172.17.0.1" # The IP of docker0 interface to access host from container
- "kibana:172.17.0.1" # The IP of docker0 interface to access host from container
- "logstash:172.17.0.1"
+ - "db:172.17.0.1"
command: metricbeat -e -system.hostfs=/hostfs -strict.perms=false
network_mode: host # Mandatory to monitor HOST filesystem, memory, processes,...