You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@senssoft.apache.org by ms...@apache.org on 2018/03/04 20:36:57 UTC
[1/4] incubator-senssoft git commit: Updated ELK stack to 6.2.2.
Seperated cluster instructions.
Repository: incubator-senssoft
Updated Branches:
refs/heads/kubernetes 6e27db527 -> bf43699ff
Updated ELK stack to 6.2.2. Seperated cluster instructions.
Project: http://git-wip-us.apache.org/repos/asf/incubator-senssoft/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-senssoft/commit/66639ab5
Tree: http://git-wip-us.apache.org/repos/asf/incubator-senssoft/tree/66639ab5
Diff: http://git-wip-us.apache.org/repos/asf/incubator-senssoft/diff/66639ab5
Branch: refs/heads/kubernetes
Commit: 66639ab52e44c81413a2a0c5528ccdd1ee1db97a
Parents: 6e27db5
Author: msbeard <ms...@apache.org>
Authored: Sun Mar 4 13:45:59 2018 -0500
Committer: msbeard <ms...@apache.org>
Committed: Sun Mar 4 13:45:59 2018 -0500
----------------------------------------------------------------------
docker/CLUSTER.md | 68 +++++++++
docker/README.md | 45 +++---
docker/docker-compose.cluster.yaml | 153 ++++++++++++++++++++
docker/docker-compose.yml | 107 +++++---------
docker/elasticsearch/Dockerfile | 6 +-
docker/kibana/Dockerfile | 9 +-
docker/logstash/Dockerfile | 19 +--
docker/logstash/config/logstash-userale.conf | 53 -------
docker/logstash/pipeline/logstash-userale.conf | 52 +++++++
docker/logstash/templates/userale.json | 11 +-
10 files changed, 347 insertions(+), 176 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-senssoft/blob/66639ab5/docker/CLUSTER.md
----------------------------------------------------------------------
diff --git a/docker/CLUSTER.md b/docker/CLUSTER.md
new file mode 100644
index 0000000..93df592
--- /dev/null
+++ b/docker/CLUSTER.md
@@ -0,0 +1,68 @@
+How to Build SensSoft Docker Containers
+---------------------------------------
+
+1. Install [``Docker``](http://docker.com) on your machine. Require Docker 1.7 and above.
+
+1. Install ``docker-compose`` in an virtual environment.
+ Full instructions can be found [``here``](https://docs.docker.com/compose/install/).
+
+ ```bash
+ $ python3 -m venv env
+ $ source env/bin/activate
+ $ pip install -r requirements.txt
+ ```
+
+1. Before launching the Docker containers, ensure your ``vm_max_map_count``
+ kernel setting is set to at least 262144.
+ Visit [``Running Elasticsearch in Production mode``](https://www.elastic.co/guide/en/elasticsearch/reference/5.5/docker.html#docker-cli-run-prod-mode) for OS specific instructions.
+
+ ```bash
+ # Example for Linux systems
+ $ sysctl -w vm.max_map_count=262144
+ ```
+
+1. Start Elasticsearch cluster:
+
+ ```bash
+ $ docker-compose -f docker-compose.cluster.yaml up -d --scale elasticsearch=3 elasticsearch
+ $ docker-compose -f docker-compose.cluster.yaml up -d loadbalancer
+ ```
+
+ The loadbalancer node exposes port 9200 on localhost and is the only node
+ that has HTTP enabled. Services such as Kibana and Logstash connect to the
+ loadbalancer node directly. Loadbalancer accepts requests from Kibana and Logstash
+ and balances them across the elasticsearch worker nodes. The elasticsearch
+ worker nodes communicate to each other and the loadbalancer via TCP on port 9300.
+
+ <aside class="warning">
+ Starting an elasticsearch cluster is not recommended on a single server. This
+ is just for demonstration purposes only. Please refer to our [Kubernetes] guide to
+ deploy an Elasticsearch cluster.
+ </aside>
+
+1. Confirm cluster state:
+ ```bash
+ $ curl -XGET http://localhost:9200/_cluster/health?pretty
+ {
+ "cluster_name" : "SensSoft",
+ "status" : "green",
+ "timed_out" : false,
+ "number_of_nodes" : 4,
+ "number_of_data_nodes" : 3,
+ "active_primary_shards" : 0,
+ "active_shards" : 0,
+ "relocating_shards" : 0,
+ "initializing_shards" : 0,
+ "unassigned_shards" : 0,
+ "delayed_unassigned_shards" : 0,
+ "number_of_pending_tasks" : 0,
+ "number_of_in_flight_fetch" : 0,
+ "task_max_waiting_in_queue_millis" : 0,
+ "active_shards_percent_as_number" : 100.0
+ }
+ ```
+ Confirm that the `number_of_nodes` is 4 and `number_of_data_nodes` is 3.
+
+1. Follow remaining instructions in README.md, starting at #6.
+
+© Copyright 2016 The Charles Stark Draper Laboratory, Inc. All rights reserved.
http://git-wip-us.apache.org/repos/asf/incubator-senssoft/blob/66639ab5/docker/README.md
----------------------------------------------------------------------
diff --git a/docker/README.md b/docker/README.md
index 036a995..4281566 100644
--- a/docker/README.md
+++ b/docker/README.md
@@ -1,7 +1,7 @@
How to Build SensSoft Docker Containers
---------------------------------------
-1. Install [``Docker``](http://docker.com) on your machine.
+1. Install [``Docker``](http://docker.com) on your machine. Require Docker 1.7 and above.
1. Install ``docker-compose`` in an virtual environment.
Full instructions can be found [``here``](https://docs.docker.com/compose/install/).
@@ -21,28 +21,22 @@ How to Build SensSoft Docker Containers
$ sysctl -w vm.max_map_count=262144
```
-1. Start Elasticsearch cluster:
-
- ```bash
- $ docker-compose up -d --scale elasticsearch=3 elasticsearch loadbalancer
- ```
-
- The loadbalancer node exposes port 9200 on localhost and is the only node
- that has HTTP enabled. Services such as Kibana and Logstash connect to the
- loadbalancer node directly. Loadbalancer accepts requests from Kibana and Logstash
- and balances them across the elasticsearch worker nodes. The elasticsearch
- worker nodes communicate to each other and the loadbalancer via TCP on port 9300.
-
-
-1. Confirm cluster state:
+1. Start Elasticsearch. Give Elasticsearch about 2 minutes to start before confirming
+ its state.
+
+ ```bash
+ $ docker-compose up -d loadbalancer
+ ```
+
+1. Confirm state:
```bash
$ curl -XGET http://localhost:9200/_cluster/health?pretty
{
"cluster_name" : "SensSoft",
"status" : "green",
"timed_out" : false,
- "number_of_nodes" : 4,
- "number_of_data_nodes" : 3,
+ "number_of_nodes" : 1,
+ "number_of_data_nodes" : 1,
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
@@ -55,7 +49,6 @@ How to Build SensSoft Docker Containers
"active_shards_percent_as_number" : 100.0
}
```
- Confirm that the `number_of_nodes` is 4 and `number_of_data_nodes` is 3.
1. Launch logging server. Give Logstash about 2 minutes to start before confirming
its state.
@@ -105,8 +98,13 @@ How to Build SensSoft Docker Containers
![alt text][dashboard]
+1. To Launch Tap and Distill
+ ```bash
+ $ docker-compose up -d distill tap
+ ```
+
1. To stop all containers.
- ```sh
+ ```bash
$ docker-compose stop
```
@@ -123,11 +121,10 @@ Having Issues?
$ docker-compose logs > err.dump
```
-Todo
-----
-- [ ] TAP docker deployment instructions.
-- [ ] Distill docker deployment instructions.
-- [ ] Apache SensSoft Docker + Kubernetes instructions.
+TODO
+----
+1. Update Kibana instructions and images.
+1. Test Kibana visualizations.
[configure_index]: ./images/configure_index.png "Configure Kibana index"
[confirmation]: ./images/confirmation.png "Confirm index pattern conflicts"
http://git-wip-us.apache.org/repos/asf/incubator-senssoft/blob/66639ab5/docker/docker-compose.cluster.yaml
----------------------------------------------------------------------
diff --git a/docker/docker-compose.cluster.yaml b/docker/docker-compose.cluster.yaml
new file mode 100644
index 0000000..5835716
--- /dev/null
+++ b/docker/docker-compose.cluster.yaml
@@ -0,0 +1,153 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Apache SensSoft Cluster Build
+
+version: "3"
+services:
+
+ # Startup
+ startup:
+ build: ./startup
+ image: senssoft/startup:latest
+ container_name: senssoft-start
+ depends_on:
+ - "loadbalancer"
+ command: bash /opt/entrypoint.sh
+
+ # # Site
+ site:
+ build: ./site
+ image: senssoft/website:latest
+ container_name: senssoft-app
+ ports:
+ - 8080:8080
+ depends_on:
+ - "logstash"
+ command: python -m SimpleHTTPServer 8080
+
+ # Elasticsearch Swarm Workers
+ elasticsearch:
+ build: ./elasticsearch
+ image: senssoft/elasticsearch:6.2.2
+ environment:
+ - NODE_MASTER=true
+ - NUMBER_OF_MASTERS=3
+ - NODE_COORDINATE=true
+ - "DISCOVERY_SERVICE=elasticsearch"
+ - NODE_DATA=true
+ - NODE_INGEST=true
+ - MAX_LOCAL_STORAGE_NODES=1
+ - NETWORK_HOST=0.0.0.0
+ - HTTP_ENABLE=false
+ - HTTP_CORS_ENABLE=false
+ - HTTP_CORS_ALLOW_ORIGIN=*
+ - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
+ networks:
+ - esnet
+
+ # Elasticsearch Loadbalancer/Entry
+ loadbalancer:
+ build: ./elasticsearch
+ image: senssoft/elasticsearch:6.2.2
+ container_name: senssoft-loadbalancer
+ environment:
+ - NODE_MASTER=false
+ - NUMBER_OF_MASTERS=1
+ - NODE_COORDINATE=true
+ - "DISCOVERY_SERVICE=elasticsearch"
+ - NODE_DATA=false
+ - NODE_INGEST=false
+ - MAX_LOCAL_STORAGE_NODES=1
+ - NETWORK_HOST=0.0.0.0
+ - HTTP_ENABLE=true
+ - HTTP_CORS_ENABLE=true
+ - HTTP_CORS_ALLOW_ORIGIN=*
+ - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
+ ports:
+ - "9200:9200"
+ - "9300"
+ networks:
+ - esnet
+
+ # Kibana
+ kibana:
+ build: ./kibana
+ image: senssoft/kibana:6.2.2
+ container_name: senssoft-kibana
+ environment:
+ - "ES_JAVA_OPTS=-Xms2g -Xmx2g"
+ - "ELASTICSEARCH_URL=http://loadbalancer:9200"
+ ports:
+ - "5601:5601"
+ depends_on:
+ - "loadbalancer"
+ networks:
+ - esnet
+
+ # Logstash
+ logstash:
+ build: ./logstash
+ image: senssoft/logstash:6.2.2
+ container_name: senssoft-logstash
+ environment:
+ - "ELASTICSEARCH_URL=loadbalancer:9200"
+ ports:
+ - 8100:8100
+ volumes:
+ - ./logstash/pipeline/logstash-userale.conf:/usr/share/logstash/pipeline/logstash-userale.conf
+ - ./logstash/templates/userale.json:/usr/share/logstash/templates/userale.json
+ depends_on:
+ - "loadbalancer"
+ networks:
+ - esnet
+
+ # Distill
+ distill:
+ build: ./distill
+ image: senssoft/distill:0.1.0
+ container_name: senssoft-distill
+ ports:
+ - 8090:8090
+ depends_on:
+ - "loadbalancer"
+ networks:
+ - esnet
+
+ # TAP's Database
+ db:
+ build: ./db
+ image: senssoft/postgres:latest
+ container_name: senssoft-postgres
+ ports:
+ - "5432:5432"
+
+ # TAP
+ tap:
+ build: ./tap
+ image: senssoft/tap:0.1.0
+ container_name: senssoft-tap
+ # command: ["bash", "./wait-for-postgres.sh"]
+ command: bash -c "/usr/src/app/wait-for-postgres.sh"
+ ports:
+ - "8010:8010"
+ depends_on:
+ - db
+ links:
+ - db:db
+
+networks:
+ esnet:
+ driver: bridge
http://git-wip-us.apache.org/repos/asf/incubator-senssoft/blob/66639ab5/docker/docker-compose.yml
----------------------------------------------------------------------
diff --git a/docker/docker-compose.yml b/docker/docker-compose.yml
index d2b5ca6..25cc8e1 100644
--- a/docker/docker-compose.yml
+++ b/docker/docker-compose.yml
@@ -13,7 +13,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-# Apache SensSoft Docker Build
+# Apache SensSoft Single-Node Build
version: "3"
services:
@@ -21,16 +21,16 @@ services:
# Startup
startup:
build: ./startup
- image: senssoft.incubator.apache.org/startup:latest
+ image: senssoft/startup:latest
container_name: senssoft-start
depends_on:
- "loadbalancer"
command: bash /opt/entrypoint.sh
- # Site
+ # # Site
site:
build: ./site
- image: senssoft.incubator.apache.org/website:latest
+ image: senssoft/website:latest
container_name: senssoft-app
ports:
- 8080:8080
@@ -38,43 +38,11 @@ services:
- "logstash"
command: python -m SimpleHTTPServer 8080
- # Elasticsearch Workers
- elasticsearch:
- build: ./elasticsearch
- image: senssoft.incubator.apache.org/elasticsearch:5.6.3
- command:
- - /bin/sh
- - -c
- - |
- elasticsearch-plugin remove --silent x-pack
- elasticsearch
- environment:
- - NODE_MASTER=true
- - NUMBER_OF_MASTERS=2
- - NODE_COORDINATE=true
- - "DISCOVERY_SERVICE=elasticsearch"
- - NODE_DATA=true
- - NODE_INGEST=true
- - MAX_LOCAL_STORAGE_NODES=1
- - NETWORK_HOST=0.0.0.0
- - HTTP_ENABLE=false
- - HTTP_CORS_ENABLE=false
- - HTTP_CORS_ALLOW_ORIGIN=*
- - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
- networks:
- - esnet
-
- # Elasticsearch Loadbalancer
+ # Elasticsearch Loadbalancer/Entry
loadbalancer:
build: ./elasticsearch
- image: senssoft.incubator.apache.org/elasticsearch:5.6.3
+ image: senssoft/elasticsearch:6.2.2
container_name: senssoft-loadbalancer
- command:
- - /bin/sh
- - -c
- - |
- elasticsearch-plugin remove --silent x-pack
- elasticsearch
environment:
- NODE_MASTER=true
- NUMBER_OF_MASTERS=1
@@ -97,10 +65,11 @@ services:
# Kibana
kibana:
build: ./kibana
- image: senssoft.incubator.apache.org/kibana:5.6.3
+ image: senssoft/kibana:6.2.2
container_name: senssoft-kibana
environment:
- "ES_JAVA_OPTS=-Xms2g -Xmx2g"
+ - "ELASTICSEARCH_URL=http://loadbalancer:9200"
ports:
- "5601:5601"
depends_on:
@@ -111,56 +80,54 @@ services:
# Logstash
logstash:
build: ./logstash
- image: senssoft.incubator.apache.org/logstash:5.6.3
+ image: senssoft/logstash:6.2.2
container_name: senssoft-logstash
environment:
- "ELASTICSEARCH_URL=loadbalancer:9200"
- command: -f /etc/logstash/conf.d
ports:
- 8100:8100
volumes:
- - ./logstash/config/logstash-userale.conf:/etc/logstash/conf.d/logstash-userale.conf
+ - ./logstash/pipeline/logstash-userale.conf:/usr/share/logstash/pipeline/logstash-userale.conf
- ./logstash/templates/userale.json:/usr/share/logstash/templates/userale.json
depends_on:
- "loadbalancer"
networks:
- esnet
- # Distill
+ # Distill
distill:
- build: ./distill
- image: senssoft.incubator.apache.org/distill:0.1.0
- container_name: senssoft-distill
- ports:
- - 8090:8090
- depends_on:
- - "loadbalancer"
- networks:
- - esnet
+ build: ./distill
+ image: senssoft/distill:0.1.0
+ container_name: senssoft-distill
+ ports:
+ - 8090:8090
+ depends_on:
+ - "loadbalancer"
+ networks:
+ - esnet
- # TAP's Database
+ # TAP's Database
db:
- build: ./db
- image: senssoft.incubator.apache.org/postgres:latest
- container_name: senssoft-postgres
- ports:
- - "5432:5432"
+ build: ./db
+ image: senssoft/postgres:latest
+ container_name: senssoft-postgres
+ ports:
+ - "5432:5432"
# TAP
tap:
- build: ./tap
- image: senssoft.incubator.apache.org/tap:0.1.0
- container_name: senssoft-tap
- # command: ["bash", "./wait-for-postgres.sh"]
- command: bash -c "/usr/src/app/wait-for-postgres.sh"
- ports:
- - "8010:8010"
- depends_on:
- - db
- links:
- - db:db
+ build: ./tap
+ image: senssoft/tap:0.1.0
+ container_name: senssoft-tap
+ # command: ["bash", "./wait-for-postgres.sh"]
+ command: bash -c "/usr/src/app/wait-for-postgres.sh"
+ ports:
+ - "8010:8010"
+ depends_on:
+ - db
+ links:
+ - db:db
networks:
-
esnet:
driver: bridge
http://git-wip-us.apache.org/repos/asf/incubator-senssoft/blob/66639ab5/docker/elasticsearch/Dockerfile
----------------------------------------------------------------------
diff --git a/docker/elasticsearch/Dockerfile b/docker/elasticsearch/Dockerfile
index 6256564..f1fe259 100644
--- a/docker/elasticsearch/Dockerfile
+++ b/docker/elasticsearch/Dockerfile
@@ -13,12 +13,13 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-FROM docker.elastic.co/elasticsearch/elasticsearch:5.6.3
+FROM docker.elastic.co/elasticsearch/elasticsearch-oss:6.2.2
MAINTAINER Michelle Beard <ms...@apache.org>
# Override config
COPY config /usr/share/elasticsearch/config
+RUN chown -R elasticsearch:elasticsearch /usr/share/elasticsearch/config
# Set environment
ENV CLUSTER_NAME SensSoft
@@ -26,7 +27,4 @@ ENV CLUSTER_NAME SensSoft
# Kubernetes requires swap is turned off
ENV MEMORY_LOCK false
-# DELETE XPACK Plugin
-RUN elasticsearch-plugin remove x-pack --silent
-
EXPOSE 9200
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-senssoft/blob/66639ab5/docker/kibana/Dockerfile
----------------------------------------------------------------------
diff --git a/docker/kibana/Dockerfile b/docker/kibana/Dockerfile
index 91027a5..630fe98 100644
--- a/docker/kibana/Dockerfile
+++ b/docker/kibana/Dockerfile
@@ -13,13 +13,8 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-FROM docker.elastic.co/kibana/kibana:5.6.3
+FROM docker.elastic.co/kibana/kibana-oss:6.2.2
MAINTAINER Michelle Beard <ms...@apache.org>
-# Override config
-COPY config /usr/share/kibana/config
-
-RUN kibana-plugin remove x-pack
-
-EXPOSE 5601
\ No newline at end of file
+EXPOSE 5601
http://git-wip-us.apache.org/repos/asf/incubator-senssoft/blob/66639ab5/docker/logstash/Dockerfile
----------------------------------------------------------------------
diff --git a/docker/logstash/Dockerfile b/docker/logstash/Dockerfile
index fb46e5c..d041e48 100644
--- a/docker/logstash/Dockerfile
+++ b/docker/logstash/Dockerfile
@@ -13,18 +13,19 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-FROM docker.elastic.co/logstash/logstash:5.6.3
+FROM docker.elastic.co/logstash/logstash-oss:6.2.2
MAINTAINER Michelle Beard <ms...@apache.org>
-RUN mkdir /usr/share/logstash/templates
-
-RUN logstash-plugin remove x-pack
+# Ensure default logstash configuration is removed
+RUN rm -f /usr/share/logstash/pipeline/logstash.conf
-# Known issue: https://github.com/elastic/logstash-docker/issues/45
-RUN sed -i '/xpack/d' /usr/share/logstash/config/logstash.yml
+# Copy Pipeline files over
+ADD pipeline/ /usr/share/logstash/pipeline/
-# Copy Configuration files over
-COPY config/logstash-userale.conf /etc/logstash/conf.d/logstash-userale.conf
+# Copy templates over
+RUN mkdir /usr/share/logstash/templates
COPY templates/userale.json /usr/share/logstash/templates/userale.json
-EXPOSE 8100
\ No newline at end of file
+EXPOSE 8100
+
+CMD ["logstash"]
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-senssoft/blob/66639ab5/docker/logstash/config/logstash-userale.conf
----------------------------------------------------------------------
diff --git a/docker/logstash/config/logstash-userale.conf b/docker/logstash/config/logstash-userale.conf
deleted file mode 100644
index 4af6ee7..0000000
--- a/docker/logstash/config/logstash-userale.conf
+++ /dev/null
@@ -1,53 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-input {
- http {
- codec => "json"
- port => 8100
- response_headers => {
- "Access-Control-Allow-Origin" => "*"
- "Access-Control-Allow-Headers" => "Origin, X-Requested-With, Content-Type, Accept"
- }
- }
-}
-
-## Add your filters / logstash plugins configuration here
-filter {
- mutate {
- remove_field => [ "message", "@version", "headers"]
- }
-
- grok {
- match => [ "useraleVersion", "(?<major_ver>\d+).(?<minor_ver>\d+)(.(?<patch_ver>\d+))?" ]
- match => [ "toolVersion", "(?<tool_major_ver>\d+).(?<tool_minor_ver>\d+)(.(?<tool_patch_ver>\d+))?" ]
- }
-}
-
-output {
- # Output data to Elasticsearch instance
- elasticsearch {
- hosts => "${ELASTICSEARCH_URL}"
- index => "userale"
- document_type => "logs"
- manage_template => true
- template_overwrite => true
- template => "/usr/share/logstash/templates/userale.json"
- template_name => "userale"
- }
-
- # Debug
- # stdout { codec => rubydebug }
-}
http://git-wip-us.apache.org/repos/asf/incubator-senssoft/blob/66639ab5/docker/logstash/pipeline/logstash-userale.conf
----------------------------------------------------------------------
diff --git a/docker/logstash/pipeline/logstash-userale.conf b/docker/logstash/pipeline/logstash-userale.conf
new file mode 100644
index 0000000..8b4313f
--- /dev/null
+++ b/docker/logstash/pipeline/logstash-userale.conf
@@ -0,0 +1,52 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+input {
+ http {
+ codec => "json"
+ port => 8100
+ response_headers => {
+ "Access-Control-Allow-Origin" => "*"
+ "Access-Control-Allow-Headers" => "Origin, X-Requested-With, Content-Type, Accept"
+ }
+ }
+}
+
+## Add your filters / logstash plugins configuration here
+filter {
+ mutate {
+ remove_field => [ "message", "@version", "headers"]
+ }
+
+ grok {
+ match => [ "useraleVersion", "(?<major_ver>\d+).(?<minor_ver>\d+)(.(?<patch_ver>\d+))?" ]
+ match => [ "toolVersion", "(?<tool_major_ver>\d+).(?<tool_minor_ver>\d+)(.(?<tool_patch_ver>\d+))?" ]
+ }
+}
+
+output {
+ # Output data to Elasticsearch instance
+ elasticsearch {
+ hosts => "${ELASTICSEARCH_URL}"
+ index => "userale"
+ manage_template => true
+ template_overwrite => true
+ template => "/usr/share/logstash/templates/userale.json"
+ template_name => "userale"
+ }
+
+ # Debug
+ stdout { codec => rubydebug }
+}
http://git-wip-us.apache.org/repos/asf/incubator-senssoft/blob/66639ab5/docker/logstash/templates/userale.json
----------------------------------------------------------------------
diff --git a/docker/logstash/templates/userale.json b/docker/logstash/templates/userale.json
index c1bdbb1..5aa92d0 100644
--- a/docker/logstash/templates/userale.json
+++ b/docker/logstash/templates/userale.json
@@ -1,5 +1,5 @@
{
- "template": "userale",
+ "index_patterns": "userale",
"order": 1,
"settings":
{
@@ -21,14 +21,7 @@
},
"mappings":
{
- "_default_":
- {
- "_all":
- {
- "enabled": false
- }
- },
- "logs":
+ "doc":
{
"dynamic_templates": [
{
[2/4] incubator-senssoft git commit: Upgraded Kubernetes build to
support ELK 6.2.2. Added Kibana and Logstash.
Posted by ms...@apache.org.
Upgraded Kubernetes build to support ELK 6.2.2. Added Kibana and Logstash.
Project: http://git-wip-us.apache.org/repos/asf/incubator-senssoft/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-senssoft/commit/ea05491f
Tree: http://git-wip-us.apache.org/repos/asf/incubator-senssoft/tree/ea05491f
Diff: http://git-wip-us.apache.org/repos/asf/incubator-senssoft/diff/ea05491f
Branch: refs/heads/kubernetes
Commit: ea05491fd0996478f31aacdbb4b2d76906266284
Parents: 66639ab
Author: msbeard <ms...@apache.org>
Authored: Sun Mar 4 15:36:16 2018 -0500
Committer: msbeard <ms...@apache.org>
Committed: Sun Mar 4 15:36:16 2018 -0500
----------------------------------------------------------------------
kubernetes/README.md | 234 +++++++++++++++--------
kubernetes/elasticsearch/es-client-svc.yaml | 7 +-
kubernetes/elasticsearch/es-client.yaml | 8 +-
kubernetes/elasticsearch/es-data.yaml | 8 +-
kubernetes/elasticsearch/es-master-svc.yaml | 6 +-
kubernetes/elasticsearch/es-master.yaml | 12 +-
kubernetes/kibana/kibana-svc.yaml | 10 +-
kubernetes/kibana/kibana.yaml | 27 ++-
kubernetes/logstash/logstash-svc.yaml | 19 ++
kubernetes/logstash/logstash.yaml | 52 +++++
10 files changed, 268 insertions(+), 115 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-senssoft/blob/ea05491f/kubernetes/README.md
----------------------------------------------------------------------
diff --git a/kubernetes/README.md b/kubernetes/README.md
index db1e8e6..b100fcc 100644
--- a/kubernetes/README.md
+++ b/kubernetes/README.md
@@ -1,7 +1,12 @@
-# Apache SensSoft on top of Kubernetes
+Apache SensSoft on top of Kubernetes
+====================================
-Apache SensSoft would like to thank [Pires] for providing excellent documentation.
-We have modified [Pires] version to fit Apache [SensSoft] requirements.
+Apache SensSoft would like to thank [Pires] for his great documentation on deploying
+ the ELK stack in Kubernetes.
+We have modified [Pires] instructions to fit Apache [SensSoft] requirements and guidelines.
+
+Introduction
+------------
The Software as a Sensor™ ([SensSoft]) Project offers an open-source (ALv2.0) software
tool usability testing platform. It includes a number of components that work together
@@ -14,9 +19,10 @@ and [Kibana] for visualizing and building custom data-viz dashboards.
This guide describes how to build and scale Elasticsearch clusters using Kubernetes.
-Current Elasticsearch version is `5.6.3`.
+Current Elasticsearch version is `6.2.2`.
-## Abstract
+Abstract
+--------
Before we start, one needs to know that Elasticsearch best-practices recommend to separate nodes in three roles:
* `Master` nodes - intended for clustering management only, no data, no HTTP API
@@ -28,7 +34,8 @@ This is enforced throughout this document.
Given this, I'm going to demonstrate how to provision a production grade scenario
consisting of 3 master, 2 client and 2 data nodes.
-##(Very) Important notes
+(Very) Important Notes
+----------------------
* Elasticsearch pods need for an init-container to run in privileged mode, so it
can set some VM options. For that to happen, the `kubelet` should be running with
@@ -43,93 +50,111 @@ args `--allow-privileged`, otherwise the init-container will fail to run.
in each data node container. This is meant to be for the sake of simplicity and
should be adapted according to one's storage needs.
-## Docker image
+Docker Images
+-------------
-This example uses [this pre-built image](https://github.com/apache/incubator-senssoft/tree/master/docker) of the custom SensSoft ELK stack.
+This example uses [this pre-built image](https://hub.docker.com/u/senssoft/) of the custom SensSoft ELK stack.
-## Perquisites (Install and Setup)
+Perquisites (Install and Setup)
+-------------------------------
Ensure that you have [Docker], [kubectl], and [minikube] (optional) installed and started.
-## Deploy Elasticsearch cluster
+Deploy Elasticsearch Cluster
+============================
+
+Rollout Elasticsearch services and pod replicas.
```bash
-kubectl create -f es-master-svc.yaml
-kubectl create -f es-client-svc.yaml
-kubectl create -f es-master.yaml
-kubectl rollout status -f es-master.yaml
-kubectl create -f es-client.yaml
-kubectl rollout status -f es-client.yaml
-kubectl create -f es-data.yaml
-kubectl rollout status -f es-data.yaml
+kubectl create -f elasticsearch/es-master-svc.yaml
+kubectl create -f elasticsearch/es-client-svc.yaml
+kubectl create -f elasticsearch/es-master.yaml
+kubectl rollout status -f elasticsearch/es-master.yaml
+kubectl create -f elasticsearch/es-client.yaml
+kubectl rollout status -f elasticsearch/es-client.yaml
+kubectl create -f elasticsearch/es-data.yaml
+kubectl rollout status -f elasticsearch/es-data.yaml
```
or
```bash
-./configure deploy
+./configure deploy elasticsearch
```
Check one of the Elasticsearch master nodes logs:
```bash
-$ kubectl get svc,deployment,pods -l component=elasticsearch
-NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
-svc/elasticsearch-client-discovery 10.0.0.107 <none> 9200/TCP 8m
-svc/elasticsearch-master-discovery 10.0.0.134 <none> 9300/TCP 9m
+$ kubectl get svc,deployment,pods -l component=elk
+NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
+svc/elasticsearch-discovery 10.103.242.73 <none> 9300/TCP 6m
+svc/elasticsearch-loadbalancer 10.111.110.60 <nodes> 9200:30510/TCP 3m
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
-deploy/es-client 2 2 2 2 4m
+deploy/es-client 2 2 2 2 3m
deploy/es-data 2 2 2 2 2m
-deploy/es-master 3 3 3 3 7m
+deploy/es-master 3 3 3 3 6m
NAME READY STATUS RESTARTS AGE
-po/es-client-1931188858-0pxhg 1/1 Running 0 4m
-po/es-client-1931188858-sk10l 1/1 Running 0 4m
-po/es-data-2969197230-dx13s 1/1 Running 0 2m
-po/es-data-2969197230-tcqgf 1/1 Running 0 2m
-po/es-master-1140560815-4j8xb 1/1 Running 0 7m
-po/es-master-1140560815-cch1r 1/1 Running 0 7m
-po/es-master-1140560815-m8ghc 1/1 Running 0 7m
+po/es-client-cbb74b6fb-tgbmp 1/1 Running 0 3m
+po/es-client-cbb74b6fb-xnp5c 1/1 Running 0 3m
+po/es-data-796d884bfb-fw8w5 1/1 Running 0 2m
+po/es-data-796d884bfb-qs675 1/1 Running 0 2m
+po/es-master-6f79799c8c-9sdtn 1/1 Running 0 6m
+po/es-master-6f79799c8c-g6g9k 1/1 Running 0 6m
+po/es-master-6f79799c8c-w2dcs 1/1 Running 0 6m
```
```bash
-$ kubectl logs po/es-master-1140560815-4j8xb
-[2018-02-21T23:40:24,539][INFO ][o.e.n.Node ] [es-master-1140560815-4j8xb] initializing ...
-[2018-02-21T23:40:24,922][INFO ][o.e.e.NodeEnvironment ] [es-master-1140560815-4j8xb] using [1] data paths, mounts [[/ (overlay)]], net usable_space [13.7gb], net total_space [16.1gb], spins? [possibly], types [overlay]
-[2018-02-21T23:40:24,923][INFO ][o.e.e.NodeEnvironment ] [es-master-1140560815-4j8xb] heap size [247.5mb], compressed ordinary object pointers [true]
-[2018-02-21T23:40:24,925][INFO ][o.e.n.Node ] [es-master-1140560815-4j8xb] node name [es-master-1140560815-4j8xb], node ID [y3hGFzDmQM-65WWRwnuCUg]
-[2018-02-21T23:40:24,930][INFO ][o.e.n.Node ] [es-master-1140560815-4j8xb] version[5.6.3], pid[1], build[1a2f265/2017-10-06T20:33:39.012Z], OS[Linux/4.9.13/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_141/25.141-b16]
-[2018-02-21T23:40:24,931][INFO ][o.e.n.Node ] [es-master-1140560815-4j8xb] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.cgroups.hierarchy.override=/, -Xms256m, -Xmx256m, -Des.path.home=/usr/share/elasticsearch]
-[2018-02-21T23:40:29,547][INFO ][o.e.p.PluginsService ] [es-master-1140560815-4j8xb] loaded module [aggs-matrix-stats]
-[2018-02-21T23:40:29,547][INFO ][o.e.p.PluginsService ] [es-master-1140560815-4j8xb] loaded module [ingest-common]
-[2018-02-21T23:40:29,548][INFO ][o.e.p.PluginsService ] [es-master-1140560815-4j8xb] loaded module [lang-expression]
-[2018-02-21T23:40:29,548][INFO ][o.e.p.PluginsService ] [es-master-1140560815-4j8xb] loaded module [lang-groovy]
-[2018-02-21T23:40:29,548][INFO ][o.e.p.PluginsService ] [es-master-1140560815-4j8xb] loaded module [lang-mustache]
-[2018-02-21T23:40:29,548][INFO ][o.e.p.PluginsService ] [es-master-1140560815-4j8xb] loaded module [lang-painless]
-[2018-02-21T23:40:29,549][INFO ][o.e.p.PluginsService ] [es-master-1140560815-4j8xb] loaded module [parent-join]
-[2018-02-21T23:40:29,550][INFO ][o.e.p.PluginsService ] [es-master-1140560815-4j8xb] loaded module [percolator]
-[2018-02-21T23:40:29,550][INFO ][o.e.p.PluginsService ] [es-master-1140560815-4j8xb] loaded module [reindex]
-[2018-02-21T23:40:29,551][INFO ][o.e.p.PluginsService ] [es-master-1140560815-4j8xb] loaded module [transport-netty3]
-[2018-02-21T23:40:29,551][INFO ][o.e.p.PluginsService ] [es-master-1140560815-4j8xb] loaded module [transport-netty4]
-[2018-02-21T23:40:29,554][INFO ][o.e.p.PluginsService ] [es-master-1140560815-4j8xb] loaded plugin [ingest-geoip]
-[2018-02-21T23:40:29,556][INFO ][o.e.p.PluginsService ] [es-master-1140560815-4j8xb] loaded plugin [ingest-user-agent]
-[2018-02-21T23:40:35,893][INFO ][o.e.d.DiscoveryModule ] [es-master-1140560815-4j8xb] using discovery type [zen]
-[2018-02-21T23:40:37,624][INFO ][o.e.n.Node ] [es-master-1140560815-4j8xb] initialized
-[2018-02-21T23:40:37,628][INFO ][o.e.n.Node ] [es-master-1140560815-4j8xb] starting ...
-[2018-02-21T23:40:38,715][INFO ][o.e.t.TransportService ] [es-master-1140560815-4j8xb] publish_address {172.17.0.5:9300}, bound_addresses {[::]:9300}
-[2018-02-21T23:40:38,752][INFO ][o.e.b.BootstrapChecks ] [es-master-1140560815-4j8xb] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
-[2018-02-21T23:40:42,861][INFO ][o.e.c.s.ClusterService ] [es-master-1140560815-4j8xb] detected_master {es-master-1140560815-m8ghc}{6aFy3IAjQqKnObD6TDuyKg}{tvZZmgLgSxGhjrOHseKdvw}{172.17.0.6}{172.17.0.6:9300}, added {{es-master-1140560815-m8ghc}{6aFy3IAjQqKnObD6TDuyKg}{tvZZmgLgSxGhjrOHseKdvw}{172.17.0.6}{172.17.0.6:9300},{es-master-1140560815-cch1r}{ItzfidgcTWyIPO3enbXxiQ}{-bUB95d2TxSj7sVos1Y_hA}{172.17.0.4}{172.17.0.4:9300},}, reason: zen-disco-receive(from master [master {es-master-1140560815-m8ghc}{6aFy3IAjQqKnObD6TDuyKg}{tvZZmgLgSxGhjrOHseKdvw}{172.17.0.6}{172.17.0.6:9300} committed version [1]])
-[2018-02-21T23:40:42,943][INFO ][o.e.n.Node ] [es-master-1140560815-4j8xb] started
-[2018-02-21T23:41:56,922][INFO ][o.e.c.s.ClusterService ] [es-master-1140560815-4j8xb] added {{es-client-1931188858-sk10l}{DWii7iInRQOFvNI9Uk8wgg}{E1gZ034YQYiD0S3qJ7iIgw}{172.17.0.7}{172.17.0.7:9300},}, reason: zen-disco-receive(from master [master {es-master-1140560815-m8ghc}{6aFy3IAjQqKnObD6TDuyKg}{tvZZmgLgSxGhjrOHseKdvw}{172.17.0.6}{172.17.0.6:9300} committed version [3]])
-[2018-02-21T23:41:57,432][INFO ][o.e.c.s.ClusterService ] [es-master-1140560815-4j8xb] added {{es-client-1931188858-0pxhg}{hKCgO7VQTSOydiUonCemEg}{ICdFKL_OQjqF8Vff00ymXw}{172.17.0.8}{172.17.0.8:9300},}, reason: zen-disco-receive(from master [master {es-master-1140560815-m8ghc}{6aFy3IAjQqKnObD6TDuyKg}{tvZZmgLgSxGhjrOHseKdvw}{172.17.0.6}{172.17.0.6:9300} committed version [4]])
-[2018-02-21T23:43:18,817][INFO ][o.e.c.s.ClusterService ] [es-master-1140560815-4j8xb] added {{es-data-2969197230-dx13s}{kZtdA7uETyKWtMKGQyheBQ}{gv093zE9R2GgLdDuDvpN1A}{172.17.0.9}{172.17.0.9:9300},}, reason: zen-disco-receive(from master [master {es-master-1140560815-m8ghc}{6aFy3IAjQqKnObD6TDuyKg}{tvZZmgLgSxGhjrOHseKdvw}{172.17.0.6}{172.17.0.6:9300} committed version [5]])
-[2018-02-21T23:43:19,899][INFO ][o.e.c.s.ClusterService ] [es-master-1140560815-4j8xb] added {{es-data-2969197230-tcqgf}{UmHRVW1MTNq5uA6nng8Nmg}{-DzJM2KRRwqkq7Hmm8oXjQ}{172.17.0.10}{172.17.0.10:9300},}, reason: zen-disco-receive(from master [master {es-master-1140560815-m8ghc}{6aFy3IAjQqKnObD6TDuyKg}{tvZZmgLgSxGhjrOHseKdvw}{172.17.0.6}{172.17.0.6:9300} committed version [6]])
+$ kubectl logs po/es-master-6f79799c8c-9sdtn
+[2018-03-04T19:59:22,520][INFO ][o.e.n.Node ] [es-master-6f79799c8c-9sdtn] initializing ...
+[2018-03-04T19:59:22,886][INFO ][o.e.e.NodeEnvironment ] [es-master-6f79799c8c-9sdtn] using [1] data paths, mounts [[/ (overlay)]], net usable_space [14gb], net total_space [16.1gb], types [overlay]
+[2018-03-04T19:59:22,889][INFO ][o.e.e.NodeEnvironment ] [es-master-6f79799c8c-9sdtn] heap size [247.5mb], compressed ordinary object pointers [true]
+[2018-03-04T19:59:22,902][INFO ][o.e.n.Node ] [es-master-6f79799c8c-9sdtn] node name [es-master-6f79799c8c-9sdtn], node ID [1720vLNASnmbPxwyElXeKQ]
+[2018-03-04T19:59:22,904][INFO ][o.e.n.Node ] [es-master-6f79799c8c-9sdtn] version[6.2.2], pid[1], build[10b1edd/2018-02-16T19:01:30.685723Z], OS[Linux/4.9.64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_161/25.161-b14]
+[2018-03-04T19:59:22,905][INFO ][o.e.n.Node ] [es-master-6f79799c8c-9sdtn] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch.F63IAHVh, -XX:+HeapDumpOnOutOfMemoryError, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:logs/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Des.cgroups.hierarchy.override=/, -Xms256m, -Xmx256m, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config]
+[2018-03-04T19:59:27,519][INFO ][o.e.p.PluginsService ] [es-master-6f79799c8c-9sdtn] loaded module [aggs-matrix-stats]
+[2018-03-04T19:59:27,530][INFO ][o.e.p.PluginsService ] [es-master-6f79799c8c-9sdtn] loaded module [analysis-common]
+[2018-03-04T19:59:27,531][INFO ][o.e.p.PluginsService ] [es-master-6f79799c8c-9sdtn] loaded module [ingest-common]
+[2018-03-04T19:59:27,533][INFO ][o.e.p.PluginsService ] [es-master-6f79799c8c-9sdtn] loaded module [lang-expression]
+[2018-03-04T19:59:27,534][INFO ][o.e.p.PluginsService ] [es-master-6f79799c8c-9sdtn] loaded module [lang-mustache]
+[2018-03-04T19:59:27,534][INFO ][o.e.p.PluginsService ] [es-master-6f79799c8c-9sdtn] loaded module [lang-painless]
+[2018-03-04T19:59:27,534][INFO ][o.e.p.PluginsService ] [es-master-6f79799c8c-9sdtn] loaded module [mapper-extras]
+[2018-03-04T19:59:27,540][INFO ][o.e.p.PluginsService ] [es-master-6f79799c8c-9sdtn] loaded module [parent-join]
+[2018-03-04T19:59:27,551][INFO ][o.e.p.PluginsService ] [es-master-6f79799c8c-9sdtn] loaded module [percolator]
+[2018-03-04T19:59:27,552][INFO ][o.e.p.PluginsService ] [es-master-6f79799c8c-9sdtn] loaded module [rank-eval]
+[2018-03-04T19:59:27,552][INFO ][o.e.p.PluginsService ] [es-master-6f79799c8c-9sdtn] loaded module [reindex]
+[2018-03-04T19:59:27,553][INFO ][o.e.p.PluginsService ] [es-master-6f79799c8c-9sdtn] loaded module [repository-url]
+[2018-03-04T19:59:27,555][INFO ][o.e.p.PluginsService ] [es-master-6f79799c8c-9sdtn] loaded module [transport-netty4]
+[2018-03-04T19:59:27,561][INFO ][o.e.p.PluginsService ] [es-master-6f79799c8c-9sdtn] loaded module [tribe]
+[2018-03-04T19:59:27,565][INFO ][o.e.p.PluginsService ] [es-master-6f79799c8c-9sdtn] loaded plugin [ingest-geoip]
+[2018-03-04T19:59:27,569][INFO ][o.e.p.PluginsService ] [es-master-6f79799c8c-9sdtn] loaded plugin [ingest-user-agent]
+[2018-03-04T19:59:37,641][INFO ][o.e.d.DiscoveryModule ] [es-master-6f79799c8c-9sdtn] using discovery type [zen]
+[2018-03-04T19:59:40,865][INFO ][o.e.n.Node ] [es-master-6f79799c8c-9sdtn] initialized
+[2018-03-04T19:59:40,869][INFO ][o.e.n.Node ] [es-master-6f79799c8c-9sdtn] starting ...
+[2018-03-04T19:59:42,001][INFO ][o.e.t.TransportService ] [es-master-6f79799c8c-9sdtn] publish_address {172.17.0.4:9300}, bound_addresses {0.0.0.0:9300}
+[2018-03-04T19:59:42,079][INFO ][o.e.b.BootstrapChecks ] [es-master-6f79799c8c-9sdtn] bound or publishing to a non-loopback address, enforcing bootstrap checks
+[2018-03-04T19:59:45,225][WARN ][o.e.d.z.ZenDiscovery ] [es-master-6f79799c8c-9sdtn] not enough master nodes discovered during pinging (found [[Candidate{node={es-master-6f79799c8c-9sdtn}{1720vLNASnmbPxwyElXeKQ}{S-_neLjjSPWj4ItZ1mv0zQ}{172.17.0.4}{172.17.0.4:9300}, clusterStateVersion=-1}]], but needed [2]), pinging again
+[2018-03-04T19:59:48,273][INFO ][o.e.c.s.MasterService ] [es-master-6f79799c8c-9sdtn] zen-disco-elected-as-master ([1] nodes joined)[{es-master-6f79799c8c-g6g9k}{9IQhhVe0Sj6reW9pPh0FsA}{FQzU-VNtRlexOnRaK9R2mQ}{172.17.0.6}{172.17.0.6:9300}], reason: new_master {es-master-6f79799c8c-9sdtn}{1720vLNASnmbPxwyElXeKQ}{S-_neLjjSPWj4ItZ1mv0zQ}{172.17.0.4}{172.17.0.4:9300}, added {{es-master-6f79799c8c-g6g9k}{9IQhhVe0Sj6reW9pPh0FsA}{FQzU-VNtRlexOnRaK9R2mQ}{172.17.0.6}{172.17.0.6:9300},}
+[2018-03-04T19:59:48,329][INFO ][o.e.c.s.ClusterApplierService] [es-master-6f79799c8c-9sdtn] new_master {es-master-6f79799c8c-9sdtn}{1720vLNASnmbPxwyElXeKQ}{S-_neLjjSPWj4ItZ1mv0zQ}{172.17.0.4}{172.17.0.4:9300}, added {{es-master-6f79799c8c-g6g9k}{9IQhhVe0Sj6reW9pPh0FsA}{FQzU-VNtRlexOnRaK9R2mQ}{172.17.0.6}{172.17.0.6:9300},}, reason: apply cluster state (from master [master {es-master-6f79799c8c-9sdtn}{1720vLNASnmbPxwyElXeKQ}{S-_neLjjSPWj4ItZ1mv0zQ}{172.17.0.4}{172.17.0.4:9300} committed version [1] source [zen-disco-elected-as-master ([1] nodes joined)[{es-master-6f79799c8c-g6g9k}{9IQhhVe0Sj6reW9pPh0FsA}{FQzU-VNtRlexOnRaK9R2mQ}{172.17.0.6}{172.17.0.6:9300}]]])
+[2018-03-04T19:59:48,349][INFO ][o.e.n.Node ] [es-master-6f79799c8c-9sdtn] started
+[2018-03-04T19:59:48,437][INFO ][o.e.g.GatewayService ] [es-master-6f79799c8c-9sdtn] recovered [0] indices into cluster_state
+[2018-03-04T19:59:50,789][INFO ][o.e.c.s.MasterService ] [es-master-6f79799c8c-9sdtn] zen-disco-node-join[{es-master-6f79799c8c-w2dcs}{WAGDODrFRQOb_G9kh4MdBw}{vnfsSU2BQQKOYPB6S8tlHg}{172.17.0.5}{172.17.0.5:9300}], reason: added {{es-master-6f79799c8c-w2dcs}{WAGDODrFRQOb_G9kh4MdBw}{vnfsSU2BQQKOYPB6S8tlHg}{172.17.0.5}{172.17.0.5:9300},}
+[2018-03-04T19:59:50,880][INFO ][o.e.c.s.ClusterApplierService] [es-master-6f79799c8c-9sdtn] added {{es-master-6f79799c8c-w2dcs}{WAGDODrFRQOb_G9kh4MdBw}{vnfsSU2BQQKOYPB6S8tlHg}{172.17.0.5}{172.17.0.5:9300},}, reason: apply cluster state (from master [master {es-master-6f79799c8c-9sdtn}{1720vLNASnmbPxwyElXeKQ}{S-_neLjjSPWj4ItZ1mv0zQ}{172.17.0.4}{172.17.0.4:9300} committed version [3] source [zen-disco-node-join[{es-master-6f79799c8c-w2dcs}{WAGDODrFRQOb_G9kh4MdBw}{vnfsSU2BQQKOYPB6S8tlHg}{172.17.0.5}{172.17.0.5:9300}]]])
+[2018-03-04T20:01:07,315][INFO ][o.e.c.s.MasterService ] [es-master-6f79799c8c-9sdtn] zen-disco-node-join[{es-client-cbb74b6fb-tgbmp}{EunRsql8Q9CqSWJf0FKQbQ}{32GSGju1QbaYe30kTbG_5Q}{172.17.0.8}{172.17.0.8:9300}], reason: added {{es-client-cbb74b6fb-tgbmp}{EunRsql8Q9CqSWJf0FKQbQ}{32GSGju1QbaYe30kTbG_5Q}{172.17.0.8}{172.17.0.8:9300},}
+[2018-03-04T20:01:07,751][INFO ][o.e.c.s.ClusterApplierService] [es-master-6f79799c8c-9sdtn] added {{es-client-cbb74b6fb-tgbmp}{EunRsql8Q9CqSWJf0FKQbQ}{32GSGju1QbaYe30kTbG_5Q}{172.17.0.8}{172.17.0.8:9300},}, reason: apply cluster state (from master [master {es-master-6f79799c8c-9sdtn}{1720vLNASnmbPxwyElXeKQ}{S-_neLjjSPWj4ItZ1mv0zQ}{172.17.0.4}{172.17.0.4:9300} committed version [4] source [zen-disco-node-join[{es-client-cbb74b6fb-tgbmp}{EunRsql8Q9CqSWJf0FKQbQ}{32GSGju1QbaYe30kTbG_5Q}{172.17.0.8}{172.17.0.8:9300}]]])
+[2018-03-04T20:01:07,822][INFO ][o.e.c.s.MasterService ] [es-master-6f79799c8c-9sdtn] zen-disco-node-join[{es-client-cbb74b6fb-xnp5c}{SP4_qYhURHuQipeiiuC35w}{ISw5yvwPTUKMIFMvpMYXMA}{172.17.0.7}{172.17.0.7:9300}], reason: added {{es-client-cbb74b6fb-xnp5c}{SP4_qYhURHuQipeiiuC35w}{ISw5yvwPTUKMIFMvpMYXMA}{172.17.0.7}{172.17.0.7:9300},}
+[2018-03-04T20:01:08,133][INFO ][o.e.c.s.ClusterApplierService] [es-master-6f79799c8c-9sdtn] added {{es-client-cbb74b6fb-xnp5c}{SP4_qYhURHuQipeiiuC35w}{ISw5yvwPTUKMIFMvpMYXMA}{172.17.0.7}{172.17.0.7:9300},}, reason: apply cluster state (from master [master {es-master-6f79799c8c-9sdtn}{1720vLNASnmbPxwyElXeKQ}{S-_neLjjSPWj4ItZ1mv0zQ}{172.17.0.4}{172.17.0.4:9300} committed version [5] source [zen-disco-node-join[{es-client-cbb74b6fb-xnp5c}{SP4_qYhURHuQipeiiuC35w}{ISw5yvwPTUKMIFMvpMYXMA}{172.17.0.7}{172.17.0.7:9300}]]])
+[2018-03-04T20:02:16,606][INFO ][o.e.c.s.MasterService ] [es-master-6f79799c8c-9sdtn] zen-disco-node-join[{es-data-796d884bfb-qs675}{ZOopEPppSrGGk1f2uB6_Pg}{M7ZPAbbxQ0mnOpRZjNtq0A}{172.17.0.10}{172.17.0.10:9300}], reason: added {{es-data-796d884bfb-qs675}{ZOopEPppSrGGk1f2uB6_Pg}{M7ZPAbbxQ0mnOpRZjNtq0A}{172.17.0.10}{172.17.0.10:9300},}
+[2018-03-04T20:02:17,310][INFO ][o.e.c.s.ClusterApplierService] [es-master-6f79799c8c-9sdtn] added {{es-data-796d884bfb-qs675}{ZOopEPppSrGGk1f2uB6_Pg}{M7ZPAbbxQ0mnOpRZjNtq0A}{172.17.0.10}{172.17.0.10:9300},}, reason: apply cluster state (from master [master {es-master-6f79799c8c-9sdtn}{1720vLNASnmbPxwyElXeKQ}{S-_neLjjSPWj4ItZ1mv0zQ}{172.17.0.4}{172.17.0.4:9300} committed version [6] source [zen-disco-node-join[{es-data-796d884bfb-qs675}{ZOopEPppSrGGk1f2uB6_Pg}{M7ZPAbbxQ0mnOpRZjNtq0A}{172.17.0.10}{172.17.0.10:9300}]]])
+[2018-03-04T20:02:17,494][INFO ][o.e.c.s.MasterService ] [es-master-6f79799c8c-9sdtn] zen-disco-node-join[{es-data-796d884bfb-fw8w5}{GAMmdne1Q02pb0tujxGMTQ}{DsiN0UeYQQa9Gzeg_B3ARw}{172.17.0.9}{172.17.0.9:9300}], reason: added {{es-data-796d884bfb-fw8w5}{GAMmdne1Q02pb0tujxGMTQ}{DsiN0UeYQQa9Gzeg_B3ARw}{172.17.0.9}{172.17.0.9:9300},}
+[2018-03-04T20:02:17,753][INFO ][o.e.c.s.ClusterApplierService] [es-master-6f79799c8c-9sdtn] added {{es-data-796d884bfb-fw8w5}{GAMmdne1Q02pb0tujxGMTQ}{DsiN0UeYQQa9Gzeg_B3ARw}{172.17.0.9}{172.17.0.9:9300},}, reason: apply cluster state (from master [master {es-master-6f79799c8c-9sdtn}{1720vLNASnmbPxwyElXeKQ}{S-_neLjjSPWj4ItZ1mv0zQ}{172.17.0.4}{172.17.0.4:9300} committed version [7] source [zen-disco-node-join[{es-data-796d884bfb-fw8w5}{GAMmdne1Q02pb0tujxGMTQ}{DsiN0UeYQQa9Gzeg_B3ARw}{172.17.0.9}{172.17.0.9:9300}]]])
```
As you can assert, the cluster is up and running.
-## Access the service
+Access the service
+------------------
Don't forget that services in Kubernetes are only accessible from containers in
the cluster. For different behavior one should configure the creation of an
@@ -141,30 +166,32 @@ balancers, setting the type field to "LoadBalancer" will provision a load
balancer for your Service. You can uncomment the field in es-client-svc.yaml.
```bash
-$ kubectl get svc elasticsearch-client-discovery
-NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
-elasticsearch-client-discovery 10.0.0.107 <none> 9200/TCP 11m
+$ kubectl get svc elasticsearch-loadbalancer
+NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
+elasticsearch-loadbalancer 10.111.110.60 <nodes> 9200:30510/TCP 6m
```
From any host on your cluster (that's running `kube-proxy`), run:
```bash
-$ curl -XGET http://10.0.0.107:9200
+$ kubectl exec -it es-data-796d884bfb-fw8w5 -- /bin/bash
+$ curl -XGET http://10.111.110.60:9200
```
You should see something similar to the following:
-
```json
{
- "name" : "es-client-1931188858-sk10l",
+ "name" : "es-client-cbb74b6fb-tgbmp",
"cluster_name" : "SensSoft",
- "cluster_uuid" : "aFJwv7ngRZWlOHK6muOeLg",
+ "cluster_uuid" : "J7-noNLVQP21dq8MZVnF-w",
"version" : {
- "number" : "5.6.3",
- "build_hash" : "1a2f265",
- "build_date" : "2017-10-06T20:33:39.012Z",
+ "number" : "6.2.2",
+ "build_hash" : "10b1edd",
+ "build_date" : "2018-02-16T19:01:30.685723Z",
"build_snapshot" : false,
- "lucene_version" : "6.6.1"
+ "lucene_version" : "7.2.1",
+ "minimum_wire_compatibility_version" : "5.6.0",
+ "minimum_index_compatibility_version" : "5.0.0"
},
"tagline" : "You Know, for Search"
}
@@ -173,7 +200,7 @@ You should see something similar to the following:
Or if you want to check cluster information:
```
-curl -XGET http://10.0.0.107:9200/_cluster/health?pretty
+curl -XGET http://10.111.110.60:9200/_cluster/health?pretty
```
You should see something similar to the following:
@@ -198,16 +225,61 @@ You should see something similar to the following:
}
```
-### Additional Notes - Internal access
+Deploy Logstash Service
+=======================
+
+Rollout Logstash service
+
+```bash
+$ ./configure deploy logstash
+```
+
+Check status
+```bash
+$ kubectl get svc,deployment,pods -l role=logstash
+NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
+svc/logstash-discovery 10.103.234.11 <nodes> 8100:31010/TCP 3m
+
+NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
+deploy/logstash 1 1 1 1 1m
+
+NAME READY STATUS RESTARTS AGE
+po/logstash-9464b7fd8-dn8gv 1/1 Running 1 1m
+```
+
+Deploy Kibana Service
+=====================
+
+Rollout Kibana service
+
+```bash
+$ ./configure.sh deploy kibana
+```
+
+Check status
+```bash
+$ kubectl get svc,deployment,pods -l role=kibana
+NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
+svc/kibana 10.103.73.161 <nodes> 5601:32422/TCP 1m
+
+NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
+deploy/kibana 1 1 1 1 1m
+
+NAME READY STATUS RESTARTS AGE
+po/kibana-b6f6b7b8f-plvkz 1/1 Running 0 1m
+```
+
+To view Kibana, either log into the VM or find your minikube VM address and
+goto the assigned external port to view Kibana.
+
```bash
-kubectl exec -it es-data-2969197230-tcqgf -- /bin/bash
+$ firefox $(minikube ip):32422
```
## Todo
-1. Upload configuration file instead of using environments
-1. Deprecate LoadBalancer service since that requires deployment on AWS/GAE
-1. Kube-proxy/port forwarding
1. Create new service `app` that will include pods tap, distill, site, db, etc.
+2. Need a better way of checking on the state of Logstash. It needs the most time to provision.
+3. Logstash is very unstable.
[Pires]: https://github.com/pires/kubernetes-elasticsearch-cluster/
[SensSoft]: http://senssoft.incubator.apache.org/
http://git-wip-us.apache.org/repos/asf/incubator-senssoft/blob/ea05491f/kubernetes/elasticsearch/es-client-svc.yaml
----------------------------------------------------------------------
diff --git a/kubernetes/elasticsearch/es-client-svc.yaml b/kubernetes/elasticsearch/es-client-svc.yaml
index 2dd8ba1..0d0622e 100644
--- a/kubernetes/elasticsearch/es-client-svc.yaml
+++ b/kubernetes/elasticsearch/es-client-svc.yaml
@@ -24,17 +24,18 @@
apiVersion: v1
kind: Service
metadata:
- name: elasticsearch-client-discovery
+ name: elasticsearch-loadbalancer
labels:
- component: elasticsearch
+ component: elk
role: client
spec:
selector:
- component: elasticsearch
+ component: elk
role: client
ports:
- name: http
port: 9200
+# targetPort: 9200
protocol: TCP
type: NodePort
# type: Loadbalancer only works with certain providers:
http://git-wip-us.apache.org/repos/asf/incubator-senssoft/blob/ea05491f/kubernetes/elasticsearch/es-client.yaml
----------------------------------------------------------------------
diff --git a/kubernetes/elasticsearch/es-client.yaml b/kubernetes/elasticsearch/es-client.yaml
index cc34822..032ba18 100644
--- a/kubernetes/elasticsearch/es-client.yaml
+++ b/kubernetes/elasticsearch/es-client.yaml
@@ -3,14 +3,14 @@ kind: Deployment
metadata:
name: es-client
labels:
- component: elasticsearch
+ component: elk
role: client
spec:
replicas: 2
template:
metadata:
labels:
- component: elasticsearch
+ component: elk
role: client
spec:
initContainers:
@@ -25,7 +25,7 @@ spec:
privileged: true
containers:
- name: es-client
- image: senssoft/elasticsearch:5.6.3
+ image: senssoft/elasticsearch:6.2.2
imagePullPolicy: IfNotPresent
env:
- name: NAMESPACE
@@ -46,7 +46,7 @@ spec:
- name: HTTP_ENABLE
value: "true"
- name: DISCOVERY_SERVICE
- value: "elasticsearch-master-discovery"
+ value: "elasticsearch-discovery"
- name: "NETWORK_HOST"
value: _site_,_lo_
- name: ES_JAVA_OPTS
http://git-wip-us.apache.org/repos/asf/incubator-senssoft/blob/ea05491f/kubernetes/elasticsearch/es-data.yaml
----------------------------------------------------------------------
diff --git a/kubernetes/elasticsearch/es-data.yaml b/kubernetes/elasticsearch/es-data.yaml
index c7f01a3..afe5617 100644
--- a/kubernetes/elasticsearch/es-data.yaml
+++ b/kubernetes/elasticsearch/es-data.yaml
@@ -3,14 +3,14 @@ kind: Deployment
metadata:
name: es-data
labels:
- component: elasticsearch
+ component: elk
role: data
spec:
replicas: 2
template:
metadata:
labels:
- component: elasticsearch
+ component: elk
role: data
spec:
initContainers:
@@ -25,7 +25,7 @@ spec:
privileged: true
containers:
- name: es-data
- image: senssoft/elasticsearch:5.6.3
+ image: senssoft/elasticsearch:6.2.2
imagePullPolicy: IfNotPresent
env:
- name: NAMESPACE
@@ -44,7 +44,7 @@ spec:
- name: HTTP_ENABLE
value: "false"
- name: DISCOVERY_SERVICE
- value: "elasticsearch-master-discovery"
+ value: "elasticsearch-discovery"
- name: ES_JAVA_OPTS
value: -Xms256m -Xmx256m
ports:
http://git-wip-us.apache.org/repos/asf/incubator-senssoft/blob/ea05491f/kubernetes/elasticsearch/es-master-svc.yaml
----------------------------------------------------------------------
diff --git a/kubernetes/elasticsearch/es-master-svc.yaml b/kubernetes/elasticsearch/es-master-svc.yaml
index 7cf9ab0..0cb1de8 100644
--- a/kubernetes/elasticsearch/es-master-svc.yaml
+++ b/kubernetes/elasticsearch/es-master-svc.yaml
@@ -24,13 +24,13 @@
apiVersion: v1
kind: Service
metadata:
- name: elasticsearch-master-discovery
+ name: elasticsearch-discovery
labels:
- component: elasticsearch
+ component: elk
role: master
spec:
selector:
- component: elasticsearch
+ component: elk
role: master
ports:
- name: transport
http://git-wip-us.apache.org/repos/asf/incubator-senssoft/blob/ea05491f/kubernetes/elasticsearch/es-master.yaml
----------------------------------------------------------------------
diff --git a/kubernetes/elasticsearch/es-master.yaml b/kubernetes/elasticsearch/es-master.yaml
index d04b2e9..592a00c 100644
--- a/kubernetes/elasticsearch/es-master.yaml
+++ b/kubernetes/elasticsearch/es-master.yaml
@@ -3,14 +3,14 @@ kind: Deployment
metadata:
name: es-master
labels:
- component: elasticsearch
+ component: elk
role: master
spec:
replicas: 3
template:
metadata:
labels:
- component: elasticsearch
+ component: elk
role: master
spec:
initContainers:
@@ -25,7 +25,7 @@ spec:
privileged: true
containers:
- name: es-master
- image: senssoft/elasticsearch:5.6.3
+ image: senssoft/elasticsearch:6.2.2
imagePullPolicy: IfNotPresent
env:
- name: NAMESPACE
@@ -46,13 +46,13 @@ spec:
- name: NODE_COORDINATE
value: "false"
- name: NUMBER_OF_MASTERS
- value: "3"
+ value: "2"
- name: HTTP_ENABLE
value: "false"
- name: DISCOVERY_SERVICE
- value: "elasticsearch-master-discovery"
+ value: "elasticsearch-discovery"
- name: DISCOVERY_TIMEOUT
- value: "20s"
+ value: "30s"
- name: ES_JAVA_OPTS
value: -Xms256m -Xmx256m
ports:
http://git-wip-us.apache.org/repos/asf/incubator-senssoft/blob/ea05491f/kubernetes/kibana/kibana-svc.yaml
----------------------------------------------------------------------
diff --git a/kubernetes/kibana/kibana-svc.yaml b/kubernetes/kibana/kibana-svc.yaml
index e60e262..fbefde0 100644
--- a/kubernetes/kibana/kibana-svc.yaml
+++ b/kubernetes/kibana/kibana-svc.yaml
@@ -3,11 +3,13 @@ kind: Service
metadata:
name: kibana
labels:
- component: kibana
+ component: elk
+ role: kibana
spec:
selector:
- component: kibana
+ component: elk
+ role: kibana
ports:
- name: http
- port: 80
- targetPort: http
+ port: 5601
+ type: NodePort
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-senssoft/blob/ea05491f/kubernetes/kibana/kibana.yaml
----------------------------------------------------------------------
diff --git a/kubernetes/kibana/kibana.yaml b/kubernetes/kibana/kibana.yaml
index 7d9366d..f0f1ed0 100644
--- a/kubernetes/kibana/kibana.yaml
+++ b/kubernetes/kibana/kibana.yaml
@@ -3,30 +3,37 @@ kind: Deployment
metadata:
name: kibana
labels:
- component: kibana
+ component: elk
+ role: kibana
spec:
replicas: 1
selector:
matchLabels:
- component: kibana
+ component: elk
+ role: kibana
template:
metadata:
labels:
- component: kibana
+ component: elk
+ role: kibana
spec:
containers:
- name: kibana
- image: docker.elastic.co/kibana/kibana:5.6.3
- env:
- - name: CLUSTER_NAME
- value: SensSoft
- - name: SERVER_BASEPATH
- value: /api/v1/proxy/namespaces/default/services/kibana
+ image: senssoft/kibana:6.2.2
+ imagePullPolicy: IfNotPresent
resources:
+ # need more cpu upon initialization, therefore burstable class
limits:
cpu: 1000m
requests:
cpu: 100m
+ env:
+ - name: CLUSTER_NAME
+ value: SensSoft
+ - name: ELASTICSEARCH_URL
+ value: "http://elasticsearch-loadbalancer:9200"
+ - name: ES_JAVA_OPTS
+ value: "-Xms512m -Xmx512m"
ports:
- containerPort: 5601
- name: http
+ name: http
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-senssoft/blob/ea05491f/kubernetes/logstash/logstash-svc.yaml
----------------------------------------------------------------------
diff --git a/kubernetes/logstash/logstash-svc.yaml b/kubernetes/logstash/logstash-svc.yaml
new file mode 100644
index 0000000..b77a032
--- /dev/null
+++ b/kubernetes/logstash/logstash-svc.yaml
@@ -0,0 +1,19 @@
+apiVersion: v1
+kind: Service
+metadata:
+ name: logstash-discovery
+ labels:
+ component: elk
+ role: logstash
+spec:
+ selector:
+ component: elk
+ role: logstash
+ ports:
+# - name: transport
+# port: 9600
+# protocol: TCP
+ - name: http
+ port: 8100
+ protocol: TCP
+ type: NodePort
http://git-wip-us.apache.org/repos/asf/incubator-senssoft/blob/ea05491f/kubernetes/logstash/logstash.yaml
----------------------------------------------------------------------
diff --git a/kubernetes/logstash/logstash.yaml b/kubernetes/logstash/logstash.yaml
new file mode 100644
index 0000000..8756731
--- /dev/null
+++ b/kubernetes/logstash/logstash.yaml
@@ -0,0 +1,52 @@
+apiVersion: apps/v1beta1
+kind: Deployment
+metadata:
+ name: logstash
+ labels:
+ component: elk
+ role: logstash
+spec:
+ replicas: 1
+ template:
+ metadata:
+ labels:
+ component: elk
+ role: logstash
+ spec:
+ containers:
+ - name: logstash
+ image: senssoft/logstash:6.2.2
+ imagePullPolicy: IfNotPresent
+ command:
+ - logstash
+ - -f
+ - /etc/logstash/conf.d
+ env:
+ - name: ELASTICSEARCH_URL
+ value: "elasticsearch-loadbalancer:9200"
+ - name: ES_JAVA_OPTS
+ value: "-Xms1g -Xmx1g"
+ resources:
+ limits:
+ cpu: 1000m
+ requests:
+ cpu: 100m
+ ports:
+ - containerPort: 8100
+ name: http
+ - containerPort: 9600
+ name: transport
+ # Need better readiness check policy than checking if port 9600 is active
+# livenessProbe:
+# tcpSocket:
+# port: transport
+# initialDelaySeconds: 20
+# periodSeconds: 10
+# failureThreshold: 5
+# readinessProbe:
+# httpGet:
+# path: /?pretty=true
+# port: http
+# initialDelaySeconds: 30
+# periodSeconds: 10
+# failureThreshold: 10
\ No newline at end of file
[3/4] incubator-senssoft git commit: Updated kubernetes bash script
Posted by ms...@apache.org.
Updated kubernetes bash script
Project: http://git-wip-us.apache.org/repos/asf/incubator-senssoft/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-senssoft/commit/4dbc26d7
Tree: http://git-wip-us.apache.org/repos/asf/incubator-senssoft/tree/4dbc26d7
Diff: http://git-wip-us.apache.org/repos/asf/incubator-senssoft/diff/4dbc26d7
Branch: refs/heads/kubernetes
Commit: 4dbc26d70745603cc02538d7e491cb328bb30d08
Parents: ea05491
Author: msbeard <ms...@apache.org>
Authored: Sun Mar 4 15:36:36 2018 -0500
Committer: msbeard <ms...@apache.org>
Committed: Sun Mar 4 15:36:36 2018 -0500
----------------------------------------------------------------------
kubernetes/configure.sh | 163 +++++++++++++++++++++++++++++++------------
1 file changed, 120 insertions(+), 43 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-senssoft/blob/4dbc26d7/kubernetes/configure.sh
----------------------------------------------------------------------
diff --git a/kubernetes/configure.sh b/kubernetes/configure.sh
index ae26dda..e6ae111 100755
--- a/kubernetes/configure.sh
+++ b/kubernetes/configure.sh
@@ -22,29 +22,32 @@
help_usage() {
echo "configure.sh"
echo "A simple utility to deploy Apache SensSoft Kubernetes build."
+ echo "Not meant to be used in production."
echo ""
- echo "Usage: $ configure.sh COMMAND"
+ echo "Usage: $ configure.sh COMMAND [OPT]"
echo ""
help_commands
echo "e.g."
- echo "$ $0 deploy"
+ echo "$ $0 deploy all"
}
# Print out commands.
help_commands() {
echo "The commands are:"
- echo " status View status of Kubernetes deployment"
- echo " deploy Deploy SensSoft stack into Kubernetes cluster"
- echo " elasticsearch Deploy Elasticsearch Kubernetes stack"
- echo " logstash Deploy Logstash Kubernetes stack"
- echo " kibana Deploy Kibana Kubernetes stack"
- echo " elk Deploy ELK Kubernetes stack"
- echo " check Check environment for release"
- echo " shutdown Shutdown Kubernetes cluster"
- echo " purge Purge all Kubernetes artifacts"
- echo " start Startup minikube"
- echo " stop Stop minikube"
- echo " delete Delete minikube"
+ echo " status View status of Kubernetes deployment"
+ echo " deploy elk Deploy ELK stack into Kubernetes cluster"
+ echo " deploy elasticsearch Deploy Elasticsearch Kubernetes"
+ echo " deploy logstash Deploy Logstash Kubernetes"
+ echo " deploy kibana Deploy Kibana Kubernetes"
+ echo " purge elk Purge all ELK Kubernetes artifacts"
+ echo " purge elasticsearch Purge all Elasticsearch Kubernetes artifacts"
+ echo " purge logstash Purge all Logstash Kubernetes artifacts"
+ echo " purge kibana Purge all Kibana Kubernetes artifacts"
+ echo " check Check environment for release"
+ echo " shutdown Shutdown Kubernetes cluster"
+ echo " start Startup minikube"
+ echo " stop Stop minikube"
+ echo " delete Delete minikube"
echo ""
}
@@ -56,13 +59,11 @@ fi
# Check for a command argument.
COMMAND=$1
+COMMAND_OPT=$2
+
if [[ -z $COMMAND ]] || \
[[ $COMMAND != "status" && \
$COMMAND != "deploy" && \
- $COMMAND != "elasticsearch" && \
- $COMMAND != "logstash" && \
- $COMMAND != "kibana" && \
- $COMMAND != "elk" && \
$COMMAND != "check" && \
$COMMAND != "shutdown" && \
$COMMAND != "purge" && \
@@ -75,13 +76,41 @@ if [[ -z $COMMAND ]] || \
exit 1
fi
+# Start minikube w/ hyperkit
+if [[ $COMMAND == "provision" ]]; then
+ minikube start --cpus 2 --memory 5120 --vm-driver=virtualbox
+ # this for loop waits until kubectl can access the api server that Minikube has created
+ for i in {1..150}; do # timeout for 5 minutes
+ ./kubectl get po &> /dev/null
+ if [ $? -ne 1 ]; then
+ break
+ fi
+ sleep 2
+ done
+ exit 0
+fi
+
+# Stop minikube w/ hyperkit
+if [[ $COMMAND == "stop" ]]; then
+ minikube stop
+ eval $(minikube docker-env -u)
+ exit 0
+fi
+
+# Start minikube w/ hyperkit
+if [[ $COMMAND == "delete" ]]; then
+ minikube delete
+ exit 0
+fi
+
# Fetch status of entire Kubernets SensSoft namespace
if [[ $COMMAND == "status" ]]; then
- kubectl get svc,deployment,pods -l component=elasticsearch
+ kubectl get svc,deployment,pods -l component=elk
exit 0
fi
function elasticsearch() {
+ echo "Elasticsearch deployment"
kubectl create -f elasticsearch/es-master-svc.yaml
kubectl create -f elasticsearch/es-client-svc.yaml
kubectl create -f elasticsearch/es-master.yaml
@@ -92,39 +121,89 @@ function elasticsearch() {
kubectl rollout status -f elasticsearch/es-data.yaml
}
-# Deploy entire Kubernetes SensSoft namespace
-if [[ $COMMAND == "deploy" ]]; then
+function logstash() {
+ echo "Logstash deployment"
+ kubectl create -f logstash/logstash-svc.yaml
+ kubectl create -f logstash/logstash-client.yaml
+ kubectl create -f logstash/logstash.yaml
+ kubectl rollout status -f logstash/logstash.yaml
+}
+
+function kibana() {
+ echo "Kibana deployment"
+ kubectl create -f kibana/kibana-svc.yaml
+ kubectl create -f kibana/kibana.yaml
+ kubectl rollout status -f kibana/kibana.yaml
+}
+
+if [[ $COMMAND == "deploy" && \
+ $COMMAND_OPT == "elasticsearch" ]]; then
elasticsearch
- exit 0
+ exit 0
fi
-# Delete entire Kubernetes SensSoft namespace
-if [[ $COMMAND == "purge" ]]; then
- for f in elasticsearch/*.yaml
+if [[ $COMMAND == "deploy" && \
+ $COMMAND_OPT == "logstash" ]]; then
+ logstash
+ exit 0
+fi
+
+if [[ $COMMAND == "deploy" && \
+ $COMMAND_OPT == "kibana" ]]; then
+ kibana
+ exit 0
+fi
+
+if [[ $COMMAND == "deploy" && \
+ $COMMAND_OPT == "elk" ]]; then
+ elasticsearch
+ logstash
+ kibana
+ exit 0
+fi
+
+# Delete entire Elasticsearch Kubernetes artifacts
+if [[ $COMMAND == 'purge' && \
+ $COMMAND_OPT == 'elasticsearch' ]]; then
+ for f in elasticsearch/*.yaml
do
kubectl delete -f $f
done
- exit 0
fi
-# Start minikube w/ hyperkit
-if [[ $COMMAND == "provision" ]]; then
- minikube start --cpus 2 --memory 4096
-
- exit 0
+# Delete entire Logstash Kubernetes artifacts
+if [[ $COMMAND == 'purge' && \
+ $COMMAND_OPT == 'logstash' ]]; then
+ for f in logstash/*.yaml
+ do
+ kubectl delete -f $f
+ done
fi
-# Stop minikube w/ hyperkit
-if [[ $COMMAND == "stop" ]]; then
- minikube stop
- eval $(minikube docker-env -u)
- exit 0
+# Delete entire Kibana Kubernetes artifacts
+if [[ $COMMAND == 'purge' && \
+ $COMMAND_OPT == 'kibana' ]]; then
+ for f in kibana/*.yaml
+ do
+ kubectl delete -f $f
+ done
fi
-# Start minikube w/ hyperkit
-if [[ $COMMAND == "delete" ]]; then
- minikube delete
- exit 0
+# Delete entire Kubernetes SensSoft namespace
+if [[ $COMMAND == 'purge' && \
+ $COMMAND_OPT == 'elk' ]]; then
+ for f in elasticsearch/*.yaml
+ do
+ kubectl delete -f $f
+ done
+ for f in logstash/*.yaml
+ do
+ kubectl delete -f $f
+ done
+ for f in kibana/*.yaml
+ do
+ kubectl delete -f $f
+ done
fi
# Prepare for Distill deployment
@@ -143,6 +222,4 @@ fi
if [[ $COMMAND == "check" ]]; then
echo "Error: Unsupported check build."
exit 1
-fi
-
-exit 0
\ No newline at end of file
+fi
\ No newline at end of file
[4/4] incubator-senssoft git commit: Added Kuberentes instructions
Posted by ms...@apache.org.
Added Kuberentes instructions
Project: http://git-wip-us.apache.org/repos/asf/incubator-senssoft/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-senssoft/commit/bf43699f
Tree: http://git-wip-us.apache.org/repos/asf/incubator-senssoft/tree/bf43699f
Diff: http://git-wip-us.apache.org/repos/asf/incubator-senssoft/diff/bf43699f
Branch: refs/heads/kubernetes
Commit: bf43699ffa49b2baf01c7827defe1ce86436bf7d
Parents: 4dbc26d
Author: msbeard <ms...@apache.org>
Authored: Sun Mar 4 15:36:58 2018 -0500
Committer: msbeard <ms...@apache.org>
Committed: Sun Mar 4 15:36:58 2018 -0500
----------------------------------------------------------------------
README.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-senssoft/blob/bf43699f/README.md
----------------------------------------------------------------------
diff --git a/README.md b/README.md
index c1558ec..37cec1d 100644
--- a/README.md
+++ b/README.md
@@ -1,5 +1,5 @@
SensSoft Docker Containers and Website
--------------------------------------
-Specific build instructions located in docker and site folders.
+Specific build instructions located in docker, kubernetes and site folders.