You are viewing a plain text version of this content. The canonical link for it is here.
Posted to pr@flagon.apache.org by "r2o3k (via GitHub)" <gi...@apache.org> on 2023/12/12 17:44:04 UTC
[PR] Checking in 8.11.1 version of ELK stack for flagon (flagon)
r2o3k opened a new pull request, #42:
URL: https://github.com/apache/flagon/pull/42
(no comment)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@flagon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@flagon.apache.org
For additional commands, e-mail: pr-help@flagon.apache.org
Re: [PR] Checking in 8.11.1 version of ELK stack for flagon (flagon)
Posted by "Jyyjy (via GitHub)" <gi...@apache.org>.
Jyyjy merged PR #42:
URL: https://github.com/apache/flagon/pull/42
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: pr-unsubscribe@flagon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: pr-unsubscribe@flagon.apache.org
For additional commands, e-mail: pr-help@flagon.apache.org
Re: [PR] Checking in 8.11.1 version of ELK stack for flagon (flagon)
Posted by "EandrewJones (via GitHub)" <gi...@apache.org>.
EandrewJones commented on code in PR #42:
URL: https://github.com/apache/flagon/pull/42#discussion_r1438366361
##########
docker/logstash/8.11.1/templates/userale.json:
##########
Review Comment:
Also, I cannot get any of these containers to run. A simple:
```
cd docker
docker network create esnet
docker compose up
```
should work, but I encounter errors or indefinite hangs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@flagon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@flagon.apache.org
For additional commands, e-mail: notifications-help@flagon.apache.org
Re: [PR] Checking in 8.11.1 version of ELK stack for flagon (flagon)
Posted by "EandrewJones (via GitHub)" <gi...@apache.org>.
EandrewJones commented on PR #42:
URL: https://github.com/apache/flagon/pull/42#issuecomment-1875610086
New network:
```
networks:
esnet:
name: esnet
external: true
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@flagon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@flagon.apache.org
For additional commands, e-mail: notifications-help@flagon.apache.org
Re: [PR] Checking in 8.11.1 version of ELK stack for flagon (flagon)
Posted by "Jyyjy (via GitHub)" <gi...@apache.org>.
Jyyjy commented on code in PR #42:
URL: https://github.com/apache/flagon/pull/42#discussion_r1433212051
##########
docker/logstash/8.11.1/templates/userale.json:
##########
Review Comment:
My bad for not catching this before, but this file is giving an error. This is considered a legacy template and as of elastic 7.8, it has to match the new [composable template format](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-templates.html).
2023-12-20 16:32:43 [2023-12-20T21:32:43,741][ERROR][logstash.outputs.elasticsearch][main] Failed to install template {:message=>"Got response code '400' contacting Elasticsearch at URL 'http://elasticsearch:9200/_index_template/userale'", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-elasticsearch-11.19.0-java/lib/logstash/outputs/elasticsearch/http_client/manticore_adapter.rb:84:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-elasticsearch-11.19.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:350:in `perform_request_to_url'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-elasticsearch-11.19.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:336:in `block in perform_request'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-elasticsearch-11.19.0-java/li
b/logstash/outputs/elasticsearch/http_client/pool.rb:424:in `with_connection'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-elasticsearch-11.19.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:335:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-elasticsearch-11.19.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:343:in `block in Pool'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-elasticsearch-11.19.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:419:in `template_put'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-elasticsearch-11.19.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:85:in `template_install'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-elasticsearch-11.19.0-java/lib/logstash/outputs/elasticsearch/template_manager.rb:44:in `install'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logs
tash-output-elasticsearch-11.19.0-java/lib/logstash/outputs/elasticsearch/template_manager.rb:32:in `install_template'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-elasticsearch-11.19.0-java/lib/logstash/outputs/elasticsearch.rb:597:in `install_template'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-elasticsearch-11.19.0-java/lib/logstash/outputs/elasticsearch.rb:354:in `finish_register'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-elasticsearch-11.19.0-java/lib/logstash/outputs/elasticsearch.rb:327:in `block in register'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-elasticsearch-11.19.0-java/lib/logstash/plugin_mixins/elasticsearch/common.rb:172:in `block in after_successful_connection'"], :body=>"{\"error\":{\"root_cause\":[{\"type\":\"x_content_parse_exception\",\"reason\":\"[1:2] [index_template] unknown field [settings]\"}],\"type\":\"x_content_parse_exception\",\"reason\":\"[1:2]
[index_template] unknown field [settings]\"},\"status\":400}"}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@flagon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@flagon.apache.org
For additional commands, e-mail: notifications-help@flagon.apache.org
Re: [PR] Checking in 8.11.1 version of ELK stack for flagon (flagon)
Posted by "Jyyjy (via GitHub)" <gi...@apache.org>.
Jyyjy commented on code in PR #42:
URL: https://github.com/apache/flagon/pull/42#discussion_r1434496039
##########
docker/logstash/8.11.1/templates/userale.json:
##########
Review Comment:
Or you can set template_api => "legacy" in logstash.conf
https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-template_api
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@flagon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@flagon.apache.org
For additional commands, e-mail: notifications-help@flagon.apache.org
Re: [PR] Checking in 8.11.1 version of ELK stack for flagon (flagon)
Posted by "EandrewJones (via GitHub)" <gi...@apache.org>.
EandrewJones commented on code in PR #42:
URL: https://github.com/apache/flagon/pull/42#discussion_r1438319285
##########
docker/logstash/8.11.1/templates/userale.json:
##########
Review Comment:
Let's use the legacy template for now.
> Also, not related to this PR, but is there a reason why each elk stack version is kept in source instead of keeping the latest and relying on git?
The reason is purely convention. But I agree it makes more sense to switch to using git tags for each version and having having the tip of the branch only show latest. However, this would require quite a bit of GIT wrangling for a low value add at this point.
We can add your two asks as tickets in the backlog:
1. migrating the logstash template to the composable template
2. Switch elk stack versioning to git tags, showing only the latest in source.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@flagon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@flagon.apache.org
For additional commands, e-mail: notifications-help@flagon.apache.org
Re: [PR] Checking in 8.11.1 version of ELK stack for flagon (flagon)
Posted by "Jyyjy (via GitHub)" <gi...@apache.org>.
Jyyjy commented on code in PR #42:
URL: https://github.com/apache/flagon/pull/42#discussion_r1433214578
##########
docker/kibana/8.11.1/Saved Objects/Apache Flagon Business Analytics Dashboard.json:
##########
@@ -0,0 +1,18 @@
+[
Review Comment:
After your changes I still can't import these files as saved objects.
`The file could not be processed due to error: "Bad Request: Unexpected end of JSON input"`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@flagon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@flagon.apache.org
For additional commands, e-mail: notifications-help@flagon.apache.org
Re: [PR] Checking in 8.11.1 version of ELK stack for flagon (flagon)
Posted by "Jyyjy (via GitHub)" <gi...@apache.org>.
Jyyjy commented on code in PR #42:
URL: https://github.com/apache/flagon/pull/42#discussion_r1428455155
##########
docker/kibana/8.11.1/Saved Objects/Apache Flagon Business Analytics Dashboard.json:
##########
@@ -0,0 +1,18 @@
+[
Review Comment:
The saved object format was changed, and this are no longer compatible with 8.11. The files in this directory should be .ndjson instead of .json
##########
docker/docker-compose-8.11.1.yml:
##########
@@ -0,0 +1,157 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Apache Flagon Single-Node Build
+
+version: "3"
+services:
+
+ # Startup
+ startup:
+ build: ./startup
+ depends_on:
+ - "elasticsearch"
+ command: bash /opt/entrypoint.sh
+
+ # Site
+ site:
+ build: ./site
+ ports:
+ - 8080:8080
+ depends_on:
+ - "logstash"
+ command: python -m SimpleHTTPServer 8080
+
+ # Elasticsearch
+ elasticsearch:
+ build: ./elasticsearch/8.11.1
+ environment:
+ - INITIAL_MASTER_NODES=elasticsearch
+ - "DISCOVERY_SERVICE=0.0.0.0"
+ - NETWORK_HOST=0.0.0.0
+ - HTTP_CORS_ENABLE=true
+ - HTTP_CORS_ALLOW_ORIGIN=*
+ - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
+ ulimits:
+ memlock:
+ soft: -1
+ hard: -1
+ healthcheck:
+ test: ["CMD", "curl","-s" ,"-f", "-u", "http://localhost:9200/_cluster/health"]
+ # Mount volumes to backup elasticsearch data
+# volumes:
+# - esdata:/usr/share/elasticsaerch/data
+ ports:
+ - "9200:9200"
+ networks:
+ - esnet
+
+ # Kibana
+ kibana:
+ build: ./kibana/8.11.1
+ environment:
+ - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
+ - "ELASTICSEARCH_URL=http://elasticsearch:9200"
+ ports:
+ - "5601:5601"
+ depends_on:
+ - "elasticsearch"
+ networks:
+ - esnet
+
+ # Logstash
+ logstash:
+ build: ./logstash/8.11.1
+ environment:
+ - "ELASTICSEARCH_URL=elasticsearch:9200"
+ ports:
+ - 8100:8100
+ volumes:
+ - ./logstash/8.11.1/pipeline/logstash-userale.conf:/usr/share/logstash/pipeline/logstash-userale.conf
+ - ./logstash/8.11.1/templates/userale.json:/usr/share/logstash/templates/userale.json
+ depends_on:
+ - "elasticsearch"
+ networks:
+ - esnet
+
+ # Distill
+ distill:
+ build: ./distill
+ ports:
+ - 8090:8090
+ depends_on:
+ - "elasticsearch"
+ networks:
+ - esnet
+
+ # TAP's Database
+ db:
+ build: ./db
+ image: senssoft/postgres:latest
+ ports:
+ - "5432:5432"
+
+ # TAP
+ tap:
+ build: ./tap
+ command: bash -c "/usr/src/app/wait-for-postgres.sh"
+ ports:
+ - "8010:8010"
+ depends_on:
+ - db
+ links:
+ - db:db
+
+ # Specific container and configuration to monitor the HOST (filesystem, memory, processes,...)
+ metricbeat:
+ image: docker.elastic.co/beats/metricbeat:8.11.1
+# restart: unless-stopped
+ user: root
+ pid: host
+ volumes:
+ #The commented sections below enable Metricbeat to monitor the Docker host
+ #rather than the Metricbeat container. These are used by the system module.
+ - /proc:/hostfs/proc:ro
+ - /sys/fs/cgroup:/hostfs/sys/fs/cgroup:ro
+ #We mount the host filesystem so we can report on disk usage with the system module
+ - /:/hostfs:ro
+ #Mount the metricbeat configuration so users can make edits
+ - ./beats/metricbeat/metricbeat.yml:/usr/share/metricbeat/metricbeat.yml
+ #Mount the modules.d directory into the container. This allows user to
+ #potentially make changes to the modules and they will be dynamically loaded.
+ - ./beats/metricbeat/modules.d/:/usr/share/metricbeat/modules.d/
+ #Allows us to report on docker from the hosts information
+ - /var/run/docker.sock:/var/run/docker.sock
+ depends_on:
+ - elasticsearch
+ environment:
+ - ELASTICSEARCH_URL=http://elasticsearch:9200
+ - KIBANA_URL=http://kibana:5601
+ - LOGSTASH_URL=http://logstash:9600
+ - POSTGRESQL_URL="postgresql://db:5432/tapdb?sslmode=disable"
+ - POSTGRESQL_USERNAME=tapuser
+ - POSTGRESQL_PASSWORD="Dr@p3rUs3r"
+ extra_hosts:
+ - "elasticsearch:172.17.0.1" # The IP of docker0 interface to access host from container
+ - "kibana:172.17.0.1" # The IP of docker0 interface to access host from container
+ - "logstash:172.17.0.1"
+ - "db:172.17.0.1"
+ command: metricbeat -e -system.hostfs=/hostfs -strict.perms=false
+ network_mode: host # Mandatory to monitor HOST filesystem, memory, processes,...
+
+networks:
Review Comment:
This is giving a warning, and has a simple fix.
`network esnet: network.external.name is deprecated. Please set network.name with external: true`
##########
docker/docker-compose-8.11.1.yml:
##########
@@ -0,0 +1,157 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Apache Flagon Single-Node Build
+
+version: "3"
+services:
+
+ # Startup
+ startup:
+ build: ./startup
+ depends_on:
+ - "elasticsearch"
+ command: bash /opt/entrypoint.sh
+
+ # Site
+ site:
+ build: ./site
+ ports:
+ - 8080:8080
+ depends_on:
+ - "logstash"
+ command: python -m SimpleHTTPServer 8080
Review Comment:
This module was renamed, the command should be `python -m http.server 8080`. For both this docker compose and the default
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@flagon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@flagon.apache.org
For additional commands, e-mail: notifications-help@flagon.apache.org