You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@ambari.apache.org by ol...@apache.org on 2019/01/04 18:30:26 UTC
[ambari-logsearch] branch master updated: AMBARI-23456. Add
developement guide markdowns & remove some old docs (#68)
This is an automated email from the ASF dual-hosted git repository.
oleewere pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/ambari-logsearch.git
The following commit(s) were added to refs/heads/master by this push:
new 1deed01 AMBARI-23456. Add developement guide markdowns & remove some old docs (#68)
1deed01 is described below
commit 1deed01e48015519c3e566762f53e2e5a38bb3eb
Author: Olivér Sz <ol...@gmail.com>
AuthorDate: Fri Jan 4 19:30:22 2019 +0100
AMBARI-23456. Add developement guide markdowns & remove some old docs (#68)
---
README.md | 95 +++++++---------------------
ambari-logsearch-logfeeder/README.md | 49 ---------------
ambari-logsearch-server/README.md | 39 ------------
docs/development.md | 119 ++++++++++++++++++++++++++++++++++-
4 files changed, 139 insertions(+), 163 deletions(-)
diff --git a/README.md b/README.md
index 21b94fc..142587e 100644
--- a/README.md
+++ b/README.md
@@ -2,89 +2,38 @@
[](https://builds.apache.org/view/A/view/Ambari/job/Ambari-LogSearch-master-Commit/)
-Log aggregation, analysis, and visualization for Ambari managed (and any other) services.
-
Log Search is a sub-project of [Apache Ambari](https://github.com/apache/ambari)
-## Development
-
-Requires JDK 8 (JDK 11 is recommended) and Maven 3.5.x
-
-### Prerequisites
-
-- Install [docker](https://docs.docker.com/)
-- For Mac OS X use [Docker for Mac](https://docs.docker.com/docker-for-mac/)
-- [Docker compose](https://docs.docker.com/compose/) is also required.
-
-### Build and start Log Search in docker container
-```bash
-# to see available commands: run start-logsearch without arguments
-cd docker
-./logsearch-docker build-and-run # build mvn project locally, build docker image, start containers
-```
-If you run the script at first time, it will generate you a new `Profile` file or an `.env` file inside docker directory (run twice if both missing and you want to generate Profile and .env as well), in .env file you should set `MAVEN_REPOSITORY_LOCATION` (point to local maven repository location, it uses `~/.m2` by default). These will be used as volumes for the docker container. Profile file holds the environment variables that are used inside the containers, the .env file is used outs [...]
-
-Then you can use the `logsearch-docker` script to start the containers (`start` command).
-Also you can use docker-compose manually to start/manage the containers.
-```bash
-docker-compose up -d
-# or start all services in one container:
-docker-compose -f all.yml up -d
-```
-After the logsearch container is started you can enter to it with following commands:
-```bash
-docker exec -it docker_logsearch_1 bash
-# or if you used all.yml for starting the logsearch docker container:
-docker exec -it logsearch bash
-```
-In case if you started the containers separately and if you would like to access Solr locally with through your external ZooKeeper container, then point `solr` to `localhost` in your `/etc/hosts` file.
-
-### Run applications from IDE / maven
-
-- [Start Log Search locally](ambari-logsearch-server/README.md)
-- [Start Log Feeder locally](ambari-logsearch-logfeeder/README.md)
-
-## Package build process
+## Introduction
-1. Check out the code from GIT repository
+Log aggregation, analysis, and visualization for Ambari managed (or any other) services.
-2. On the logsearch root folder (ambari/ambari-logsearch), please execute the following make command to build RPM/DPKG:
-```bash
-make rpm
-# or for jdk11
-export LOGSEARCH_JDK_11=true
-make rpm
-```
- or
-```bash
-make deb
-# or for jdk11
-export LOGSEARCH_JDK_11=true
-make deb
-```
-3. Generated RPM/DPKG files will be found in ambari-logsearch-assembly/target folder
+## Features
-## Running Integration Tests
+- Parse / aggregate and ship logs
+- Send and index logs in Solr
+- Store logs (structured or un-sructured format) in Cloud Storage (S3 / GCS / ADLS / WASB)
+- Fultext Search in logs (if the logs are shipped to Solr)
+- JWT/SSO support
+- Support testing the log parsing on the UI
-By default integration tests are not a part of the build process, you need to set -Dbackend-tests or -Dselenium-tests (or you can use -Dall-tests to run both). To running the tests you will need docker here as well (right now docker-for-mac and unix are supported by default, for boot2docker you need to pass -Ddocker.host parameter to the build).
+## Architecture
-```bash
-# from ambari-logsearch folder
-mvn clean integration-test -Dbackend-tests failsafe:verify
-# or run selenium tests with docker for mac, but before that you nedd to start xquartz
-open -a XQuartz
-# then in an another window you can start ui tests
-mvn clean integration-test -Dselenium-tests failsafe:verify
-# you can specify story file folde location with -Dbackend.stories.location and -Dui.stories.location (absolute file path) in the commands
-```
-Also you can run from the IDE, but make sure all of the ambari logsearch modules are built.
+- Log Feeder: agent component on all hosts to monitor and shipping logs.
+- Log Search Portal: REST API + UI for rendering logs
+- Solr (Optional - default): storage for logs, used by both Log Search Portal and Log Feeder
+- ZooKeeper (Optional - default): configuration service for Solr, Log Search and Log Feeder
+- HDFS / S3 / GCS / ADLS: storage for logs (write only), used by Log Feeder [cloud mode](docs/cloud_mode.md)
+
-## Update version (for release or specific builds)
+## Contents
-```bash
-make update-version new-version="2.8.0.0-11"
-```
+- [1, Installation](docs/installation.md)
+- [2. Collections](docs/collections.md)
+- [3. Adding new logs to monitor](docs/add_new_input.md)
+- [4. Development guide](docs/development.md)
+- [5. Using Log Feeder in Cloud mode](docs/cloud_mode.md)
## Contributing
diff --git a/ambari-logsearch-logfeeder/README.md b/ambari-logsearch-logfeeder/README.md
deleted file mode 100644
index 48cb81b..0000000
--- a/ambari-logsearch-logfeeder/README.md
+++ /dev/null
@@ -1,49 +0,0 @@
-<!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-
-# Log Search - Log Feeder:
-
-Log Feeder is a component of the Log Search service that reads logs, parses them and stores them in Apache Solr for the purpose
-of later analysis.
-
-## Start locally from maven / IDE
-
-First you need to start every required service (except logfeeder), go to `ambari-logsearch/docker` folder and run:
-```bash
-docker-compose up -d zookeeper solr logsearch
-```
-
-Secondly, if you are planning to run Log Feeder from an IDE, for running the LogFeeder main methoud, you will need to set the working directory to `ambari/ambari-logsearch/ambari-logsearch-logfeeder` or set `LOGFEEDER_RELATIVE_LOCATION` env variable.
-With Maven, you won't need these steps, just run this command from the ambari-logsearch-logfeeder folder:
-
-```bash
-mvn clean package -DskipTests spring-boot:run
-```
-
-# Input Configuration
-
-The configuration for the log feeder contains
-* description of the log files
-* description of the filters that parse the data of the log entries
-* description of the mappers that modify the parsed fields
-
-The element description can be found [here](docs/inputConfig.md)
-
-All these data are stored in json files, which should be named in the directory /etc/ambari-logsearch-logfeeder/conf, and the
-name of the files should be input.config-<service\_name>.json
\ No newline at end of file
diff --git a/ambari-logsearch-server/README.md b/ambari-logsearch-server/README.md
deleted file mode 100644
index 7432344..0000000
--- a/ambari-logsearch-server/README.md
+++ /dev/null
@@ -1,39 +0,0 @@
-<!--
-{% comment %}
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to you under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-{% endcomment %}
--->
-
-# Log Search Server
-
-## Start locally from maven / IDE
-
-Other services (like zookeeper, solr, logfeeder) can be started with `docker-compose`
-```bash
-cd ambari/ambari-logsearch/docker
-docker-compose up -d zookeeper solr logfeeder
-```
-
-Then you can start Log Search server from maven
-
-```bash
-cd ambari/ambari-logsearch/ambari-logsearch-server
-./run.sh
-# or
-mvn clean package -DskipTests spring-boot:run
-```
-
-You can also start Log Search server from an IDE as well. One thing is important: the config set location that the server tries to upload to ZooKeeper. By default config sets are located at `${LOGSEARCH_SERVER_RELATIVE_LOCATION:}src/main/configsets` in `logsearch.properties`. Based or from where you run `LogSearch.java`, you need to set `LOGSEARCH_SERVER_RELATIVE_LOCATION` env variable properly.
diff --git a/docs/development.md b/docs/development.md
index 9a27bef..866d435 100644
--- a/docs/development.md
+++ b/docs/development.md
@@ -15,9 +15,124 @@ See the License for the specific language governing permissions and
limitations under the License.
-->
-### Backend development
+## Developement guide
-TODO
+## Requirements
+
+- JDK 8 (JDK 11 is recommended)
+- Maven 3.5.x
+
+## Backend development
+
+### Prerequisites
+
+- Install [docker](https://docs.docker.com/)
+- For Mac OS X use [Docker for Mac](https://docs.docker.com/docker-for-mac/)
+- [Docker compose](https://docs.docker.com/compose/) is also required.
+
+### Build and start Log Search in docker container
+```bash
+# to see available commands: run start-logsearch without arguments
+cd docker
+./logsearch-docker build-and-run # build mvn project locally, build docker image, start containers
+```
+If you run the script at first time, it will generate you a new `Profile` file or an `.env` file inside docker directory (run twice if both missing and you want to generate Profile and .env as well), in .env file you should set `MAVEN_REPOSITORY_LOCATION` (point to local maven repository location, it uses `~/.m2` by default). These will be used as volumes for the docker container. Profile file holds the environment variables that are used inside the containers, the .env file is used outs [...]
+
+Then you can use the `logsearch-docker` script to start the containers (`start` command).
+Also you can use docker-compose manually to start/manage the containers.
+```bash
+docker-compose up -d
+```
+After the logsearch container is started you can enter to it with following commands:
+```bash
+docker exec -it docker_logsearch_1 bash
+```
+In case if you started the containers separately and if you would like to access Solr locally with through your external ZooKeeper container, then point `solr` to `localhost` in your `/etc/hosts` file.
+
+### Enable debug for Log Search Server or Log Feeder
+
+By default remote debug is enabled in docker containers for Log Search server and Log Feeder (in order to use IDEs for debugging). Log Search server port for debug is `5005` and `5006` for Log Feeder.
+
+To suspend components in debug mode, you need to edit your `Profile` file (that was generated in docker folder) and set `LOGSEARCH_DEBUG_SUSPEND` or `LOGFEEDER_DEBUG_SUSPEND` to `true`.
+
+### Run applications from IDE / maven
+
+#### Start Log Search server locally from maven / IDE
+
+Other services (like zookeeper, solr, logfeeder) can be started with `docker-compose`
+```bash
+cd ambari/ambari-logsearch/docker
+docker-compose up -d zookeeper solr logfeeder
+```
+
+Then you can start Log Search server from maven
+
+```bash
+cd ambari/ambari-logsearch/ambari-logsearch-server
+./run.sh
+# or
+mvn clean package -DskipTests spring-boot:run
+```
+
+You can also start Log Search server from an IDE as well. One thing is important: the config set location that the server tries to upload to ZooKeeper. By default config sets are located at `${LOGSEARCH_SERVER_RELATIVE_LOCATION:}src/main/configsets` in `logsearch.properties`. Based or from where you run `LogSearch.java`, you need to set `LOGSEARCH_SERVER_RELATIVE_LOCATION` env variable properly. (or just simply use the ambari-logsearch-server as the working directory)
+
+#### Start Log Feeder locally from maven / IDE
+
+First you need to start every required service (except logfeeder), go to `ambari-logsearch/docker` folder and run:
+```bash
+docker-compose up -d zookeeper solr logsearch
+```
+
+Secondly, if you are planning to run Log Feeder from an IDE, for running the LogFeeder main methoud, you will need to set the working directory to `ambari/ambari-logsearch/ambari-logsearch-logfeeder` or set `LOGFEEDER_RELATIVE_LOCATION` env variable.
+With Maven, you won't need these steps, just run this command from the ambari-logsearch-logfeeder folder:
+
+```bash
+mvn clean package -DskipTests spring-boot:run
+```
+
+For Log Feeder, it is also important to use the ambari-logsearch-logfeeder as a working directory if you are trying to run the application from an IDE.
+
+### Package build process
+
+1. Check out the code from GIT repository
+
+2. On the logsearch root folder (ambari/ambari-logsearch), please execute the following make command to build RPM/DPKG:
+```bash
+make rpm
+# or for jdk11
+export LOGSEARCH_JDK_11=true
+make rpm
+```
+ or
+```bash
+make deb
+# or for jdk11
+export LOGSEARCH_JDK_11=true
+make deb
+```
+3. Generated RPM/DPKG files will be found in ambari-logsearch-assembly/target folder
+
+### Running Integration Tests
+
+By default integration tests are not a part of the build process, you need to set -Dbackend-tests or -Dselenium-tests (or you can use -Dall-tests to run both). To running the tests you will need docker here as well (right now docker-for-mac and unix are supported by default, for boot2docker you need to pass -Ddocker.host parameter to the build).
+
+```bash
+# from ambari-logsearch folder
+mvn clean integration-test -Dbackend-tests failsafe:verify
+# or run selenium tests with docker for mac, but before that you nedd to start xquartz
+open -a XQuartz
+# then in an another window you can start ui tests
+mvn clean integration-test -Dselenium-tests failsafe:verify
+# you can specify story file folde location with -Dbackend.stories.location and -Dui.stories.location (absolute file path) in the commands
+```
+Also you can run from the IDE, but make sure all of the ambari logsearch modules are built.
+
+
+### Update version (for release or specific builds)
+
+```bash
+make update-version new-version="2.8.0.0-11"
+```
### UI development