You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by lu...@apache.org on 2024/07/04 10:32:50 UTC
(doris-website) branch master updated: [doc] Add the Log Storage and Analysis doc & fix some issues (#821)
This is an automated email from the ASF dual-hosted git repository.
luzhijing pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new 941946b4e5a [doc] Add the Log Storage and Analysis doc & fix some issues (#821)
941946b4e5a is described below
commit 941946b4e5a62b0fe9e18d6fc0bb7347df7ca5db
Author: lishiqi_amy <am...@163.com>
AuthorDate: Thu Jul 4 18:32:44 2024 +0800
[doc] Add the Log Storage and Analysis doc & fix some issues (#821)
---
.../update/update-of-aggregate-model.md | 2 +-
docs/data-operate/update/update-of-unique-model.md | 2 +-
docs/ecosystem/flink-doris-connector.md | 8 +-
docs/practical-guide/log-storage-analysis.md | 593 +++++++++++++++++++++
docs/table-design/data-partition.md | 4 +-
.../docusaurus-plugin-content-docs/current.json | 6 +-
.../current/admin-manual/fe/debug-point-action.md | 12 +-
.../practical-guide/log-storage-analysis.md | 564 ++++++++++++++++++++
.../version-2.0.json | 4 +
.../practical-guide/log-storage-analysis.md | 564 ++++++++++++++++++++
.../version-2.1.json | 4 +
.../practical-guide/log-storage-analysis.md | 564 ++++++++++++++++++++
sidebars.json | 7 +
static/images/WebUI-CN.jpeg | Bin 0 -> 595835 bytes
static/images/WebUI-EN.jpeg | Bin 0 -> 601164 bytes
static/images/doris-overall-architecture.png | Bin 0 -> 184926 bytes
.../version-2.0/ecosystem/flink-doris-connector.md | 8 +-
.../practical-guide/log-storage-analysis.md | 593 +++++++++++++++++++++
.../version-2.0/table-design/data-partition.md | 4 +-
.../update/update-of-aggregate-model.md | 2 +-
.../data-operate/update/update-of-unique-model.md | 2 +-
.../version-2.1/ecosystem/flink-doris-connector.md | 8 +-
.../practical-guide/log-storage-analysis.md | 593 +++++++++++++++++++++
.../version-2.1/table-design/data-partition.md | 4 +-
versioned_sidebars/version-2.0-sidebars.json | 7 +
versioned_sidebars/version-2.1-sidebars.json | 7 +
26 files changed, 3533 insertions(+), 29 deletions(-)
diff --git a/docs/data-operate/update/update-of-aggregate-model.md b/docs/data-operate/update/update-of-aggregate-model.md
index 27ea77192d3..945bc013c47 100644
--- a/docs/data-operate/update/update-of-aggregate-model.md
+++ b/docs/data-operate/update/update-of-aggregate-model.md
@@ -1,6 +1,6 @@
---
{
- "title": "Updaing Data on Aggregate Key Model",
+ "title": "Updating Data on Aggregate Key Model",
"language": "en"
}
---
diff --git a/docs/data-operate/update/update-of-unique-model.md b/docs/data-operate/update/update-of-unique-model.md
index 1d537dfa6d5..3b35dc6dd35 100644
--- a/docs/data-operate/update/update-of-unique-model.md
+++ b/docs/data-operate/update/update-of-unique-model.md
@@ -1,6 +1,6 @@
---
{
- "title": "Updaing Data on Unique Key Model",
+ "title": "Updating Data on Unique Key Model",
"language": "en"
}
---
diff --git a/docs/ecosystem/flink-doris-connector.md b/docs/ecosystem/flink-doris-connector.md
index cc2957eae2c..264bc8b1a4e 100644
--- a/docs/ecosystem/flink-doris-connector.md
+++ b/docs/ecosystem/flink-doris-connector.md
@@ -79,7 +79,7 @@ Copy this file to `classpath` of `Flink` to use `Flink-Doris-Connector`. For exa
### read
-####SQL
+**SQL**
```sql
-- doris source
@@ -98,7 +98,7 @@ CREATE TABLE flink_doris_source (
);
```
-####DataStream
+**DataStream**
```java
DorisOptions.Builder builder = DorisOptions.builder()
@@ -118,7 +118,7 @@ env.fromSource(dorisSource, WatermarkStrategy.noWatermarks(), "doris source").pr
### write
-####SQL
+**SQL**
```sql
--enable checkpoint
@@ -144,7 +144,7 @@ CREATE TABLE flink_doris_sink (
INSERT INTO flink_doris_sink select name,age,price,sale from flink_doris_source
```
-####DataStream
+**DataStream**
DorisSink writes data to Doris through StreamLoad, and DataStream supports different serialization methods when writing
diff --git a/docs/practical-guide/log-storage-analysis.md b/docs/practical-guide/log-storage-analysis.md
new file mode 100644
index 00000000000..db36bec1b1a
--- /dev/null
+++ b/docs/practical-guide/log-storage-analysis.md
@@ -0,0 +1,593 @@
+---
+{
+ "title": "Log Storage and Analysis",
+ "language": "en"
+}
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Logs record key events in the system and contain crucial information such as the events' subject, time, location, and content. To meet the diverse needs of observability in operations, network security monitoring, and business analysis, enterprises might need to collect scattered logs for centralized storage, querying, and analysis to extract valuable content from the log data further.
+
+In this scenario, Apache Doris provides a corresponding solution. With the characteristics of log scenarios in mind, Apache Doris added inverted-index and ultra-fast full-text search capabilities, optimizing write performance and storage space to the extreme. This allows users to build an open, high-performance, cost-effective, and unified log storage and analysis platform based on Apache Doris.
+
+Focused on this solution, this chapter contains the following 3 sections:
+
+- **Overall architecture**: This section explains the core components and architecture of the log storage and analysis platform built on Apache Doris.
+
+- **Features and advantages**: This section explains the features and advantages of the log storage and analysis platform built on Apache Doris.
+
+- **Operational guide**: This section explains how to build a log storage and analysis platform based on Apache Doris.
+
+## Overall architecture
+
+The following figure illustrates the architecture of the log storage and analysis platform built on Apache Doris:
+
+
+
+The architecture contains the following 3 parts:
+
+- **Log collection and preprocessing**: Various log collection tools can write log data into Apache Doris through HTTP APIs.
+
+- **Log storage and analysis engine**: Apache Doris provides high-performance, low-cost unified log storage, offering rich retrieval and analysis capabilities through SQL interfaces.
+
+- **Log analysis and alert interface**: Various log retrieval and analysis tools can query Apache Doris through standard SQL interfaces, providing users with a simple and user-friendly interface.
+
+## Features and advantages
+
+The following figure illustrates the architecture of the log storage and analysis platform built on Apache Doris::
+
+- **High throughput, low latency log writing**: Supports stable writing of log data at the level of hundreds of TBs and GB/s per day, while maintaining latency within 1 second.
+
+- **Cost-effective storage of massive log data**: Supports petabyte-scale storage, saving 60% to 80% in storage costs compared to Elasticsearch, and further reducing storage costs by 50% by storing cold data in S3/HDFS.
+
+- **High-performance log full-text search and analysis**: Supports inverted indexes and full-text search, providing second-level response times for common log queries (keyword searches, trend analysis, etc.).
+
+- **Open and user-friendly upstream and downstream ecosystem**: Upstream integration with common log collection systems and data sources such as Logstash, Filebeat, Fluentbit, and Kafka through Stream Load's universal HTTP APIs, and downstream integration with various visual analytics UIs using standard MySQL protocol and syntax, such as observability platform Grafana, BI analytics Superset, and log retrieval Doris WebUI similar to Kibana.
+
+### Cost-effective performance
+
+After Benchmark testing and production validation, the log storage and analysis platform built on Apache Doris has a 5 to 10 times cost-performance advantage over Elasticsearch. Apache Doris's performance benefits are mainly due to its globally leading high-performance storage and query engine, as well as specialized optimizations for log scenarios:
+
+- **Improved write throughput**: Elasticsearch's write performance bottleneck lies in CPU consumption for parsing data and building inverted indexes. In comparison, Apache Doris has optimized writes in two aspects: using SIMD and other CPU vector instructions to improve JSON data parsing speed and index-building performance and simplifying the inverted index structure for log scenarios by removing unnecessary data structures like forward indexes, effectively reducing index build complexi [...]
+
+- **Reduced storage costs**: Elasticsearch's storage bottleneck comes from multiple storage of forward, inverted, Docvalue columns, and lower compression ratios with general compression algorithms. Conversely, Apache Doris has optimized storage by eliminating forward indexes, reducing index data volume by 30%; adopting columnar storage and Zstandard compression algorithm, achieving compression ratios 5-10 times higher than Elasticsearch's 1.5 times; with low access frequency of cold data [...]
+
+### Strong analytical capabilities
+
+Apache Doris supports standard SQL and is compatible with MySQL protocol and syntax. Therefore, log systems built on Apache Doris can use SQL for log analysis, giving the following advantages to log systems:
+
+- **Easy to use**: Engineers and data analysts are very familiar with SQL, their expertise can be reused, no need to learn new technology stacks to quickly get started.
+
+- **Rich ecosystem**: The MySQL ecosystem is the most widely used language in the database field, seamlessly integrating with and applying to the MySQL ecosystem. Doris can leverage MySQL command line and various GUI tools, BI tools, and other big data ecosystem tools for more complex and diverse data processing and analysis needs.
+
+- **Strong analytical capabilities**: SQL has become the de facto standard for database and big data analysis, with powerful expressive capabilities and functions supporting retrieval, aggregation, multi-table JOIN, subqueries, UDFs, logical views, materialized views, and various data analysis capabilities.
+
+### Flexible Schema
+
+Here is a typical example of a semi-structured log in JSON format. The top-level fields are some fixed fields, such as timestamp, source, node, component, level, clientRequestID, message, and properties, which are present in every log entry. The nested fields of the properties , such as properties.size and properties.format are more dynamic, and the fields of each log may vary.
+
+```JSON
+{
+"timestamp": "2014-03-08T00:50:03.8432810Z",
+"source": "ADOPTIONCUSTOMERS81",
+"node": "Engine000000000405",
+"level": "Information",
+"component": "DOWNLOADER",
+"clientRequestId": "671db15d-abad-94f6-dd93-b3a2e6000672",
+"message": "Downloading file path: benchmark/2014/ADOPTIONCUSTOMERS81_94_0.parquet.gz",
+"properties": {
+"size": 1495636750,
+"format": "parquet",
+"rowCount": 855138,
+"downloadDuration": "00:01:58.3520561"
+}
+}
+```
+
+
+Apache Doris provides several aspects of support for Flexible Schema log data:
+
+- For minor changes to top-level fields, Light Schema Change can be used to add or remove columns and to add or remove indexes, enabling schema changes to be completed in seconds. When planning a log platform, users only need to consider which fields need to be indexed.
+
+- For extension fields similar to properties, the native semi-structured data type `VARIANT` is provided, which can write any JSON data, automatically recognize field names and types in JSON, and automatically split frequently occurring fields for columnar storage for subsequent analysis. Additionally, `VARIANT` can create inverted indexes to accelerate internal field queries and retrievals.
+
+Compared to Elasticsearch's Dynamic Mapping, Apache Doris's Flexible Schema has the following advantages:
+
+- Allows a field to have multiple types, `VARIANT` automatically handles conflicts and type promotion for fields, better adapting to iterative changes in log data.
+
+- `VARIANT` automatically merges infrequently occurring fields into a column store to avoid performance issues caused by excessive fields, metadata, or columns.
+
+- Not only can columns be dynamically added, but they can also be dynamically deleted, and indexes can be dynamically added or removed, eliminating the need to index all fields at the beginning like Elasticsearch, reducing unnecessary costs.
+
+## Operational guide
+
+### Step 1: Estimate resources
+
+Before deploying the cluster, you need to estimate the hardware resources required for the servers. Follow the steps below:
+
+1. Estimate the resources for data writing by the following calculation formulas:
+
+- Daily data increment / 86400 s = Average write throughput
+
+- Average write throughput \* Ratio of the peak write throughput to the average write throughput = Peak write throughput
+
+- Peak write throughput / Write throughput of a single-core CPU = Number of CPU cores for the peak write throughput
+
+2. Estimate the resources for data storage by the calculation formula: Daily data increment / Data compression ratio \* Number of data copies \* Data storage duration = Data storage space.
+
+3. Estimate the resources for data querying. The resources for data querying depend on the query volume and complexity. It is recommended to reserve 50% of CPU resources for data query initially and then adjust according to the actual test results.
+
+4. Integrate the calculation results as follows:
+
+ 1. Divide the number of CPU cores calculated in Step 1 and Step 3 by the number of CPU cores of a BE server, and you can get the number of BE servers.
+
+ 2. Based on the number of BE servers and the calculation result of Step 2, estimate the storage space required for each BE server.
+
+ 3. Allocate the storage space required for each BE server to 4 to 12 data disks, and you can get the storage capacity required for a single data disk.
+
+For example, suppose that the daily data increment is 100 TB, the data compression ratio is 7, the number of data copies is 1, the storage duration of hot data is 3 days, the storage duration of cold data is 30 days, the ratio of the peak write throughput to the average write throughput is 200%, the write throughput of a single-core CUP is 10 MB/s, and 50% of CPU resources are reserved for data querying, one can estimate that:
+
+- 3 FE servers are required, each configured with a 16-core CPU, 64 GB memory, and an 1100 GB SSD disk.
+
+- 15 BE servers are required, each configured with a 32-core CPU, 256 GB memory, and 8 500 GB SSD disks.
+
+- S3:430 TB
+
+Refer to the following table to learn about the values of indicators in the example above and how they are calculated.
+
+| Indicator (Unit) | Value | Description |
+| --- | --- | --- |
+| Daily data increment (TB) | 100 | Specify the value according to your actual needs. |
+| Data compression ratio | 7 | Specify the value according to your actual needs, which is often between 5 to 10. Note that the data contains index data. |
+| Number of data copies | 1 | Specify the value according to your actual needs, which can be 1, 2, or 3. The default value is 1. |
+| Storage duration of hot data (day) | 3 | Specify the value according to your actual needs. |
+| Storage duration of cold data (day) | 30 | Specify the value according to your actual needs. |
+| Data storage duration | 33 | Calculation formula: Storage duration of hot data + Storage duration of cold data |
+| Estimated storage space for hot data (TB) | 42.9 | Calculation formula: Daily data increment / Data compression ratios \* Number of data copies \* Storage duration of hot data |
+| Estimated storage space for cold data (TB) | 428.6 | Calculation formula: Daily data increment / Data compression ratios \* Number of data copies \* Storage duration of cold data |
+| Ratio of the peak write throughput to the average write throughput | 200% | Specify the value according to your actual needs. The default value is 200%. |
+| Number of CPU cores of a BE server | 32 | Specify the value according to your actual needs. The default value is 32. |
+| Average write throughput (MB/s) | 1214 | Calculation formula: Daily data increment / 86400 s |
+| Peak write throughput (MB/s) | 2427 | Calculation formula: Average write throughput \* Ratio of the peak write throughput to the average write throughput |
+| Number of CPU cores for the peak write throughput | 242.7 | Calculation formula: Peak write throughput / Write throughput of a single-core CPU |
+| Percent of CPU resources reserved for data querying | 50% | Specify the value according to your actual needs. The default value is 50%. |
+| Estimated number of BE servers | 15.2 | Calculation formula: Number of CPU cores for the peak write throughput / Number of CPU cores of a BE server /（1 - Percent of CPU resources reserved for data querying） |
+| Rounded number of BE servers | 15 | Calculation formula: MAX (Number of data copies, Estimated number of BE servers) |
+| Estimated data storage space for each BE server (TB) | 4.03 | Calculation formula: Estimated storage space for hot data / Estimated number of BE servers /（1 - 30%）, where 30% represents the percent of reserved storage space.<br /><br />It is recommended to mount 4 to 12 data disks on each BE server to enhance I/O capabilities. |
+
+### Step 2: Deploy the cluster
+
+After estimating the resources, you need to deploy the cluster. It is recommended to deploy in both physical and virtual environments manually. For manual deployment, refer to [Manual Deployment](../install/cluster-deployment/standard-deployment.md).
+
+Alternatively, it is recommended to use VeloDB Manager provided by VeloDB Enterprise to deploy the cluster, reducing overall deployment costs. For more information about the VeloDB Manager, please refer to the following documents:
+
+- [VeloDB Manager 24.x Installation Manual](https://docs.velodb.io/enterprise/velodb-manager-guide/deployment-guide/deployment-guide-24.x)
+
+- [VeloDB Manager 24.x User Guide](https://docs.velodb.io/enterprise/velodb-manager-guide/management-guide/management-guide-24.x)
+
+### Step 3: Optimize FE and BE configurations
+
+After completing the cluster deployment, it is necessary to optimize the configuration parameters for both the front-end and back-end separately, so as to better suit the scenario of log storage and analysis.
+
+**Optimize FE configurations**
+
+You can find FE configuration fields in `fe/conf/fe.conf`. Refer to the following table to optimize FE configurations.
+
+| Configuration fields to be optimized | Description |
+| :----------------------------------------------------------- | :----------------------------------------------------------- |
+| `max_running_txn_num_per_db = 10000` | Increase the parameter value to adapt to high-concurrency import transactions. |
+| `streaming_lable_keep_max_second = 3600``label_keep_max_second = 7200` | Increase the retention time to handle high-frequency import transactions with high memory usage. |
+| `enable_round_robin_create_tablet = true` | When creating Tablets, use a Round Robin strategy to distribute evenly. |
+| `tablet_rebalancer_type = partition` | When balancing Tablets, use a strategy to evenly distribute within each partition. |
+| `enable_single_replica_load = true` | Enable single-replica import, where multiple replicas only need to build an index once to reduce CPU consumption. |
+| `autobucket_min_buckets = 10` | Increase the minimum number of automatically bucketed buckets from 1 to 10 to avoid insufficient buckets when the log volume increases. |
+| `max_backend_heartbeat_failure_tolerance_count = 10` | In log scenarios, the BE server may experience high pressure, leading to short-term timeouts, so increase the tolerance count from 1 to 10. |
+
+For more information, refer to [FE Configuration](../admin-manual/config/fe-config.md).
+
+**Optimize BE configurations**
+
+You can find BE configuration fields in `be/conf/be.conf`. Refer to the following table to optimize BE configurations.
+
+| Module | Configuration fields to be optimized | Description |
+| :--------- | :----------------------------------------------------------- | :----------------------------------------------------------- |
+| Storage | `storage_root_path = /path/to/dir1;/path/to/dir2;...;/path/to/dir12` | Configure the storage path for hot data on disk directories. |
+| - | `enable_file_cache = true` | Enable file caching. |
+| - | `file_cache_path = [{"path": "/mnt/datadisk0/file_cache", "total_size":53687091200, "query_limit": "10737418240"},{"path": "/mnt/datadisk1/file_cache", "total_size":53687091200,"query_limit": "10737418240"}]` | Configure the cache path and related settings for cold data with the following specific configurations:<br/>`path`: cache path<br/>`total_size`: total size of the cache path in bytes, where 53687091200 bytes equals 50 GB<br/>`query_limit`: maximum amount of data tha [...]
+| Write | `write_buffer_size = 1073741824` | Increase the file size of the write buffer to reduce small files and random I/O operations, improving performance. |
+| - | `max_tablet_version_num = 20000` | In coordination with the time_series compaction strategy for table creation, allow more versions to remain temporarily unmerged |
+| - | `enable_single_replica_load = true` | Enable single replica import to reduce CPU consumption, consistent with FE configuration. |
+| Compaction | `max_cumu_compaction_threads = 8` | Set to CPU core count / 4, indicating that 1/4 of CPU resources are used for writing, 1/4 for background compaction, and 2/1 for queries and other operations. |
+| - | `inverted_index_compaction_enable = true` | Enable inverted index compaction to reduce CPU consumption during compaction. |
+| - | `enable_segcompaction = false` `enable_ordered_data_compaction = false` | Disable two compaction features that are unnecessary for log scenarios. |
+| Cache | `disable_storage_page_cache = true` `inverted_index_searcher_cache_limit = 30%` | Due to the large volume of log data and limited caching effect, switch from data caching to index caching. |
+| - | `inverted_index_cache_stale_sweep_time_sec = 3600` `index_cache_entry_stay_time_after_lookup_s = 3600` | Maintain index caching in memory for up to 1 hour. |
+| - | `enable_inverted_index_cache_on_cooldown = true`<br />`enable_write_index_searcher_cache = false` | Enable automatic caching of cold data storage during index uploading. |
+| - | `tablet_schema_cache_recycle_interval = 3600` `segment_cache_capacity = 20000` | Reduce memory usage by other caches. |
+| Thread | `pipeline_executor_size = 24` `doris_scanner_thread_pool_thread_num = 48` | Configure computing threads and I/O threads for a 32-core CPU in proportion to core count. |
+| - | `scan_thread_nice_value = 5` | Lower the priority of query I/O threads to ensure writing performance and timeliness. |
+| Other | `string_type_length_soft_limit_bytes = 10485760` | Increase the length limit of string-type data to 10 MB. |
+| - | `trash_file_expire_time_sec = 300` `path_gc_check_interval_second = 900` `path_scan_interval_second = 900` | Accelerate the recycling of trash files. |
+
+For more information, refer to [BE Configuration](../admin-manual/config/be-config.md).
+
+### Step 4: Create tables
+
+Due to the distinct characteristics of both writing and querying log data, it is recommended to configure tables with targeted settings to enhance performance.
+
+**Configure data partitioning and bucketing**
+
+- For data partitioning:
+
+ - Enable [range partitioning](https://doris.apache.org/docs/table-design/data-partition#range-partition) with [dynamic partitions](https://doris.apache.org/docs/table-design/data-partition#dynamic-partition) managed automatically by day.
+
+ - Use a field in the DATETIME type as the key for accelerated retrieval of the latest N log entries.
+
+- For data bucketing:
+
+ - Configure the number of buckets to be roughly three times the total number of disks in the cluster.
+
+ - Use the Random strategy to optimize batch writing efficiency when paired with single tablet imports.
+
+For more information, refer to [Data Partitioning](../table-design/data-partition.md).
+
+**Configure compaction fileds**
+
+Configure compaction fields as follows:
+
+- Use the time_series strategy to reduce write amplification, which is crucial for high-throughput log writes.
+
+- Use single-replica compaction to minimize the overhead of multi-replica compaction.
+
+**Configure index fields**
+
+Configuring index fields as follows:
+
+- Create indexes for fields that are frequently queried.
+
+- For fields that require full-text search, specify the parser field as unicode, which satisfies most requirements. If there is a need to support phrase queries, set the support_phrase field to true; if not needed, set it to false to reduce storage space.
+
+**Configure storage policies**
+
+Configure storage policies as follows:
+
+- For storage of hot data, if using cloud storage, configure the number of data copies as 1; if using physical disks, configure the number of data copies as at least 2.
+
+- Configure the storage location for log_s3 and set the log_policy_3day policy, where the data is cooled and moved to the specified storage location of log_s3 after 3 days. Refer to the code below.
+
+```Go
+CREATE DATABASE log_db;
+USE log_db;
+
+CREATE RESOURCE "log_s3"
+PROPERTIES
+(
+ "type" = "s3",
+ "s3.endpoint" = "your_endpoint_url",
+ "s3.region" = "your_region",
+ "s3.bucket" = "your_bucket",
+ "s3.root.path" = "your_path",
+ "s3.access_key" = "your_ak",
+ "s3.secret_key" = "your_sk"
+);
+
+CREATE STORAGE POLICY log_policy_3day
+PROPERTIES(
+ "storage_resource" = "log_s3",
+ "cooldown_ttl" = "259200"
+);
+
+CREATE TABLE log_table
+(
+ `ts` DATETIME,
+ `host` TEXT,
+ `path` TEXT,
+ `message` TEXT,
+ INDEX idx_host (`host`) USING INVERTED,
+ INDEX idx_path (`path`) USING INVERTED,
+ INDEX idx_message (`message`) USING INVERTED PROPERTIES("parser" = "unicode", "support_phrase" = "true")
+)
+ENGINE = OLAP
+DUPLICATE KEY(`ts`)
+PARTITION BY RANGE(`ts`) ()
+DISTRIBUTED BY RANDOM BUCKETS 250
+PROPERTIES (
+"dynamic_partition.enable" = "true",
+"dynamic_partition.create_history_partition" = "true",
+"dynamic_partition.time_unit" = "DAY",
+"dynamic_partition.start" = "-30",
+"dynamic_partition.end" = "1",
+"dynamic_partition.prefix" = "p",
+"dynamic_partition.buckets" = "250",
+"dynamic_partition.replication_num" = "1", -- unneccessary for the compute-storage coupled mode
+"replication_num" = "1" -- unneccessary for the compute-storage coupled mode
+"enable_single_replica_compaction" = "true", -- unneccessary for the compute-storage coupled mode
+"storage_policy" = "log_policy_3day", -- unneccessary for the compute-storage coupled mode
+"compaction_policy" = "time_series"
+);
+```
+
+### Step 5: Collect, query, and analyze logs
+
+After completing table creation, you can proceed with log collection, querying, and analysis.
+
+**Collect logs**
+
+Apache Doris provides open and versatile Stream HTTP APIs, through which you can connect with popular log collectors such as Logstash, Filebeat, Kafka, and others to carry out log collection work. This section explains how to integrate these log collectors using the Stream HTTP APIs.
+
+**Integrating Logstash**
+
+Follow these steps:
+
+1. Download and install the Logstash Doris Output plugin. You can choose one of the following two methods:
+
+ - [Click to download](https://apache-doris-releases.oss-accelerate.aliyuncs.com/logstash-output-doris-1.0.0.gem) and install.
+
+ - Compile from the source code and run the following command to install:
+
+```markdown
+./bin/logstash-plugin install logstash-output-doris-1.0.0.gem
+```
+
+2. Configure Logstash. Specify the following fields:
+
+ - `logstash.yml`: Used to configure Logstash batch processing log sizes and timings for improved data writing performance.
+
+ ```Plain Text
+ pipeline.batch.size: 1000000
+ pipeline.batch.delay: 10000
+ ```
+
+ - `logstash_demo.conf`: Used to configure the specific input path of the collected logs and the settings for output to Apache Doris.
+
+ ```markdown
+ input {
+ file {
+ path => "/path/to/your/log"
+ }
+ }
+ <br />output {
+ doris {
+ http_hosts => \[ "<http://fehost1:http_port>", "<http://fehost2:http_port>", "<http://fehost3:http_port"\>]
+ user => "your_username"
+ password => "your_password"
+ db => "your_db"
+ table => "your_table"
+ \# doris stream load http headers
+ headers => {
+ "format" => "json"
+ "read_json_by_line" => "true"
+ "load_to_single_tablet" => "true"
+ }
+ \# field mapping: doris fileld name => logstash field name
+ \# %{} to get a logstash field, \[\] for nested field such as \[host\]\[name\] for host.name
+ mapping => {
+ "ts" => "%{@timestamp}"
+ "host" => "%{\[host\]\[name\]}"
+ "path" => "%{\[log\]\[file\]\[path\]}"
+ "message" => "%{message}"
+ }
+ log_request => true
+ log_speed_interval => 10
+ }
+ }
+ ```
+
+3. Run Logstash according to the command below, collect logs, and output to Apache Doris.
+
+ ```Bash
+ ./bin/logstash -f logstash_demo.conf
+ ```
+
+For more information about the Logstash Doris Output plugin, see [Logstash Doris Output Plugin](../ecosystem/logstash.md).
+
+**Integrating Filebeat**
+
+Follow these steps:
+
+1. Obtain the Filebeat binary file that supports output to Apache Doris. You can [click to download](https://apache-doris-releases.oss-accelerate.aliyuncs.com/filebeat-doris-1.0.0) or compile it from the Apache Doris source code.
+
+2. Configure Filebeat. Specify the filebeat_demo.yml field that is used to configure the specific input path of the collected logs and the settings for output to Apache Doris.
+
+
+ ```YAML
+ # input
+ filebeat.inputs:
+ - type: log
+ enabled: true
+ paths:
+ - /path/to/your/log
+ multiline:
+ type: pattern
+ pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}'
+ negate: true
+ match: after
+ skip_newline: true
+
+ processors:
+ - script:
+ lang: javascript
+ source: >
+ function process(event) {
+ var msg = event.Get("message");
+ msg = msg.replace(/\t/g, " ");
+ event.Put("message", msg);
+ }
+ - dissect:
+ # 2024-06-08 18:26:25,481 INFO (report-thread|199) [ReportHandler.cpuReport():617] begin to handle
+ tokenizer: "%{day} %{time} %{log_level} (%{thread}) [%{position}] %{content}"
+ target_prefix: ""
+ ignore_failure: true
+ overwrite_keys: true
+
+ # queue and batch
+ queue.mem:
+ events: 1000000
+ flush.min_events: 100000
+ flush.timeout: 10s
+
+ # output
+ output.doris:
+ fenodes: [ "http://fehost1:http_port", "http://fehost2:http_port", "http://fehost3:http_port" ]
+ user: "your_username"
+ password: "your_password"
+ database: "your_db"
+ table: "your_table"
+ # output string format
+ codec_format_string: '{"ts": "%{[day]} %{[time]}", "host": "%{[agent][hostname]}", "path": "%{[log][file][path]}", "message": "%{[message]}"}'
+ headers:
+ format: "json"
+ read_json_by_line: "true"
+ load_to_single_tablet: "true"
+ ```
+
+3. Run Filebeat according to the command below, collect logs, and output to Apache Doris.
+
+ ```Bash
+ chmod +x filebeat-doris-1.0.0
+ ./filebeat-doris-1.0.0 -c filebeat_demo.yml
+ ```
+
+For more information about Filebeat, refer to [Beats Doris Output Plugin](../ecosystem/beats.md).
+
+**Integrating Kafka**
+
+Write JSON formatted logs to Kafka's message queue, create a Kafka Routine Load, and allow Apache Doris to actively pull data from Kafka.
+
+You can refer to the example below, where `property.\*` represents Librdkafka client-related configurations and needs to be adjusted according to the actual Kafka cluster situation.
+
+```SQL
+CREATE ROUTINE LOAD load_log_kafka ON log_db.log_table
+COLUMNS(ts, clientip, request, status, size)
+PROPERTIES (
+"max_batch_interval" = "10",
+"max_batch_rows" = "1000000",
+"max_batch_size" = "109715200",
+"load_to_single_tablet" = "true",
+"timeout" = "600",
+"strict_mode" = "false",
+"format" = "json"
+)
+FROM KAFKA (
+"kafka_broker_list" = "host:port",
+"kafka_topic" = "log_\_topic_",
+"property.group.id" = "your_group_id",
+"property.security.protocol"="SASL_PLAINTEXT",
+"property.sasl.mechanism"="GSSAPI",
+"property.sasl.kerberos.service.name"="kafka",
+"property.sasl.kerberos.keytab"="/path/to/xxx.keytab",
+"property.sasl.kerberos.principal"="<xx...@yyy.com>"
+);
+<br />SHOW ROUTINE LOAD;
+```
+
+For more information about Kafka, see [Routine Load](../data-operate/import/routine-load-manual.md)。
+
+**Using customized programs to collect logs**
+
+In addition to integrating common log collectors, you can also customize programs to import log data into Apache Doris using the Stream Load HTTP API. Refer to the following code:
+
+```Bash
+curl \\
+\--location-trusted \\
+\-u username:password \\
+\-H "format:json" \\
+\-H "read_json_by_line:true" \\
+\-H "load_to_single_tablet:true" \\
+\-H "timeout:600" \\
+\-T logfile.json \\
+http://fe_host:fe_http_port/api/log_db/log_table/\_stream_load
+```
+
+When using custom programs, pay attention to the following key points:
+
+- Use Basic Auth for HTTP authentication, calculate using the command echo -n 'username:password' | base64.
+
+- Set HTTP header "format:json" to specify the data format as JSON.
+
+- Set HTTP header "read_json_by_line:true" to specify one JSON per line.
+
+- Set HTTP header "load_to_single_tablet:true" to import data into one bucket at a time to reduce small file imports.
+
+- It is recommended to write batches whose sizes are between 100MB to 1GB on the client side. For Apache Doris version 2.1 and higher, you need to reduce batch sizes on the client side through the Group Commit function.
+
+**Query logs**
+
+Apache Doris supports standard SQL, so you can connect to the cluster through MySQL client or JDBC to execute SQL for log queries.
+
+```Plain Text
+mysql -h fe_host -P fe_mysql_port -u your_username -Dyour_db_name
+```
+
+Here are 5 common SQL query commands for reference:
+
+- View the latest 10 log entries
+
+```SQL
+SELECT \* FROM your_table_name ORDER BY ts DESC LIMIT 10;
+```
+
+- Query the latest 10 log entries with the host as 8.8.8.8
+
+```SQL
+SELECT \* FROM your_table_name WHERE host = '8.8.8.8' ORDER BY ts DESC LIMIT 10;
+```
+
+- Retrieve the latest 10 log entries with error or 404 in the request field. In the command below, MATCH_ANY is a full-text search SQL syntax used by Apache Doris for matching any keyword in the fields.
+
+```SQL
+SELECT \* FROM your_table_name WHERE message **MATCH_ANY** 'error 404'
+ORDER BY ts DESC LIMIT 10;
+```
+
+- Retrieve the latest 10 log entries with image and faq in the request field. In the command below, MATCH_ALL is a full-text search SQL syntax used by Apache Doris for matching all keywords in the fields.
+
+```SQL
+SELECT \* FROM your_table_name WHERE message **MATCH_ALL** 'image faq'
+ORDER BY ts DESC LIMIT 10;
+```
+
+- Retrieve the latest 10 entries with image and faq in the request field. In the following command, MATCH_PHRASE is a full-text search SQL syntax used by Apache Doris for matching all keywords in the fields and requiring consistent order. In the example below, a image faq b can match, but a faq image b cannot match because the order of image and faq does not match the syntax.
+
+```SQL
+SELECT \* FROM your_table_name WHERE message **MATCH_PHRASE** 'image faq'
+ORDER BY ts DESC LIMIT 10;
+```
+
+**Analyze logs visually**
+
+VeloDB Enterprise Core, built on Apache Doris, provides a data development platform called Velo Enterprise WebUI ("WebUI"), featuring a Kibana Discover-like log retrieval and analysis interface for intuitive and easy exploratory log analysis interaction as shown in the image below:
+
+
+
+On this interface, WebUI supports the following operations:
+
+- Support for full-text search and SQL modes
+
+- Support for selecting query log timeframes with time boxes and histograms
+
+- Display of detailed log information, expandable into JSON or tables
+
+- Interactive clicking to add and remove filter conditions in the log data context
+
+- Display of top field values in search results for finding anomalies and further drilling down for analysis
+
+You can [click to download Velo Enterprise Core](https://www.velodb.io/download/enterprise) and [install it](https://docs.velodb.io/enterprise/enterprise-core-guide/velodb-distribution-doris-core-deployment-guide) to use WebUI. For more information about the main functions and how to use WebUI, see [WebUI](https://docs.velodb.io/enterprise/enterprise-core-guide/velodb-webui-guide).
\ No newline at end of file
diff --git a/docs/table-design/data-partition.md b/docs/table-design/data-partition.md
index 5805199f219..2e2a1a9fdc0 100644
--- a/docs/table-design/data-partition.md
+++ b/docs/table-design/data-partition.md
@@ -792,8 +792,8 @@ Suppose our table DDL is as follows:
```sql
CREATE TABLE `DAILY_TRADE_VALUE`
(
- `TRADE_DATE` datev2 NOT NULL COMMENT '交易日期',
- `TRADE_ID` varchar(40) NOT NULL COMMENT '交易编号',
+ `TRADE_DATE` datev2 NOT NULL COMMENT 'TRADE_DATE',
+ `TRADE_ID` varchar(40) NOT NULL COMMENT 'TRADE_ID',
......
)
UNIQUE KEY(`TRADE_DATE`, `TRADE_ID`)
diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current.json b/i18n/zh-CN/docusaurus-plugin-content-docs/current.json
index 975f4db1044..ce2444c762d 100644
--- a/i18n/zh-CN/docusaurus-plugin-content-docs/current.json
+++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current.json
@@ -378,5 +378,9 @@
"sidebar.docs.category.Compute-Storage Decoupled": {
"message": "存算分离",
"description": "Compute-Storage Decoupled"
+ },
+ "sidebar.docs.category.Practical Guide": {
+ "message": "实践教程",
+ "description": "The label for category Practical Guide in sidebar docs"
}
-}
+}
\ No newline at end of file
diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/fe/debug-point-action.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/fe/debug-point-action.md
index df68ac003c8..d8bb53f2e78 100644
--- a/i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/fe/debug-point-action.md
+++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/admin-manual/fe/debug-point-action.md
@@ -41,7 +41,7 @@ FE 桩子示例代码
```java
private Status foo() {
// dbug_fe_foo_do_nothing 是一个木桩名字，
- // 打开这个木桩之后，DebugPointUtil.isEnable("dbug_fe_foo_do_nothing") 将会返回true
+ // 打开这个木桩之后，DebugPointUtil.isEnable("dbug_fe_foo_do_nothing") 将会返回 true
if (DebugPointUtil.isEnable("dbug_fe_foo_do_nothing")) {
return Status.Nothing;
}
@@ -90,10 +90,10 @@ POST /api/debug_point/add/{debug_point_name}[?timeout=<int>&execute=<int>]
木桩名字。必填。
* `timeout`
- 超时时间，单位为秒。超时之后，木桩失活。默认值-1表示永远不超时。可选。
+ 超时时间，单位为秒。超时之后，木桩失活。默认值 -1 表示永远不超时。可选。
* `execute`
- 木桩最大执行次数。默认值-1表示不限执行次数。可选。
+ 木桩最大执行次数。默认值 -1 表示不限执行次数。可选。
### Request body
@@ -112,7 +112,7 @@ POST /api/debug_point/add/{debug_point_name}[?timeout=<int>&execute=<int>]
### Examples
-打开木桩 `foo`，最多执行5次。
+打开木桩 `foo`，最多执行 5 次。
```
@@ -132,7 +132,7 @@ curl -X POST "http://127.0.0.1:8030/api/debug_point/add/foo?execute=5"
POST /api/debug_point/add/{debug_point_name}[?k1=v1&k2=v2&k3=v3...]
```
* `k1=v1`
- k1为参数名称，v1为参数值，多个参数用&分隔。
+ k1 为参数名称，v1 为参数值，多个参数用&分隔。
### Request body
@@ -269,7 +269,7 @@ curl -X POST "http://127.0.0.1:8030/api/debug_point/clear"
## 在回归测试中使用木桩
-> 提交PR时，社区 CI 系统默认开启 FE 和 BE 的`enable_debug_points`配置。
+> 提交 PR 时，社区 CI 系统默认开启 FE 和 BE 的`enable_debug_points`配置。
回归测试框架提供方法函数来开关指定的木桩，它们声明如下：
diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/current/practical-guide/log-storage-analysis.md b/i18n/zh-CN/docusaurus-plugin-content-docs/current/practical-guide/log-storage-analysis.md
new file mode 100644
index 00000000000..1dec9e8019f
--- /dev/null
+++ b/i18n/zh-CN/docusaurus-plugin-content-docs/current/practical-guide/log-storage-analysis.md
@@ -0,0 +1,564 @@
+---
+{
+ "title": "日志存储与分析",
+ "language": "zh-CN"
+}
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+日志是系统运行的详细记录，包含各种事件发生的主体、时间、位置、内容等关键信息。出于运维可观测、网络安全监控及业务分析等多重需求，企业通常需要将分散的日志采集起来，进行集中存储、查询和分析，以进一步从日志数据里挖掘出有价值的内容。
+
+针对此场景，Apache Doris 提供了相应解决方案，针对日志场景的特点，增加了倒排索引和极速全文检索能力，极致优化写入性能和存储空间，使得用户可以基于 Apache Doris 构建开放、高性能、低成本、统一的日志存储与分析平台。
+
+本文将围绕这一解决方案，介绍以下内容：
+
+- **整体架构**：说明基于 Apache Doris 构建的日志存储与分析平台的核心组成部分和基础架构。
+- **特点与优势**：说明基于 Apache Doris 构建的日志存储与分析平台的特点和优势。
+- **操作指南**：说明如何基于 Apache Doris 构建日志存储分析平台。
+
+## 整体架构
+
+基于 Apache Doris 构建的日志存储与分析平台的架构如下图：
+
+
+
+此架构主要由 3 大部分组成：
+
+- **日志采集和预处理**：多种日志采集工具可以通过 HTTP APIs 将日志数据写入 Apache Doris。
+- **日志存储和分析引擎**：Apache Doris 提供高性能、低成本的统一日志存储，通过 SQL 接口提供丰富的检索分析能力。
+- **日志分析和告警界面**：多种日志检索分析通工具通过标准 SQL 接口查询 Apache Doris，为用户提供简单易用的界面。
+
+## 特点与优势
+
+基于 Apache Doris 构建的日志存储与分析平台的特点和优势如下：
+
+- **高吞吐、低延迟日志写入**：支持每天百 TB 级、GB/s 级日志数据持续稳定写入，同时保持延迟 1s 以内。
+- **海量日志数据低成本存储**：支持 PB 级海量存储，相对于 Elasticsearch 存储成本节省 60% 到 80%，支持冷数据存储到 S3/HDFS，存储成本再降 50%。
+- **高性能日志全文检索分析**：支持倒排索引和全文检索，日志场景常见查询（关键词检索明细、趋势分析等）秒级响应。
+- **开放、易用的上下游生态**：上游通过 Stream Load 通用 HTTP APIs 对接常见的日志采集系统和数据源 Logstash、Filebeat、Fluentbit、Kafka 等，下游通过标准 MySQL 协议和语法对接各种可视化分析 UI，比如可观测性 Grafana、BI 分析 Superset、类 Kibana 的日志检索 Doris WebUI。
+
+### 高性能、低成本
+
+经过 Benchmark 测试及生产验证，基于 Apache Doris 构建的日志存储与分析平台，性价比相对于 Elasticsearch 具有 5～10 倍的提升。Apache Doris 的性能优势，主要得益于全球领先的高性能存储和查询引擎，以及下面一些针对日志场景的专门优化：
+
+- **写入吞吐提升**：Elasticsearch 写入的性能瓶颈在于解析数据和构建倒排索引的 CPU 消耗。相比之下，Apache Doris 进行了两方面的写入优化：一方面利用 SIMD 等 CPU 向量化指令提升了 JSON 数据解析速度和索引构建性能；另一方面针对日志场景简化倒了排索引结构，去掉日志场景不需要的正排等数据结构，有效降低了索引构建的复杂度。同样的资源，Apache Doris 的写入性能是 Elasticsearch 的 3～5 倍。
+- **存储成本降低**：Elasticsearch 存储瓶颈在于正排、倒排、Docvalue 列存多份存储和通用压缩算法压缩率较低。相比之下，Apache Doris 在存储上进行了以下优化：去掉正排，缩减了 30% 的索引数据量；采用列式存储和 Zstandard 压缩算法，压缩比可达到 5～10 倍，远高于 Elasticsearch 的 1.5 倍；日志数据中冷数据访问频率很低，Apache Doris 冷热分层功能可以将超过定义时间段的日志自动存储到更低的对象存储中，冷数据的存储成本可降低 70% 以上。同样的原始数据，SelectDB 的存储成本只需要 Elasticsearch 的 20% 左右。
+- **查询性能提升**：Apache Doris 将全文检索的流程简化，跳过了相关性打分等日志场景不需要的算法，加速基础的检索性能。同时针对日志场景常见的查询，比如查询包含某个关键字的最新 100 条日志，在查询规划和执行上做专门的 TopN 动态剪枝等优化。
+
+### 分析能力强
+
+Apache Doris 支持标准 SQL、兼容 MySQL 协议和语法，因此基于 Apache Doris 构建的日志系统能够使用 SQL 进行日志分析，这使得日志系统具备以下优势：
+
+- **简单易用**：工程师和数据分析师对于 SQL 非常熟悉，经验可以复用，不需要学习新的技术栈即可快速上手。
+- **生态丰富**：MySQL 生态是数据库领域使用最广泛的语言，因此可以与 MySQL 生态的集成和应用无缝衔接。Doris 可以利用 MySQL 命令行与各种 GUI 工具、BI 工具等大数据生态结合，实现更复杂及多样化的数据处理分析需求。
+- **分析能力强**：SQL 语言已经成为数据库和大数据分析的事实标准，它具有强大的表达能力和功能，支持检索、聚合、多表 JOIN、子查询、UDF、逻辑视图、物化视图等多种数据分析能力。
+
+### Flexible Schema
+
+下面是一个典型的 JSON 格式半结构化日志样例。顶层字段是一些比较固定的字段，比如日志时间戳（`timestamp`），日志来源（`source`），志所在机器（`node`），打日志的模块（`component`），日志级别（`level`），客户端请求标识（`clientRequestId`），日志内容（`message`），日志扩展属性（`properties`），基本上每条日志都会有。而扩展属性 `properties` 的内部嵌套字段 `properties.size`、`properties.format` 等是比较动态的，每条日志的字段可能不一样。
+
+```JSON
+{
+"timestamp": "2014-03-08T00:50:03.8432810Z",
+"source": "ADOPTIONCUSTOMERS81",
+"node": "Engine000000000405",
+"level": "Information",
+"component": "DOWNLOADER",
+"clientRequestId": "671db15d-abad-94f6-dd93-b3a2e6000672",
+"message": "Downloading file path: benchmark/2014/ADOPTIONCUSTOMERS81_94_0.parquet.gz",
+"properties": {
+"size": 1495636750,
+"format": "parquet",
+"rowCount": 855138,
+"downloadDuration": "00:01:58.3520561"
+}
+}
+```
+
+Apache Doris 对 Flexible Schema 的日志数据提供了几个方面的支持：
+
+- 对于顶层字段的少量变化，可以通过 Light Schema Change 发起 ADD / DROP COLUMN 增加 / 删除列，ADD / DROP INDEX 增加 / 删除索引，能够在秒级完成 Schema 变更。用户在日志平台规划时只需考虑当前需要哪些字段创建索引。
+- 对于类似 `properties` 的扩展字段，提供了原生半结构化数据类型 `VARIANT`，可以写入任何 JSON 数据，自动识别 JSON 中的字段名和类型，并自动拆分频繁出现的字段采用列式存储，以便于后续的分析，还可以对 `VARIANT` 创建倒排索引，加快内部字段的查询和检索。
+
+相对于 Elasticsearch 的 Dynamic Mapping，Apache Doris 的 Flexible Schema 有以下优势：
+
+- 允许一个字段有多种类型，`VARIANT` 自动对字段类型做冲突处理和类型提升，更好地适应日志数据的迭代变化。
+- `VARIANT` 自动将不频繁出现的字段合并成一个列存储，可避免字段、元数据、列过多导致性能问题。
+- 不仅可以动态加列，还可以动态删列、动态增加索引、动态删索引，无需像 Elasticsearch 在一开始对所有字段建索引，减少不必要的成本。
+
+## 操作指南
+
+### 第 1 步：评估资源
+
+在部署集群之前，首先应评估所需服务器硬件资源，包括以下几个关键步骤：
+
+1. **评估写入资源**：计算公式如下：
+
+ - 日增数据量 / 86400 s = 平均写入吞吐
+ - 平均写入吞吐 \* 写入吞吐峰值 / 均值比 = 峰值写入吞吐
+ - 峰值写入吞吐 / 单核写入吞吐 = 峰值写入所需 CPU 核数
+
+2. **评估存储资源**：计算公式为「日增数据量 / 压缩率 \* 副本数 \* 数据存储周期 = 所需存储空间」。
+
+3. **评估查询资源**：查询的资源消耗随查询量和复杂度而异，建议初始预留 50% 的 CPU 资源用于查询，再根据实际测试情况进行调整。
+
+4. **汇总整合资源**：由第 1 步和第 3 步估算出所需 CPU 核数后，除以单机 CPU 核数，估算出 BE 服务器数量，再根据 BE 服务器数量和第 2 步的结果，估算出每台 BE 服务器所需存储空间，然后分摊到 4～12 块数据盘，计算出单盘存储容量。
+
+以每天新增 100 TB 数据量（压缩前）、7 倍压缩率、1 副本、热数据存储 3 天、冷数据存储 30 天、写入吞吐峰值 / 均值比 200%、单核写入吞吐 10 MB/s、查询预留 50% CPU 资源为例，可估算出：
+
+- FE：3 台服务器，每台配置 16 核 CPU、64 GB 内存、1 块 100 GB SSD 盘
+- BE：15 台服务器，每台配置 32 核 CPU、256 GB 内存、8 块 500 GB SSD 盘
+- Amazon S3 存储空间：即为预估冷数据存储空间，430 TB
+
+该例子中，各关键指标的值及具体计算方法可见下表：
+
+| 关键指标（单位） | 值 | 说明 |
+| :------------------------------- | :---- | :----------------------------------------------------------- |
+| 日增数据量（TB） | 100 | 根据实际需求填写 |
+| 压缩率 | 7 | 一般为 5～10 倍（含索引），根据实际需求填写 |
+| 副本数 | 1 | 根据实际需求填写，默认 1 副本，可选值：1，2，3 |
+| 热数据存储周期（天） | 3 | 根据实际需求填写 |
+| 冷数据存储周期（天） | 30 | 根据实际需求填写 |
+| 总存储周期（天） | 33 | 算法：热数据存储周期 + 冷数据存储周期 |
+| 预估热数据存储空间（TB） | 42.9 | 算法：日增数据量 / 压缩率 * 副本数 * 热数据存储周期 |
+| 预估冷数据存储空间（TB） | 428.6 | 算法：日增数据量 / 压缩率 * 副本数 * 冷数据存储周期 |
+| 写入吞吐峰值 / 均值比 | 200% | 根据实际需求填写，默认 200% |
+| 单机 CPU 核数 | 32 | 根据实际需求填写，默认 32 核 |
+| 平均写入吞吐（MB/s） | 1214 | 算法：日增数据量 / 86400 s |
+| 峰值写入吞吐（MB/s） | 2427 | 算法：平均写入吞吐 * 写入吞吐峰值 / 均值比 |
+| 峰值写入所需 CPU 核数 | 242.7 | 算法：峰值写入吞吐 / 单机 CPU 核数 |
+| 查询预留 CPU 百分比 | 50% | 根据实际需求填写，默认 50% |
+| 预估 BE 服务器数 | 15.2 | 算法：峰值写入所需 CPU 核数 / 单机 CPU 核数 /（1 - 查询预留 CPU 百分比） |
+| 预估 BE 服务器数取整 | 15 | 算法：MAX (副本数，预估 BE 服务器数上取整) |
+| 预估每台 BE 服务器存储空间（TB） | 4.03 | 算法：预估热数据存储空间 / 预估 BE 服务器数 /（1 - 30%），其中，30% 是存储空间预留值。建议每台 BE 服务器挂载 4～12 块数据盘，以提高 I/O 能力。 |
+
+### 第 2 步：部署集群
+
+完成资源评估后，可以开始部署 Apache Doris 集群，推荐在物理机及虚拟机环境中进行部署。手动部署集群，可参考 [手动部署](../install/cluster-deployment/standard-deployment.md)。
+
+另，推荐使用 SelectDB Enterprise 推出的 Cluster Manager 工具部署集群，以降低整体部署成本。更多关于 Cluster Manager 的信息，可参考以下文档：
+
+- [Cluster Manager for Apache Doris 24.x 安装手册](https://docs.selectdb.com/docs/enterprise/cluster-manager-guide/deployment-guide/deployment-guide-24.x)
+- [Cluster Manager for Apache Doris 24.x 使用手册](https://docs.selectdb.com/docs/enterprise/cluster-manager-guide/management-guide/management-guide-24.x)
+
+### 第 3 步：优化 FE 和 BE 配置
+
+完成集群部署后，需分别优化 FE 和 BE 配置参数，以更加契合日志存储与分析的场景。
+
+**优化 FE 配置**
+
+在 `fe/conf/fe.conf` 目录下找到 FE 的相关配置项，并按照以下表格，调整 FE 配置。
+
+| 需调整参数 | 说明 |
+| :----------------------------------------------------------- | :----------------------------------------------------------- |
+| `max_running_txn_num_per_db = 10000` | 高并发导入运行事务数较多，需调高参数。 |
+| `streaming_lable_keep_max_second = 3600``label_keep_max_second = 7200` | 高频导入事务标签内存占用多，保留时间调短。 |
+| `enable_round_robin_create_tablet = true` | 创建 Tablet 时，采用 Round Robin 策略，尽量均匀。 |
+| `tablet_rebalancer_type = partition` | 均衡 Tablet 时，采用每个分区内尽量均匀的策略。 |
+| `enable_single_replica_load = true` | 开启单副本导入，多个副本只需构建一次索引，减少 CPU 消耗。 |
+| `autobucket_min_buckets = 10` | 将自动分桶的最小分桶数从 1 调大到 10，避免日志量增加时分桶不够。 |
+| `max_backend_heartbeat_failure_tolerance_count = 10` | 日志场景下 BE 服务器压力较大，可能短时间心跳超时，因此将容忍次数从 1 调大到 10。 |
+
+更多关于 FE 配置项的信息，可参考 [FE 配置项](../admin-manual/config/fe-config.md)。
+
+**优化 BE 配置**
+
+在 `be/conf/be.conf` 目录下找到 BE 的相关配置项，并按照以下表格，调整 BE 配置。
+
+| 模块 | 需调整参数 | 说明 |
+| :--------- | :----------------------------------------------------------- | :----------------------------------------------------------- |
+| 存储 | `storage_root_path = /path/to/dir1;/path/to/dir2;...;/path/to/dir12` | 配置热数据在磁盘目录上的存储路径。 |
+| - | `enable_file_cache = true` | 开启文件缓存。 |
+| - | `file_cache_path = [{"path": "/mnt/datadisk0/file_cache", "total_size":53687091200, "query_limit": "10737418240"},{"path": "/mnt/datadisk1/file_cache", "total_size":53687091200,"query_limit": "10737418240"}]` | 配置冷数据的缓存路径和相关设置，具体配置说明如下：<br />`path`：缓存路径<br />`total_size`：该缓存路径的总大小，单位为字节，53687091200 字节等于 50 GB<br />`query_limit`：单次查询可以从缓存路径中查询的最大数据量，单位为字节，10737418240 字节等于 10 GB |
+| 写入 | `write_buffer_size = 1073741824` | 增加写入缓冲区（buffer）的文件大小，减少小文件和随机 I/O 操作，提升性能。 |
+| - | `max_tablet_version_num = 20000` | 配合建表的 time_series compaction 策略，允许更多版本暂时未合并。 |
+| - | `enable_single_replica_load = true` | 开启单副本写入，减少 CPU 消耗。与 FE 配置相同。 |
+| Compaction | `max_cumu_compaction_threads = 8` | 设置为 CPU 核数 / 4，意味着 CPU 资源的 1/4 用于写入，1/4 用于后台 Compaction，2/1 留给查询和其他操作。 |
+| - | `inverted_index_compaction_enable = true` | 开启索引合并（index compaction），减少 Compaction 时的 CPU 消耗。 |
+| - | `enable_segcompaction = false` `enable_ordered_data_compaction = false` | 关闭日志场景不需要的两个 Compaction 功能。 |
+| 缓存 | `disable_storage_page_cache = true` `inverted_index_searcher_cache_limit = 30%` | 因为日志数据量较大，缓存（cache）作用有限，因此关闭数据缓存，调换为索引缓存（index cache）的方式。 |
+| - | `inverted_index_cache_stale_sweep_time_sec = 3600` `index_cache_entry_stay_time_after_lookup_s = 3600` | 让索引缓存在内存中尽量保留 1 小时。 |
+| - | `enable_inverted_index_cache_on_cooldown = true` <br />`enable_write_index_searcher_cache = false` | 开启索引上传冷数据存储时自动缓存的功能。 |
+| - | `tablet_schema_cache_recycle_interval = 3600` `segment_cache_capacity = 20000` | 减少其他缓存对内存的占用。 |
+| 线程 | `pipeline_executor_size = 24` `doris_scanner_thread_pool_thread_num = 48` | 32 核 CPU 的计算线程和 I/O 线程配置，根据核数等比扩缩。 |
+| - | `scan_thread_nice_value = 5` | 降低查询 I/O 线程的优先级，保证写入性能和时效性。 |
+| 其他 | `string_type_length_soft_limit_bytes = 10485760` | 将 String 类型数据的长度限制调高至 10 MB。 |
+| - | `trash_file_expire_time_sec = 300` `path_gc_check_interval_second = 900` `path_scan_interval_second = 900` | 调快垃圾文件的回收时间。 |
+
+更多关于 BE 配置项的信息，可参考 [BE 配置项](../admin-manual/config/be-config.md)。
+
+### 第 4 步：建表
+
+由于日志数据的写入和查询都具备明显的特征，因此，在建表时按照本节说明进行针对性配置，以提升性能表现。
+
+**配置分区分桶参数**
+
+- 分区时，按照以下说明配置：
+- 使用时间字段上的 [Range 分区](https://doris.apache.org/zh-CN/docs/dev/table-design/data-partition/#range-%E5%88%86%E5%8C%BA)，并开启 [动态分区](https://doris.apache.org/zh-CN/docs/dev/table-design/data-partition?_highlight=%E8%87%AA%E5%8A%A8&_highlight=%E5%88%86&_highlight=%E6%A1%B6#%E5%8A%A8%E6%80%81%E5%88%86%E5%8C%BA)，按天自动管理分区。
+- 使用 Datetime 类型的时间字段作为 Key，在查询最新 N 条日志时有数倍加速。
+- 分桶时，按照以下说明配置：
+- 分桶数量大致为集群磁盘总数的 3 倍。
+- 使用 Random 策略，配合写入时的 Single Tablet 导入，可以提升批量（Batch）写入的效率。
+
+更多关于分区分桶的信息，可参考 [分区分桶](../table-design/data-partition.md)。
+
+**配置 Compaction 参数**
+
+按照以下说明配置 Compaction 参数：
+
+- 使用 time_series 策略，以减轻写放大效应，对于高吞吐日志写入的资源写入很重要。
+- 使用单副本 Compaction，减少多副本 Compaction 的开销。
+
+**建立和配置索引参数**
+
+按照以下说明操作：
+
+- 对经常查询的字段建立索引。
+- 对需要全文检索的字段，将分词器（parser）参数赋值为 unicode，一般能满足大部分需求。如有支持短语查询的需求，将 support_phrase 参数赋值为 true；如不需要，则设置为 false，以降低存储空间。
+
+**配置存储策略**
+
+按照以下说明操作：
+
+- 对于热存储数据，如果使用云盘，可配置 1 副本；如果使用物理盘，则至少配置 2 副本。
+- 配置 `log_s3` 的存储位置，并设置 `log_policy_3day` 冷热数据分层策略，即在超过 3 天后将数据冷却至 `log_s3` 指定的存储位置。可参考以下代码：
+
+```Go
+CREATE DATABASE log_db;
+USE log_db;
+
+CREATE RESOURCE "log_s3"
+PROPERTIES
+(
+ "type" = "s3",
+ "s3.endpoint" = "your_endpoint_url",
+ "s3.region" = "your_region",
+ "s3.bucket" = "your_bucket",
+ "s3.root.path" = "your_path",
+ "s3.access_key" = "your_ak",
+ "s3.secret_key" = "your_sk"
+);
+
+CREATE STORAGE POLICY log_policy_3day
+PROPERTIES(
+ "storage_resource" = "log_s3",
+ "cooldown_ttl" = "259200"
+);
+
+CREATE TABLE log_table
+(
+ `ts` DATETIME,
+ `host` TEXT,
+ `path` TEXT,
+ `message` TEXT,
+ INDEX idx_host (`host`) USING INVERTED,
+ INDEX idx_path (`path`) USING INVERTED,
+ INDEX idx_message (`message`) USING INVERTED PROPERTIES("parser" = "unicode", "support_phrase" = "true")
+)
+ENGINE = OLAP
+DUPLICATE KEY(`ts`)
+PARTITION BY RANGE(`ts`) ()
+DISTRIBUTED BY RANDOM BUCKETS 250
+PROPERTIES (
+"dynamic_partition.enable" = "true",
+"dynamic_partition.create_history_partition" = "true",
+"dynamic_partition.time_unit" = "DAY",
+"dynamic_partition.start" = "-30",
+"dynamic_partition.end" = "1",
+"dynamic_partition.prefix" = "p",
+"dynamic_partition.buckets" = "250",
+"dynamic_partition.replication_num" = "1", -- 存算分离不需要
+"replication_num" = "1" -- 存算分离不需要
+"enable_single_replica_compaction" = "true", -- 存算分离不需要
+"storage_policy" = "log_policy_3day", -- 存算分离不需要
+"compaction_policy" = "time_series"
+);
+```
+
+### 第 5 步：采集和查询日志
+
+完成建表后，可进行日志采集、查询和分析。
+
+**日志采集**
+
+Apache Doris 提供开放、通用的 Stream HTTP APIs，通过这些 APIs，你可与常用的日志采集器打通，包括 Logstash、Filebeat、Kafka 等，从而开展日志采集工作。本节介绍了如何使用 Stream HTTP APIs 对接日志采集器。
+
+**对接 Logstash**
+
+按照以下步骤操作：
+
+1. 下载并安装 Logstash Doris Output 插件。你可选择以下两种方式之一：
+
+- 直接下载：[点此下载](https://apache-doris-releases.oss-accelerate.aliyuncs.com/logstash-output-doris-1.0.0.gem)。
+
+- 从源码编译，并运行下方命令安装：
+
+```markdown
+./bin/logstash-plugin install logstash-output-doris-1.0.0.gem
+```
+
+2. 配置 Logstash。需配置以下参数：
+
+- `logstash.yml`：配置 Logstash 批处理日志的条数和时间，用于提升数据写入性能。
+
+```markdown
+pipeline.batch.size: 1000000
+pipeline.batch.delay: 10000
+```
+
+
+- `logstash_demo.conf`：配置所采集日志的具体输入路径和输出到 Apache Doris 的设置。
+
+```markdown
+input {
+file {
+path => "/path/to/your/log"
+}
+}
+<br />output {
+doris {
+http_hosts => \[ "<http://fehost1:http_port>", "<http://fehost2:http_port>", "<http://fehost3:http_port"\>]
+user => "your_username"
+password => "your_password"
+db => "your_db"
+table => "your_table"
+\# doris stream load http headers
+headers => {
+"format" => "json"
+"read_json_by_line" => "true"
+"load_to_single_tablet" => "true"
+}
+\# field mapping: doris fileld name => logstash field name
+\# %{} to get a logstash field, \[\] for nested field such as \[host\]\[name\] for host.name
+mapping => {
+"ts" => "%{@timestamp}"
+"host" => "%{\[host\]\[name\]}"
+"path" => "%{\[log\]\[file\]\[path\]}"
+"message" => "%{message}"
+}
+log_request => true
+log_speed_interval => 10
+}
+}
+```
+
+3. 按照下方命令运行 Logstash，采集日志并输出至 Apache Doris。
+
+```Bash
+./bin/logstash -f logstash_demo.conf
+```
+
+更多关于 Logstash 配置和使用的说明，可参考 [Logstash Doris Output Plugin](../ecosystem/logstash.md)。
+
+**对接 Filebeat**
+
+按照以下步骤操作：
+
+1. 获取支持输出至 Apache Doris 的 Filebeat 二进制文件。可 [点此下载](https://apache-doris-releases.oss-accelerate.aliyuncs.com/filebeat-doris-1.0.0) 或者从 Apache Doris 源码编译。
+2. 配置 Filebeat。需配置以下参数：
+
+- `filebeat_demo.yml`：配置所采集日志的具体输入路径和输出到 Apache Doris 的设置。
+
+ ```YAML
+ # input
+ filebeat.inputs:
+ - type: log
+ enabled: true
+ paths:
+ - /path/to/your/log
+ # multiline 可以将跨行的日志（比如 Java stacktrace）拼接起来
+ multiline:
+ type: pattern
+ # 效果：以 yyyy-mm-dd HH:MM:SS 开头的行认为是一条新的日志，其他都拼接到上一条日志
+ pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}'
+ negate: true
+ match: after
+ skip_newline: true
+
+ processors:
+ # 用 js script 插件将日志中的 \t 替换成空格，避免 JSON 解析报错
+ - script:
+ lang: javascript
+ source: >
+ function process(event) {
+ var msg = event.Get("message");
+ msg = msg.replace(/\t/g, " ");
+ event.Put("message", msg);
+ }
+ # 用 dissect 插件做简单的日志解析
+ - dissect:
+ # 2024-06-08 18:26:25,481 INFO (report-thread|199) [ReportHandler.cpuReport():617] begin to handle
+ tokenizer: "%{day} %{time} %{log_level} (%{thread}) [%{position}] %{content}"
+ target_prefix: ""
+ ignore_failure: true
+ overwrite_keys: true
+
+ # queue and batch
+ queue.mem:
+ events: 1000000
+ flush.min_events: 100000
+ flush.timeout: 10s
+
+ # output
+ output.doris:
+ fenodes: [ "http://fehost1:http_port", "http://fehost2:http_port", "http://fehost3:http_port" ]
+ user: "your_username"
+ password: "your_password"
+ database: "your_db"
+ table: "your_table"
+ # output string format
+ ## %{[agent][hostname]} %{[log][file][path]} 是filebeat自带的metadata
+ ## 常用的 filebeat metadata 还是有采集时间戳 %{[@timestamp]}
+ ## %{[day]} %{[time]} 是上面 dissect 解析得到字段
+ codec_format_string: '{"ts": "%{[day]} %{[time]}", "host": "%{[agent][hostname]}", "path": "%{[log][file][path]}", "message": "%{[message]}"}'
+ headers:
+ format: "json"
+ read_json_by_line: "true"
+ load_to_single_tablet: "true"
+ ```
+
+3. 按照下方命令运行 Filebeat，采集日志并输出至 Apache Doris。
+
+```Bash
+chmod +x filebeat-doris-1.0.0
+./filebeat-doris-1.0.0 -c filebeat_demo.yml
+```
+
+更多关于 Filebeat 配置和使用的说明，可参考 [Beats Doris Output Plugin](../ecosystem/beats.md)。
+
+**对接 Kafka**
+
+将 JSON 格式的日志写入 Kafka 的消息队列，创建 Kafka Routine Load，即可让 Apache Doris 从 Kafka 主动拉取数据。
+
+可参考如下示例。其中，`property.\*` 是 Librdkafka 客户端相关配置，根据实际 Kafka 集群情况配置。
+
+```SQL
+\-- 准备好kafka集群和topic log_\_topic_
+\-- 创建routine load，从kafka log_\_topic_将数据导入log_table表
+CREATE ROUTINE LOAD load_log_kafka ON log_db.log_table
+COLUMNS(ts, clientip, request, status, size)
+PROPERTIES (
+"max_batch_interval" = "10",
+"max_batch_rows" = "1000000",
+"max_batch_size" = "109715200",
+"load_to_single_tablet" = "true",
+"timeout" = "600",
+"strict_mode" = "false",
+"format" = "json"
+)
+FROM KAFKA (
+"kafka_broker_list" = "host:port",
+"kafka_topic" = "log_\_topic_",
+"property.group.id" = "your_group_id",
+"property.security.protocol"="SASL_PLAINTEXT",
+"property.sasl.mechanism"="GSSAPI",
+"property.sasl.kerberos.service.name"="kafka",
+"property.sasl.kerberos.keytab"="/path/to/xxx.keytab",
+"property.sasl.kerberos.principal"="<xx...@yyy.com>"
+);
+-- 查看routine的状态
+SHOW ROUTINE LOAD;
+```
+
+更多关于 Kafka 配置和使用的说明，可参考 [Routine Load](../data-operate/import/routine-load-manual.md)。
+
+**使用自定义程序采集日志**
+
+除了对接常用的日志采集器以外，你也可以自定义程序，通过 HTTP API Stream Load 将日志数据导入 Apache Doris。参考以下代码：
+
+```Bash
+curl \\
+\--location-trusted \\
+\-u username:password \\
+\-H "format:json" \\
+\-H "read_json_by_line:true" \\
+\-H "load_to_single_tablet:true" \\
+\-H "timeout:600" \\
+\-T logfile.json \\
+http://fe_host:fe_http_port/api/log_db/log_table/\_stream_load
+```
+
+在使用自定义程序时，需注意以下关键点：
+
+- 使用 Basic Auth 进行 HTTP 鉴权，用命令 `echo -n 'username:password' | base64` 进行计算。
+- 设置 HTTP header "format:json"，指定数据格式为 JSON。
+- 设置 HTTP header "read_json_by_line:true"，指定每行一个 JSON。
+- 设置 HTTP header "load_to_single_tablet:true"，指定一次导入写入一个分桶减少导入的小文件。
+- 建议写入客户端一个 Batch 的大小为 100MB ～ 1GB。如果你使用的是 Apache Doris 2.1 及更高版本，需通过服务端 Group Commit 功能，降低客户端 Batch 大小。
+
+**日志查询**
+
+Apache Doris 支持标准 SQL，因此，你可以通过 MySQL 客户端或者 JDBC 等方式连接到集群，执行 SQL 进行日志查询。参考以下命令：
+
+```
+mysql -h fe_host -P fe_mysql_port -u your_username -Dyour_db_name
+```
+
+下方列出常见的 5 条 SQL 查询命令，以供参考：
+
+- 查看最新的 10 条数据
+
+```SQL
+SELECT \* FROM your_table_name ORDER BY ts DESC LIMIT 10;
+```
+
+- 查询 `host` 为 `8.8.8.8` 的最新 10 条数据
+
+```SQL
+SELECT \* FROM your_table_name WHERE host = '8.8.8.8' ORDER BY ts DESC LIMIT 10;
+```
+
+- 检索请求字段中有 `error` 或者 `404` 的最新 10 条数据。其中，`MATCH_ANY` 是 Apache Doris 全文检索的 SQL 语法，用于匹配参数中任一关键字。
+
+```SQL
+SELECT \* FROM your_table_name WHERE message MATCH_ANY 'error 404'
+ORDER BY ts DESC LIMIT 10;
+```
+
+- 检索请求字段中有 `image` 和 `faq` 的最新 10 条数据。其中，`MATCH_ALL` 是 Apache Doris 全文检索的 SQL 语法，用于匹配参数中所有关键字。
+
+```SQL
+SELECT \* FROM your_table_name WHERE message MATCH_ALL 'image faq'
+ORDER BY ts DESC LIMIT 10;
+```
+
+- 检索请求字段中有 `image` 和 `faq` 的最新 10 条数据。其中，`MATCH_PHRASE` 是 Apache Doris 全文检索的 SQL 语法，用于匹配参数中所有关键字，并且要求顺序一致。在下方例子中，`a image faq b` 能匹配，但是 `a faq image b` 不能匹配，因为 `image` 和 `faq` 的顺序与查询不一致。
+
+```SQL
+SELECT \* FROM your_table_name WHERE message MATCH_PHRASE 'image faq'
+ORDER BY ts DESC LIMIT 10;
+```
+
+**可视化日志分析**
+
+基于 Apache Doris 构建的 SelectDB Enterprise Core 提供了名为 Doris WebUI 的数据开发平台，Doris WebUI 包含了类 Kibana Discover 的日志检索分析界面，提供直观、易用的探索式日志分析交互，如下图所示：
+
+
+
+在此界面上，Doris WebUI 主要支持以下功能：
+
+- 支持全文检索和 SQL 两种模式
+- 支持时间框和直方图上选择查询日志的时间段
+- 支持信息丰富的日志明细展示，还可以展开成 JSON 或表格
+- 在日志数据上下文交互式点击增加和删除筛选条件
+- 搜索结果的字段 Top 值展示，便于发现异常值和进一步下钻分析
+
+你可以 [点此下载 SelectDB Enterprise Core](https://www.selectdb.com/download/enterprise#core)，完成 [安装](https://docs.selectdb.com/docs/enterprise/enterprise-core-guide/selectdb-distribution-doris-core-deployment-guide) 后，即可使用 Doris WebUI 登录数据库。更多关于如何使用 Doris WebUI 的信息，可参考 [WebUI](https://docs.selectdb.com/docs/enterprise/enterprise-core-guide/selectdb-webui-guide)。
\ No newline at end of file
diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.0.json b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.0.json
index 44a2d22e1e3..0d847df2227 100644
--- a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.0.json
+++ b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.0.json
@@ -326,5 +326,9 @@
"sidebar.docs.category.Release notes": {
"message": "版本发布",
"description": "The label for category Release notes in sidebar docs"
+ },
+ "sidebar.docs.category.Practical Guide": {
+ "message": "实践教程",
+ "description": "The label for category Practical Guide in sidebar docs"
}
}
diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.0/practical-guide/log-storage-analysis.md b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.0/practical-guide/log-storage-analysis.md
new file mode 100644
index 00000000000..74595e26fea
--- /dev/null
+++ b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.0/practical-guide/log-storage-analysis.md
@@ -0,0 +1,564 @@
+---
+{
+ "title": "日志存储与分析",
+ "language": "zh-CN"
+}
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+日志是系统运行的详细记录，包含各种事件发生的主体、时间、位置、内容等关键信息。出于运维可观测、网络安全监控及业务分析等多重需求，企业通常需要将分散的日志采集起来，进行集中存储、查询和分析，以进一步从日志数据里挖掘出有价值的内容。
+
+针对此场景，Apache Doris 提供了相应解决方案，针对日志场景的特点，增加了倒排索引和极速全文检索能力，极致优化写入性能和存储空间，使得用户可以基于 Apache Doris 构建开放、高性能、低成本、统一的日志存储与分析平台。
+
+本文将围绕这一解决方案，介绍以下内容：
+
+- **整体架构**：说明基于 Apache Doris 构建的日志存储与分析平台的核心组成部分和基础架构。
+- **特点与优势**：说明基于 Apache Doris 构建的日志存储与分析平台的特点和优势。
+- **操作指南**：说明如何基于 Apache Doris 构建日志存储分析平台。
+
+## 整体架构
+
+基于 Apache Doris 构建的日志存储与分析平台的架构如下图：
+
+
+
+此架构主要由 3 大部分组成：
+
+- **日志采集和预处理**：多种日志采集工具可以通过 HTTP APIs 将日志数据写入 Apache Doris。
+- **日志存储和分析引擎**：Apache Doris 提供高性能、低成本的统一日志存储，通过 SQL 接口提供丰富的检索分析能力。
+- **日志分析和告警界面**：多种日志检索分析通工具通过标准 SQL 接口查询 Apache Doris，为用户提供简单易用的界面。
+
+## 特点与优势
+
+基于 Apache Doris 构建的日志存储与分析平台的特点和优势如下：
+
+- **高吞吐、低延迟日志写入**：支持每天百 TB 级、GB/s 级日志数据持续稳定写入，同时保持延迟 1s 以内。
+- **海量日志数据低成本存储**：支持 PB 级海量存储，相对于 Elasticsearch 存储成本节省 60% 到 80%，支持冷数据存储到 S3/HDFS，存储成本再降 50%。
+- **高性能日志全文检索分析**：支持倒排索引和全文检索，日志场景常见查询（关键词检索明细、趋势分析等）秒级响应。
+- **开放、易用的上下游生态**：上游通过 Stream Load 通用 HTTP APIs 对接常见的日志采集系统和数据源 Logstash、Filebeat、Fluentbit、Kafka 等，下游通过标准 MySQL 协议和语法对接各种可视化分析 UI，比如可观测性 Grafana、BI 分析 Superset、类 Kibana 的日志检索 Doris WebUI。
+
+### 高性能、低成本
+
+经过 Benchmark 测试及生产验证，基于 Apache Doris 构建的日志存储与分析平台，性价比相对于 Elasticsearch 具有 5～10 倍的提升。Apache Doris 的性能优势，主要得益于全球领先的高性能存储和查询引擎，以及下面一些针对日志场景的专门优化：
+
+- **写入吞吐提升**：Elasticsearch 写入的性能瓶颈在于解析数据和构建倒排索引的 CPU 消耗。相比之下，Apache Doris 进行了两方面的写入优化：一方面利用 SIMD 等 CPU 向量化指令提升了 JSON 数据解析速度和索引构建性能；另一方面针对日志场景简化倒了排索引结构，去掉日志场景不需要的正排等数据结构，有效降低了索引构建的复杂度。同样的资源，Apache Doris 的写入性能是 Elasticsearch 的 3～5 倍。
+- **存储成本降低**：Elasticsearch 存储瓶颈在于正排、倒排、Docvalue 列存多份存储和通用压缩算法压缩率较低。相比之下，Apache Doris 在存储上进行了以下优化：去掉正排，缩减了 30% 的索引数据量；采用列式存储和 Zstandard 压缩算法，压缩比可达到 5～10 倍，远高于 Elasticsearch 的 1.5 倍；日志数据中冷数据访问频率很低，Apache Doris 冷热分层功能可以将超过定义时间段的日志自动存储到更低的对象存储中，冷数据的存储成本可降低 70% 以上。同样的原始数据，SelectDB 的存储成本只需要 Elasticsearch 的 20% 左右。
+- **查询性能提升**：Apache Doris 将全文检索的流程简化，跳过了相关性打分等日志场景不需要的算法，加速基础的检索性能。同时针对日志场景常见的查询，比如查询包含某个关键字的最新 100 条日志，在查询规划和执行上做专门的 TopN 动态剪枝等优化。
+
+### 分析能力强
+
+Apache Doris 支持标准 SQL、兼容 MySQL 协议和语法，因此基于 Apache Doris 构建的日志系统能够使用 SQL 进行日志分析，这使得日志系统具备以下优势：
+
+- **简单易用**：工程师和数据分析师对于 SQL 非常熟悉，经验可以复用，不需要学习新的技术栈即可快速上手。
+- **生态丰富**：MySQL 生态是数据库领域使用最广泛的语言，因此可以与 MySQL 生态的集成和应用无缝衔接。Doris 可以利用 MySQL 命令行与各种 GUI 工具、BI 工具等大数据生态结合，实现更复杂及多样化的数据处理分析需求。
+- **分析能力强**：SQL 语言已经成为数据库和大数据分析的事实标准，它具有强大的表达能力和功能，支持检索、聚合、多表 JOIN、子查询、UDF、逻辑视图、物化视图等多种数据分析能力。
+
+### Flexible Schema
+
+下面是一个典型的 JSON 格式半结构化日志样例。顶层字段是一些比较固定的字段，比如日志时间戳（`timestamp`），日志来源（`source`），志所在机器（`node`），打日志的模块（`component`），日志级别（`level`），客户端请求标识（`clientRequestId`），日志内容（`message`），日志扩展属性（`properties`），基本上每条日志都会有。而扩展属性 `properties` 的内部嵌套字段 `properties.size`、`properties.format` 等是比较动态的，每条日志的字段可能不一样。
+
+```JSON
+{
+"timestamp": "2014-03-08T00:50:03.8432810Z",
+"source": "ADOPTIONCUSTOMERS81",
+"node": "Engine000000000405",
+"level": "Information",
+"component": "DOWNLOADER",
+"clientRequestId": "671db15d-abad-94f6-dd93-b3a2e6000672",
+"message": "Downloading file path: benchmark/2014/ADOPTIONCUSTOMERS81_94_0.parquet.gz",
+"properties": {
+"size": 1495636750,
+"format": "parquet",
+"rowCount": 855138,
+"downloadDuration": "00:01:58.3520561"
+}
+}
+```
+
+Apache Doris 对 Flexible Schema 的日志数据提供了几个方面的支持：
+
+- 对于顶层字段的少量变化，可以通过 Light Schema Change 发起 ADD / DROP COLUMN 增加 / 删除列，ADD / DROP INDEX 增加 / 删除索引，能够在秒级完成 Schema 变更。用户在日志平台规划时只需考虑当前需要哪些字段创建索引。
+- 对于类似 `properties` 的扩展字段，提供了原生半结构化数据类型 `VARIANT`，可以写入任何 JSON 数据，自动识别 JSON 中的字段名和类型，并自动拆分频繁出现的字段采用列式存储，以便于后续的分析，还可以对 `VARIANT` 创建倒排索引，加快内部字段的查询和检索。
+
+相对于 Elasticsearch 的 Dynamic Mapping，Apache Doris 的 Flexible Schema 有以下优势：
+
+- 允许一个字段有多种类型，`VARIANT` 自动对字段类型做冲突处理和类型提升，更好地适应日志数据的迭代变化。
+- `VARIANT` 自动将不频繁出现的字段合并成一个列存储，可避免字段、元数据、列过多导致性能问题。
+- 不仅可以动态加列，还可以动态删列、动态增加索引、动态删索引，无需像 Elasticsearch 在一开始对所有字段建索引，减少不必要的成本。
+
+## 操作指南
+
+### 第 1 步：评估资源
+
+在部署集群之前，首先应评估所需服务器硬件资源，包括以下几个关键步骤：
+
+1. **评估写入资源**：计算公式如下：
+
+ - 日增数据量 / 86400 s = 平均写入吞吐
+ - 平均写入吞吐 \* 写入吞吐峰值 / 均值比 = 峰值写入吞吐
+ - 峰值写入吞吐 / 单核写入吞吐 = 峰值写入所需 CPU 核数
+
+2. **评估存储资源**：计算公式为「日增数据量 / 压缩率 \* 副本数 \* 数据存储周期 = 所需存储空间」。
+
+3. **评估查询资源**：查询的资源消耗随查询量和复杂度而异，建议初始预留 50% 的 CPU 资源用于查询，再根据实际测试情况进行调整。
+
+4. **汇总整合资源**：由第 1 步和第 3 步估算出所需 CPU 核数后，除以单机 CPU 核数，估算出 BE 服务器数量，再根据 BE 服务器数量和第 2 步的结果，估算出每台 BE 服务器所需存储空间，然后分摊到 4～12 块数据盘，计算出单盘存储容量。
+
+以每天新增 100 TB 数据量（压缩前）、7 倍压缩率、1 副本、热数据存储 3 天、冷数据存储 30 天、写入吞吐峰值 / 均值比 200%、单核写入吞吐 10 MB/s、查询预留 50% CPU 资源为例，可估算出：
+
+- FE：3 台服务器，每台配置 16 核 CPU、64 GB 内存、1 块 100 GB SSD 盘
+- BE：15 台服务器，每台配置 32 核 CPU、256 GB 内存、8 块 500 GB SSD 盘
+- Amazon S3 存储空间：即为预估冷数据存储空间，430 TB
+
+该例子中，各关键指标的值及具体计算方法可见下表：
+
+| 关键指标（单位） | 值 | 说明 |
+| :------------------------------- | :---- | :----------------------------------------------------------- |
+| 日增数据量（TB） | 100 | 根据实际需求填写 |
+| 压缩率 | 7 | 一般为 5～10 倍（含索引），根据实际需求填写 |
+| 副本数 | 1 | 根据实际需求填写，默认 1 副本，可选值：1，2，3 |
+| 热数据存储周期（天） | 3 | 根据实际需求填写 |
+| 冷数据存储周期（天） | 30 | 根据实际需求填写 |
+| 总存储周期（天） | 33 | 算法：热数据存储周期 + 冷数据存储周期 |
+| 预估热数据存储空间（TB） | 42.9 | 算法：日增数据量 / 压缩率 * 副本数 * 热数据存储周期 |
+| 预估冷数据存储空间（TB） | 428.6 | 算法：日增数据量 / 压缩率 * 副本数 * 冷数据存储周期 |
+| 写入吞吐峰值 / 均值比 | 200% | 根据实际需求填写，默认 200% |
+| 单机 CPU 核数 | 32 | 根据实际需求填写，默认 32 核 |
+| 平均写入吞吐（MB/s） | 1214 | 算法：日增数据量 / 86400 s |
+| 峰值写入吞吐（MB/s） | 2427 | 算法：平均写入吞吐 * 写入吞吐峰值 / 均值比 |
+| 峰值写入所需 CPU 核数 | 242.7 | 算法：峰值写入吞吐 / 单机 CPU 核数 |
+| 查询预留 CPU 百分比 | 50% | 根据实际需求填写，默认 50% |
+| 预估 BE 服务器数 | 15.2 | 算法：峰值写入所需 CPU 核数 / 单机 CPU 核数 /（1 - 查询预留 CPU 百分比） |
+| 预估 BE 服务器数取整 | 15 | 算法：MAX (副本数，预估 BE 服务器数上取整) |
+| 预估每台 BE 服务器存储空间（TB） | 4.03 | 算法：预估热数据存储空间 / 预估 BE 服务器数 /（1 - 30%），其中，30% 是存储空间预留值。建议每台 BE 服务器挂载 4～12 块数据盘，以提高 I/O 能力。 |
+
+### 第 2 步：部署集群
+
+完成资源评估后，可以开始部署 Apache Doris 集群，推荐在物理机及虚拟机环境中进行部署。手动部署集群，可参考 [手动部署](../install/cluster-deployment/standard-deployment.md)。
+
+另，推荐使用 SelectDB Enterprise 推出的 Cluster Manager 工具部署集群，以降低整体部署成本。更多关于 Cluster Manager 的信息，可参考以下文档：
+
+- [Cluster Manager for Apache Doris 24.x 安装手册](https://docs.selectdb.com/docs/enterprise/cluster-manager-guide/deployment-guide/deployment-guide-24.x)
+- [Cluster Manager for Apache Doris 24.x 使用手册](https://docs.selectdb.com/docs/enterprise/cluster-manager-guide/management-guide/management-guide-24.x)
+
+### 第 3 步：优化 FE 和 BE 配置
+
+完成集群部署后，需分别优化 FE 和 BE 配置参数，以更加契合日志存储与分析的场景。
+
+**优化 FE 配置**
+
+在 `fe/conf/fe.conf` 目录下找到 FE 的相关配置项，并按照以下表格，调整 FE 配置。
+
+| 需调整参数 | 说明 |
+| :----------------------------------------------------------- | :----------------------------------------------------------- |
+| `max_running_txn_num_per_db = 10000` | 高并发导入运行事务数较多，需调高参数。 |
+| `streaming_lable_keep_max_second = 3600``label_keep_max_second = 7200` | 高频导入事务标签内存占用多，保留时间调短。 |
+| `enable_round_robin_create_tablet = true` | 创建 Tablet 时，采用 Round Robin 策略，尽量均匀。 |
+| `tablet_rebalancer_type = partition` | 均衡 Tablet 时，采用每个分区内尽量均匀的策略。 |
+| `enable_single_replica_load = true` | 开启单副本导入，多个副本只需构建一次索引，减少 CPU 消耗。 |
+| `autobucket_min_buckets = 10` | 将自动分桶的最小分桶数从 1 调大到 10，避免日志量增加时分桶不够。 |
+| `max_backend_heartbeat_failure_tolerance_count = 10` | 日志场景下 BE 服务器压力较大，可能短时间心跳超时，因此将容忍次数从 1 调大到 10。 |
+
+更多关于 FE 配置项的信息，可参考 [FE 配置项](../admin-manual/config/fe-config.md)。
+
+**优化 BE 配置**
+
+在 `be/conf/be.conf` 目录下找到 BE 的相关配置项，并按照以下表格，调整 BE 配置。
+
+| 模块 | 需调整参数 | 说明 |
+| :--------- | :----------------------------------------------------------- | :----------------------------------------------------------- |
+| 存储 | `storage_root_path = /path/to/dir1;/path/to/dir2;...;/path/to/dir12` | 配置热数据在磁盘目录上的存储路径。 |
+| - | `enable_file_cache = true` | 开启文件缓存。 |
+| - | `file_cache_path = [{"path": "/mnt/datadisk0/file_cache", "total_size":53687091200, "query_limit": "10737418240"},{"path": "/mnt/datadisk1/file_cache", "total_size":53687091200,"query_limit": "10737418240"}]` | 配置冷数据的缓存路径和相关设置，具体配置说明如下：<br/>`path`：缓存路径<br/>`total_size`：该缓存路径的总大小，单位为字节，53687091200 字节等于 50 GB<br/>`query_limit`：单次查询可以从缓存路径中查询的最大数据量，单位为字节，10737418240 字节等于 10 GB |
+| 写入 | `write_buffer_size = 1073741824` | 增加写入缓冲区（buffer）的文件大小，减少小文件和随机 I/O 操作，提升性能。 |
+| - | `max_tablet_version_num = 20000` | 配合建表的 time_series compaction 策略，允许更多版本暂时未合并。 |
+| - | `enable_single_replica_load = true` | 开启单副本写入，减少 CPU 消耗。与 FE 配置相同。 |
+| Compaction | `max_cumu_compaction_threads = 8` | 设置为 CPU 核数 / 4，意味着 CPU 资源的 1/4 用于写入，1/4 用于后台 Compaction，2/1 留给查询和其他操作。 |
+| - | `inverted_index_compaction_enable = true` | 开启索引合并（index compaction），减少 Compaction 时的 CPU 消耗。 |
+| - | `enable_segcompaction = false` `enable_ordered_data_compaction = false` | 关闭日志场景不需要的两个 Compaction 功能。 |
+| 缓存 | `disable_storage_page_cache = true` `inverted_index_searcher_cache_limit = 30%` | 因为日志数据量较大，缓存（cache）作用有限，因此关闭数据缓存，调换为索引缓存（index cache）的方式。 |
+| - | `inverted_index_cache_stale_sweep_time_sec = 3600` `index_cache_entry_stay_time_after_lookup_s = 3600` | 让索引缓存在内存中尽量保留 1 小时。 |
+| - | `enable_inverted_index_cache_on_cooldown = true` <br />`enable_write_index_searcher_cache = false` | 开启索引上传冷数据存储时自动缓存的功能。 |
+| - | `tablet_schema_cache_recycle_interval = 3600` `segment_cache_capacity = 20000` | 减少其他缓存对内存的占用。 |
+| 线程 | `pipeline_executor_size = 24` `doris_scanner_thread_pool_thread_num = 48` | 32 核 CPU 的计算线程和 I/O 线程配置，根据核数等比扩缩。 |
+| - | `scan_thread_nice_value = 5` | 降低查询 I/O 线程的优先级，保证写入性能和时效性。 |
+| 其他 | `string_type_length_soft_limit_bytes = 10485760` | 将 String 类型数据的长度限制调高至 10 MB。 |
+| - | `trash_file_expire_time_sec = 300` `path_gc_check_interval_second = 900` `path_scan_interval_second = 900` | 调快垃圾文件的回收时间。 |
+
+更多关于 BE 配置项的信息，可参考 [BE 配置项](../admin-manual/config/be-config.md)。
+
+### 第 4 步：建表
+
+由于日志数据的写入和查询都具备明显的特征，因此，在建表时按照本节说明进行针对性配置，以提升性能表现。
+
+**配置分区分桶参数**
+
+- 分区时，按照以下说明配置：
+- 使用时间字段上的 [Range 分区](https://doris.apache.org/zh-CN/docs/2.0/table-design/data-partition/#range-%E5%88%86%E5%8C%BA)，并开启 [动态分区](https://doris.apache.org/zh-CN/docs/2.0/table-design/data-partition#%E5%8A%A8%E6%80%81%E5%88%86%E5%8C%BA)，按天自动管理分区。
+- 使用 Datetime 类型的时间字段作为 Key，在查询最新 N 条日志时有数倍加速。
+- 分桶时，按照以下说明配置：
+- 分桶数量大致为集群磁盘总数的 3 倍。
+- 使用 Random 策略，配合写入时的 Single Tablet 导入，可以提升批量（Batch）写入的效率。
+
+更多关于分区分桶的信息，可参考 [分区分桶](../table-design/data-partition.md)。
+
+**配置 Compaction 参数**
+
+按照以下说明配置 Compaction 参数：
+
+- 使用 time_series 策略，以减轻写放大效应，对于高吞吐日志写入的资源写入很重要。
+- 使用单副本 Compaction，减少多副本 Compaction 的开销。
+
+**建立和配置索引参数**
+
+按照以下说明操作：
+
+- 对经常查询的字段建立索引。
+- 对需要全文检索的字段，将分词器（parser）参数赋值为 unicode，一般能满足大部分需求。如有支持短语查询的需求，将 support_phrase 参数赋值为 true；如不需要，则设置为 false，以降低存储空间。
+
+**配置存储策略**
+
+按照以下说明操作：
+
+- 对于热存储数据，如果使用云盘，可配置 1 副本；如果使用物理盘，则至少配置 2 副本。
+- 配置 `log_s3` 的存储位置，并设置 `log_policy_3day` 冷热数据分层策略，即在超过 3 天后将数据冷却至 `log_s3` 指定的存储位置。可参考以下代码：
+
+```Go
+CREATE DATABASE log_db;
+USE log_db;
+
+CREATE RESOURCE "log_s3"
+PROPERTIES
+(
+ "type" = "s3",
+ "s3.endpoint" = "your_endpoint_url",
+ "s3.region" = "your_region",
+ "s3.bucket" = "your_bucket",
+ "s3.root.path" = "your_path",
+ "s3.access_key" = "your_ak",
+ "s3.secret_key" = "your_sk"
+);
+
+CREATE STORAGE POLICY log_policy_3day
+PROPERTIES(
+ "storage_resource" = "log_s3",
+ "cooldown_ttl" = "259200"
+);
+
+CREATE TABLE log_table
+(
+ `ts` DATETIME,
+ `host` TEXT,
+ `path` TEXT,
+ `message` TEXT,
+ INDEX idx_host (`host`) USING INVERTED,
+ INDEX idx_path (`path`) USING INVERTED,
+ INDEX idx_message (`message`) USING INVERTED PROPERTIES("parser" = "unicode", "support_phrase" = "true")
+)
+ENGINE = OLAP
+DUPLICATE KEY(`ts`)
+PARTITION BY RANGE(`ts`) ()
+DISTRIBUTED BY RANDOM BUCKETS 250
+PROPERTIES (
+"dynamic_partition.enable" = "true",
+"dynamic_partition.create_history_partition" = "true",
+"dynamic_partition.time_unit" = "DAY",
+"dynamic_partition.start" = "-30",
+"dynamic_partition.end" = "1",
+"dynamic_partition.prefix" = "p",
+"dynamic_partition.buckets" = "250",
+"dynamic_partition.replication_num" = "1", -- 存算分离不需要
+"replication_num" = "1" -- 存算分离不需要
+"enable_single_replica_compaction" = "true", -- 存算分离不需要
+"storage_policy" = "log_policy_3day", -- 存算分离不需要
+"compaction_policy" = "time_series"
+);
+```
+
+### 第 5 步：采集和查询日志
+
+完成建表后，可进行日志采集、查询和分析。
+
+**日志采集**
+
+Apache Doris 提供开放、通用的 Stream HTTP APIs，通过这些 APIs，你可与常用的日志采集器打通，包括 Logstash、Filebeat、Kafka 等，从而开展日志采集工作。本节介绍了如何使用 Stream HTTP APIs 对接日志采集器。
+
+**对接 Logstash**
+
+按照以下步骤操作：
+
+1. 下载并安装 Logstash Doris Output 插件。你可选择以下两种方式之一：
+
+- 直接下载：[点此下载](https://apache-doris-releases.oss-accelerate.aliyuncs.com/logstash-output-doris-1.0.0.gem)。
+
+- 从源码编译，并运行下方命令安装：
+
+```markdown
+./bin/logstash-plugin install logstash-output-doris-1.0.0.gem
+```
+
+2. 配置 Logstash。需配置以下参数：
+
+- `logstash.yml`：配置 Logstash 批处理日志的条数和时间，用于提升数据写入性能。
+
+```markdown
+pipeline.batch.size: 1000000
+pipeline.batch.delay: 10000
+```
+
+
+- `logstash_demo.conf`：配置所采集日志的具体输入路径和输出到 Apache Doris 的设置。
+
+```markdown
+input {
+file {
+path => "/path/to/your/log"
+}
+}
+<br />output {
+doris {
+http_hosts => \[ "<http://fehost1:http_port>", "<http://fehost2:http_port>", "<http://fehost3:http_port"\>]
+user => "your_username"
+password => "your_password"
+db => "your_db"
+table => "your_table"
+\# doris stream load http headers
+headers => {
+"format" => "json"
+"read_json_by_line" => "true"
+"load_to_single_tablet" => "true"
+}
+\# field mapping: doris fileld name => logstash field name
+\# %{} to get a logstash field, \[\] for nested field such as \[host\]\[name\] for host.name
+mapping => {
+"ts" => "%{@timestamp}"
+"host" => "%{\[host\]\[name\]}"
+"path" => "%{\[log\]\[file\]\[path\]}"
+"message" => "%{message}"
+}
+log_request => true
+log_speed_interval => 10
+}
+}
+```
+
+3. 按照下方命令运行 Logstash，采集日志并输出至 Apache Doris。
+
+```Bash
+./bin/logstash -f logstash_demo.conf
+```
+
+更多关于 Logstash 配置和使用的说明，可参考 [Logstash Doris Output Plugin](../ecosystem/logstash.md)。
+
+**对接 Filebeat**
+
+按照以下步骤操作：
+
+1. 获取支持输出至 Apache Doris 的 Filebeat 二进制文件。可 [点此下载](https://apache-doris-releases.oss-accelerate.aliyuncs.com/filebeat-doris-1.0.0) 或者从 Apache Doris 源码编译。
+2. 配置 Filebeat。需配置以下参数：
+
+- `filebeat_demo.yml`：配置所采集日志的具体输入路径和输出到 Apache Doris 的设置。
+
+```YAML
+ # input
+ filebeat.inputs:
+ - type: log
+ enabled: true
+ paths:
+ - /path/to/your/log
+ # multiline 可以将跨行的日志（比如 Java stacktrace）拼接起来
+ multiline:
+ type: pattern
+ # 效果：以 yyyy-mm-dd HH:MM:SS 开头的行认为是一条新的日志，其他都拼接到上一条日志
+ pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}'
+ negate: true
+ match: after
+ skip_newline: true
+
+ processors:
+ # 用 js script 插件将日志中的 \t 替换成空格，避免 JSON 解析报错
+ - script:
+ lang: javascript
+ source: >
+ function process(event) {
+ var msg = event.Get("message");
+ msg = msg.replace(/\t/g, " ");
+ event.Put("message", msg);
+ }
+ # 用 dissect 插件做简单的日志解析
+ - dissect:
+ # 2024-06-08 18:26:25,481 INFO (report-thread|199) [ReportHandler.cpuReport():617] begin to handle
+ tokenizer: "%{day} %{time} %{log_level} (%{thread}) [%{position}] %{content}"
+ target_prefix: ""
+ ignore_failure: true
+ overwrite_keys: true
+
+ # queue and batch
+ queue.mem:
+ events: 1000000
+ flush.min_events: 100000
+ flush.timeout: 10s
+
+ # output
+ output.doris:
+ fenodes: [ "http://fehost1:http_port", "http://fehost2:http_port", "http://fehost3:http_port" ]
+ user: "your_username"
+ password: "your_password"
+ database: "your_db"
+ table: "your_table"
+ # output string format
+ ## %{[agent][hostname]} %{[log][file][path]} 是filebeat自带的metadata
+ ## 常用的 filebeat metadata 还是有采集时间戳 %{[@timestamp]}
+ ## %{[day]} %{[time]} 是上面 dissect 解析得到字段
+ codec_format_string: '{"ts": "%{[day]} %{[time]}", "host": "%{[agent][hostname]}", "path": "%{[log][file][path]}", "message": "%{[message]}"}'
+ headers:
+ format: "json"
+ read_json_by_line: "true"
+ load_to_single_tablet: "true"
+ ```
+
+3. 按照下方命令运行 Filebeat，采集日志并输出至 Apache Doris。
+
+```Bash
+chmod +x filebeat-doris-1.0.0
+./filebeat-doris-1.0.0 -c filebeat_demo.yml
+```
+
+更多关于 Filebeat 配置和使用的说明，可参考 [Beats Doris Output Plugin](../ecosystem/beats.md)。
+
+**对接 Kafka**
+
+将 JSON 格式的日志写入 Kafka 的消息队列，创建 Kafka Routine Load，即可让 Apache Doris 从 Kafka 主动拉取数据。
+
+可参考如下示例。其中，`property.\*` 是 Librdkafka 客户端相关配置，根据实际 Kafka 集群情况配置。
+
+```SQL
+\-- 准备好kafka集群和topic log_\_topic_
+\-- 创建routine load，从kafka log_\_topic_将数据导入log_table表
+CREATE ROUTINE LOAD load_log_kafka ON log_db.log_table
+COLUMNS(ts, clientip, request, status, size)
+PROPERTIES (
+"max_batch_interval" = "10",
+"max_batch_rows" = "1000000",
+"max_batch_size" = "109715200",
+"load_to_single_tablet" = "true",
+"timeout" = "600",
+"strict_mode" = "false",
+"format" = "json"
+)
+FROM KAFKA (
+"kafka_broker_list" = "host:port",
+"kafka_topic" = "log_\_topic_",
+"property.group.id" = "your_group_id",
+"property.security.protocol"="SASL_PLAINTEXT",
+"property.sasl.mechanism"="GSSAPI",
+"property.sasl.kerberos.service.name"="kafka",
+"property.sasl.kerberos.keytab"="/path/to/xxx.keytab",
+"property.sasl.kerberos.principal"="<xx...@yyy.com>"
+);
+\-- 查看routine的状态
+SHOW ROUTINE LOAD;
+```
+
+更多关于 Kafka 配置和使用的说明，可参考 [Routine Load](../data-operate/import/routine-load-manual.md)。
+
+**使用自定义程序采集日志**
+
+除了对接常用的日志采集器以外，你也可以自定义程序，通过 HTTP API Stream Load 将日志数据导入 Apache Doris。参考以下代码：
+
+```Bash
+curl \\
+\--location-trusted \\
+\-u username:password \\
+\-H "format:json" \\
+\-H "read_json_by_line:true" \\
+\-H "load_to_single_tablet:true" \\
+\-H "timeout:600" \\
+\-T logfile.json \\
+http://fe_host:fe_http_port/api/log_db/log_table/\_stream_load
+```
+
+在使用自定义程序时，需注意以下关键点：
+
+- 使用 Basic Auth 进行 HTTP 鉴权，用命令 `echo -n 'username:password' | base64` 进行计算。
+- 设置 HTTP header "format:json"，指定数据格式为 JSON。
+- 设置 HTTP header "read_json_by_line:true"，指定每行一个 JSON。
+- 设置 HTTP header "load_to_single_tablet:true"，指定一次导入写入一个分桶减少导入的小文件。
+- 建议写入客户端一个 Batch 的大小为 100MB ～ 1GB。如果你使用的是 Apache Doris 2.1 及更高版本，需通过服务端 Group Commit 功能，降低客户端 Batch 大小。
+
+**日志查询**
+
+Apache Doris 支持标准 SQL，因此，你可以通过 MySQL 客户端或者 JDBC 等方式连接到集群，执行 SQL 进行日志查询。参考以下命令：
+
+```
+mysql -h fe_host -P fe_mysql_port -u your_username -Dyour_db_name
+```
+
+下方列出常见的 5 条 SQL 查询命令，以供参考：
+
+- 查看最新的 10 条数据
+
+```SQL
+SELECT \* FROM your_table_name ORDER BY ts DESC LIMIT 10;
+```
+
+- 查询 `host` 为 `8.8.8.8` 的最新 10 条数据
+
+```SQL
+SELECT \* FROM your_table_name WHERE host = '8.8.8.8' ORDER BY ts DESC LIMIT 10;
+```
+
+- 检索请求字段中有 `error` 或者 `404` 的最新 10 条数据。其中，`MATCH_ANY` 是 Apache Doris 全文检索的 SQL 语法，用于匹配参数中任一关键字。
+
+```SQL
+SELECT \* FROM your_table_name WHERE message MATCH_ANY 'error 404'
+ORDER BY ts DESC LIMIT 10;
+```
+
+- 检索请求字段中有 `image` 和 `faq` 的最新 10 条数据。其中，`MATCH_ALL` 是 Apache Doris 全文检索的 SQL 语法，用于匹配参数中所有关键字。
+
+```SQL
+SELECT \* FROM your_table_name WHERE message MATCH_ALL 'image faq'
+ORDER BY ts DESC LIMIT 10;
+```
+
+- 检索请求字段中有 `image` 和 `faq` 的最新 10 条数据。其中，`MATCH_PHRASE` 是 Apache Doris 全文检索的 SQL 语法，用于匹配参数中所有关键字，并且要求顺序一致。在下方例子中，`a image faq b` 能匹配，但是 `a faq image b` 不能匹配，因为 `image` 和 `faq` 的顺序与查询不一致。
+
+```SQL
+SELECT \* FROM your_table_name WHERE message MATCH_PHRASE 'image faq'
+ORDER BY ts DESC LIMIT 10;
+```
+
+**可视化日志分析**
+
+基于 Apache Doris 构建的 SelectDB Enterprise Core 提供了名为 Doris WebUI 的数据开发平台，Doris WebUI 包含了类 Kibana Discover 的日志检索分析界面，提供直观、易用的探索式日志分析交互，如下图所示：
+
+
+
+在此界面上，Doris WebUI 主要支持以下功能：
+
+- 支持全文检索和 SQL 两种模式
+- 支持时间框和直方图上选择查询日志的时间段
+- 支持信息丰富的日志明细展示，还可以展开成 JSON 或表格
+- 在日志数据上下文交互式点击增加和删除筛选条件
+- 搜索结果的字段 Top 值展示，便于发现异常值和进一步下钻分析
+
+你可以 [点此下载 SelectDB Enterprise Core](https://www.selectdb.com/download/enterprise#core)，完成 [安装](https://docs.selectdb.com/docs/enterprise/enterprise-core-guide/selectdb-distribution-doris-core-deployment-guide) 后，即可使用 Doris WebUI 登录数据库。更多关于如何使用 Doris WebUI 的信息，可参考 [WebUI](https://docs.selectdb.com/docs/enterprise/enterprise-core-guide/selectdb-webui-guide)。
\ No newline at end of file
diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1.json b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1.json
index d4ceeeb030d..ee82fd1bcfe 100644
--- a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1.json
+++ b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1.json
@@ -370,5 +370,9 @@
"sidebar.docs.category.External Table": {
"message": "外部表",
"description": "The label for category Lakehouse.External Table in sidebar docs"
+ },
+ "sidebar.docs.category.Practical Guide": {
+ "message": "实践教程",
+ "description": "The label for category Practical Guide in sidebar docs"
}
}
diff --git a/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/practical-guide/log-storage-analysis.md b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/practical-guide/log-storage-analysis.md
new file mode 100644
index 00000000000..df77b71dd65
--- /dev/null
+++ b/i18n/zh-CN/docusaurus-plugin-content-docs/version-2.1/practical-guide/log-storage-analysis.md
@@ -0,0 +1,564 @@
+---
+{
+ "title": "日志存储与分析",
+ "language": "zh-CN"
+}
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+日志是系统运行的详细记录，包含各种事件发生的主体、时间、位置、内容等关键信息。出于运维可观测、网络安全监控及业务分析等多重需求，企业通常需要将分散的日志采集起来，进行集中存储、查询和分析，以进一步从日志数据里挖掘出有价值的内容。
+
+针对此场景，Apache Doris 提供了相应解决方案，针对日志场景的特点，增加了倒排索引和极速全文检索能力，极致优化写入性能和存储空间，使得用户可以基于 Apache Doris 构建开放、高性能、低成本、统一的日志存储与分析平台。
+
+本文将围绕这一解决方案，介绍以下内容：
+
+- **整体架构**：说明基于 Apache Doris 构建的日志存储与分析平台的核心组成部分和基础架构。
+- **特点与优势**：说明基于 Apache Doris 构建的日志存储与分析平台的特点和优势。
+- **操作指南**：说明如何基于 Apache Doris 构建日志存储分析平台。
+
+## 整体架构
+
+基于 Apache Doris 构建的日志存储与分析平台的架构如下图：
+
+
+
+此架构主要由 3 大部分组成：
+
+- **日志采集和预处理**：多种日志采集工具可以通过 HTTP APIs 将日志数据写入 Apache Doris。
+- **日志存储和分析引擎**：Apache Doris 提供高性能、低成本的统一日志存储，通过 SQL 接口提供丰富的检索分析能力。
+- **日志分析和告警界面**：多种日志检索分析通工具通过标准 SQL 接口查询 Apache Doris，为用户提供简单易用的界面。
+
+## 特点与优势
+
+基于 Apache Doris 构建的日志存储与分析平台的特点和优势如下：
+
+- **高吞吐、低延迟日志写入**：支持每天百 TB 级、GB/s 级日志数据持续稳定写入，同时保持延迟 1s 以内。
+- **海量日志数据低成本存储**：支持 PB 级海量存储，相对于 Elasticsearch 存储成本节省 60% 到 80%，支持冷数据存储到 S3/HDFS，存储成本再降 50%。
+- **高性能日志全文检索分析**：支持倒排索引和全文检索，日志场景常见查询（关键词检索明细、趋势分析等）秒级响应。
+- **开放、易用的上下游生态**：上游通过 Stream Load 通用 HTTP APIs 对接常见的日志采集系统和数据源 Logstash、Filebeat、Fluentbit、Kafka 等，下游通过标准 MySQL 协议和语法对接各种可视化分析 UI，比如可观测性 Grafana、BI 分析 Superset、类 Kibana 的日志检索 Doris WebUI。
+
+### 高性能、低成本
+
+经过 Benchmark 测试及生产验证，基于 Apache Doris 构建的日志存储与分析平台，性价比相对于 Elasticsearch 具有 5～10 倍的提升。Apache Doris 的性能优势，主要得益于全球领先的高性能存储和查询引擎，以及下面一些针对日志场景的专门优化：
+
+- **写入吞吐提升**：Elasticsearch 写入的性能瓶颈在于解析数据和构建倒排索引的 CPU 消耗。相比之下，Apache Doris 进行了两方面的写入优化：一方面利用 SIMD 等 CPU 向量化指令提升了 JSON 数据解析速度和索引构建性能；另一方面针对日志场景简化倒了排索引结构，去掉日志场景不需要的正排等数据结构，有效降低了索引构建的复杂度。同样的资源，Apache Doris 的写入性能是 Elasticsearch 的 3～5 倍。
+- **存储成本降低**：Elasticsearch 存储瓶颈在于正排、倒排、Docvalue 列存多份存储和通用压缩算法压缩率较低。相比之下，Apache Doris 在存储上进行了以下优化：去掉正排，缩减了 30% 的索引数据量；采用列式存储和 Zstandard 压缩算法，压缩比可达到 5～10 倍，远高于 Elasticsearch 的 1.5 倍；日志数据中冷数据访问频率很低，Apache Doris 冷热分层功能可以将超过定义时间段的日志自动存储到更低的对象存储中，冷数据的存储成本可降低 70% 以上。同样的原始数据，SelectDB 的存储成本只需要 Elasticsearch 的 20% 左右。
+- **查询性能提升**：Apache Doris 将全文检索的流程简化，跳过了相关性打分等日志场景不需要的算法，加速基础的检索性能。同时针对日志场景常见的查询，比如查询包含某个关键字的最新 100 条日志，在查询规划和执行上做专门的 TopN 动态剪枝等优化。
+
+### 分析能力强
+
+Apache Doris 支持标准 SQL、兼容 MySQL 协议和语法，因此基于 Apache Doris 构建的日志系统能够使用 SQL 进行日志分析，这使得日志系统具备以下优势：
+
+- **简单易用**：工程师和数据分析师对于 SQL 非常熟悉，经验可以复用，不需要学习新的技术栈即可快速上手。
+- **生态丰富**：MySQL 生态是数据库领域使用最广泛的语言，因此可以与 MySQL 生态的集成和应用无缝衔接。Doris 可以利用 MySQL 命令行与各种 GUI 工具、BI 工具等大数据生态结合，实现更复杂及多样化的数据处理分析需求。
+- **分析能力强**：SQL 语言已经成为数据库和大数据分析的事实标准，它具有强大的表达能力和功能，支持检索、聚合、多表 JOIN、子查询、UDF、逻辑视图、物化视图等多种数据分析能力。
+
+### Flexible Schema
+
+下面是一个典型的 JSON 格式半结构化日志样例。顶层字段是一些比较固定的字段，比如日志时间戳（`timestamp`），日志来源（`source`），志所在机器（`node`），打日志的模块（`component`），日志级别（`level`），客户端请求标识（`clientRequestId`），日志内容（`message`），日志扩展属性（`properties`），基本上每条日志都会有。而扩展属性 `properties` 的内部嵌套字段 `properties.size`、`properties.format` 等是比较动态的，每条日志的字段可能不一样。
+
+```JSON
+{
+"timestamp": "2014-03-08T00:50:03.8432810Z",
+"source": "ADOPTIONCUSTOMERS81",
+"node": "Engine000000000405",
+"level": "Information",
+"component": "DOWNLOADER",
+"clientRequestId": "671db15d-abad-94f6-dd93-b3a2e6000672",
+"message": "Downloading file path: benchmark/2014/ADOPTIONCUSTOMERS81_94_0.parquet.gz",
+"properties": {
+"size": 1495636750,
+"format": "parquet",
+"rowCount": 855138,
+"downloadDuration": "00:01:58.3520561"
+}
+}
+```
+
+Apache Doris 对 Flexible Schema 的日志数据提供了几个方面的支持：
+
+- 对于顶层字段的少量变化，可以通过 Light Schema Change 发起 ADD / DROP COLUMN 增加 / 删除列，ADD / DROP INDEX 增加 / 删除索引，能够在秒级完成 Schema 变更。用户在日志平台规划时只需考虑当前需要哪些字段创建索引。
+- 对于类似 `properties` 的扩展字段，提供了原生半结构化数据类型 `VARIANT`，可以写入任何 JSON 数据，自动识别 JSON 中的字段名和类型，并自动拆分频繁出现的字段采用列式存储，以便于后续的分析，还可以对 `VARIANT` 创建倒排索引，加快内部字段的查询和检索。
+
+相对于 Elasticsearch 的 Dynamic Mapping，Apache Doris 的 Flexible Schema 有以下优势：
+
+- 允许一个字段有多种类型，`VARIANT` 自动对字段类型做冲突处理和类型提升，更好地适应日志数据的迭代变化。
+- `VARIANT` 自动将不频繁出现的字段合并成一个列存储，可避免字段、元数据、列过多导致性能问题。
+- 不仅可以动态加列，还可以动态删列、动态增加索引、动态删索引，无需像 Elasticsearch 在一开始对所有字段建索引，减少不必要的成本。
+
+## 操作指南
+
+### 第 1 步：评估资源
+
+在部署集群之前，首先应评估所需服务器硬件资源，包括以下几个关键步骤：
+
+1. **评估写入资源**：计算公式如下：
+
+ - 日增数据量 / 86400 s = 平均写入吞吐
+ - 平均写入吞吐 \* 写入吞吐峰值 / 均值比 = 峰值写入吞吐
+ - 峰值写入吞吐 / 单核写入吞吐 = 峰值写入所需 CPU 核数
+
+2. **评估存储资源**：计算公式为「日增数据量 / 压缩率 \* 副本数 \* 数据存储周期 = 所需存储空间」。
+
+3. **评估查询资源**：查询的资源消耗随查询量和复杂度而异，建议初始预留 50% 的 CPU 资源用于查询，再根据实际测试情况进行调整。
+
+4. **汇总整合资源**：由第 1 步和第 3 步估算出所需 CPU 核数后，除以单机 CPU 核数，估算出 BE 服务器数量，再根据 BE 服务器数量和第 2 步的结果，估算出每台 BE 服务器所需存储空间，然后分摊到 4～12 块数据盘，计算出单盘存储容量。
+
+以每天新增 100 TB 数据量（压缩前）、7 倍压缩率、1 副本、热数据存储 3 天、冷数据存储 30 天、写入吞吐峰值 / 均值比 200%、单核写入吞吐 10 MB/s、查询预留 50% CPU 资源为例，可估算出：
+
+- FE：3 台服务器，每台配置 16 核 CPU、64 GB 内存、1 块 100 GB SSD 盘
+- BE：15 台服务器，每台配置 32 核 CPU、256 GB 内存、8 块 500 GB SSD 盘
+- Amazon S3 存储空间：即为预估冷数据存储空间，430 TB
+
+该例子中，各关键指标的值及具体计算方法可见下表：
+
+| 关键指标（单位） | 值 | 说明 |
+| :------------------------------- | :---- | :----------------------------------------------------------- |
+| 日增数据量（TB） | 100 | 根据实际需求填写 |
+| 压缩率 | 7 | 一般为 5～10 倍（含索引），根据实际需求填写 |
+| 副本数 | 1 | 根据实际需求填写，默认 1 副本，可选值：1，2，3 |
+| 热数据存储周期（天） | 3 | 根据实际需求填写 |
+| 冷数据存储周期（天） | 30 | 根据实际需求填写 |
+| 总存储周期（天） | 33 | 算法：热数据存储周期 + 冷数据存储周期 |
+| 预估热数据存储空间（TB） | 42.9 | 算法：日增数据量 / 压缩率 * 副本数 * 热数据存储周期 |
+| 预估冷数据存储空间（TB） | 428.6 | 算法：日增数据量 / 压缩率 * 副本数 * 冷数据存储周期 |
+| 写入吞吐峰值 / 均值比 | 200% | 根据实际需求填写，默认 200% |
+| 单机 CPU 核数 | 32 | 根据实际需求填写，默认 32 核 |
+| 平均写入吞吐（MB/s） | 1214 | 算法：日增数据量 / 86400 s |
+| 峰值写入吞吐（MB/s） | 2427 | 算法：平均写入吞吐 * 写入吞吐峰值 / 均值比 |
+| 峰值写入所需 CPU 核数 | 242.7 | 算法：峰值写入吞吐 / 单机 CPU 核数 |
+| 查询预留 CPU 百分比 | 50% | 根据实际需求填写，默认 50% |
+| 预估 BE 服务器数 | 15.2 | 算法：峰值写入所需 CPU 核数 / 单机 CPU 核数 /（1 - 查询预留 CPU 百分比） |
+| 预估 BE 服务器数取整 | 15 | 算法：MAX (副本数，预估 BE 服务器数上取整) |
+| 预估每台 BE 服务器存储空间（TB） | 4.03 | 算法：预估热数据存储空间 / 预估 BE 服务器数 /（1 - 30%），其中，30% 是存储空间预留值。建议每台 BE 服务器挂载 4～12 块数据盘，以提高 I/O 能力。 |
+
+### 第 2 步：部署集群
+
+完成资源评估后，可以开始部署 Apache Doris 集群，推荐在物理机及虚拟机环境中进行部署。手动部署集群，可参考 [手动部署](../install/cluster-deployment/standard-deployment.md)。
+
+另，推荐使用 SelectDB Enterprise 推出的 Cluster Manager 工具部署集群，以降低整体部署成本。更多关于 Cluster Manager 的信息，可参考以下文档：
+
+- [Cluster Manager for Apache Doris 24.x 安装手册](https://docs.selectdb.com/docs/enterprise/cluster-manager-guide/deployment-guide/deployment-guide-24.x)
+- [Cluster Manager for Apache Doris 24.x 使用手册](https://docs.selectdb.com/docs/enterprise/cluster-manager-guide/management-guide/management-guide-24.x)
+
+### 第 3 步：优化 FE 和 BE 配置
+
+完成集群部署后，需分别优化 FE 和 BE 配置参数，以更加契合日志存储与分析的场景。
+
+**优化 FE 配置**
+
+在 `fe/conf/fe.conf` 目录下找到 FE 的相关配置项，并按照以下表格，调整 FE 配置。
+
+| 需调整参数 | 说明 |
+| :----------------------------------------------------------- | :----------------------------------------------------------- |
+| `max_running_txn_num_per_db = 10000` | 高并发导入运行事务数较多，需调高参数。 |
+| `streaming_lable_keep_max_second = 3600``label_keep_max_second = 7200` | 高频导入事务标签内存占用多，保留时间调短。 |
+| `enable_round_robin_create_tablet = true` | 创建 Tablet 时，采用 Round Robin 策略，尽量均匀。 |
+| `tablet_rebalancer_type = partition` | 均衡 Tablet 时，采用每个分区内尽量均匀的策略。 |
+| `enable_single_replica_load = true` | 开启单副本导入，多个副本只需构建一次索引，减少 CPU 消耗。 |
+| `autobucket_min_buckets = 10` | 将自动分桶的最小分桶数从 1 调大到 10，避免日志量增加时分桶不够。 |
+| `max_backend_heartbeat_failure_tolerance_count = 10` | 日志场景下 BE 服务器压力较大，可能短时间心跳超时，因此将容忍次数从 1 调大到 10。 |
+
+更多关于 FE 配置项的信息，可参考 [FE 配置项](../admin-manual/config/fe-config.md)。
+
+**优化 BE 配置**
+
+在 `be/conf/be.conf` 目录下找到 BE 的相关配置项，并按照以下表格，调整 BE 配置。
+
+| 模块 | 需调整参数 | 说明 |
+| :--------- | :----------------------------------------------------------- | :----------------------------------------------------------- |
+| 存储 | `storage_root_path = /path/to/dir1;/path/to/dir2;...;/path/to/dir12` | 配置热数据在磁盘目录上的存储路径。 |
+| - | `enable_file_cache = true` | 开启文件缓存。 |
+| - | `file_cache_path = [{"path": "/mnt/datadisk0/file_cache", "total_size":53687091200, "query_limit": "10737418240"},{"path": "/mnt/datadisk1/file_cache", "total_size":53687091200,"query_limit": "10737418240"}]` | 配置冷数据的缓存路径和相关设置，具体配置说明如下：<br/>`path`：缓存路径<br/>`total_size`：该缓存路径的总大小，单位为字节，53687091200 字节等于 50 GB<br/>`query_limit`：单次查询可以从缓存路径中查询的最大数据量，单位为字节，10737418240 字节等于 10 GB |
+| 写入 | `write_buffer_size = 1073741824` | 增加写入缓冲区（buffer）的文件大小，减少小文件和随机 I/O 操作，提升性能。 |
+| - | `max_tablet_version_num = 20000` | 配合建表的 time_series compaction 策略，允许更多版本暂时未合并。 |
+| - | `enable_single_replica_load = true` | 开启单副本写入，减少 CPU 消耗。与 FE 配置相同。 |
+| Compaction | `max_cumu_compaction_threads = 8` | 设置为 CPU 核数 / 4，意味着 CPU 资源的 1/4 用于写入，1/4 用于后台 Compaction，2/1 留给查询和其他操作。 |
+| - | `inverted_index_compaction_enable = true` | 开启索引合并（index compaction），减少 Compaction 时的 CPU 消耗。 |
+| - | `enable_segcompaction = false` `enable_ordered_data_compaction = false` | 关闭日志场景不需要的两个 Compaction 功能。 |
+| 缓存 | `disable_storage_page_cache = true` `inverted_index_searcher_cache_limit = 30%` | 因为日志数据量较大，缓存（cache）作用有限，因此关闭数据缓存，调换为索引缓存（index cache）的方式。 |
+| - | `inverted_index_cache_stale_sweep_time_sec = 3600` `index_cache_entry_stay_time_after_lookup_s = 3600` | 让索引缓存在内存中尽量保留 1 小时。 |
+| - | `enable_inverted_index_cache_on_cooldown = true`<br />`enable_write_index_searcher_cache = false` | 开启索引上传冷数据存储时自动缓存的功能。 |
+| - | `tablet_schema_cache_recycle_interval = 3600` `segment_cache_capacity = 20000` | 减少其他缓存对内存的占用。 |
+| 线程 | `pipeline_executor_size = 24` `doris_scanner_thread_pool_thread_num = 48` | 32 核 CPU 的计算线程和 I/O 线程配置，根据核数等比扩缩。 |
+| - | `scan_thread_nice_value = 5` | 降低查询 I/O 线程的优先级，保证写入性能和时效性。 |
+| 其他 | `string_type_length_soft_limit_bytes = 10485760` | 将 String 类型数据的长度限制调高至 10 MB。 |
+| - | `trash_file_expire_time_sec = 300` `path_gc_check_interval_second = 900` `path_scan_interval_second = 900` | 调快垃圾文件的回收时间。 |
+
+更多关于 BE 配置项的信息，可参考 [BE 配置项](../admin-manual/config/be-config.md)。
+
+### 第 4 步：建表
+
+由于日志数据的写入和查询都具备明显的特征，因此，在建表时按照本节说明进行针对性配置，以提升性能表现。
+
+**配置分区分桶参数**
+
+- 分区时，按照以下说明配置：
+- 使用时间字段上的 [Range 分区](https://doris.apache.org/zh-CN/docs/table-design/data-partition/#range-%E5%88%86%E5%8C%BA)，并开启 [动态分区](https://doris.apache.org/zh-CN/docs/table-design/data-partition#%E5%8A%A8%E6%80%81%E5%88%86%E5%8C%BA)，按天自动管理分区。
+- 使用 Datetime 类型的时间字段作为 Key，在查询最新 N 条日志时有数倍加速。
+- 分桶时，按照以下说明配置：
+- 分桶数量大致为集群磁盘总数的 3 倍。
+- 使用 Random 策略，配合写入时的 Single Tablet 导入，可以提升批量（Batch）写入的效率。
+
+更多关于分区分桶的信息，可参考 [分区分桶](../table-design/data-partition.md)。
+
+**配置 Compaction 参数**
+
+按照以下说明配置 Compaction 参数：
+
+- 使用 time_series 策略，以减轻写放大效应，对于高吞吐日志写入的资源写入很重要。
+- 使用单副本 Compaction，减少多副本 Compaction 的开销。
+
+**建立和配置索引参数**
+
+按照以下说明操作：
+
+- 对经常查询的字段建立索引。
+- 对需要全文检索的字段，将分词器（parser）参数赋值为 unicode，一般能满足大部分需求。如有支持短语查询的需求，将 support_phrase 参数赋值为 true；如不需要，则设置为 false，以降低存储空间。
+
+**配置存储策略**
+
+按照以下说明操作：
+
+- 对于热存储数据，如果使用云盘，可配置 1 副本；如果使用物理盘，则至少配置 2 副本。
+- 配置 `log_s3` 的存储位置，并设置 `log_policy_3day` 冷热数据分层策略，即在超过 3 天后将数据冷却至 `log_s3` 指定的存储位置。可参考以下代码：
+
+```Go
+CREATE DATABASE log_db;
+USE log_db;
+
+CREATE RESOURCE "log_s3"
+PROPERTIES
+(
+ "type" = "s3",
+ "s3.endpoint" = "your_endpoint_url",
+ "s3.region" = "your_region",
+ "s3.bucket" = "your_bucket",
+ "s3.root.path" = "your_path",
+ "s3.access_key" = "your_ak",
+ "s3.secret_key" = "your_sk"
+);
+
+CREATE STORAGE POLICY log_policy_3day
+PROPERTIES(
+ "storage_resource" = "log_s3",
+ "cooldown_ttl" = "259200"
+);
+
+CREATE TABLE log_table
+(
+ `ts` DATETIME,
+ `host` TEXT,
+ `path` TEXT,
+ `message` TEXT,
+ INDEX idx_host (`host`) USING INVERTED,
+ INDEX idx_path (`path`) USING INVERTED,
+ INDEX idx_message (`message`) USING INVERTED PROPERTIES("parser" = "unicode", "support_phrase" = "true")
+)
+ENGINE = OLAP
+DUPLICATE KEY(`ts`)
+PARTITION BY RANGE(`ts`) ()
+DISTRIBUTED BY RANDOM BUCKETS 250
+PROPERTIES (
+"dynamic_partition.enable" = "true",
+"dynamic_partition.create_history_partition" = "true",
+"dynamic_partition.time_unit" = "DAY",
+"dynamic_partition.start" = "-30",
+"dynamic_partition.end" = "1",
+"dynamic_partition.prefix" = "p",
+"dynamic_partition.buckets" = "250",
+"dynamic_partition.replication_num" = "1", -- 存算分离不需要
+"replication_num" = "1" -- 存算分离不需要
+"enable_single_replica_compaction" = "true", -- 存算分离不需要
+"storage_policy" = "log_policy_3day", -- 存算分离不需要
+"compaction_policy" = "time_series"
+);
+```
+
+### 第 5 步：采集和查询日志
+
+完成建表后，可进行日志采集、查询和分析。
+
+**日志采集**
+
+Apache Doris 提供开放、通用的 Stream HTTP APIs，通过这些 APIs，你可与常用的日志采集器打通，包括 Logstash、Filebeat、Kafka 等，从而开展日志采集工作。本节介绍了如何使用 Stream HTTP APIs 对接日志采集器。
+
+**对接 Logstash**
+
+按照以下步骤操作：
+
+1. 下载并安装 Logstash Doris Output 插件。你可选择以下两种方式之一：
+
+- 直接下载：[点此下载](https://apache-doris-releases.oss-accelerate.aliyuncs.com/logstash-output-doris-1.0.0.gem)。
+
+- 从源码编译，并运行下方命令安装：
+
+```markdown
+./bin/logstash-plugin install logstash-output-doris-1.0.0.gem
+```
+
+2. 配置 Logstash。需配置以下参数：
+
+- `logstash.yml`：配置 Logstash 批处理日志的条数和时间，用于提升数据写入性能。
+
+```markdown
+pipeline.batch.size: 1000000
+pipeline.batch.delay: 10000
+```
+
+
+- `logstash_demo.conf`：配置所采集日志的具体输入路径和输出到 Apache Doris 的设置。
+
+```markdown
+input {
+file {
+path => "/path/to/your/log"
+}
+}
+<br />output {
+doris {
+http_hosts => \[ "<http://fehost1:http_port>", "<http://fehost2:http_port>", "<http://fehost3:http_port"\>]
+user => "your_username"
+password => "your_password"
+db => "your_db"
+table => "your_table"
+\# doris stream load http headers
+headers => {
+"format" => "json"
+"read_json_by_line" => "true"
+"load_to_single_tablet" => "true"
+}
+\# field mapping: doris fileld name => logstash field name
+\# %{} to get a logstash field, \[\] for nested field such as \[host\]\[name\] for host.name
+mapping => {
+"ts" => "%{@timestamp}"
+"host" => "%{\[host\]\[name\]}"
+"path" => "%{\[log\]\[file\]\[path\]}"
+"message" => "%{message}"
+}
+log_request => true
+log_speed_interval => 10
+}
+}
+```
+
+3. 按照下方命令运行 Logstash，采集日志并输出至 Apache Doris。
+
+```Bash
+./bin/logstash -f logstash_demo.conf
+```
+
+更多关于 Logstash 配置和使用的说明，可参考 [Logstash Doris Output Plugin](../ecosystem/logstash.md)。
+
+**对接 Filebeat**
+
+按照以下步骤操作：
+
+1. 获取支持输出至 Apache Doris 的 Filebeat 二进制文件。可 [点此下载](https://apache-doris-releases.oss-accelerate.aliyuncs.com/filebeat-doris-1.0.0) 或者从 Apache Doris 源码编译。
+2. 配置 Filebeat。需配置以下参数：
+
+- `filebeat_demo.yml`：配置所采集日志的具体输入路径和输出到 Apache Doris 的设置。
+
+```YAML
+ # input
+ filebeat.inputs:
+ - type: log
+ enabled: true
+ paths:
+ - /path/to/your/log
+ # multiline 可以将跨行的日志（比如 Java stacktrace）拼接起来
+ multiline:
+ type: pattern
+ # 效果：以 yyyy-mm-dd HH:MM:SS 开头的行认为是一条新的日志，其他都拼接到上一条日志
+ pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}'
+ negate: true
+ match: after
+ skip_newline: true
+
+ processors:
+ # 用 js script 插件将日志中的 \t 替换成空格，避免 JSON 解析报错
+ - script:
+ lang: javascript
+ source: >
+ function process(event) {
+ var msg = event.Get("message");
+ msg = msg.replace(/\t/g, " ");
+ event.Put("message", msg);
+ }
+ # 用 dissect 插件做简单的日志解析
+ - dissect:
+ # 2024-06-08 18:26:25,481 INFO (report-thread|199) [ReportHandler.cpuReport():617] begin to handle
+ tokenizer: "%{day} %{time} %{log_level} (%{thread}) [%{position}] %{content}"
+ target_prefix: ""
+ ignore_failure: true
+ overwrite_keys: true
+
+ # queue and batch
+ queue.mem:
+ events: 1000000
+ flush.min_events: 100000
+ flush.timeout: 10s
+
+ # output
+ output.doris:
+ fenodes: [ "http://fehost1:http_port", "http://fehost2:http_port", "http://fehost3:http_port" ]
+ user: "your_username"
+ password: "your_password"
+ database: "your_db"
+ table: "your_table"
+ # output string format
+ ## %{[agent][hostname]} %{[log][file][path]} 是filebeat自带的metadata
+ ## 常用的 filebeat metadata 还是有采集时间戳 %{[@timestamp]}
+ ## %{[day]} %{[time]} 是上面 dissect 解析得到字段
+ codec_format_string: '{"ts": "%{[day]} %{[time]}", "host": "%{[agent][hostname]}", "path": "%{[log][file][path]}", "message": "%{[message]}"}'
+ headers:
+ format: "json"
+ read_json_by_line: "true"
+ load_to_single_tablet: "true"
+ ```
+
+3. 按照下方命令运行 Filebeat，采集日志并输出至 Apache Doris。
+
+```Bash
+chmod +x filebeat-doris-1.0.0
+./filebeat-doris-1.0.0 -c filebeat_demo.yml
+```
+
+更多关于 Filebeat 配置和使用的说明，可参考 [Beats Doris Output Plugin](../ecosystem/beats.md)。
+
+**对接 Kafka**
+
+将 JSON 格式的日志写入 Kafka 的消息队列，创建 Kafka Routine Load，即可让 Apache Doris 从 Kafka 主动拉取数据。
+
+可参考如下示例。其中，`property.\*` 是 Librdkafka 客户端相关配置，根据实际 Kafka 集群情况配置。
+
+```SQL
+\-- 准备好kafka集群和topic log_\_topic_
+\-- 创建routine load，从kafka log_\_topic_将数据导入log_table表
+CREATE ROUTINE LOAD load_log_kafka ON log_db.log_table
+COLUMNS(ts, clientip, request, status, size)
+PROPERTIES (
+"max_batch_interval" = "10",
+"max_batch_rows" = "1000000",
+"max_batch_size" = "109715200",
+"load_to_single_tablet" = "true",
+"timeout" = "600",
+"strict_mode" = "false",
+"format" = "json"
+)
+FROM KAFKA (
+"kafka_broker_list" = "host:port",
+"kafka_topic" = "log_\_topic_",
+"property.group.id" = "your_group_id",
+"property.security.protocol"="SASL_PLAINTEXT",
+"property.sasl.mechanism"="GSSAPI",
+"property.sasl.kerberos.service.name"="kafka",
+"property.sasl.kerberos.keytab"="/path/to/xxx.keytab",
+"property.sasl.kerberos.principal"="<xx...@yyy.com>"
+);
+\-- 查看routine的状态
+SHOW ROUTINE LOAD;
+```
+
+更多关于 Kafka 配置和使用的说明，可参考 [Routine Load](../data-operate/import/routine-load-manual.md)。
+
+**使用自定义程序采集日志**
+
+除了对接常用的日志采集器以外，你也可以自定义程序，通过 HTTP API Stream Load 将日志数据导入 Apache Doris。参考以下代码：
+
+```Bash
+curl \\
+\--location-trusted \\
+\-u username:password \\
+\-H "format:json" \\
+\-H "read_json_by_line:true" \\
+\-H "load_to_single_tablet:true" \\
+\-H "timeout:600" \\
+\-T logfile.json \\
+http://fe_host:fe_http_port/api/log_db/log_table/\_stream_load
+```
+
+在使用自定义程序时，需注意以下关键点：
+
+- 使用 Basic Auth 进行 HTTP 鉴权，用命令 `echo -n 'username:password' | base64` 进行计算。
+- 设置 HTTP header "format:json"，指定数据格式为 JSON。
+- 设置 HTTP header "read_json_by_line:true"，指定每行一个 JSON。
+- 设置 HTTP header "load_to_single_tablet:true"，指定一次导入写入一个分桶减少导入的小文件。
+- 建议写入客户端一个 Batch 的大小为 100MB ～ 1GB。如果你使用的是 Apache Doris 2.1 及更高版本，需通过服务端 Group Commit 功能，降低客户端 Batch 大小。
+
+**日志查询**
+
+Apache Doris 支持标准 SQL，因此，你可以通过 MySQL 客户端或者 JDBC 等方式连接到集群，执行 SQL 进行日志查询。参考以下命令：
+
+```
+mysql -h fe_host -P fe_mysql_port -u your_username -Dyour_db_name
+```
+
+下方列出常见的 5 条 SQL 查询命令，以供参考：
+
+- 查看最新的 10 条数据
+
+```SQL
+SELECT \* FROM your_table_name ORDER BY ts DESC LIMIT 10;
+```
+
+- 查询 `host` 为 `8.8.8.8` 的最新 10 条数据
+
+```SQL
+SELECT \* FROM your_table_name WHERE host = '8.8.8.8' ORDER BY ts DESC LIMIT 10;
+```
+
+- 检索请求字段中有 `error` 或者 `404` 的最新 10 条数据。其中，`MATCH_ANY` 是 Apache Doris 全文检索的 SQL 语法，用于匹配参数中任一关键字。
+
+```SQL
+SELECT \* FROM your_table_name WHERE message MATCH_ANY 'error 404'
+ORDER BY ts DESC LIMIT 10;
+```
+
+- 检索请求字段中有 `image` 和 `faq` 的最新 10 条数据。其中，`MATCH_ALL` 是 Apache Doris 全文检索的 SQL 语法，用于匹配参数中所有关键字。
+
+```SQL
+SELECT \* FROM your_table_name WHERE message MATCH_ALL 'image faq'
+ORDER BY ts DESC LIMIT 10;
+```
+
+- 检索请求字段中有 `image` 和 `faq` 的最新 10 条数据。其中，`MATCH_PHRASE` 是 Apache Doris 全文检索的 SQL 语法，用于匹配参数中所有关键字，并且要求顺序一致。在下方例子中，`a image faq b` 能匹配，但是 `a faq image b` 不能匹配，因为 `image` 和 `faq` 的顺序与查询不一致。
+
+```SQL
+SELECT \* FROM your_table_name WHERE message MATCH_PHRASE 'image faq'
+ORDER BY ts DESC LIMIT 10;
+```
+
+**可视化日志分析**
+
+基于 Apache Doris 构建的 SelectDB Enterprise Core 提供了名为 Doris WebUI 的数据开发平台，Doris WebUI 包含了类 Kibana Discover 的日志检索分析界面，提供直观、易用的探索式日志分析交互，如下图所示：
+
+
+
+在此界面上，Doris WebUI 主要支持以下功能：
+
+- 支持全文检索和 SQL 两种模式
+- 支持时间框和直方图上选择查询日志的时间段
+- 支持信息丰富的日志明细展示，还可以展开成 JSON 或表格
+- 在日志数据上下文交互式点击增加和删除筛选条件
+- 搜索结果的字段 Top 值展示，便于发现异常值和进一步下钻分析
+
+你可以 [点此下载 SelectDB Enterprise Core](https://www.selectdb.com/download/enterprise#core)，完成 [安装](https://docs.selectdb.com/docs/enterprise/enterprise-core-guide/selectdb-distribution-doris-core-deployment-guide) 后，即可使用 Doris WebUI 登录数据库。更多关于如何使用 Doris WebUI 的信息，可参考 [WebUI](https://docs.selectdb.com/docs/enterprise/enterprise-core-guide/selectdb-webui-guide)。
\ No newline at end of file
diff --git a/sidebars.json b/sidebars.json
index 2c665ebd1b6..b9fa62b5485 100644
--- a/sidebars.json
+++ b/sidebars.json
@@ -510,6 +510,13 @@
"admin-manual/rowsets"
]
},
+ {
+ "type": "category",
+ "label": "Practical Guide",
+ "items": [
+ "practical-guide/log-storage-analysis"
+ ]
+ },
{
"type": "category",
"label": "Ecosystem",
diff --git a/static/images/WebUI-CN.jpeg b/static/images/WebUI-CN.jpeg
new file mode 100644
index 00000000000..43c17e0dc2b
Binary files /dev/null and b/static/images/WebUI-CN.jpeg differ
diff --git a/static/images/WebUI-EN.jpeg b/static/images/WebUI-EN.jpeg
new file mode 100644
index 00000000000..681019e5bc4
Binary files /dev/null and b/static/images/WebUI-EN.jpeg differ
diff --git a/static/images/doris-overall-architecture.png b/static/images/doris-overall-architecture.png
new file mode 100644
index 00000000000..b20d3314e52
Binary files /dev/null and b/static/images/doris-overall-architecture.png differ
diff --git a/versioned_docs/version-2.0/ecosystem/flink-doris-connector.md b/versioned_docs/version-2.0/ecosystem/flink-doris-connector.md
index 9051ae49764..e7bfe7d97ff 100644
--- a/versioned_docs/version-2.0/ecosystem/flink-doris-connector.md
+++ b/versioned_docs/version-2.0/ecosystem/flink-doris-connector.md
@@ -79,7 +79,7 @@ Copy this file to `classpath` of `Flink` to use `Flink-Doris-Connector`. For exa
### read
-####SQL
+**SQL**
```sql
-- doris source
@@ -98,7 +98,7 @@ CREATE TABLE flink_doris_source (
);
```
-####DataStream
+**DataStream**
```java
DorisOptions.Builder builder = DorisOptions.builder()
@@ -118,7 +118,7 @@ env.fromSource(dorisSource, WatermarkStrategy.noWatermarks(), "doris source").pr
### write
-####SQL
+**SQL**
```sql
--enable checkpoint
@@ -144,7 +144,7 @@ CREATE TABLE flink_doris_sink (
INSERT INTO flink_doris_sink select name,age,price,sale from flink_doris_source
```
-####DataStream
+**DataStream**
DorisSink writes data to Doris through StreamLoad, and DataStream supports different serialization methods when writing
diff --git a/versioned_docs/version-2.0/practical-guide/log-storage-analysis.md b/versioned_docs/version-2.0/practical-guide/log-storage-analysis.md
new file mode 100644
index 00000000000..36962f2e138
--- /dev/null
+++ b/versioned_docs/version-2.0/practical-guide/log-storage-analysis.md
@@ -0,0 +1,593 @@
+---
+{
+ "title": "Log Storage and Analysis",
+ "language": "en"
+}
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Logs record key events in the system and contain crucial information such as the events' subject, time, location, and content. To meet the diverse needs of observability in operations, network security monitoring, and business analysis, enterprises might need to collect scattered logs for centralized storage, querying, and analysis to extract valuable content from the log data further.
+
+In this scenario, Apache Doris provides a corresponding solution. With the characteristics of log scenarios in mind, Apache Doris added inverted-index and ultra-fast full-text search capabilities, optimizing write performance and storage space to the extreme. This allows users to build an open, high-performance, cost-effective, and unified log storage and analysis platform based on Apache Doris.
+
+Focused on this solution, this chapter contains the following 3 sections:
+
+- **Overall architecture**: This section explains the core components and architecture of the log storage and analysis platform built on Apache Doris.
+
+- **Features and advantages**: This section explains the features and advantages of the log storage and analysis platform built on Apache Doris.
+
+- **Operational guide**: This section explains how to build a log storage and analysis platform based on Apache Doris.
+
+## Overall architecture
+
+The following figure illustrates the architecture of the log storage and analysis platform built on Apache Doris:
+
+
+
+The architecture contains the following 3 parts:
+
+- **Log collection and preprocessing**: Various log collection tools can write log data into Apache Doris through HTTP APIs.
+
+- **Log storage and analysis engine**: Apache Doris provides high-performance, low-cost unified log storage, offering rich retrieval and analysis capabilities through SQL interfaces.
+
+- **Log analysis and alert interface**: Various log retrieval and analysis tools can query Apache Doris through standard SQL interfaces, providing users with a simple and user-friendly interface.
+
+## Features and advantages
+
+The following figure illustrates the architecture of the log storage and analysis platform built on Apache Doris::
+
+- **High throughput, low latency log writing**: Supports stable writing of log data at the level of hundreds of TBs and GB/s per day, while maintaining latency within 1 second.
+
+- **Cost-effective storage of massive log data**: Supports petabyte-scale storage, saving 60% to 80% in storage costs compared to Elasticsearch, and further reducing storage costs by 50% by storing cold data in S3/HDFS.
+
+- **High-performance log full-text search and analysis**: Supports inverted indexes and full-text search, providing second-level response times for common log queries (keyword searches, trend analysis, etc.).
+
+- **Open and user-friendly upstream and downstream ecosystem**: Upstream integration with common log collection systems and data sources such as Logstash, Filebeat, Fluentbit, and Kafka through Stream Load's universal HTTP APIs, and downstream integration with various visual analytics UIs using standard MySQL protocol and syntax, such as observability platform Grafana, BI analytics Superset, and log retrieval Doris WebUI similar to Kibana.
+
+### Cost-effective performance
+
+After Benchmark testing and production validation, the log storage and analysis platform built on Apache Doris has a 5 to 10 times cost-performance advantage over Elasticsearch. Apache Doris's performance benefits are mainly due to its globally leading high-performance storage and query engine, as well as specialized optimizations for log scenarios:
+
+- **Improved write throughput**: Elasticsearch's write performance bottleneck lies in CPU consumption for parsing data and building inverted indexes. In comparison, Apache Doris has optimized writes in two aspects: using SIMD and other CPU vector instructions to improve JSON data parsing speed and index-building performance and simplifying the inverted index structure for log scenarios by removing unnecessary data structures like forward indexes, effectively reducing index build complexi [...]
+
+- **Reduced storage costs**: Elasticsearch's storage bottleneck comes from multiple storage of forward, inverted, Docvalue columns, and lower compression ratios with general compression algorithms. Conversely, Apache Doris has optimized storage by eliminating forward indexes, reducing index data volume by 30%; adopting columnar storage and Zstandard compression algorithm, achieving compression ratios 5-10 times higher than Elasticsearch's 1.5 times; with low access frequency of cold data [...]
+
+### Strong analytical capabilities
+
+Apache Doris supports standard SQL and is compatible with MySQL protocol and syntax. Therefore, log systems built on Apache Doris can use SQL for log analysis, giving the following advantages to log systems:
+
+- **Easy to use**: Engineers and data analysts are very familiar with SQL, their expertise can be reused, no need to learn new technology stacks to quickly get started.
+
+- **Rich ecosystem**: The MySQL ecosystem is the most widely used language in the database field, seamlessly integrating with and applying to the MySQL ecosystem. Doris can leverage MySQL command line and various GUI tools, BI tools, and other big data ecosystem tools for more complex and diverse data processing and analysis needs.
+
+- **Strong analytical capabilities**: SQL has become the de facto standard for database and big data analysis, with powerful expressive capabilities and functions supporting retrieval, aggregation, multi-table JOIN, subqueries, UDFs, logical views, materialized views, and various data analysis capabilities.
+
+### Flexible Schema
+
+Here is a typical example of a semi-structured log in JSON format. The top-level fields are some fixed fields, such as timestamp, source, node, component, level, clientRequestID, message, and properties, which are present in every log entry. The nested fields of the properties , such as properties.size and properties.format are more dynamic, and the fields of each log may vary.
+
+```JSON
+{
+"timestamp": "2014-03-08T00:50:03.8432810Z",
+"source": "ADOPTIONCUSTOMERS81",
+"node": "Engine000000000405",
+"level": "Information",
+"component": "DOWNLOADER",
+"clientRequestId": "671db15d-abad-94f6-dd93-b3a2e6000672",
+"message": "Downloading file path: benchmark/2014/ADOPTIONCUSTOMERS81_94_0.parquet.gz",
+"properties": {
+"size": 1495636750,
+"format": "parquet",
+"rowCount": 855138,
+"downloadDuration": "00:01:58.3520561"
+}
+}
+```
+
+
+Apache Doris provides several aspects of support for Flexible Schema log data:
+
+- For minor changes to top-level fields, Light Schema Change can be used to add or remove columns and to add or remove indexes, enabling schema changes to be completed in seconds. When planning a log platform, users only need to consider which fields need to be indexed.
+
+- For extension fields similar to properties, the native semi-structured data type `VARIANT` is provided, which can write any JSON data, automatically recognize field names and types in JSON, and automatically split frequently occurring fields for columnar storage for subsequent analysis. Additionally, `VARIANT` can create inverted indexes to accelerate internal field queries and retrievals.
+
+Compared to Elasticsearch's Dynamic Mapping, Apache Doris's Flexible Schema has the following advantages:
+
+- Allows a field to have multiple types, `VARIANT` automatically handles conflicts and type promotion for fields, better adapting to iterative changes in log data.
+
+- `VARIANT` automatically merges infrequently occurring fields into a column store to avoid performance issues caused by excessive fields, metadata, or columns.
+
+- Not only can columns be dynamically added, but they can also be dynamically deleted, and indexes can be dynamically added or removed, eliminating the need to index all fields at the beginning like Elasticsearch, reducing unnecessary costs.
+
+## Operational guide
+
+### Step 1: Estimate resources
+
+Before deploying the cluster, you need to estimate the hardware resources required for the servers. Follow the steps below:
+
+1. Estimate the resources for data writing by the following calculation formulas:
+
+- Daily data increment / 86400 s = Average write throughput
+
+- Average write throughput \* Ratio of the peak write throughput to the average write throughput = Peak write throughput
+
+- Peak write throughput / Write throughput of a single-core CPU = Number of CPU cores for the peak write throughput
+
+2. Estimate the resources for data storage by the calculation formula: Daily data increment / Data compression ratio \* Number of data copies \* Data storage duration = Data storage space.
+
+3. Estimate the resources for data querying. The resources for data querying depend on the query volume and complexity. It is recommended to reserve 50% of CPU resources for data query initially and then adjust according to the actual test results.
+
+4. Integrate the calculation results as follows:
+
+ 1. Divide the number of CPU cores calculated in Step 1 and Step 3 by the number of CPU cores of a BE server, and you can get the number of BE servers.
+
+ 2. Based on the number of BE servers and the calculation result of Step 2, estimate the storage space required for each BE server.
+
+ 3. Allocate the storage space required for each BE server to 4 to 12 data disks, and you can get the storage capacity required for a single data disk.
+
+For example, suppose that the daily data increment is 100 TB, the data compression ratio is 7, the number of data copies is 1, the storage duration of hot data is 3 days, the storage duration of cold data is 30 days, the ratio of the peak write throughput to the average write throughput is 200%, the write throughput of a single-core CUP is 10 MB/s, and 50% of CPU resources are reserved for data querying, one can estimate that:
+
+- 3 FE servers are required, each configured with a 16-core CPU, 64 GB memory, and an 1100 GB SSD disk.
+
+- 15 BE servers are required, each configured with a 32-core CPU, 256 GB memory, and 8 500 GB SSD disks.
+
+- S3:430 TB
+
+Refer to the following table to learn about the values of indicators in the example above and how they are calculated.
+
+| Indicator (Unit) | Value | Description |
+| --- | --- | --- |
+| Daily data increment (TB) | 100 | Specify the value according to your actual needs. |
+| Data compression ratio | 7 | Specify the value according to your actual needs, which is often between 5 to 10. Note that the data contains index data. |
+| Number of data copies | 1 | Specify the value according to your actual needs, which can be 1, 2, or 3. The default value is 1. |
+| Storage duration of hot data (day) | 3 | Specify the value according to your actual needs. |
+| Storage duration of cold data (day) | 30 | Specify the value according to your actual needs. |
+| Data storage duration | 33 | Calculation formula: Storage duration of hot data + Storage duration of cold data |
+| Estimated storage space for hot data (TB) | 42.9 | Calculation formula: Daily data increment / Data compression ratios \* Number of data copies \* Storage duration of hot data |
+| Estimated storage space for cold data (TB) | 428.6 | Calculation formula: Daily data increment / Data compression ratios \* Number of data copies \* Storage duration of cold data |
+| Ratio of the peak write throughput to the average write throughput | 200% | Specify the value according to your actual needs. The default value is 200%. |
+| Number of CPU cores of a BE server | 32 | Specify the value according to your actual needs. The default value is 32. |
+| Average write throughput (MB/s) | 1214 | Calculation formula: Daily data increment / 86400 s |
+| Peak write throughput (MB/s) | 2427 | Calculation formula: Average write throughput \* Ratio of the peak write throughput to the average write throughput |
+| Number of CPU cores for the peak write throughput | 242.7 | Calculation formula: Peak write throughput / Write throughput of a single-core CPU |
+| Percent of CPU resources reserved for data querying | 50% | Specify the value according to your actual needs. The default value is 50%. |
+| Estimated number of BE servers | 15.2 | Calculation formula: Number of CPU cores for the peak write throughput / Number of CPU cores of a BE server /（1 - Percent of CPU resources reserved for data querying） |
+| Rounded number of BE servers | 15 | Calculation formula: MAX (Number of data copies, Estimated number of BE servers) |
+| Estimated data storage space for each BE server (TB) | 4.03 | Calculation formula: Estimated storage space for hot data / Estimated number of BE servers /（1 - 30%）, where 30% represents the percent of reserved storage space.<br/><br/>It is recommended to mount 4 to 12 data disks on each BE server to enhance I/O capabilities. |
+
+### Step 2: Deploy the cluster
+
+After estimating the resources, you need to deploy the cluster. It is recommended to deploy in both physical and virtual environments manually. For manual deployment, refer to [Manual Deployment](../install/cluster-deployment/standard-deployment.md).
+
+Alternatively, it is recommended to use VeloDB Manager provided by VeloDB Enterprise to deploy the cluster, reducing overall deployment costs. For more information about the VeloDB Manager, please refer to the following documents:
+
+- [VeloDB Manager 24.x Installation Manual](https://docs.velodb.io/enterprise/velodb-manager-guide/deployment-guide/deployment-guide-24.x)
+
+- [VeloDB Manager 24.x User Guide](https://docs.velodb.io/enterprise/velodb-manager-guide/management-guide/management-guide-24.x)
+
+### Step 3: Optimize FE and BE configurations
+
+After completing the cluster deployment, it is necessary to optimize the configuration parameters for both the front-end and back-end separately, so as to better suit the scenario of log storage and analysis.
+
+**Optimize FE configurations**
+
+You can find FE configuration fields in `fe/conf/fe.conf`. Refer to the following table to optimize FE configurations.
+
+| Configuration fields to be optimized | Description |
+| :----------------------------------------------------------- | :----------------------------------------------------------- |
+| `max_running_txn_num_per_db = 10000` | Increase the parameter value to adapt to high-concurrency import transactions. |
+| `streaming_lable_keep_max_second = 3600``label_keep_max_second = 7200` | Increase the retention time to handle high-frequency import transactions with high memory usage. |
+| `enable_round_robin_create_tablet = true` | When creating Tablets, use a Round Robin strategy to distribute evenly. |
+| `tablet_rebalancer_type = partition` | When balancing Tablets, use a strategy to evenly distribute within each partition. |
+| `enable_single_replica_load = true` | Enable single-replica import, where multiple replicas only need to build an index once to reduce CPU consumption. |
+| `autobucket_min_buckets = 10` | Increase the minimum number of automatically bucketed buckets from 1 to 10 to avoid insufficient buckets when the log volume increases. |
+| `max_backend_heartbeat_failure_tolerance_count = 10` | In log scenarios, the BE server may experience high pressure, leading to short-term timeouts, so increase the tolerance count from 1 to 10. |
+
+For more information, refer to [FE Configuration](../admin-manual/config/fe-config.md).
+
+**Optimize BE configurations**
+
+You can find BE configuration fields in `be/conf/be.conf`. Refer to the following table to optimize BE configurations.
+
+| Module | Configuration fields to be optimized | Description |
+| :--------- | :----------------------------------------------------------- | :----------------------------------------------------------- |
+| Storage | `storage_root_path = /path/to/dir1;/path/to/dir2;...;/path/to/dir12` | Configure the storage path for hot data on disk directories. |
+| - | `enable_file_cache = true` | Enable file caching. |
+| - | `file_cache_path = [{"path": "/mnt/datadisk0/file_cache", "total_size":53687091200, "query_limit": "10737418240"},{"path": "/mnt/datadisk1/file_cache", "total_size":53687091200,"query_limit": "10737418240"}]` | Configure the cache path and related settings for cold data with the following specific configurations:<br/>`path`: cache path<br/>`total_size`: total size of the cache path in bytes, where 53687091200 bytes equals 50 GB<br/>`query_limit`: maximum amount of data tha [...]
+| Write | `write_buffer_size = 1073741824` | Increase the file size of the write buffer to reduce small files and random I/O operations, improving performance. |
+| - | `max_tablet_version_num = 20000` | In coordination with the time_series compaction strategy for table creation, allow more versions to remain temporarily unmerged |
+| - | `enable_single_replica_load = true` | Enable single replica import to reduce CPU consumption, consistent with FE configuration. |
+| Compaction | `max_cumu_compaction_threads = 8` | Set to CPU core count / 4, indicating that 1/4 of CPU resources are used for writing, 1/4 for background compaction, and 2/1 for queries and other operations. |
+| - | `inverted_index_compaction_enable = true` | Enable inverted index compaction to reduce CPU consumption during compaction. |
+| - | `enable_segcompaction = false` `enable_ordered_data_compaction = false` | Disable two compaction features that are unnecessary for log scenarios. |
+| Cache | `disable_storage_page_cache = true` `inverted_index_searcher_cache_limit = 30%` | Due to the large volume of log data and limited caching effect, switch from data caching to index caching. |
+| - | `inverted_index_cache_stale_sweep_time_sec = 3600` `index_cache_entry_stay_time_after_lookup_s = 3600` | Maintain index caching in memory for up to 1 hour. |
+| - | `enable_inverted_index_cache_on_cooldown = true`<br />`enable_write_index_searcher_cache = false` | Enable automatic caching of cold data storage during index uploading. |
+| - | `tablet_schema_cache_recycle_interval = 3600` `segment_cache_capacity = 20000` | Reduce memory usage by other caches. |
+| Thread | `pipeline_executor_size = 24` `doris_scanner_thread_pool_thread_num = 48` | Configure computing threads and I/O threads for a 32-core CPU in proportion to core count. |
+| - | `scan_thread_nice_value = 5` | Lower the priority of query I/O threads to ensure writing performance and timeliness. |
+| Other | `string_type_length_soft_limit_bytes = 10485760` | Increase the length limit of string-type data to 10 MB. |
+| - | `trash_file_expire_time_sec = 300` `path_gc_check_interval_second = 900` `path_scan_interval_second = 900` | Accelerate the recycling of trash files. |
+
+For more information, refer to [BE Configuration](../admin-manual/config/be-config.md).
+
+### Step 4: Create tables
+
+Due to the distinct characteristics of both writing and querying log data, it is recommended to configure tables with targeted settings to enhance performance.
+
+**Configure data partitioning and bucketing**
+
+- For data partitioning:
+
+ - Enable [range partitioning](https://doris.apache.org/docs/2.0/table-design/data-partition#range-partition) with [dynamic partitions](https://doris.apache.org/docs/2.0/table-design/data-partition#dynamic-partition) managed automatically by day.
+
+ - Use a field in the DATETIME type as the key for accelerated retrieval of the latest N log entries.
+
+- For data bucketing:
+
+ - Configure the number of buckets to be roughly three times the total number of disks in the cluster.
+
+ - Use the Random strategy to optimize batch writing efficiency when paired with single tablet imports.
+
+For more information, refer to [Data Partitioning](../table-design/data-partition.md).
+
+**Configure compaction fileds**
+
+Configure compaction fields as follows:
+
+- Use the time_series strategy to reduce write amplification, which is crucial for high-throughput log writes.
+
+- Use single-replica compaction to minimize the overhead of multi-replica compaction.
+
+**Configure index fields**
+
+Configuring index fields as follows:
+
+- Create indexes for fields that are frequently queried.
+
+- For fields that require full-text search, specify the parser field as unicode, which satisfies most requirements. If there is a need to support phrase queries, set the support_phrase field to true; if not needed, set it to false to reduce storage space.
+
+**Configure storage policies**
+
+Configure storage policies as follows:
+
+- For storage of hot data, if using cloud storage, configure the number of data copies as 1; if using physical disks, configure the number of data copies as at least 2.
+
+- Configure the storage location for log_s3 and set the log_policy_3day policy, where the data is cooled and moved to the specified storage location of log_s3 after 3 days. Refer to the code below.
+
+```Go
+CREATE DATABASE log_db;
+USE log_db;
+
+CREATE RESOURCE "log_s3"
+PROPERTIES
+(
+ "type" = "s3",
+ "s3.endpoint" = "your_endpoint_url",
+ "s3.region" = "your_region",
+ "s3.bucket" = "your_bucket",
+ "s3.root.path" = "your_path",
+ "s3.access_key" = "your_ak",
+ "s3.secret_key" = "your_sk"
+);
+
+CREATE STORAGE POLICY log_policy_3day
+PROPERTIES(
+ "storage_resource" = "log_s3",
+ "cooldown_ttl" = "259200"
+);
+
+CREATE TABLE log_table
+(
+ `ts` DATETIME,
+ `host` TEXT,
+ `path` TEXT,
+ `message` TEXT,
+ INDEX idx_host (`host`) USING INVERTED,
+ INDEX idx_path (`path`) USING INVERTED,
+ INDEX idx_message (`message`) USING INVERTED PROPERTIES("parser" = "unicode", "support_phrase" = "true")
+)
+ENGINE = OLAP
+DUPLICATE KEY(`ts`)
+PARTITION BY RANGE(`ts`) ()
+DISTRIBUTED BY RANDOM BUCKETS 250
+PROPERTIES (
+"dynamic_partition.enable" = "true",
+"dynamic_partition.create_history_partition" = "true",
+"dynamic_partition.time_unit" = "DAY",
+"dynamic_partition.start" = "-30",
+"dynamic_partition.end" = "1",
+"dynamic_partition.prefix" = "p",
+"dynamic_partition.buckets" = "250",
+"dynamic_partition.replication_num" = "1", -- unneccessary for the compute-storage coupled mode
+"replication_num" = "1" -- unneccessary for the compute-storage coupled mode
+"enable_single_replica_compaction" = "true", -- unneccessary for the compute-storage coupled mode
+"storage_policy" = "log_policy_3day", -- unneccessary for the compute-storage coupled mode
+"compaction_policy" = "time_series"
+);
+```
+
+### Step 5: Collect, query, and analyze logs
+
+After completing table creation, you can proceed with log collection, querying, and analysis.
+
+**Collect logs**
+
+Apache Doris provides open and versatile Stream HTTP APIs, through which you can connect with popular log collectors such as Logstash, Filebeat, Kafka, and others to carry out log collection work. This section explains how to integrate these log collectors using the Stream HTTP APIs.
+
+**Integrating Logstash**
+
+Follow these steps:
+
+1. Download and install the Logstash Doris Output plugin. You can choose one of the following two methods:
+
+ - [Click to download](https://apache-doris-releases.oss-accelerate.aliyuncs.com/logstash-output-doris-1.0.0.gem) and install.
+
+ - Compile from the source code and run the following command to install:
+
+```markdown
+./bin/logstash-plugin install logstash-output-doris-1.0.0.gem
+```
+
+2. Configure Logstash. Specify the following fields:
+
+ - `logstash.yml`: Used to configure Logstash batch processing log sizes and timings for improved data writing performance.
+
+ ```Plain Text
+ pipeline.batch.size: 1000000
+ pipeline.batch.delay: 10000
+ ```
+
+ - `logstash_demo.conf`: Used to configure the specific input path of the collected logs and the settings for output to Apache Doris.
+
+ ```markdown
+ input {
+ file {
+ path => "/path/to/your/log"
+ }
+ }
+ <br />output {
+ doris {
+ http_hosts => \[ "<http://fehost1:http_port>", "<http://fehost2:http_port>", "<http://fehost3:http_port"\>]
+ user => "your_username"
+ password => "your_password"
+ db => "your_db"
+ table => "your_table"
+ \# doris stream load http headers
+ headers => {
+ "format" => "json"
+ "read_json_by_line" => "true"
+ "load_to_single_tablet" => "true"
+ }
+ \# field mapping: doris fileld name => logstash field name
+ \# %{} to get a logstash field, \[\] for nested field such as \[host\]\[name\] for host.name
+ mapping => {
+ "ts" => "%{@timestamp}"
+ "host" => "%{\[host\]\[name\]}"
+ "path" => "%{\[log\]\[file\]\[path\]}"
+ "message" => "%{message}"
+ }
+ log_request => true
+ log_speed_interval => 10
+ }
+ }
+ ```
+
+3. Run Logstash according to the command below, collect logs, and output to Apache Doris.
+
+ ```Bash
+ ./bin/logstash -f logstash_demo.conf
+ ```
+
+For more information about the Logstash Doris Output plugin, see [Logstash Doris Output Plugin](../ecosystem/logstash.md).
+
+**Integrating Filebeat**
+
+Follow these steps:
+
+1. Obtain the Filebeat binary file that supports output to Apache Doris. You can [click to download](https://apache-doris-releases.oss-accelerate.aliyuncs.com/filebeat-doris-1.0.0) or compile it from the Apache Doris source code.
+
+2. Configure Filebeat. Specify the filebeat_demo.yml field that is used to configure the specific input path of the collected logs and the settings for output to Apache Doris.
+
+
+ ```YAML
+ # input
+ filebeat.inputs:
+ - type: log
+ enabled: true
+ paths:
+ - /path/to/your/log
+ multiline:
+ type: pattern
+ pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}'
+ negate: true
+ match: after
+ skip_newline: true
+
+ processors:
+ - script:
+ lang: javascript
+ source: >
+ function process(event) {
+ var msg = event.Get("message");
+ msg = msg.replace(/\t/g, " ");
+ event.Put("message", msg);
+ }
+ - dissect:
+ # 2024-06-08 18:26:25,481 INFO (report-thread|199) [ReportHandler.cpuReport():617] begin to handle
+ tokenizer: "%{day} %{time} %{log_level} (%{thread}) [%{position}] %{content}"
+ target_prefix: ""
+ ignore_failure: true
+ overwrite_keys: true
+
+ # queue and batch
+ queue.mem:
+ events: 1000000
+ flush.min_events: 100000
+ flush.timeout: 10s
+
+ # output
+ output.doris:
+ fenodes: [ "http://fehost1:http_port", "http://fehost2:http_port", "http://fehost3:http_port" ]
+ user: "your_username"
+ password: "your_password"
+ database: "your_db"
+ table: "your_table"
+ # output string format
+ codec_format_string: '{"ts": "%{[day]} %{[time]}", "host": "%{[agent][hostname]}", "path": "%{[log][file][path]}", "message": "%{[message]}"}'
+ headers:
+ format: "json"
+ read_json_by_line: "true"
+ load_to_single_tablet: "true"
+ ```
+
+3. Run Filebeat according to the command below, collect logs, and output to Apache Doris.
+
+ ```Bash
+ chmod +x filebeat-doris-1.0.0
+ ./filebeat-doris-1.0.0 -c filebeat_demo.yml
+ ```
+
+For more information about Filebeat, refer to [Beats Doris Output Plugin](../ecosystem/beats.md).
+
+**Integrating Kafka**
+
+Write JSON formatted logs to Kafka's message queue, create a Kafka Routine Load, and allow Apache Doris to actively pull data from Kafka.
+
+You can refer to the example below, where `property.\*` represents Librdkafka client-related configurations and needs to be adjusted according to the actual Kafka cluster situation.
+
+```SQL
+CREATE ROUTINE LOAD load_log_kafka ON log_db.log_table
+COLUMNS(ts, clientip, request, status, size)
+PROPERTIES (
+"max_batch_interval" = "10",
+"max_batch_rows" = "1000000",
+"max_batch_size" = "109715200",
+"load_to_single_tablet" = "true",
+"timeout" = "600",
+"strict_mode" = "false",
+"format" = "json"
+)
+FROM KAFKA (
+"kafka_broker_list" = "host:port",
+"kafka_topic" = "log_\_topic_",
+"property.group.id" = "your_group_id",
+"property.security.protocol"="SASL_PLAINTEXT",
+"property.sasl.mechanism"="GSSAPI",
+"property.sasl.kerberos.service.name"="kafka",
+"property.sasl.kerberos.keytab"="/path/to/xxx.keytab",
+"property.sasl.kerberos.principal"="<xx...@yyy.com>"
+);
+<br/>SHOW ROUTINE LOAD;
+```
+
+For more information about Kafka, see [Routine Load](../data-operate/import/routine-load-manual.md)。
+
+**Using customized programs to collect logs**
+
+In addition to integrating common log collectors, you can also customize programs to import log data into Apache Doris using the Stream Load HTTP API. Refer to the following code:
+
+```Bash
+curl \\
+\--location-trusted \\
+\-u username:password \\
+\-H "format:json" \\
+\-H "read_json_by_line:true" \\
+\-H "load_to_single_tablet:true" \\
+\-H "timeout:600" \\
+\-T logfile.json \\
+http://fe_host:fe_http_port/api/log_db/log_table/\_stream_load
+```
+
+When using custom programs, pay attention to the following key points:
+
+- Use Basic Auth for HTTP authentication, calculate using the command echo -n 'username:password' | base64.
+
+- Set HTTP header "format:json" to specify the data format as JSON.
+
+- Set HTTP header "read_json_by_line:true" to specify one JSON per line.
+
+- Set HTTP header "load_to_single_tablet:true" to import data into one bucket at a time to reduce small file imports.
+
+- It is recommended to write batches whose sizes are between 100MB to 1GB on the client side. For Apache Doris version 2.1 and higher, you need to reduce batch sizes on the client side through the Group Commit function.
+
+**Query logs**
+
+Apache Doris supports standard SQL, so you can connect to the cluster through MySQL client or JDBC to execute SQL for log queries.
+
+```Plain Text
+mysql -h fe_host -P fe_mysql_port -u your_username -Dyour_db_name
+```
+
+Here are 5 common SQL query commands for reference:
+
+- View the latest 10 log entries
+
+```SQL
+SELECT \* FROM your_table_name ORDER BY ts DESC LIMIT 10;
+```
+
+- Query the latest 10 log entries with the host as 8.8.8.8
+
+```SQL
+SELECT \* FROM your_table_name WHERE host = '8.8.8.8' ORDER BY ts DESC LIMIT 10;
+```
+
+- Retrieve the latest 10 log entries with error or 404 in the request field. In the command below, MATCH_ANY is a full-text search SQL syntax used by Apache Doris for matching any keyword in the fields.
+
+```SQL
+SELECT \* FROM your_table_name WHERE message **MATCH_ANY** 'error 404'
+ORDER BY ts DESC LIMIT 10;
+```
+
+- Retrieve the latest 10 log entries with image and faq in the request field. In the command below, MATCH_ALL is a full-text search SQL syntax used by Apache Doris for matching all keywords in the fields.
+
+```SQL
+SELECT \* FROM your_table_name WHERE message **MATCH_ALL** 'image faq'
+ORDER BY ts DESC LIMIT 10;
+```
+
+- Retrieve the latest 10 entries with image and faq in the request field. In the following command, MATCH_PHRASE is a full-text search SQL syntax used by Apache Doris for matching all keywords in the fields and requiring consistent order. In the example below, a image faq b can match, but a faq image b cannot match because the order of image and faq does not match the syntax.
+
+```SQL
+SELECT \* FROM your_table_name WHERE message **MATCH_PHRASE** 'image faq'
+ORDER BY ts DESC LIMIT 10;
+```
+
+**Analyze logs visually**
+
+VeloDB Enterprise Core, built on Apache Doris, provides a data development platform called Velo Enterprise WebUI ("WebUI"), featuring a Kibana Discover-like log retrieval and analysis interface for intuitive and easy exploratory log analysis interaction as shown in the image below:
+
+
+
+On this interface, WebUI supports the following operations:
+
+- Support for full-text search and SQL modes
+
+- Support for selecting query log timeframes with time boxes and histograms
+
+- Display of detailed log information, expandable into JSON or tables
+
+- Interactive clicking to add and remove filter conditions in the log data context
+
+- Display of top field values in search results for finding anomalies and further drilling down for analysis
+
+You can [click to download Velo Enterprise Core](https://www.velodb.io/download/enterprise) and [install it](https://docs.velodb.io/enterprise/enterprise-core-guide/velodb-distribution-doris-core-deployment-guide) to use WebUI. For more information about the main functions and how to use WebUI, see [WebUI](https://docs.velodb.io/enterprise/enterprise-core-guide/velodb-webui-guide).
\ No newline at end of file
diff --git a/versioned_docs/version-2.0/table-design/data-partition.md b/versioned_docs/version-2.0/table-design/data-partition.md
index 1b0cfba9e0c..e0fab7e88fe 100644
--- a/versioned_docs/version-2.0/table-design/data-partition.md
+++ b/versioned_docs/version-2.0/table-design/data-partition.md
@@ -777,8 +777,8 @@ Suppose our table DDL is as follows:
```sql
CREATE TABLE `DAILY_TRADE_VALUE`
(
- `TRADE_DATE` datev2 NOT NULL COMMENT '交易日期',
- `TRADE_ID` varchar(40) NOT NULL COMMENT '交易编号',
+ `TRADE_DATE` datev2 NOT NULL COMMENT 'TRADE_DATE',
+ `TRADE_ID` varchar(40) NOT NULL COMMENT 'TRADE_ID',
......
)
UNIQUE KEY(`TRADE_DATE`, `TRADE_ID`)
diff --git a/versioned_docs/version-2.1/data-operate/update/update-of-aggregate-model.md b/versioned_docs/version-2.1/data-operate/update/update-of-aggregate-model.md
index d4822da1211..373a890ad49 100644
--- a/versioned_docs/version-2.1/data-operate/update/update-of-aggregate-model.md
+++ b/versioned_docs/version-2.1/data-operate/update/update-of-aggregate-model.md
@@ -1,6 +1,6 @@
---
{
- "title": "Updaing Data on Aggregate Key Model",
+ "title": "Updating Data on Aggregate Key Model",
"language": "en"
}
---
diff --git a/versioned_docs/version-2.1/data-operate/update/update-of-unique-model.md b/versioned_docs/version-2.1/data-operate/update/update-of-unique-model.md
index 1d537dfa6d5..3b35dc6dd35 100644
--- a/versioned_docs/version-2.1/data-operate/update/update-of-unique-model.md
+++ b/versioned_docs/version-2.1/data-operate/update/update-of-unique-model.md
@@ -1,6 +1,6 @@
---
{
- "title": "Updaing Data on Unique Key Model",
+ "title": "Updating Data on Unique Key Model",
"language": "en"
}
---
diff --git a/versioned_docs/version-2.1/ecosystem/flink-doris-connector.md b/versioned_docs/version-2.1/ecosystem/flink-doris-connector.md
index f453b4d87b9..ad705a1ba7f 100644
--- a/versioned_docs/version-2.1/ecosystem/flink-doris-connector.md
+++ b/versioned_docs/version-2.1/ecosystem/flink-doris-connector.md
@@ -79,7 +79,7 @@ Copy this file to `classpath` of `Flink` to use `Flink-Doris-Connector`. For exa
### read
-####SQL
+**SQL**
```sql
-- doris source
@@ -98,7 +98,7 @@ CREATE TABLE flink_doris_source (
);
```
-####DataStream
+**DataStream**
```java
DorisOptions.Builder builder = DorisOptions.builder()
@@ -118,7 +118,7 @@ env.fromSource(dorisSource, WatermarkStrategy.noWatermarks(), "doris source").pr
### write
-####SQL
+**SQL**
```sql
--enable checkpoint
@@ -144,7 +144,7 @@ CREATE TABLE flink_doris_sink (
INSERT INTO flink_doris_sink select name,age,price,sale from flink_doris_source
```
-####DataStream
+**DataStream**
DorisSink writes data to Doris through StreamLoad, and DataStream supports different serialization methods when writing
diff --git a/versioned_docs/version-2.1/practical-guide/log-storage-analysis.md b/versioned_docs/version-2.1/practical-guide/log-storage-analysis.md
new file mode 100644
index 00000000000..60d6d891737
--- /dev/null
+++ b/versioned_docs/version-2.1/practical-guide/log-storage-analysis.md
@@ -0,0 +1,593 @@
+---
+{
+ "title": "Log Storage and Analysis",
+ "language": "en"
+}
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Logs record key events in the system and contain crucial information such as the events' subject, time, location, and content. To meet the diverse needs of observability in operations, network security monitoring, and business analysis, enterprises might need to collect scattered logs for centralized storage, querying, and analysis to extract valuable content from the log data further.
+
+In this scenario, Apache Doris provides a corresponding solution. With the characteristics of log scenarios in mind, Apache Doris added inverted-index and ultra-fast full-text search capabilities, optimizing write performance and storage space to the extreme. This allows users to build an open, high-performance, cost-effective, and unified log storage and analysis platform based on Apache Doris.
+
+Focused on this solution, this chapter contains the following 3 sections:
+
+- **Overall architecture**: This section explains the core components and architecture of the log storage and analysis platform built on Apache Doris.
+
+- **Features and advantages**: This section explains the features and advantages of the log storage and analysis platform built on Apache Doris.
+
+- **Operational guide**: This section explains how to build a log storage and analysis platform based on Apache Doris.
+
+## Overall architecture
+
+The following figure illustrates the architecture of the log storage and analysis platform built on Apache Doris:
+
+
+
+The architecture contains the following 3 parts:
+
+- **Log collection and preprocessing**: Various log collection tools can write log data into Apache Doris through HTTP APIs.
+
+- **Log storage and analysis engine**: Apache Doris provides high-performance, low-cost unified log storage, offering rich retrieval and analysis capabilities through SQL interfaces.
+
+- **Log analysis and alert interface**: Various log retrieval and analysis tools can query Apache Doris through standard SQL interfaces, providing users with a simple and user-friendly interface.
+
+## Features and advantages
+
+The following figure illustrates the architecture of the log storage and analysis platform built on Apache Doris::
+
+- **High throughput, low latency log writing**: Supports stable writing of log data at the level of hundreds of TBs and GB/s per day, while maintaining latency within 1 second.
+
+- **Cost-effective storage of massive log data**: Supports petabyte-scale storage, saving 60% to 80% in storage costs compared to Elasticsearch, and further reducing storage costs by 50% by storing cold data in S3/HDFS.
+
+- **High-performance log full-text search and analysis**: Supports inverted indexes and full-text search, providing second-level response times for common log queries (keyword searches, trend analysis, etc.).
+
+- **Open and user-friendly upstream and downstream ecosystem**: Upstream integration with common log collection systems and data sources such as Logstash, Filebeat, Fluentbit, and Kafka through Stream Load's universal HTTP APIs, and downstream integration with various visual analytics UIs using standard MySQL protocol and syntax, such as observability platform Grafana, BI analytics Superset, and log retrieval Doris WebUI similar to Kibana.
+
+### Cost-effective performance
+
+After Benchmark testing and production validation, the log storage and analysis platform built on Apache Doris has a 5 to 10 times cost-performance advantage over Elasticsearch. Apache Doris's performance benefits are mainly due to its globally leading high-performance storage and query engine, as well as specialized optimizations for log scenarios:
+
+- **Improved write throughput**: Elasticsearch's write performance bottleneck lies in CPU consumption for parsing data and building inverted indexes. In comparison, Apache Doris has optimized writes in two aspects: using SIMD and other CPU vector instructions to improve JSON data parsing speed and index-building performance and simplifying the inverted index structure for log scenarios by removing unnecessary data structures like forward indexes, effectively reducing index build complexi [...]
+
+- **Reduced storage costs**: Elasticsearch's storage bottleneck comes from multiple storage of forward, inverted, Docvalue columns, and lower compression ratios with general compression algorithms. Conversely, Apache Doris has optimized storage by eliminating forward indexes, reducing index data volume by 30%; adopting columnar storage and Zstandard compression algorithm, achieving compression ratios 5-10 times higher than Elasticsearch's 1.5 times; with low access frequency of cold data [...]
+
+### Strong analytical capabilities
+
+Apache Doris supports standard SQL and is compatible with MySQL protocol and syntax. Therefore, log systems built on Apache Doris can use SQL for log analysis, giving the following advantages to log systems:
+
+- **Easy to use**: Engineers and data analysts are very familiar with SQL, their expertise can be reused, no need to learn new technology stacks to quickly get started.
+
+- **Rich ecosystem**: The MySQL ecosystem is the most widely used language in the database field, seamlessly integrating with and applying to the MySQL ecosystem. Doris can leverage MySQL command line and various GUI tools, BI tools, and other big data ecosystem tools for more complex and diverse data processing and analysis needs.
+
+- **Strong analytical capabilities**: SQL has become the de facto standard for database and big data analysis, with powerful expressive capabilities and functions supporting retrieval, aggregation, multi-table JOIN, subqueries, UDFs, logical views, materialized views, and various data analysis capabilities.
+
+### Flexible Schema
+
+Here is a typical example of a semi-structured log in JSON format. The top-level fields are some fixed fields, such as timestamp, source, node, component, level, clientRequestID, message, and properties, which are present in every log entry. The nested fields of the properties , such as properties.size and properties.format are more dynamic, and the fields of each log may vary.
+
+```JSON
+{
+"timestamp": "2014-03-08T00:50:03.8432810Z",
+"source": "ADOPTIONCUSTOMERS81",
+"node": "Engine000000000405",
+"level": "Information",
+"component": "DOWNLOADER",
+"clientRequestId": "671db15d-abad-94f6-dd93-b3a2e6000672",
+"message": "Downloading file path: benchmark/2014/ADOPTIONCUSTOMERS81_94_0.parquet.gz",
+"properties": {
+"size": 1495636750,
+"format": "parquet",
+"rowCount": 855138,
+"downloadDuration": "00:01:58.3520561"
+}
+}
+```
+
+
+Apache Doris provides several aspects of support for Flexible Schema log data:
+
+- For minor changes to top-level fields, Light Schema Change can be used to add or remove columns and to add or remove indexes, enabling schema changes to be completed in seconds. When planning a log platform, users only need to consider which fields need to be indexed.
+
+- For extension fields similar to properties, the native semi-structured data type `VARIANT` is provided, which can write any JSON data, automatically recognize field names and types in JSON, and automatically split frequently occurring fields for columnar storage for subsequent analysis. Additionally, `VARIANT` can create inverted indexes to accelerate internal field queries and retrievals.
+
+Compared to Elasticsearch's Dynamic Mapping, Apache Doris's Flexible Schema has the following advantages:
+
+- Allows a field to have multiple types, `VARIANT` automatically handles conflicts and type promotion for fields, better adapting to iterative changes in log data.
+
+- `VARIANT` automatically merges infrequently occurring fields into a column store to avoid performance issues caused by excessive fields, metadata, or columns.
+
+- Not only can columns be dynamically added, but they can also be dynamically deleted, and indexes can be dynamically added or removed, eliminating the need to index all fields at the beginning like Elasticsearch, reducing unnecessary costs.
+
+## Operational guide
+
+### Step 1: Estimate resources
+
+Before deploying the cluster, you need to estimate the hardware resources required for the servers. Follow the steps below:
+
+1. Estimate the resources for data writing by the following calculation formulas:
+
+- Daily data increment / 86400 s = Average write throughput
+
+- Average write throughput \* Ratio of the peak write throughput to the average write throughput = Peak write throughput
+
+- Peak write throughput / Write throughput of a single-core CPU = Number of CPU cores for the peak write throughput
+
+2. Estimate the resources for data storage by the calculation formula: Daily data increment / Data compression ratio \* Number of data copies \* Data storage duration = Data storage space.
+
+3. Estimate the resources for data querying. The resources for data querying depend on the query volume and complexity. It is recommended to reserve 50% of CPU resources for data query initially and then adjust according to the actual test results.
+
+4. Integrate the calculation results as follows:
+
+ 1. Divide the number of CPU cores calculated in Step 1 and Step 3 by the number of CPU cores of a BE server, and you can get the number of BE servers.
+
+ 2. Based on the number of BE servers and the calculation result of Step 2, estimate the storage space required for each BE server.
+
+ 3. Allocate the storage space required for each BE server to 4 to 12 data disks, and you can get the storage capacity required for a single data disk.
+
+For example, suppose that the daily data increment is 100 TB, the data compression ratio is 7, the number of data copies is 1, the storage duration of hot data is 3 days, the storage duration of cold data is 30 days, the ratio of the peak write throughput to the average write throughput is 200%, the write throughput of a single-core CUP is 10 MB/s, and 50% of CPU resources are reserved for data querying, one can estimate that:
+
+- 3 FE servers are required, each configured with a 16-core CPU, 64 GB memory, and an 1100 GB SSD disk.
+
+- 15 BE servers are required, each configured with a 32-core CPU, 256 GB memory, and 8 500 GB SSD disks.
+
+- S3:430 TB
+
+Refer to the following table to learn about the values of indicators in the example above and how they are calculated.
+
+| Indicator (Unit) | Value | Description |
+| --- | --- | --- |
+| Daily data increment (TB) | 100 | Specify the value according to your actual needs. |
+| Data compression ratio | 7 | Specify the value according to your actual needs, which is often between 5 to 10. Note that the data contains index data. |
+| Number of data copies | 1 | Specify the value according to your actual needs, which can be 1, 2, or 3. The default value is 1. |
+| Storage duration of hot data (day) | 3 | Specify the value according to your actual needs. |
+| Storage duration of cold data (day) | 30 | Specify the value according to your actual needs. |
+| Data storage duration | 33 | Calculation formula: Storage duration of hot data + Storage duration of cold data |
+| Estimated storage space for hot data (TB) | 42.9 | Calculation formula: Daily data increment / Data compression ratios \* Number of data copies \* Storage duration of hot data |
+| Estimated storage space for cold data (TB) | 428.6 | Calculation formula: Daily data increment / Data compression ratios \* Number of data copies \* Storage duration of cold data |
+| Ratio of the peak write throughput to the average write throughput | 200% | Specify the value according to your actual needs. The default value is 200%. |
+| Number of CPU cores of a BE server | 32 | Specify the value according to your actual needs. The default value is 32. |
+| Average write throughput (MB/s) | 1214 | Calculation formula: Daily data increment / 86400 s |
+| Peak write throughput (MB/s) | 2427 | Calculation formula: Average write throughput \* Ratio of the peak write throughput to the average write throughput |
+| Number of CPU cores for the peak write throughput | 242.7 | Calculation formula: Peak write throughput / Write throughput of a single-core CPU |
+| Percent of CPU resources reserved for data querying | 50% | Specify the value according to your actual needs. The default value is 50%. |
+| Estimated number of BE servers | 15.2 | Calculation formula: Number of CPU cores for the peak write throughput / Number of CPU cores of a BE server /（1 - Percent of CPU resources reserved for data querying） |
+| Rounded number of BE servers | 15 | Calculation formula: MAX (Number of data copies, Estimated number of BE servers) |
+| Estimated data storage space for each BE server (TB) | 4.03 | Calculation formula: Estimated storage space for hot data / Estimated number of BE servers /（1 - 30%）, where 30% represents the percent of reserved storage space.<br/><br/>It is recommended to mount 4 to 12 data disks on each BE server to enhance I/O capabilities. |
+
+### Step 2: Deploy the cluster
+
+After estimating the resources, you need to deploy the cluster. It is recommended to deploy in both physical and virtual environments manually. For manual deployment, refer to [Manual Deployment](../install/cluster-deployment/standard-deployment.md).
+
+Alternatively, it is recommended to use VeloDB Manager provided by VeloDB Enterprise to deploy the cluster, reducing overall deployment costs. For more information about the VeloDB Manager, please refer to the following documents:
+
+- [VeloDB Manager 24.x Installation Manual](https://docs.velodb.io/enterprise/velodb-manager-guide/deployment-guide/deployment-guide-24.x)
+
+- [VeloDB Manager 24.x User Guide](https://docs.velodb.io/enterprise/velodb-manager-guide/management-guide/management-guide-24.x)
+
+### Step 3: Optimize FE and BE configurations
+
+After completing the cluster deployment, it is necessary to optimize the configuration parameters for both the front-end and back-end separately, so as to better suit the scenario of log storage and analysis.
+
+**Optimize FE configurations**
+
+You can find FE configuration fields in `fe/conf/fe.conf`. Refer to the following table to optimize FE configurations.
+
+| Configuration fields to be optimized | Description |
+| :----------------------------------------------------------- | :----------------------------------------------------------- |
+| `max_running_txn_num_per_db = 10000` | Increase the parameter value to adapt to high-concurrency import transactions. |
+| `streaming_lable_keep_max_second = 3600``label_keep_max_second = 7200` | Increase the retention time to handle high-frequency import transactions with high memory usage. |
+| `enable_round_robin_create_tablet = true` | When creating Tablets, use a Round Robin strategy to distribute evenly. |
+| `tablet_rebalancer_type = partition` | When balancing Tablets, use a strategy to evenly distribute within each partition. |
+| `enable_single_replica_load = true` | Enable single-replica import, where multiple replicas only need to build an index once to reduce CPU consumption. |
+| `autobucket_min_buckets = 10` | Increase the minimum number of automatically bucketed buckets from 1 to 10 to avoid insufficient buckets when the log volume increases. |
+| `max_backend_heartbeat_failure_tolerance_count = 10` | In log scenarios, the BE server may experience high pressure, leading to short-term timeouts, so increase the tolerance count from 1 to 10. |
+
+For more information, refer to [FE Configuration](../admin-manual/config/fe-config.md).
+
+**Optimize BE configurations**
+
+You can find BE configuration fields in `be/conf/be.conf`. Refer to the following table to optimize BE configurations.
+
+| Module | Configuration fields to be optimized | Description |
+| :--------- | :----------------------------------------------------------- | :----------------------------------------------------------- |
+| Storage | `storage_root_path = /path/to/dir1;/path/to/dir2;...;/path/to/dir12` | Configure the storage path for hot data on disk directories. |
+| - | `enable_file_cache = true` | Enable file caching. |
+| - | `file_cache_path = [{"path": "/mnt/datadisk0/file_cache", "total_size":53687091200, "query_limit": "10737418240"},{"path": "/mnt/datadisk1/file_cache", "total_size":53687091200,"query_limit": "10737418240"}]` | Configure the cache path and related settings for cold data with the following specific configurations:<br/>`path`: cache path<br/>`total_size`: total size of the cache path in bytes, where 53687091200 bytes equals 50 GB<br/>`query_limit`: maximum amount of data tha [...]
+| Write | `write_buffer_size = 1073741824` | Increase the file size of the write buffer to reduce small files and random I/O operations, improving performance. |
+| - | `max_tablet_version_num = 20000` | In coordination with the time_series compaction strategy for table creation, allow more versions to remain temporarily unmerged |
+| - | `enable_single_replica_load = true` | Enable single replica import to reduce CPU consumption, consistent with FE configuration. |
+| Compaction | `max_cumu_compaction_threads = 8` | Set to CPU core count / 4, indicating that 1/4 of CPU resources are used for writing, 1/4 for background compaction, and 2/1 for queries and other operations. |
+| - | `inverted_index_compaction_enable = true` | Enable inverted index compaction to reduce CPU consumption during compaction. |
+| - | `enable_segcompaction = false` `enable_ordered_data_compaction = false` | Disable two compaction features that are unnecessary for log scenarios. |
+| Cache | `disable_storage_page_cache = true` `inverted_index_searcher_cache_limit = 30%` | Due to the large volume of log data and limited caching effect, switch from data caching to index caching. |
+| - | `inverted_index_cache_stale_sweep_time_sec = 3600` `index_cache_entry_stay_time_after_lookup_s = 3600` | Maintain index caching in memory for up to 1 hour. |
+| - | `enable_inverted_index_cache_on_cooldown = true`<br />`enable_write_index_searcher_cache = false` | Enable automatic caching of cold data storage during index uploading. |
+| - | `tablet_schema_cache_recycle_interval = 3600` `segment_cache_capacity = 20000` | Reduce memory usage by other caches. |
+| Thread | `pipeline_executor_size = 24` `doris_scanner_thread_pool_thread_num = 48` | Configure computing threads and I/O threads for a 32-core CPU in proportion to core count. |
+| - | `scan_thread_nice_value = 5` | Lower the priority of query I/O threads to ensure writing performance and timeliness. |
+| Other | `string_type_length_soft_limit_bytes = 10485760` | Increase the length limit of string-type data to 10 MB. |
+| - | `trash_file_expire_time_sec = 300` `path_gc_check_interval_second = 900` `path_scan_interval_second = 900` | Accelerate the recycling of trash files. |
+
+
+For more information, refer to [BE Configuration](../admin-manual/config/be-config.md).
+
+### Step 4: Create tables
+
+Due to the distinct characteristics of both writing and querying log data, it is recommended to configure tables with targeted settings to enhance performance.
+
+**Configure data partitioning and bucketing**
+
+- For data partitioning:
+
+ - Enable [range partitioning](https://doris.apache.org/docs/table-design/data-partition#range-partition) with [dynamic partitions](https://doris.apache.org/docs/table-design/data-partition#dynamic-partition) managed automatically by day.
+
+ - Use a field in the DATETIME type as the key for accelerated retrieval of the latest N log entries.
+
+- For data bucketing:
+
+ - Configure the number of buckets to be roughly three times the total number of disks in the cluster.
+
+ - Use the Random strategy to optimize batch writing efficiency when paired with single tablet imports.
+
+For more information, refer to [Data Partitioning](../table-design/data-partition.md).
+
+**Configure compaction fileds**
+
+Configure compaction fields as follows:
+
+- Use the time_series strategy to reduce write amplification, which is crucial for high-throughput log writes.
+
+- Use single-replica compaction to minimize the overhead of multi-replica compaction.
+
+**Configure index fields**
+
+Configuring index fields as follows:
+
+- Create indexes for fields that are frequently queried.
+
+- For fields that require full-text search, specify the parser field as unicode, which satisfies most requirements. If there is a need to support phrase queries, set the support_phrase field to true; if not needed, set it to false to reduce storage space.
+
+**Configure storage policies**
+
+Configure storage policies as follows:
+
+- For storage of hot data, if using cloud storage, configure the number of data copies as 1; if using physical disks, configure the number of data copies as at least 2.
+
+- Configure the storage location for log_s3 and set the log_policy_3day policy, where the data is cooled and moved to the specified storage location of log_s3 after 3 days. Refer to the code below.
+
+```Go
+CREATE DATABASE log_db;
+USE log_db;
+
+CREATE RESOURCE "log_s3"
+PROPERTIES
+(
+ "type" = "s3",
+ "s3.endpoint" = "your_endpoint_url",
+ "s3.region" = "your_region",
+ "s3.bucket" = "your_bucket",
+ "s3.root.path" = "your_path",
+ "s3.access_key" = "your_ak",
+ "s3.secret_key" = "your_sk"
+);
+
+CREATE STORAGE POLICY log_policy_3day
+PROPERTIES(
+ "storage_resource" = "log_s3",
+ "cooldown_ttl" = "259200"
+);
+
+CREATE TABLE log_table
+(
+ `ts` DATETIME,
+ `host` TEXT,
+ `path` TEXT,
+ `message` TEXT,
+ INDEX idx_host (`host`) USING INVERTED,
+ INDEX idx_path (`path`) USING INVERTED,
+ INDEX idx_message (`message`) USING INVERTED PROPERTIES("parser" = "unicode", "support_phrase" = "true")
+)
+ENGINE = OLAP
+DUPLICATE KEY(`ts`)
+PARTITION BY RANGE(`ts`) ()
+DISTRIBUTED BY RANDOM BUCKETS 250
+PROPERTIES (
+"dynamic_partition.enable" = "true",
+"dynamic_partition.create_history_partition" = "true",
+"dynamic_partition.time_unit" = "DAY",
+"dynamic_partition.start" = "-30",
+"dynamic_partition.end" = "1",
+"dynamic_partition.prefix" = "p",
+"dynamic_partition.buckets" = "250",
+"dynamic_partition.replication_num" = "1", -- unneccessary for the compute-storage coupled mode
+"replication_num" = "1" -- unneccessary for the compute-storage coupled mode
+"enable_single_replica_compaction" = "true", -- unneccessary for the compute-storage coupled mode
+"storage_policy" = "log_policy_3day", -- unneccessary for the compute-storage coupled mode
+"compaction_policy" = "time_series"
+);
+```
+
+### Step 5: Collect, query, and analyze logs
+
+After completing table creation, you can proceed with log collection, querying, and analysis.
+
+**Collect logs**
+
+Apache Doris provides open and versatile Stream HTTP APIs, through which you can connect with popular log collectors such as Logstash, Filebeat, Kafka, and others to carry out log collection work. This section explains how to integrate these log collectors using the Stream HTTP APIs.
+
+**Integrating Logstash**
+
+Follow these steps:
+
+1. Download and install the Logstash Doris Output plugin. You can choose one of the following two methods:
+
+ - [Click to download](https://apache-doris-releases.oss-accelerate.aliyuncs.com/logstash-output-doris-1.0.0.gem) and install.
+
+ - Compile from the source code and run the following command to install:
+
+```markdown
+./bin/logstash-plugin install logstash-output-doris-1.0.0.gem
+```
+
+2. Configure Logstash. Specify the following fields:
+
+ - `logstash.yml`: Used to configure Logstash batch processing log sizes and timings for improved data writing performance.
+
+ ```Plain Text
+ pipeline.batch.size: 1000000
+ pipeline.batch.delay: 10000
+ ```
+
+ - `logstash_demo.conf`: Used to configure the specific input path of the collected logs and the settings for output to Apache Doris.
+
+ ```markdown
+ input {
+ file {
+ path => "/path/to/your/log"
+ }
+ }
+ <br />output {
+ doris {
+ http_hosts => \[ "<http://fehost1:http_port>", "<http://fehost2:http_port>", "<http://fehost3:http_port"\>]
+ user => "your_username"
+ password => "your_password"
+ db => "your_db"
+ table => "your_table"
+ \# doris stream load http headers
+ headers => {
+ "format" => "json"
+ "read_json_by_line" => "true"
+ "load_to_single_tablet" => "true"
+ }
+ \# field mapping: doris fileld name => logstash field name
+ \# %{} to get a logstash field, \[\] for nested field such as \[host\]\[name\] for host.name
+ mapping => {
+ "ts" => "%{@timestamp}"
+ "host" => "%{\[host\]\[name\]}"
+ "path" => "%{\[log\]\[file\]\[path\]}"
+ "message" => "%{message}"
+ }
+ log_request => true
+ log_speed_interval => 10
+ }
+ }
+ ```
+
+3. Run Logstash according to the command below, collect logs, and output to Apache Doris.
+
+ ```Bash
+ ./bin/logstash -f logstash_demo.conf
+ ```
+
+For more information about the Logstash Doris Output plugin, see [Logstash Doris Output Plugin](../ecosystem/logstash.md).
+
+**Integrating Filebeat**
+
+Follow these steps:
+
+1. Obtain the Filebeat binary file that supports output to Apache Doris. You can [click to download](https://apache-doris-releases.oss-accelerate.aliyuncs.com/filebeat-doris-1.0.0) or compile it from the Apache Doris source code.
+
+2. Configure Filebeat. Specify the filebeat_demo.yml field that is used to configure the specific input path of the collected logs and the settings for output to Apache Doris.
+
+ ```YAML
+ # input
+ filebeat.inputs:
+ - type: log
+ enabled: true
+ paths:
+ - /path/to/your/log
+ multiline:
+ type: pattern
+ pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}'
+ negate: true
+ match: after
+ skip_newline: true
+
+ processors:
+ - script:
+ lang: javascript
+ source: >
+ function process(event) {
+ var msg = event.Get("message");
+ msg = msg.replace(/\t/g, " ");
+ event.Put("message", msg);
+ }
+ - dissect:
+ # 2024-06-08 18:26:25,481 INFO (report-thread|199) [ReportHandler.cpuReport():617] begin to handle
+ tokenizer: "%{day} %{time} %{log_level} (%{thread}) [%{position}] %{content}"
+ target_prefix: ""
+ ignore_failure: true
+ overwrite_keys: true
+
+ # queue and batch
+ queue.mem:
+ events: 1000000
+ flush.min_events: 100000
+ flush.timeout: 10s
+
+ # output
+ output.doris:
+ fenodes: [ "http://fehost1:http_port", "http://fehost2:http_port", "http://fehost3:http_port" ]
+ user: "your_username"
+ password: "your_password"
+ database: "your_db"
+ table: "your_table"
+ # output string format
+ codec_format_string: '{"ts": "%{[day]} %{[time]}", "host": "%{[agent][hostname]}", "path": "%{[log][file][path]}", "message": "%{[message]}"}'
+ headers:
+ format: "json"
+ read_json_by_line: "true"
+ load_to_single_tablet: "true"
+ ```
+
+3. Run Filebeat according to the command below, collect logs, and output to Apache Doris.
+
+ ```Bash
+ chmod +x filebeat-doris-1.0.0
+ ./filebeat-doris-1.0.0 -c filebeat_demo.yml
+ ```
+
+For more information about Filebeat, refer to [Beats Doris Output Plugin](../ecosystem/beats.md).
+
+**Integrating Kafka**
+
+Write JSON formatted logs to Kafka's message queue, create a Kafka Routine Load, and allow Apache Doris to actively pull data from Kafka.
+
+You can refer to the example below, where `property.\*` represents Librdkafka client-related configurations and needs to be adjusted according to the actual Kafka cluster situation.
+
+```SQL
+CREATE ROUTINE LOAD load_log_kafka ON log_db.log_table
+COLUMNS(ts, clientip, request, status, size)
+PROPERTIES (
+"max_batch_interval" = "10",
+"max_batch_rows" = "1000000",
+"max_batch_size" = "109715200",
+"load_to_single_tablet" = "true",
+"timeout" = "600",
+"strict_mode" = "false",
+"format" = "json"
+)
+FROM KAFKA (
+"kafka_broker_list" = "host:port",
+"kafka_topic" = "log_\_topic_",
+"property.group.id" = "your_group_id",
+"property.security.protocol"="SASL_PLAINTEXT",
+"property.sasl.mechanism"="GSSAPI",
+"property.sasl.kerberos.service.name"="kafka",
+"property.sasl.kerberos.keytab"="/path/to/xxx.keytab",
+"property.sasl.kerberos.principal"="<xx...@yyy.com>"
+);
+<br/>SHOW ROUTINE LOAD;
+```
+
+For more information about Kafka, see [Routine Load](../data-operate/import/routine-load-manual.md)。
+
+**Using customized programs to collect logs**
+
+In addition to integrating common log collectors, you can also customize programs to import log data into Apache Doris using the Stream Load HTTP API. Refer to the following code:
+
+```Bash
+curl \\
+\--location-trusted \\
+\-u username:password \\
+\-H "format:json" \\
+\-H "read_json_by_line:true" \\
+\-H "load_to_single_tablet:true" \\
+\-H "timeout:600" \\
+\-T logfile.json \\
+http://fe_host:fe_http_port/api/log_db/log_table/\_stream_load
+```
+
+When using custom programs, pay attention to the following key points:
+
+- Use Basic Auth for HTTP authentication, calculate using the command echo -n 'username:password' | base64.
+
+- Set HTTP header "format:json" to specify the data format as JSON.
+
+- Set HTTP header "read_json_by_line:true" to specify one JSON per line.
+
+- Set HTTP header "load_to_single_tablet:true" to import data into one bucket at a time to reduce small file imports.
+
+- It is recommended to write batches whose sizes are between 100MB to 1GB on the client side. For Apache Doris version 2.1 and higher, you need to reduce batch sizes on the client side through the Group Commit function.
+
+**Query logs**
+
+Apache Doris supports standard SQL, so you can connect to the cluster through MySQL client or JDBC to execute SQL for log queries.
+
+```Plain Text
+mysql -h fe_host -P fe_mysql_port -u your_username -Dyour_db_name
+```
+
+Here are 5 common SQL query commands for reference:
+
+- View the latest 10 log entries
+
+```SQL
+SELECT \* FROM your_table_name ORDER BY ts DESC LIMIT 10;
+```
+
+- Query the latest 10 log entries with the host as 8.8.8.8
+
+```SQL
+SELECT \* FROM your_table_name WHERE host = '8.8.8.8' ORDER BY ts DESC LIMIT 10;
+```
+
+- Retrieve the latest 10 log entries with error or 404 in the request field. In the command below, MATCH_ANY is a full-text search SQL syntax used by Apache Doris for matching any keyword in the fields.
+
+```SQL
+SELECT \* FROM your_table_name WHERE message **MATCH_ANY** 'error 404'
+ORDER BY ts DESC LIMIT 10;
+```
+
+- Retrieve the latest 10 log entries with image and faq in the request field. In the command below, MATCH_ALL is a full-text search SQL syntax used by Apache Doris for matching all keywords in the fields.
+
+```SQL
+SELECT \* FROM your_table_name WHERE message **MATCH_ALL** 'image faq'
+ORDER BY ts DESC LIMIT 10;
+```
+
+- Retrieve the latest 10 entries with image and faq in the request field. In the following command, MATCH_PHRASE is a full-text search SQL syntax used by Apache Doris for matching all keywords in the fields and requiring consistent order. In the example below, a image faq b can match, but a faq image b cannot match because the order of image and faq does not match the syntax.
+
+```SQL
+SELECT \* FROM your_table_name WHERE message **MATCH_PHRASE** 'image faq'
+ORDER BY ts DESC LIMIT 10;
+```
+
+**Analyze logs visually**
+
+VeloDB Enterprise Core, built on Apache Doris, provides a data development platform called Velo Enterprise WebUI ("WebUI"), featuring a Kibana Discover-like log retrieval and analysis interface for intuitive and easy exploratory log analysis interaction as shown in the image below:
+
+
+
+On this interface, WebUI supports the following operations:
+
+- Support for full-text search and SQL modes
+
+- Support for selecting query log timeframes with time boxes and histograms
+
+- Display of detailed log information, expandable into JSON or tables
+
+- Interactive clicking to add and remove filter conditions in the log data context
+
+- Display of top field values in search results for finding anomalies and further drilling down for analysis
+
+You can [click to download Velo Enterprise Core](https://www.velodb.io/download/enterprise) and [install it](https://docs.velodb.io/enterprise/enterprise-core-guide/velodb-distribution-doris-core-deployment-guide) to use WebUI. For more information about the main functions and how to use WebUI, see [WebUI](https://docs.velodb.io/enterprise/enterprise-core-guide/velodb-webui-guide).
\ No newline at end of file
diff --git a/versioned_docs/version-2.1/table-design/data-partition.md b/versioned_docs/version-2.1/table-design/data-partition.md
index 5805199f219..2e2a1a9fdc0 100644
--- a/versioned_docs/version-2.1/table-design/data-partition.md
+++ b/versioned_docs/version-2.1/table-design/data-partition.md
@@ -792,8 +792,8 @@ Suppose our table DDL is as follows:
```sql
CREATE TABLE `DAILY_TRADE_VALUE`
(
- `TRADE_DATE` datev2 NOT NULL COMMENT '交易日期',
- `TRADE_ID` varchar(40) NOT NULL COMMENT '交易编号',
+ `TRADE_DATE` datev2 NOT NULL COMMENT 'TRADE_DATE',
+ `TRADE_ID` varchar(40) NOT NULL COMMENT 'TRADE_ID',
......
)
UNIQUE KEY(`TRADE_DATE`, `TRADE_ID`)
diff --git a/versioned_sidebars/version-2.0-sidebars.json b/versioned_sidebars/version-2.0-sidebars.json
index d95be4b13b0..a5d65b43653 100644
--- a/versioned_sidebars/version-2.0-sidebars.json
+++ b/versioned_sidebars/version-2.0-sidebars.json
@@ -459,6 +459,13 @@
"admin-manual/compaction",
"admin-manual/rowsets"
]
+ },
+ {
+ "type": "category",
+ "label": "Practical Guide",
+ "items": [
+ "practical-guide/log-storage-analysis"
+ ]
},
{
"type": "category",
diff --git a/versioned_sidebars/version-2.1-sidebars.json b/versioned_sidebars/version-2.1-sidebars.json
index 9712b339aee..131008531be 100644
--- a/versioned_sidebars/version-2.1-sidebars.json
+++ b/versioned_sidebars/version-2.1-sidebars.json
@@ -499,6 +499,13 @@
"admin-manual/rowsets"
]
},
+ {
+ "type": "category",
+ "label": "Practical Guide",
+ "items": [
+ "practical-guide/log-storage-analysis"
+ ]
+ },
{
"type": "category",
"label": "Ecosystem",
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org