You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "xiongcp (via GitHub)" <gi...@apache.org> on 2024/09/24 09:36:56 UTC
[I] [Bug] 2.1.6版本连续的SQL查询导致BE内存持续上涨,停止查询后内存不释放 [doris]
xiongcp opened a new issue, #41223:
URL: https://github.com/apache/doris/issues/41223
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.
### Version
doris 2.1.6
### What's Wrong?
doris版本:2.1.6
FE和BE混合部署
大约十多张表,数据导入后大小总计为500M,单张表数据最大为百万级别
机器信息


```
测试代码 并发执行SQL,SQL基本都是秒内返回
func main() {
// 设置日志文件
logFile, err := os.OpenFile("run.log", os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0666)
if err != nil {
fmt.Println("Error opening log file:", err)
return
}
defer logFile.Close()
log.SetOutput(logFile)
args := os.Args[1:]
if len(args) < 1 {
fmt.Println("Usage: main run_count")
}
count, err := strconv.Atoi(args[0])
if err != nil {
fmt.Println("Usage: main run_count need int val")
}
for i := 0; i < (count); i++ {
fmt.Printf("run time :%d \n", i)
run()
}
}
func run() {
// 数据库连接信息
dsn := "root:test@tcp(xxxxxxxx:9030)/test"
// 连接到 MySQL 数据库
db, err := sql.Open("mysql", dsn)
if err != nil {
log.Println("Error connecting to database:", err)
fmt.Println("Error occurred while connecting to the database. Check error.log for details.")
return
}
defer db.Close()
// 设置最大空闲连接数和最大打开连接数
db.SetMaxOpenConns(10)
db.SetMaxIdleConns(5)
// 检查数据库连接
if err := db.Ping(); err != nil {
log.Println("Error pinging database:", err)
fmt.Println("Error occurred while pinging the database. Check error.log for details.")
return
}
// 打开 SQL 文件 SQL文件中有3万条业务SQL
file, err := os.Open("test.sql")
if err != nil {
log.Println("Error opening SQL file:", err)
fmt.Println("Error occurred while opening the SQL file. Check error.log for details.")
return
}
defer file.Close()
// 使用缓冲读取 SQL 文件
reader := bufio.NewReader(file)
// 控制并发执行的 goroutine 数量,设定最大并发数
maxConcurrency := 5
sem := make(chan struct{}, maxConcurrency)
var wg sync.WaitGroup
var total, successCount, failureCount int
startTime := time.Now()
// 按行读取并执行 SQL 语句
for {
line, err := reader.ReadString('\n')
if err != nil {
break
}
sqlStatement := strings.TrimSpace(line)
if sqlStatement == "" {
continue
}
total++
wg.Add(1)
sem <- struct{}{} // 占用一个 goroutine
go func(query string) {
defer wg.Done()
defer func() { <-sem }() // 释放 goroutine
//start := time.Now()
_, err := db.Exec(query)
if err != nil {
log.Printf("Error executing SQL query: %s, Error: %v", query, err)
failureCount++
} else {
successCount++
}
// 记录每条 SQL 执行的耗时
//log.Printf("Executed SQL query: %s, Duration: %s", query, time.Since(start))
}(sqlStatement)
}
// 等待所有的 goroutine 完成
wg.Wait()
// 打印并记录执行结果
log.Printf("Execution completed. Total: %d %d, Failure: %d, Total Duration: %s",
total, successCount, failureCount, time.Since(startTime))
fmt.Printf("Execution completed. Total: %d, Success: %d, Failure: %d\n", total, successCount, failureCount)
}
```
在持续进行查询时,BE内存不断上涨,导致使用超限内存的SQL无法再执行。但是在停止SQL执行后。保持在26.5G


停止查询SQL后的BE的info日志持续打印

be.warning日志

be mem_tracker


mem_tracker?type=global



### What You Expected?
持续的查询不会导致内存持续的上升,停止查询后应该释放部分内存。
### How to Reproduce?
_No response_
### Anything Else?
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Bug] 2.1.6版本连续的SQL查询导致BE内存持续上涨,停止查询后内存不释放 [doris]
Posted by "zy-kkk (via GitHub)" <gi...@apache.org>.
zy-kkk commented on issue #41223:
URL: https://github.com/apache/doris/issues/41223#issuecomment-2374392942
Is there a query for the Jdbc Catalog in the query?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org