You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "liuneng1994 (via GitHub)" <gi...@apache.org> on 2024/07/23 08:12:17 UTC

[PR] [CH] Refactor off heap memory management, clean shuffle write code [incubator-gluten]

liuneng1994 opened a new pull request, #6558:
URL: https://github.com/apache/incubator-gluten/pull/6558

   ## What changes were proposed in this pull request?
   
   将现有的通过spark管理内存并触发spill的机制改为由ClickHouse托管。shuffle spill触发由当前内存使用量和固定阈值决定。
   
   修改原因:
   1. 统计不准确,代码运行过程中如果存在异步线程,容易出现内存使用量统计失真
   2. 无法统一管理各个算子的spill,目前已有sort ,aggregate,join,shuffle支持spill,spark task memory manager无法管理众多算子的spill触发
   3. shuffle通过spark触发spill容易出错,因为内存分配会出现在任何时候,但不是任何时机都可以触发spill,容易导致运行错误。难以根本解决。
   
   ### 删除多余的配置项
   1. spark.gluten.sql.columnar.backend.ch.shuffle.preferSpill
   不再需要选择不同的模式,保留preferSpill=false的模式,对celeborn没有影响
   4. spark.gluten.sql.columnar.backend.ch.throwIfMemoryExceed
   offheap内存管理切换至clickhouse MemoryTracker,不在需要该配置
   5. spark.gluten.sql.columnar.backend.ch.flushBlockBufferBeforeEvict
   保留flushBlockBufferBeforeEvict=true的行为,移除配置项
   6. spark.gluten.sql.columnar.backend.ch.spillFirstlyBeforeStop
   7. spark.gluten.sql.columnar.backend.ch.forceExternalSortShuffle
   移除冗余的external sort shuffle 及配置
   
   ### 新增配置
   1. spark.gluten.sql.columnar.backend.ch.runtime_config.spill_mem_ratio=0.9
   shuffle spill触发控制,默认当前task内存使用量超过90%时触发
   2. spark.gluten.sql.columnar.backend.ch.runtime_config.extra_memory_hard_limit=0
   为task内存增加额外的hard limit限制,单位为byte,当task内存使用超过hardlimit时会产生异常。如果希望task能适度超过offheap配置可增大该配置项
   内存的hard_limit = soft_limit + extra_memory_hard_limit, soft_limit为spark配置的offheap size。
   
   新增performance counter 日志输出,方便对task进行性能分析。
   ![image](https://github.com/user-attachments/assets/1b6d9bab-cd75-4b7a-9bea-2abbfdd997ad)
   
   
   ## How was this patch tested?
   
   unit tests
   
   
   (If this patch involves UI changes, please attach a screenshot; otherwise, remove this)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [CH] Refactor off heap memory management, clean shuffle write code [incubator-gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6558:
URL: https://github.com/apache/incubator-gluten/pull/6558#issuecomment-2244821101

   Run Gluten Clickhouse CI


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [CH] Refactor off heap memory management, clean shuffle write code [incubator-gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6558:
URL: https://github.com/apache/incubator-gluten/pull/6558#issuecomment-2249266510

   Run Gluten Clickhouse CI


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [CH] Refactor off heap memory management, clean shuffle write code [incubator-gluten]

Posted by "lgbo-ustc (via GitHub)" <gi...@apache.org>.
lgbo-ustc commented on code in PR #6558:
URL: https://github.com/apache/incubator-gluten/pull/6558#discussion_r1687688074


##########
cpp-ch/local-engine/Common/QueryContext.cpp:
##########
@@ -33,14 +36,23 @@ namespace ErrorCodes
 namespace local_engine
 {
 using namespace DB;
-thread_local std::shared_ptr<CurrentThread::QueryScope> query_scope;
 thread_local std::shared_ptr<ThreadStatus> thread_status;
-ConcurrentMap<int64_t, NativeAllocatorContextPtr> allocator_map;
+thread_local std::shared_ptr<ThreadGroup> thread_group;
+thread_local ContextMutablePtr query_context;
 
-int64_t initializeQuery(ReservationListenerWrapperPtr listener)
+struct QueryContextSettings

Review Comment:
   `QueryContextSettings`的目的是什么? 这里只有一个字段,容易让人迷惑



##########
cpp-ch/local-engine/Common/QueryContext.h:
##########
@@ -15,30 +15,29 @@
  * limitations under the License.
  */
 #pragma once
-#include <unordered_map>
 #include <Interpreters/Context_fwd.h>
-#include <jni/ReservationListenerWrapper.h>
-#include <Common/CurrentThread.h>
 #include <Common/ThreadStatus.h>
 
 namespace local_engine
 {
-int64_t initializeQuery(ReservationListenerWrapperPtr listener);
-
-void releaseAllocator(int64_t allocator_id);
-
-int64_t allocatorMemoryUsage(int64_t allocator_id);
-
-struct NativeAllocatorContext
+class QueryContextManager
 {
-    std::shared_ptr<DB::CurrentThread::QueryScope> query_scope;
-    std::shared_ptr<DB::ThreadStatus> thread_status;
-    DB::ContextMutablePtr query_context;
-    std::shared_ptr<DB::ThreadGroup> group;
-    ReservationListenerWrapperPtr listener;
-};
+public:
+    static QueryContextManager & instance()
+    {
+        static QueryContextManager instance;
+        return instance;
+    }
+    DB::ContextMutablePtr initializeQuery();
+    DB::ContextMutablePtr currentQueryContext();
+    void logCurrentPerformanceCounters();
+    size_t currentPeakMemory();
+    void finalizeQuery();
 
-using NativeAllocatorContextPtr = std::shared_ptr<NativeAllocatorContext>;
+private:

Review Comment:
   make the constructor be a protected method



##########
cpp-ch/local-engine/Common/QueryContext.cpp:
##########
@@ -33,14 +36,23 @@ namespace ErrorCodes
 namespace local_engine
 {
 using namespace DB;
-thread_local std::shared_ptr<CurrentThread::QueryScope> query_scope;
 thread_local std::shared_ptr<ThreadStatus> thread_status;
-ConcurrentMap<int64_t, NativeAllocatorContextPtr> allocator_map;
+thread_local std::shared_ptr<ThreadGroup> thread_group;
+thread_local ContextMutablePtr query_context;
 
-int64_t initializeQuery(ReservationListenerWrapperPtr listener)
+struct QueryContextSettings
 {
-    if (thread_status) return -1;
-    auto query_context = Context::createCopy(SerializedPlanParser::global_context);
+    size_t extra_memory_hard_limit = 0;
+
+    void loadFromContext(ContextPtr context)

Review Comment:
   实现为静态函数是不是更好
   ```c++
   static QueryContextSettingsPtr loadFromContext(ContextPtr context)
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [CH] Refactor off heap memory management, clean shuffle write code [incubator-gluten]

Posted by "liuneng1994 (via GitHub)" <gi...@apache.org>.
liuneng1994 commented on code in PR #6558:
URL: https://github.com/apache/incubator-gluten/pull/6558#discussion_r1687806566


##########
backends-clickhouse/src/main/java/org/apache/gluten/vectorized/BatchIterator.java:
##########
@@ -77,8 +80,17 @@ public IMetrics getMetricsInternal() {
     return new NativeMetrics(nativeFetchMetrics(handle));
   }
 
+  public void updatePeakMemory() {
+    if (TaskContext.get() != null) {
+      TaskContext.get().taskMetrics().setPeakExecutionMemory(nativeCurrentPeakMemory());
+    }
+  }
+
   @Override
   public void closeInternal() {
+    // CH backend uses Clickhouse to manage off heap and updates memory statistics at the end of the

Review Comment:
   没什么影响,之前spark的管理方式也是在task结束的时候更新,ui也只能在结束后查看到



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [CH] Refactor off heap memory management, clean shuffle write code [incubator-gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6558:
URL: https://github.com/apache/incubator-gluten/pull/6558#issuecomment-2247301401

   Run Gluten Clickhouse CI


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [CH] Refactor off heap memory management, clean shuffle write code [incubator-gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6558:
URL: https://github.com/apache/incubator-gluten/pull/6558#issuecomment-2252066055

   Run Gluten Clickhouse CI


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [CH] Refactor off heap memory management, clean shuffle write code [incubator-gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6558:
URL: https://github.com/apache/incubator-gluten/pull/6558#issuecomment-2252313157

   Run Gluten Clickhouse CI


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [CH] Refactor off heap memory management, clean shuffle write code [incubator-gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6558:
URL: https://github.com/apache/incubator-gluten/pull/6558#issuecomment-2244645889

   Run Gluten Clickhouse CI


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [CH] Refactor off heap memory management, clean shuffle write code [incubator-gluten]

Posted by "liuneng1994 (via GitHub)" <gi...@apache.org>.
liuneng1994 commented on code in PR #6558:
URL: https://github.com/apache/incubator-gluten/pull/6558#discussion_r1687806566


##########
backends-clickhouse/src/main/java/org/apache/gluten/vectorized/BatchIterator.java:
##########
@@ -77,8 +80,17 @@ public IMetrics getMetricsInternal() {
     return new NativeMetrics(nativeFetchMetrics(handle));
   }
 
+  public void updatePeakMemory() {
+    if (TaskContext.get() != null) {
+      TaskContext.get().taskMetrics().setPeakExecutionMemory(nativeCurrentPeakMemory());
+    }
+  }
+
   @Override
   public void closeInternal() {
+    // CH backend uses Clickhouse to manage off heap and updates memory statistics at the end of the

Review Comment:
   没什么影响,之前spark的管理方式也是在task结束的时候更新,ui也只能在结束后查看到,唯一的区别是 spark的 offheap execution memory, storage memory在统计上会是零,不过对native来说也不会区分这两种类型的内存。这个两个值没啥作用



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [CH] Refactor off heap memory management, clean shuffle write code [incubator-gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6558:
URL: https://github.com/apache/incubator-gluten/pull/6558#issuecomment-2246725229

   Run Gluten Clickhouse CI


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [CH] Refactor off heap memory management, clean shuffle write code [incubator-gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6558:
URL: https://github.com/apache/incubator-gluten/pull/6558#issuecomment-2244546065

   <!--
     Licensed to the Apache Software Foundation (ASF) under one
     or more contributor license agreements.  See the NOTICE file
     distributed with this work for additional information
     regarding copyright ownership.  The ASF licenses this file
     to you under the Apache License, Version 2.0 (the
     "License"); you may not use this file except in compliance
     with the License.  You may obtain a copy of the License at
   
       http://www.apache.org/licenses/LICEGJ-2.0
   
     Unless required by applicable law or agreed to in writing,
     software distributed under the License is distributed on an
     "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
     KIND, either express or implied.  See the License for the
     specific language governing permissions and limitations
     under the License.
   -->
   
   Thanks for opening a pull request!
   
   Could you open an issue for this pull request on Github Issues?
   
   https://github.com/apache/incubator-gluten/issues
   
   Then could you also rename ***commit message*** and ***pull request title*** in the following format?
   
       [GLUTEN-${ISSUES_ID}][COMPONENT]feat/fix: ${detailed message}
   
   See also:
   
     * [Other pull requests](https://github.com/apache/incubator-gluten/pulls/)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [CH] Refactor off heap memory management, clean shuffle write code [incubator-gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6558:
URL: https://github.com/apache/incubator-gluten/pull/6558#issuecomment-2247400815

   Run Gluten Clickhouse CI


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [CH] Refactor off heap memory management, clean shuffle write code [incubator-gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6558:
URL: https://github.com/apache/incubator-gluten/pull/6558#issuecomment-2252198065

   Run Gluten Clickhouse CI


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [CH] Refactor off heap memory management, clean shuffle write code [incubator-gluten]

Posted by "liuneng1994 (via GitHub)" <gi...@apache.org>.
liuneng1994 commented on code in PR #6558:
URL: https://github.com/apache/incubator-gluten/pull/6558#discussion_r1696433460


##########
cpp-ch/local-engine/Common/QueryContext.cpp:
##########
@@ -15,90 +15,163 @@
  * limitations under the License.
  */
 #include "QueryContext.h"
+
+#include <format>
+
 #include <Interpreters/Context.h>
 #include <Parser/SerializedPlanParser.h>
-#include <Common/ConcurrentMap.h>
 #include <Common/CurrentThread.h>
 #include <Common/ThreadStatus.h>
+#include <Common/CHUtil.h>
+#include <Common/GlutenConfig.h>
+#include <base/unit.h>
+#include <sstream>
+#include <iomanip>
 
 
 namespace DB
 {
 namespace ErrorCodes
 {
-    extern const int LOGICAL_ERROR;
+extern const int LOGICAL_ERROR;
 }
 }
 
 namespace local_engine
 {
 using namespace DB;
-thread_local std::shared_ptr<CurrentThread::QueryScope> query_scope;
-thread_local std::shared_ptr<ThreadStatus> thread_status;
-ConcurrentMap<int64_t, NativeAllocatorContextPtr> allocator_map;
 
-int64_t initializeQuery(ReservationListenerWrapperPtr listener)
+struct QueryContext
+{
+    std::shared_ptr<ThreadStatus> thread_status;
+    std::shared_ptr<ThreadGroup> thread_group;
+    ContextMutablePtr query_context;
+};
+
+std::unordered_map<int64_t, std::shared_ptr<QueryContext>> query_map;
+std::mutex query_map_mutex;

Review Comment:
   应该需要,这里会有多个task并发初始化和释放的情况



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [CH] Refactor off heap memory management, clean shuffle write code [incubator-gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6558:
URL: https://github.com/apache/incubator-gluten/pull/6558#issuecomment-2244580681

   Run Gluten Clickhouse CI


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [CH] Refactor off heap memory management, clean shuffle write code [incubator-gluten]

Posted by "liuneng1994 (via GitHub)" <gi...@apache.org>.
liuneng1994 commented on code in PR #6558:
URL: https://github.com/apache/incubator-gluten/pull/6558#discussion_r1687811809


##########
cpp-ch/local-engine/Shuffle/ShuffleCommon.h:
##########
@@ -45,16 +45,10 @@ struct SplitOptions
     std::string out_exprs;
     std::string compress_method = "zstd";
     int compress_level;
-    size_t spill_threshold = 300 * 1024 * 1024;
+    size_t spill_threshold = 500 * 1024 * 1024;

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [CH] Refactor off heap memory management, clean shuffle write code [incubator-gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6558:
URL: https://github.com/apache/incubator-gluten/pull/6558#issuecomment-2244546580

   Run Gluten Clickhouse CI


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [CH] Refactor off heap memory management, clean shuffle write code [incubator-gluten]

Posted by "liuneng1994 (via GitHub)" <gi...@apache.org>.
liuneng1994 merged PR #6558:
URL: https://github.com/apache/incubator-gluten/pull/6558


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [CH] Refactor off heap memory management, clean shuffle write code [incubator-gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6558:
URL: https://github.com/apache/incubator-gluten/pull/6558#issuecomment-2252271209

   Run Gluten Clickhouse CI


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [CH] Refactor off heap memory management, clean shuffle write code [incubator-gluten]

Posted by "lgbo-ustc (via GitHub)" <gi...@apache.org>.
lgbo-ustc commented on code in PR #6558:
URL: https://github.com/apache/incubator-gluten/pull/6558#discussion_r1687698636


##########
backends-clickhouse/src/main/java/org/apache/gluten/vectorized/BatchIterator.java:
##########
@@ -77,8 +80,17 @@ public IMetrics getMetricsInternal() {
     return new NativeMetrics(nativeFetchMetrics(handle));
   }
 
+  public void updatePeakMemory() {
+    if (TaskContext.get() != null) {
+      TaskContext.get().taskMetrics().setPeakExecutionMemory(nativeCurrentPeakMemory());
+    }
+  }
+
   @Override
   public void closeInternal() {
+    // CH backend uses Clickhouse to manage off heap and updates memory statistics at the end of the

Review Comment:
   只在结束时更新这个信息会有什么影响吗?



##########
cpp-ch/local-engine/Shuffle/ShuffleCommon.h:
##########
@@ -45,16 +45,10 @@ struct SplitOptions
     std::string out_exprs;
     std::string compress_method = "zstd";
     int compress_level;
-    size_t spill_threshold = 300 * 1024 * 1024;
+    size_t spill_threshold = 500 * 1024 * 1024;

Review Comment:
   调大默认值可能有风险。shuffle占据更大的内存空间,会对上游的聚合/join/sort造成内存压力,引发更多的磁盘spill



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [CH] Refactor off heap memory management, clean shuffle write code [incubator-gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6558:
URL: https://github.com/apache/incubator-gluten/pull/6558#issuecomment-2244864796

   Run Gluten Clickhouse CI


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [CH] Refactor off heap memory management, clean shuffle write code [incubator-gluten]

Posted by "lgbo-ustc (via GitHub)" <gi...@apache.org>.
lgbo-ustc commented on code in PR #6558:
URL: https://github.com/apache/incubator-gluten/pull/6558#discussion_r1696412373


##########
cpp-ch/local-engine/Common/QueryContext.cpp:
##########
@@ -15,90 +15,163 @@
  * limitations under the License.
  */
 #include "QueryContext.h"
+
+#include <format>
+
 #include <Interpreters/Context.h>
 #include <Parser/SerializedPlanParser.h>
-#include <Common/ConcurrentMap.h>
 #include <Common/CurrentThread.h>
 #include <Common/ThreadStatus.h>
+#include <Common/CHUtil.h>
+#include <Common/GlutenConfig.h>
+#include <base/unit.h>
+#include <sstream>
+#include <iomanip>
 
 
 namespace DB
 {
 namespace ErrorCodes
 {
-    extern const int LOGICAL_ERROR;
+extern const int LOGICAL_ERROR;
 }
 }
 
 namespace local_engine
 {
 using namespace DB;
-thread_local std::shared_ptr<CurrentThread::QueryScope> query_scope;
-thread_local std::shared_ptr<ThreadStatus> thread_status;
-ConcurrentMap<int64_t, NativeAllocatorContextPtr> allocator_map;
 
-int64_t initializeQuery(ReservationListenerWrapperPtr listener)
+struct QueryContext
+{
+    std::shared_ptr<ThreadStatus> thread_status;
+    std::shared_ptr<ThreadGroup> thread_group;
+    ContextMutablePtr query_context;
+};
+
+std::unordered_map<int64_t, std::shared_ptr<QueryContext>> query_map;
+std::mutex query_map_mutex;

Review Comment:
   这个需要锁?



##########
cpp-ch/local-engine/Common/QueryContext.cpp:
##########
@@ -15,90 +15,163 @@
  * limitations under the License.
  */
 #include "QueryContext.h"
+
+#include <format>
+
 #include <Interpreters/Context.h>
 #include <Parser/SerializedPlanParser.h>
-#include <Common/ConcurrentMap.h>
 #include <Common/CurrentThread.h>
 #include <Common/ThreadStatus.h>
+#include <Common/CHUtil.h>
+#include <Common/GlutenConfig.h>
+#include <base/unit.h>
+#include <sstream>
+#include <iomanip>
 
 
 namespace DB
 {
 namespace ErrorCodes
 {
-    extern const int LOGICAL_ERROR;
+extern const int LOGICAL_ERROR;
 }
 }
 
 namespace local_engine
 {
 using namespace DB;
-thread_local std::shared_ptr<CurrentThread::QueryScope> query_scope;
-thread_local std::shared_ptr<ThreadStatus> thread_status;
-ConcurrentMap<int64_t, NativeAllocatorContextPtr> allocator_map;
 
-int64_t initializeQuery(ReservationListenerWrapperPtr listener)
+struct QueryContext
+{
+    std::shared_ptr<ThreadStatus> thread_status;
+    std::shared_ptr<ThreadGroup> thread_group;
+    ContextMutablePtr query_context;
+};
+
+std::unordered_map<int64_t, std::shared_ptr<QueryContext>> query_map;
+std::mutex query_map_mutex;
+
+int64_t QueryContextManager::initializeQuery()
 {
-    if (thread_status) return -1;
-    auto query_context = Context::createCopy(SerializedPlanParser::global_context);
-    query_context->makeQueryContext();
+    std::shared_ptr<QueryContext> query_context = std::make_shared<QueryContext>();
+    query_context->query_context = Context::createCopy(SerializedPlanParser::global_context);
+    query_context->query_context->makeQueryContext();
 
     // empty input will trigger random query id to be set
     // FileCache will check if query id is set to decide whether to skip cache or not
     // check FileCache.isQueryInitialized()
     //
     // Notice:
     // this generated random query id a qualified global queryid for the spark query
-    query_context->setCurrentQueryId("");
-
-    auto allocator_context = std::make_shared<NativeAllocatorContext>();
-    allocator_context->thread_status = std::make_shared<ThreadStatus>(true);
-    allocator_context->query_scope = std::make_shared<CurrentThread::QueryScope>(query_context);
-    allocator_context->group = std::make_shared<ThreadGroup>(query_context);
-    allocator_context->query_context = query_context;
-    allocator_context->listener = listener;
-    thread_status = allocator_context->thread_status;
-    query_scope = allocator_context->query_scope;
-    auto allocator_id = reinterpret_cast<int64_t>(allocator_context.get());
-    CurrentMemoryTracker::before_alloc = [listener](Int64 size, bool throw_if_memory_exceed) -> void
+    query_context->query_context->setCurrentQueryId(toString(UUIDHelpers::generateV4()));
+    auto config = MemoryConfig::loadFromContext(query_context->query_context);
+    query_context->thread_status = std::make_shared<ThreadStatus>(false);
+    query_context->thread_group = std::make_shared<ThreadGroup>(query_context->query_context);
+    CurrentThread::attachToGroup(query_context->thread_group);
+    auto memory_limit = config.off_heap_per_task;
+
+    query_context->thread_group->memory_tracker.setSoftLimit(memory_limit);
+    query_context->thread_group->memory_tracker.setHardLimit(memory_limit + config.extra_memory_hard_limit);
+    std::lock_guard<std::mutex> lock_guard(query_map_mutex);
+    int64_t id = reinterpret_cast<int64_t>(query_context->thread_group.get());
+    query_map.emplace(id, query_context);
+    return id;
+}
+
+DB::ContextMutablePtr QueryContextManager::currentQueryContext()
+{
+    if (!CurrentThread::getGroup())
     {
-        if (throw_if_memory_exceed)
-            listener->reserveOrThrow(size);
-        else
-            listener->reserve(size);
-    };
-    CurrentMemoryTracker::before_free = [listener](Int64 size) -> void { listener->tryFree(size); };
-    CurrentMemoryTracker::current_memory = [listener]() -> Int64 { return listener->currentMemory(); };
-    allocator_map.insert(allocator_id, allocator_context);
-    return allocator_id;
+        throw DB::Exception(ErrorCodes::LOGICAL_ERROR, "Thread group not found.");
+    }
+    std::lock_guard lock_guard(query_map_mutex);
+    int64_t id = reinterpret_cast<int64_t>(CurrentThread::getGroup().get());
+    return query_map[id]->query_context;
 }
 
-void releaseAllocator(int64_t allocator_id)
+void QueryContextManager::logCurrentPerformanceCounters(ProfileEvents::Counters & counters)
 {
-    if (!allocator_map.get(allocator_id))
+    if (!CurrentThread::getGroup())
+    {
+        return;
+    }
+    if (logger->information())
     {
-        throw DB::Exception(ErrorCodes::LOGICAL_ERROR, "allocator {} not found", allocator_id);
+        std::ostringstream msg;

Review Comment:
   ostringstream is not recommended in CH



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [CH] Refactor off heap memory management, clean shuffle write code [incubator-gluten]

Posted by "liuneng1994 (via GitHub)" <gi...@apache.org>.
liuneng1994 commented on code in PR #6558:
URL: https://github.com/apache/incubator-gluten/pull/6558#discussion_r1696437935


##########
cpp-ch/local-engine/Common/QueryContext.cpp:
##########
@@ -15,90 +15,163 @@
  * limitations under the License.
  */
 #include "QueryContext.h"
+
+#include <format>
+
 #include <Interpreters/Context.h>
 #include <Parser/SerializedPlanParser.h>
-#include <Common/ConcurrentMap.h>
 #include <Common/CurrentThread.h>
 #include <Common/ThreadStatus.h>
+#include <Common/CHUtil.h>
+#include <Common/GlutenConfig.h>
+#include <base/unit.h>
+#include <sstream>
+#include <iomanip>
 
 
 namespace DB
 {
 namespace ErrorCodes
 {
-    extern const int LOGICAL_ERROR;
+extern const int LOGICAL_ERROR;
 }
 }
 
 namespace local_engine
 {
 using namespace DB;
-thread_local std::shared_ptr<CurrentThread::QueryScope> query_scope;
-thread_local std::shared_ptr<ThreadStatus> thread_status;
-ConcurrentMap<int64_t, NativeAllocatorContextPtr> allocator_map;
 
-int64_t initializeQuery(ReservationListenerWrapperPtr listener)
+struct QueryContext
+{
+    std::shared_ptr<ThreadStatus> thread_status;
+    std::shared_ptr<ThreadGroup> thread_group;
+    ContextMutablePtr query_context;
+};
+
+std::unordered_map<int64_t, std::shared_ptr<QueryContext>> query_map;
+std::mutex query_map_mutex;
+
+int64_t QueryContextManager::initializeQuery()
 {
-    if (thread_status) return -1;
-    auto query_context = Context::createCopy(SerializedPlanParser::global_context);
-    query_context->makeQueryContext();
+    std::shared_ptr<QueryContext> query_context = std::make_shared<QueryContext>();
+    query_context->query_context = Context::createCopy(SerializedPlanParser::global_context);
+    query_context->query_context->makeQueryContext();
 
     // empty input will trigger random query id to be set
     // FileCache will check if query id is set to decide whether to skip cache or not
     // check FileCache.isQueryInitialized()
     //
     // Notice:
     // this generated random query id a qualified global queryid for the spark query
-    query_context->setCurrentQueryId("");
-
-    auto allocator_context = std::make_shared<NativeAllocatorContext>();
-    allocator_context->thread_status = std::make_shared<ThreadStatus>(true);
-    allocator_context->query_scope = std::make_shared<CurrentThread::QueryScope>(query_context);
-    allocator_context->group = std::make_shared<ThreadGroup>(query_context);
-    allocator_context->query_context = query_context;
-    allocator_context->listener = listener;
-    thread_status = allocator_context->thread_status;
-    query_scope = allocator_context->query_scope;
-    auto allocator_id = reinterpret_cast<int64_t>(allocator_context.get());
-    CurrentMemoryTracker::before_alloc = [listener](Int64 size, bool throw_if_memory_exceed) -> void
+    query_context->query_context->setCurrentQueryId(toString(UUIDHelpers::generateV4()));
+    auto config = MemoryConfig::loadFromContext(query_context->query_context);
+    query_context->thread_status = std::make_shared<ThreadStatus>(false);
+    query_context->thread_group = std::make_shared<ThreadGroup>(query_context->query_context);
+    CurrentThread::attachToGroup(query_context->thread_group);
+    auto memory_limit = config.off_heap_per_task;
+
+    query_context->thread_group->memory_tracker.setSoftLimit(memory_limit);
+    query_context->thread_group->memory_tracker.setHardLimit(memory_limit + config.extra_memory_hard_limit);
+    std::lock_guard<std::mutex> lock_guard(query_map_mutex);
+    int64_t id = reinterpret_cast<int64_t>(query_context->thread_group.get());
+    query_map.emplace(id, query_context);
+    return id;
+}
+
+DB::ContextMutablePtr QueryContextManager::currentQueryContext()
+{
+    if (!CurrentThread::getGroup())
     {
-        if (throw_if_memory_exceed)
-            listener->reserveOrThrow(size);
-        else
-            listener->reserve(size);
-    };
-    CurrentMemoryTracker::before_free = [listener](Int64 size) -> void { listener->tryFree(size); };
-    CurrentMemoryTracker::current_memory = [listener]() -> Int64 { return listener->currentMemory(); };
-    allocator_map.insert(allocator_id, allocator_context);
-    return allocator_id;
+        throw DB::Exception(ErrorCodes::LOGICAL_ERROR, "Thread group not found.");
+    }
+    std::lock_guard lock_guard(query_map_mutex);
+    int64_t id = reinterpret_cast<int64_t>(CurrentThread::getGroup().get());
+    return query_map[id]->query_context;
 }
 
-void releaseAllocator(int64_t allocator_id)
+void QueryContextManager::logCurrentPerformanceCounters(ProfileEvents::Counters & counters)
 {
-    if (!allocator_map.get(allocator_id))
+    if (!CurrentThread::getGroup())
+    {
+        return;
+    }
+    if (logger->information())
     {
-        throw DB::Exception(ErrorCodes::LOGICAL_ERROR, "allocator {} not found", allocator_id);
+        std::ostringstream msg;

Review Comment:
   这里主要是需要利用到setw 和 setfill来实现一个输出内容的对齐,提高可读性。没有找到其他更好的方式



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [CH] Refactor off heap memory management, clean shuffle write code [incubator-gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6558:
URL: https://github.com/apache/incubator-gluten/pull/6558#issuecomment-2249287847

   Run Gluten Clickhouse CI


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [CH] Refactor off heap memory management, clean shuffle write code [incubator-gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6558:
URL: https://github.com/apache/incubator-gluten/pull/6558#issuecomment-2255577685

   Run Gluten Clickhouse CI


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [CH] Refactor off heap memory management, clean shuffle write code [incubator-gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6558:
URL: https://github.com/apache/incubator-gluten/pull/6558#issuecomment-2257331379

   Run Gluten Clickhouse CI


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [CH] Refactor off heap memory management, clean shuffle write code [incubator-gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6558:
URL: https://github.com/apache/incubator-gluten/pull/6558#issuecomment-2255830791

   Run Gluten Clickhouse CI


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [CH] Refactor off heap memory management, clean shuffle write code [incubator-gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6558:
URL: https://github.com/apache/incubator-gluten/pull/6558#issuecomment-2252445727

   Run Gluten Clickhouse CI


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [CH] Refactor off heap memory management, clean shuffle write code [incubator-gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6558:
URL: https://github.com/apache/incubator-gluten/pull/6558#issuecomment-2247465657

   Run Gluten Clickhouse CI


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [CH] Refactor off heap memory management, clean shuffle write code [incubator-gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6558:
URL: https://github.com/apache/incubator-gluten/pull/6558#issuecomment-2246765823

   Run Gluten Clickhouse CI


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [CH] Refactor off heap memory management, clean shuffle write code [incubator-gluten]

Posted by "liuneng1994 (via GitHub)" <gi...@apache.org>.
liuneng1994 commented on code in PR #6558:
URL: https://github.com/apache/incubator-gluten/pull/6558#discussion_r1687702440


##########
cpp-ch/local-engine/Common/QueryContext.cpp:
##########
@@ -33,14 +36,23 @@ namespace ErrorCodes
 namespace local_engine
 {
 using namespace DB;
-thread_local std::shared_ptr<CurrentThread::QueryScope> query_scope;
 thread_local std::shared_ptr<ThreadStatus> thread_status;
-ConcurrentMap<int64_t, NativeAllocatorContextPtr> allocator_map;
+thread_local std::shared_ptr<ThreadGroup> thread_group;
+thread_local ContextMutablePtr query_context;
 
-int64_t initializeQuery(ReservationListenerWrapperPtr listener)
+struct QueryContextSettings

Review Comment:
   我再修改一下,统一一下Config的获取入口



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [CH] Refactor off heap memory management, clean shuffle write code [incubator-gluten]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #6558:
URL: https://github.com/apache/incubator-gluten/pull/6558#issuecomment-2249637370

   Run Gluten Clickhouse CI


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [CH] Refactor off heap memory management, clean shuffle write code [incubator-gluten]

Posted by "liuneng1994 (via GitHub)" <gi...@apache.org>.
liuneng1994 commented on code in PR #6558:
URL: https://github.com/apache/incubator-gluten/pull/6558#discussion_r1696433460


##########
cpp-ch/local-engine/Common/QueryContext.cpp:
##########
@@ -15,90 +15,163 @@
  * limitations under the License.
  */
 #include "QueryContext.h"
+
+#include <format>
+
 #include <Interpreters/Context.h>
 #include <Parser/SerializedPlanParser.h>
-#include <Common/ConcurrentMap.h>
 #include <Common/CurrentThread.h>
 #include <Common/ThreadStatus.h>
+#include <Common/CHUtil.h>
+#include <Common/GlutenConfig.h>
+#include <base/unit.h>
+#include <sstream>
+#include <iomanip>
 
 
 namespace DB
 {
 namespace ErrorCodes
 {
-    extern const int LOGICAL_ERROR;
+extern const int LOGICAL_ERROR;
 }
 }
 
 namespace local_engine
 {
 using namespace DB;
-thread_local std::shared_ptr<CurrentThread::QueryScope> query_scope;
-thread_local std::shared_ptr<ThreadStatus> thread_status;
-ConcurrentMap<int64_t, NativeAllocatorContextPtr> allocator_map;
 
-int64_t initializeQuery(ReservationListenerWrapperPtr listener)
+struct QueryContext
+{
+    std::shared_ptr<ThreadStatus> thread_status;
+    std::shared_ptr<ThreadGroup> thread_group;
+    ContextMutablePtr query_context;
+};
+
+std::unordered_map<int64_t, std::shared_ptr<QueryContext>> query_map;
+std::mutex query_map_mutex;

Review Comment:
   应该需要,这里会有多个task并发初始化和释放的情况,在一些频繁调用的接口,例如currentMemoryUsage上是不需要锁的,总的来说应该不会造成影响。



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [PR] [CH] Refactor off heap memory management, clean shuffle write code [incubator-gluten]

Posted by "liuneng1994 (via GitHub)" <gi...@apache.org>.
liuneng1994 commented on code in PR #6558:
URL: https://github.com/apache/incubator-gluten/pull/6558#discussion_r1696437935


##########
cpp-ch/local-engine/Common/QueryContext.cpp:
##########
@@ -15,90 +15,163 @@
  * limitations under the License.
  */
 #include "QueryContext.h"
+
+#include <format>
+
 #include <Interpreters/Context.h>
 #include <Parser/SerializedPlanParser.h>
-#include <Common/ConcurrentMap.h>
 #include <Common/CurrentThread.h>
 #include <Common/ThreadStatus.h>
+#include <Common/CHUtil.h>
+#include <Common/GlutenConfig.h>
+#include <base/unit.h>
+#include <sstream>
+#include <iomanip>
 
 
 namespace DB
 {
 namespace ErrorCodes
 {
-    extern const int LOGICAL_ERROR;
+extern const int LOGICAL_ERROR;
 }
 }
 
 namespace local_engine
 {
 using namespace DB;
-thread_local std::shared_ptr<CurrentThread::QueryScope> query_scope;
-thread_local std::shared_ptr<ThreadStatus> thread_status;
-ConcurrentMap<int64_t, NativeAllocatorContextPtr> allocator_map;
 
-int64_t initializeQuery(ReservationListenerWrapperPtr listener)
+struct QueryContext
+{
+    std::shared_ptr<ThreadStatus> thread_status;
+    std::shared_ptr<ThreadGroup> thread_group;
+    ContextMutablePtr query_context;
+};
+
+std::unordered_map<int64_t, std::shared_ptr<QueryContext>> query_map;
+std::mutex query_map_mutex;
+
+int64_t QueryContextManager::initializeQuery()
 {
-    if (thread_status) return -1;
-    auto query_context = Context::createCopy(SerializedPlanParser::global_context);
-    query_context->makeQueryContext();
+    std::shared_ptr<QueryContext> query_context = std::make_shared<QueryContext>();
+    query_context->query_context = Context::createCopy(SerializedPlanParser::global_context);
+    query_context->query_context->makeQueryContext();
 
     // empty input will trigger random query id to be set
     // FileCache will check if query id is set to decide whether to skip cache or not
     // check FileCache.isQueryInitialized()
     //
     // Notice:
     // this generated random query id a qualified global queryid for the spark query
-    query_context->setCurrentQueryId("");
-
-    auto allocator_context = std::make_shared<NativeAllocatorContext>();
-    allocator_context->thread_status = std::make_shared<ThreadStatus>(true);
-    allocator_context->query_scope = std::make_shared<CurrentThread::QueryScope>(query_context);
-    allocator_context->group = std::make_shared<ThreadGroup>(query_context);
-    allocator_context->query_context = query_context;
-    allocator_context->listener = listener;
-    thread_status = allocator_context->thread_status;
-    query_scope = allocator_context->query_scope;
-    auto allocator_id = reinterpret_cast<int64_t>(allocator_context.get());
-    CurrentMemoryTracker::before_alloc = [listener](Int64 size, bool throw_if_memory_exceed) -> void
+    query_context->query_context->setCurrentQueryId(toString(UUIDHelpers::generateV4()));
+    auto config = MemoryConfig::loadFromContext(query_context->query_context);
+    query_context->thread_status = std::make_shared<ThreadStatus>(false);
+    query_context->thread_group = std::make_shared<ThreadGroup>(query_context->query_context);
+    CurrentThread::attachToGroup(query_context->thread_group);
+    auto memory_limit = config.off_heap_per_task;
+
+    query_context->thread_group->memory_tracker.setSoftLimit(memory_limit);
+    query_context->thread_group->memory_tracker.setHardLimit(memory_limit + config.extra_memory_hard_limit);
+    std::lock_guard<std::mutex> lock_guard(query_map_mutex);
+    int64_t id = reinterpret_cast<int64_t>(query_context->thread_group.get());
+    query_map.emplace(id, query_context);
+    return id;
+}
+
+DB::ContextMutablePtr QueryContextManager::currentQueryContext()
+{
+    if (!CurrentThread::getGroup())
     {
-        if (throw_if_memory_exceed)
-            listener->reserveOrThrow(size);
-        else
-            listener->reserve(size);
-    };
-    CurrentMemoryTracker::before_free = [listener](Int64 size) -> void { listener->tryFree(size); };
-    CurrentMemoryTracker::current_memory = [listener]() -> Int64 { return listener->currentMemory(); };
-    allocator_map.insert(allocator_id, allocator_context);
-    return allocator_id;
+        throw DB::Exception(ErrorCodes::LOGICAL_ERROR, "Thread group not found.");
+    }
+    std::lock_guard lock_guard(query_map_mutex);
+    int64_t id = reinterpret_cast<int64_t>(CurrentThread::getGroup().get());
+    return query_map[id]->query_context;
 }
 
-void releaseAllocator(int64_t allocator_id)
+void QueryContextManager::logCurrentPerformanceCounters(ProfileEvents::Counters & counters)
 {
-    if (!allocator_map.get(allocator_id))
+    if (!CurrentThread::getGroup())
+    {
+        return;
+    }
+    if (logger->information())
     {
-        throw DB::Exception(ErrorCodes::LOGICAL_ERROR, "allocator {} not found", allocator_id);
+        std::ostringstream msg;

Review Comment:
   这里主要是需要利用到setw 和 setfill来实现一个输出内容的对齐,提高可读性。



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org