[GitHub] [incubator-doris] caiconghui edited a comment on pull request #4766: Make some debug log settings configurable and change some log level from info to debug to avoid performance bottlenecks

[GitBox <gi...@apache.org>]

caiconghui edited a comment on pull request #4766:
URL: https://github.com/apache/incubator-doris/pull/4766#issuecomment-714980396


   > > Change some log level from info to debug to escape performance bottlenecks
   > 
   > Are you sure the log is the FE performance bottlenecks? did you do any test or benchmark?
   
   the heavy work is still stream load and tablet schedule in fe,  but log in FrontendServiceImpl make thing worse when cpu load is higher, we found that all stream load request failed when qps is higher because all threads(the limit is 4096) in fe blocked in db lock or log lock, and cannot process any request. just like the issue(#4765) mentioned. 
   
   and I test the log performance in local machine with 8 logic core.
   ```
              LOG.info("start to test bench mark for log info");
               int totalThread = 4000;
               TStreamLoadPutRequest request = new TStreamLoadPutRequest("test", "test", "test", "test", null, 1111L, null, null);
               CountDownLatch cdl = new CountDownLatch(totalThread);
               long startTime = System.currentTimeMillis();
               for (int i = 0; i < totalThread; i++) {
                   new Thread(new Runnable() {
                       @Override
                       public void run() {
                           for (int j = 0; j < 5000; j++) {
                               LOG.info("receive stream load put request. db:{}, tbl: {}, txn id: {}, load id: {}, backend: {}",
                                       request.getDb(), request.getTbl(), request.getTxnId(), DebugUtil.printId(request.getLoadId()),
                                       "localhost");
                           }
                           cdl.countDown();
                       }
                   }).start();
               }
               cdl.await();
               long endTime = System.currentTimeMillis();
               LOG.info("finish to test bench mark log info cost {} ms", endTime - startTime);
               System.exit(-1);
   ```
   it cost about 149388ms, every log cost about 0.06ms.
   by the way, our fe log is heavy because of frequent stream load request which may have many rpc request, so we set the log in 
   FrontendServiceImpl from info level to debug level, but still keep it open, so that other user can still see the log, but we can close Close the log switch in FrontendServiceImpl which is useless for use at most of time.
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org