[jira] [Commented] (ARTEMIS-2216) Use a specific executor for pageSyncTimer

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ARTEMIS-2216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16733000#comment-16733000 ] 

ASF GitHub Bot commented on ARTEMIS-2216:
-----------------------------------------

Github user qihongxu commented on the issue:

    https://github.com/apache/activemq-artemis/pull/2484
  
    > @qihongxu,
    > 
    > re:
    > "
    > According to this we have an internal branch that removes isPaging() method’s readlock in PagingStoreImpl, along with adding pageSyncTimer’s specific executor. After that we performed a same test and get a result of 21k/25k Send&Receive–in total 46k TPS. This version runs smoothly in our use case but we are still exploring potential effects.
    > "
    > 
    > Agree that as the QueueImpl:checkDepage which is called from deliver method, only schedules another depage asynchronously that the read barrier lock for that is over kill as it doesn't have to be strongly consistent correct.
    > 
    > I think changing the current method could be dangerous, unless we do a fuller analysis on all its uses, but i think we could easily add a new method named something like "isPagingDirtyRead" which can return without the readlock, it will mean the returned value isn't always 100% accurate but in cases such as checkDepage, we don't need it to be, its just a single, and we can call it where we know its safe to.
    > 
    > Would you be happy adding this, (will need to be exposed via PageSubscription)? And updating checkDepage to use that one?
    
    
    As you suggested we tried update checkDepage with a new "isPagingDirtyRead" method which can return without the readlock. But after running same tests on this version it seems that checkDepage did not affect receive performance too much. Instead, CursorIterator#hasNext in PageSubscriptionImpl also called isPaging() and had a significant impact on receive performance. According to this we updated hasNext() and forced it to use the new “isPagingDirtyRead”. Below is the result chart.
    (P.S the original ver is a control group, not exactly the “original” master-branch build. It has been applied with specific pageSyncTimer executor, and cache durable in PageReference - See PR Artemis-2214.)
    
    
      | Send&Receive | In Total
    -- | -- | --
    Ver Original | 30k/12.5k | 32.5k
    Ver Modified checkDepage | 31.1k/11.3k | 42.4k
    Ver Modified hasNext | 23.8k/23.5k | 47.3k
    
    Detailed TPS chart can be seen in attachment. (Sorry we dont have enough time for running tests. Just got back to work today)
    [TPS chart.docx](https://github.com/apache/activemq-artemis/files/2723731/TPS.chart.docx)



> Use a specific executor for pageSyncTimer
> -----------------------------------------
>
>                 Key: ARTEMIS-2216
>                 URL: https://issues.apache.org/jira/browse/ARTEMIS-2216
>             Project: ActiveMQ Artemis
>          Issue Type: Improvement
>    Affects Versions: 2.6.3
>            Reporter: Qihong Xu
>            Priority: Major
>
> Improving throughput on paging mode is one of our concerns since our cluster uses paging a lot.
> We found that pageSyncTimer in PagingStoreImpl shared the same executor with pageCursorProvider from thread pool. In heavy load scenario like hundreds of consumers receiving messages simultaneously, it became difficult for pageSyncTimer to get the executor due to race condition. Therefore page sync was delayed and producers suffered low throughput.
>  
> To achieve higher performance we assign a specific executor to pageSyncTimer to avoid racing. And we run a small-scale test on a single modified broker.
>  
> Broker: 4C/8G/500G SSD
> Producer: 200 threads, non-transactional send
> Consumer 200 threads, transactional receive
> Message text size: 100-200 bytes randomly
> AddressFullPolicy: PAGE
>  
> Test result：
> | |Only Send TPS|Only Receive TPS|Send&Receive TPS|
> |Original ver|38k|33k|3k/30k|
> |Modified ver|38k|34k|30k/12.5k|
>  
> The chart above shows that on modified broker send TPS improves from “poor” to “extremely fast”, while receive TPS drops from “extremely fast” to “not-bad” under heavy load. Considering consumer systems usually have a long processing chain after receiving messages, we don’t need too fast receive TPS. Instead, we want to guarantee send TPS to cope with traffic peak and lower producer’s delay time. Moreover, send and receive TPS in total raises from 33k to about 43k. From all above this trade-off seems beneficial and acceptable.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)