[jira] [Comment Edited] (ARTEMIS-839) Support multiple backend data stores

["Michael Andre Pearce (IG) (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ARTEMIS-839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15952448#comment-15952448 ] 

Michael Andre Pearce (IG) edited comment on ARTEMIS-839 at 4/1/17 11:22 PM:
----------------------------------------------------------------------------

[~jbertram] we're running durable persistent messages in sync send over many clients, ramping up the clients as it progresses, what we note is essentially the disk is bottle neck, surprise surprise. Because of the disks we have the raid stack overhead actually degrades the performance we're getting.
This is exact reason why other tech like the one i noted have started to look to support JBOD disks for their persistent areas and itself stripe the data it needs to write.

Also aside from performance improvements, this would give a functional benefit of flow isolation, that you can segregate flows so that very latency sensitive flows could goto one disk and less sensitive can goto another (or other isolation reasons), so that at the disk level they can't impact one another.

[~clebertsuconic] you note you had some initial work in this area, is this something you could share?


was (Author: michael.andre.pearce):
[~jbertram] we're running durable persistent messages in sync send over many clients, ramping up the clients as it progresses, what we note is essentially the disk is bottle neck, surprise surprise. Because of the disks we have the raid stack overhead actually degrades the performance we're getting.
This is exact reason why other tech like the one i noted have started to look to support JBOD disks for their persistent areas and itself stripe the data it needs to write.

[~clebertsuconic] you note you had some initial work in this area, is this something you could share?

> Support multiple backend data stores
> ------------------------------------
>
>                 Key: ARTEMIS-839
>                 URL: https://issues.apache.org/jira/browse/ARTEMIS-839
>             Project: ActiveMQ Artemis
>          Issue Type: New Feature
>          Components: Broker
>            Reporter: Matt Pavlovich
>
> ActiveMQ 5.x supports multi-kahahdb where destinations matching a certain naming convention are stored in separate data stores.
> Artemis should support this to achieve parity with ActiveMQ and other commercial brokers, such as Tibco EMS that support multiple back-end data stores.
> The 2 primary use cases:
> 1. Increase overall disk I/O by having multiple mount points to allow busy destinations to have separate disks
> 2. Separate *.*.*.DLQ destinations in order to avoid the journal-cannot-be-cleaned-up issue due to sporadic messages still present
> Note: It was discussed on IRC that #2 may be mitigated with Artemis' compaction thread. However, with very large data stores the performance would be better with a partitioned data store vs one really large one.
> Enhancement: It would be great if the full path could be specified. Currently, in ActiveMQ 5.x the path is automatically generated based on the destination name filter and complicates the ability to separate destinations due to funky character names on all OS's and filesystems.
> For example, support multiple journal entries ad add an attribute:
>    
>   *  "destination-filter" or similar
> Examples:
>     
>   *  < ... destination-filter="Order.>"
>   * < .. destination-filter="Billing.>"
>   * < .. destination-filter=">"
>    
>     



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)