[jira] [Updated] (ARTEMIS-2746) Reduce memory footprint of journal loading

["Francesco Nigro (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ARTEMIS-2746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Francesco Nigro updated ARTEMIS-2746:
-------------------------------------
    Description: 
While loading the journal, before reloading the messages into each queue, we store the reference to the message and some additional information (scheduledDeliveryTime and deliveryCount) in a LinkedHashMap per-queue.

These information are using a AddMessageRecord class that can weigth 28 bytes:
12 object header + 4 for Message + 8 for scheduledDeliveryTime + 4 deliveryCount + 4 padding on a JDK 64 bits bits with COOPS.

Often both scheduledDeliveryTime and deliveryCount are not used, hence we could save 12 bytes per-message by using a separate data-structure to hold such additional records.
Such records could be stored in a separate LongObjectHasMap that would cost approximately (it depends on the load factor) per-entry: 
8 (for the messageID key) + 4 (for the additional record reference) + 16 (for the record instance) = 28 bytes

In the worst case, given that the additional informations require the same amount of memory (28 bytes), the only downside would be some lookup cost while walking/updating both the data-structures, but in the best case (and most common cases) it means eg

100_000 elements = saved ~2.8 MB 
1_000_000 elements = saved ~28 MB 
10_000_000 elements = saved ~280 MB 



  was:
While loading the journal, before reloading the messages into each queue, we store the reference to the message and some additional information (scheduledDeliveryTime and deliveryCount) on a LinkedHashMap per-queue.

These information are using a AddMessageRecord class that can weigth 28 bytes:
12 object header + 4 for Message + 8 for scheduledDeliveryTime + 4 deliveryCount + 4 padding on a JDK 64 bits bits with COOPS.

Often both scheduledDeliveryTime and deliveryCount are not used, hence we could save 12 bytes per-message by using a separate data-structure to hold such additional records.
Such records could be stored in a separate LongObjectHasMap that would cost approximately (it depends on the load factor) per-entry: 
8 (for the messageID key) + 4 (for the additional record reference) + 16 (for the record instance) = 28 bytes

In the worst case, given that the additional informations require the same amount of memory (28 bytes), the only downside would be some lookup cost while walking/updating both the data-structures, but in the best case (and most common cases) it means eg

100_000 elements = saved ~2.8 MB 
1_000_000 elements = saved ~28 MB 
10_000_000 elements = saved ~280 MB 




> Reduce memory footprint of journal loading
> ------------------------------------------
>
>                 Key: ARTEMIS-2746
>                 URL: https://issues.apache.org/jira/browse/ARTEMIS-2746
>             Project: ActiveMQ Artemis
>          Issue Type: Improvement
>          Components: Broker
>            Reporter: Francesco Nigro
>            Priority: Minor
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> While loading the journal, before reloading the messages into each queue, we store the reference to the message and some additional information (scheduledDeliveryTime and deliveryCount) in a LinkedHashMap per-queue.
> These information are using a AddMessageRecord class that can weigth 28 bytes:
> 12 object header + 4 for Message + 8 for scheduledDeliveryTime + 4 deliveryCount + 4 padding on a JDK 64 bits bits with COOPS.
> Often both scheduledDeliveryTime and deliveryCount are not used, hence we could save 12 bytes per-message by using a separate data-structure to hold such additional records.
> Such records could be stored in a separate LongObjectHasMap that would cost approximately (it depends on the load factor) per-entry: 
> 8 (for the messageID key) + 4 (for the additional record reference) + 16 (for the record instance) = 28 bytes
> In the worst case, given that the additional informations require the same amount of memory (28 bytes), the only downside would be some lookup cost while walking/updating both the data-structures, but in the best case (and most common cases) it means eg
> 100_000 elements = saved ~2.8 MB 
> 1_000_000 elements = saved ~28 MB 
> 10_000_000 elements = saved ~280 MB 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)