You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ignite.apache.org by "Ivan Bessonov (Jira)" <ji...@apache.org> on 2024/09/20 07:40:00 UTC
[jira] [Updated] (IGNITE-23240) Ignite 3 new log storage
[ https://issues.apache.org/jira/browse/IGNITE-23240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ivan Bessonov updated IGNITE-23240:
-----------------------------------
Attachment: image-2024-09-20-10-39-22-043.png
> Ignite 3 new log storage
> ------------------------
>
> Key: IGNITE-23240
> URL: https://issues.apache.org/jira/browse/IGNITE-23240
> Project: Ignite
> Issue Type: Epic
> Reporter: Ivan Bessonov
> Priority: Major
> Labels: ignite-3
> Attachments: image-2024-09-20-10-39-22-043.png
>
>
> h1. Preface
> Current implementation, based on {{{}RocksDB{}}}, is known to be way slower then it should be. There are multiple obvious reasons for that:
> * Writing into WAL +and+ memtable
> * Creating unique keys for every record
> * Inability to efficiently serialize data, we must have an intermediate state before we pass data into {{{}RocksDB{}}}'s API.
> h1. Benchmarks
> h3. Local benchmarks
> Local benchmarks ({{{}LogStorageBenchmarks{}}}) have been performed on my local environment with fsync disabled. I got the following results:
> * {{{}Logit{}}}:
>
> {noformat}
> Test write:
> Log number : 1024000
> Log Size : 16384
> Batch Size : 100
> Cost time(s) : 23.541
> Total size : 16777216000
> Throughput(bps) : 712680684
> Throughput(rps) : 43498
> Test read:
> Log number : 1024000
> Log Size : 16384
> Batch Size : 100
> Cost time(s) : 3.808
> Total size : 16777216000
> Throughput(bps) : 4405781512
> Throughput(rps) : 268907
> Test done!{noformat}
> * {{{}RocksDB{}}}:
> {noformat}
> Test write:
> Log number : 1024000
> Log Size : 16384
> Batch Size : 100
> Cost time(s) : 178.785
> Total size : 16777216000
> Throughput(bps) : 93840176
> Throughput(rps) : 5727
> Test read:
> Log number : 1024000
> Log Size : 16384
> Batch Size : 100
> Cost time(s) : 13.572
> Total size : 16777216000
> Throughput(bps) : 1236163866
> Throughput(rps) : 75449
> Test done!{noformat}
> While testing on local environment is not optimal, is still shows a huge improvement in writing speed (7.5x) and reading speed (3.5x). Enabling {{fsync}} sort-of equalizes writing speed, but we still expect that simpler log implementation would be faster dues to smaller overall overhead.
> h3. Integration testing
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)