[jira] [Commented] (CASSANDRA-18464) Enable Direct I/O For CommitLog Files

["Brandon Williams (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-18464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17713775#comment-17713775 ] 

Brandon Williams commented on CASSANDRA-18464:
----------------------------------------------

I applied this to a branch and had circle take a first pass.

||Branch||CI||
|[trunk|https://github.com/driftx/cassandra/tree/CASSANDRA-18464-trunk]|[j8|https://app.circleci.com/pipelines/github/driftx/cassandra/977/workflows/677ded2a-f1bf-416c-ae8b-bdf32f18a2eb], [j11|https://app.circleci.com/pipelines/github/driftx/cassandra/977/workflows/9554a2d4-0322-492d-9f6b-b7e70eb8dab0]|


> Enable Direct I/O For CommitLog Files
> -------------------------------------
>
>                 Key: CASSANDRA-18464
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-18464
>             Project: Cassandra
>          Issue Type: New Feature
>          Components: Local/Commit Log
>            Reporter: Josh McKenzie
>            Assignee: Amit Pawar
>            Priority: Normal
>             Fix For: 6.x
>
>         Attachments: UseDirectIOFeatureForCommitLogFiles.patch
>
>
> Relocating from [dev@ email thread.|https://lists.apache.org/thread/j6ny17q2rhkp7jxvwxm69dd6v1dozjrg]
>  
> I shared my investigation about Commitlog I/O issue on large core count system in my previous email dated July-22 and link to the thread is given below.
> [https://lists.apache.org/thread/xc5ocog2qz2v2gnj4xlw5hbthfqytx2n]
> Basically, two solutions looked possible to improve the CommitLog I/O.
>  # Multi-threaded syncing
>  # Using Direct-IO through JNA
> I worked on 2nd option considering the following benefit compared to the first one
>  # Direct I/O read/write throughput is very high compared to non-Direct I/O. Learnt through FIO benchmarking.
>  # Reduces kernel file cache uses which in-turn reduces kernel I/O activity for Commitlog files only.
>  # Overall CPU usage reduced for flush activity. JVisualvm shows CPU usage < 30% for Commitlog syncer thread with Direct I/O feature
>  # Direct I/O implementation is easier compared to multi-threaded
> As per the community suggestion, less in code complex is good to have. Direct I/O enablement looked promising but there was one issue. 
> Java version 8 does not have native support to enable Direct I/O. So, JNA library usage is must. The same implementation should also work across other versions of Java (like 11 and beyond).
> I have completed Direct I/O implementation and summary of the attached patch changes are given below.
>  # This implementation is not using Java file channels and file is opened through JNA to use Direct I/O feature.
>  # New Segment are defined named “DirectIOSegment”  for Direct I/O and “NonDirectIOSegment” for non-direct I/O (NonDirectIOSegment is test purpose only).
>  # JNA write call is used to flush the changes.
>  # New helper functions are defined in NativeLibrary.java and platform specific file. Currently tested on Linux only.
>  # Patch allows user to configure optimum block size  and alignment if default values are not OK for CommitLog disk.
>  # Following configuration options are provided in Cassandra.yaml file
> a. use_jna_for_commitlog_io : to use jna feature
> b. use_direct_io_for_commitlog : to use Direct I/O feature.
> c. direct_io_minimum_block_alignment: 512 (default)
> d. nvme_disk_block_size: 32MiB (default and can be changed as per the required size)
>  Test matrix is complex so CommitLog related testcases and TPCx-IOT benchmark was tested. It works with both Java 8 and 11 versions. Compressed and Encrypted based segments are not supported yet and it can be enabled later based on the Community feedback.
>  Following improvement are seen with Direct I/O enablement.
>  # 32 cores >= ~15%
>  # 64 cores >= ~80%
>  Also, another observation would like to share here. Reading Commitlog files with Direct I/O might help in reducing node bring-up time after the node crash.
>  Tested with commit ID: 91f6a9aca8d3c22a03e68aa901a0b154d960ab07
>  The attached patch enables Direct I/O feature for Commitlog files. Please check and share your feedback.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org