[jira] [Comment Edited] (KUDU-2707) Improve the performance of the block cache under contention

["Alexey Serbin (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KUDU-2707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17736308#comment-17736308 ] 

Alexey Serbin edited comment on KUDU-2707 at 6/29/23 4:27 PM:
--------------------------------------------------------------

The stack mentioned is related to the lock that guards a shard's lookup table and updating LRU-related stats.  As a first improvement step we might try to:
* perform the lookup and the update of the LRU-related stats in separate critical sections
* use a concurrent dictionary instead of guarding the non-concurrent container with the lock.

There are a few implementations of concurrent dictionaries in C++, at least the following are available:
* concurrent_hash_map and concurrent_unordered_map of [Intel's TBB|https://github.com/oneapi-src/oneTBB] (Apache 2.0 license)
* concurrent hash maps from the [Junction project|https://github.com/preshing/junction] (BSD license), see [this article|https://preshing.com/20160201/new-concurrent-hash-maps-for-cpp/] for more details

Both might be used as a third-party components in Kudu at least based on their license types.


was (Author: aserbin):
The stack mentioned is related to the lock that guards the shards dictionary.  As a first improvement step we might try to use a concurrent dictionary there instead of guarding the non-concurrent map with the lock.

There are a few implementations of concurrent dictionaries in C++, at least the following are available:
* concurrent_hash_map and concurrent_unordered_map of [Intel's TBB|https://github.com/oneapi-src/oneTBB] (Apache 2.0 license)
* concurrent hash maps from the [Junction project|https://github.com/preshing/junction] (BSD license), see [this article|https://preshing.com/20160201/new-concurrent-hash-maps-for-cpp/] for more details

Both might be used as a third-party components in Kudu at least based on their license types.

> Improve the performance of the block cache under contention
> -----------------------------------------------------------
>
>                 Key: KUDU-2707
>                 URL: https://issues.apache.org/jira/browse/KUDU-2707
>             Project: Kudu
>          Issue Type: Improvement
>    Affects Versions: 1.10.0
>            Reporter: William Berkeley
>            Priority: Major
>             Fix For: NA
>
>
> While looking at a random write workload where flushes outpace compactions (i.e. the typical case when inserting as fast as possible), there are occasional consensus service queue overflows. Analyzing the stacks of the service threads when this occurs (using the diagnostics log), I see many stacks like
> {noformat}
> 0x3b6720f710 <unknown>
>            0x1fb900a base::internal::SpinLockDelay()
>            0x1fb8ea7 base::SpinLock::SlowLock()
>            0x1ef7394 kudu::(anonymous namespace)::ShardedLRUCache::Lookup()
>            0x1ce379f kudu::cfile::BlockCache::Lookup()
>            0x1cec948 kudu::cfile::CFileReader::ReadBlock()
>            0x1ce5d36 kudu::cfile::BloomFileReader::CheckKeyPresent()
>             0xb311a1 kudu::tablet::CFileSet::CheckRowPresent()
>             0xac46c4 kudu::tablet::DiskRowSet::CheckRowPresent()
>             0xa6b017 _ZZN4kudu6tablet6Tablet17BulkCheckPresenceEPKNS_2fs9IOContextEPNS0_21WriteTransactionStateEENKUlvE1_clEv
>             0xa7427e _ZNSt17_Function_handlerIFvPN4kudu6tablet6RowSetEiEZNS1_6Tablet17BulkCheckPresenceEPKNS0_2fs9IOContextEPNS1_21WriteTransactionStateEEUlS3_iE2_E9_M_invokeERKSt9_Any_dataS3_i
>             0xaee074 _ZNK4kudu22interval_tree_internal6ITNodeINS_6tablet20RowSetIntervalTraitsEE31ForEachIntervalContainingPointsIZNKS2_10RowSetTree27ForEachRowSetContainingKeysERKSt6vectorINS_5SliceESaIS8_EERKSt8functionIFvPNS2_6RowSetEiEEEUlRKNS2_12_GLOBAL__N_111QueryStructEPNS2_16RowSetWithBoundsEE_N9__gnu_cxx17__normal_iteratorIPSM_S7_ISL_SaISL_EEEEEEvT0_SX_RKT_
>             0xaee1b3 _ZNK4kudu22interval_tree_internal6ITNodeINS_6tablet20RowSetIntervalTraitsEE31ForEachIntervalContainingPointsIZNKS2_10RowSetTree27ForEachRowSetContainingKeysERKSt6vectorINS_5SliceESaIS8_EERKSt8functionIFvPNS2_6RowSetEiEEEUlRKNS2_12_GLOBAL__N_111QueryStructEPNS2_16RowSetWithBoundsEE_N9__gnu_cxx17__normal_iteratorIPSM_S7_ISL_SaISL_EEEEEEvT0_SX_RKT_
>             0xaee3a3 kudu::tablet::RowSetTree::ForEachRowSetContainingKeys()
>             0xa80c17 kudu::tablet::Tablet::BulkCheckPresence()
>             0xa8108a kudu::tablet::Tablet::ApplyRowOperations()
> {noformat}
> Note that the slow step in writes for these workloads is generally CPU usage in the apply phase, once they have been running for a while.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)