You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "George Jahad (Jira)" <ji...@apache.org> on 2023/03/09 02:00:00 UTC
[jira] [Commented] (HDDS-7110) Race Condition in TestKeyManagerImpl::testListStatusWithTableCacheRecursive()
[ https://issues.apache.org/jira/browse/HDDS-7110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17698141#comment-17698141 ]
George Jahad commented on HDDS-7110:
------------------------------------
Fixed in [https://github.com/apache/ozone/pull/3653]
Closing
> Race Condition in TestKeyManagerImpl::testListStatusWithTableCacheRecursive()
> -----------------------------------------------------------------------------
>
> Key: HDDS-7110
> URL: https://issues.apache.org/jira/browse/HDDS-7110
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: George Jahad
> Assignee: George Jahad
> Priority: Minor
>
> There is a race condition this test: TestKeyManagerImpl::testListStatusWithTableCacheRecursive(). The test depends on the timing of the OM double buffer flush. If the flush happens quickly enough, the test passes.
>
> *The Problem:*
> The purpose of the test is to confirm that KeyManagerImpl::listStatus() correctly handles a mix of keytable cache and db entries.
> The test uses the "writeClient.createDirectory()" method to create some directories. That method uses the normal rpc client protocol request path, which writes to the cache and the double buffer. Then testListStatusWithTableCacheRecursive() writes some keys to those directories using addKeyToTable() which bypasses the cache.
> Then it runs keyManager.listStatus(). If the flush hasn't yet occurred, the directories are still in the cache, while the keys that belong to those directories are in the keytable. The listStatus() algorithm can't handle that inconsistency. It expects that directories are added to the table before any elements of those directories.
> This race was actually introduced by me 6 months ago in this PR:
> "HDDS-3231. Cleanup KeyManagerImpl"
>
> *The Fix:*
> I've confirmed this is the problem by disabling the flush. Then the problem happens consistently, and I can step through it with the debugger.
> The fix is to bypass the double buffer, (as the test did originally,) by writing the directories straight to the table using addKeyToTable, instead of writeClient.createDirectory(). I have confirmed the fix and will create a PR shortly.
> For some reason, this race condition, (which is in master,) happens more frequently in the snapshotting branch, which is how I noticed it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org