[jira] [Commented] (HBASE-26605) TestHStore#testRefreshStoreFiles broken due to unqualified and qualified paths

["Josh Elser (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-26605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17462002#comment-17462002 ] 

Josh Elser commented on HBASE-26605:
------------------------------------

Also, have not evaluated other active branches for relevance. Will do that after I have a fix in mind for 2.4.

> TestHStore#testRefreshStoreFiles broken due to unqualified and qualified paths
> ------------------------------------------------------------------------------
>
>                 Key: HBASE-26605
>                 URL: https://issues.apache.org/jira/browse/HBASE-26605
>             Project: HBase
>          Issue Type: Bug
>          Components: test
>    Affects Versions: 2.4.9
>            Reporter: Josh Elser
>            Assignee: Josh Elser
>            Priority: Minor
>
> Was looking at a failures of this method where 
> {noformat}
> [ERROR] org.apache.hadoop.hbase.regionserver.TestHStore.testRefreshStoreFiles  Time elapsed: 4.2 s  <<< ERROR!
> java.util.NoSuchElementException
>     at org.apache.hbase.thirdparty.com.google.common.collect.AbstractIndexedListIterator.next(AbstractIndexedListIterator.java:75)
>     at org.apache.hadoop.hbase.regionserver.TestHStore.closeCompactedFile(TestHStore.java:962)
>     at org.apache.hadoop.hbase.regionserver.TestHStore.testRefreshStoreFiles(TestHStore.java:1000) {noformat}
> This was on a branch where I had some HBASE-26067 changes backported, so I thought the problem _was_ those changes. After a bit of digging, I believe the test case itself is "broken" (the test passes, but for the wrong reasons).
> This test methods adds some files to a Store (via memstore flush or direct addition of a file) and eventually tries to get the first file which is candidate to be removed. The test {*}never compacted any files{*}. This was the first sign that the test itself was wrong.
> After lots of comparison with the HBASE-26067 logging to compare against, I found that the Store was listing a file which was created by the memstore flush as a file to retain AND a file to remove. Second warning. Upon closer inspection, I finally noticed that one of the files was qualified with the filesystem URI and the other was not.
> {noformat}
> 2021-12-18 16:57:10,903 INFO  [Time-limited test] regionserver.HStore(675): toBeAddedFiles=[file:/Users/jelser/projects/cldr/hbase-copy.git/hbase-server/target/test-data/f4ed7913-e62a-8d5f-a968-bb4c94d5494a/TestStoretestRefreshStoreFiles/data/default/table/297ad8361c3326bfb1520dbc54b1c3bd/family/dd8a430b391546d8b9bdc39bb77d447b, file:/Users/jelser/projects/cldr/hbase-copy.git/hbase-server/target/test-data/f4ed7913-e62a-8d5f-a968-bb4c94d5494a/TestStoretestRefreshStoreFiles/data/default/table/297ad8361c3326bfb1520dbc54b1c3bd/family/d4c5442b772c43fd9ebdfed1a11c0e73], toBeRemovedFiles=[/Users/jelser/projects/cldr/hbase-copy.git/hbase-server/target/test-data/f4ed7913-e62a-8d5f-a968-bb4c94d5494a/TestStoretestRefreshStoreFiles/data/default/table/297ad8361c3326bfb1520dbc54b1c3bd/family/d4c5442b772c43fd9ebdfed1a11c0e73] {noformat}
> {{d4c5442b772c43fd9ebdfed1a11c0e73}} how are we both adding and removing this file! Turns out, this is because one of them is "/..." and the other is "file:/...". Either the problem is in TestHStore in how it is creating/adding these files behind the scenes or we should be qualifying the Path inside of StoreFileInfo with the filesystem that we're using.
> I remember too vividly the problems when trying to separate the rootdir and waldir from each other and am cautious against adding something to StoreFileInfo to {{{}fs.qualifyPath(p){}}}. Need to look some more, but will get a patch up to fix.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)