[jira] [Created] (HBASE-26605) TestHStore#testRefreshStoreFiles broken due to unqualified and qualified paths

["Josh Elser (Jira)" <ji...@apache.org>]

Josh Elser created HBASE-26605:
----------------------------------

             Summary: TestHStore#testRefreshStoreFiles broken due to unqualified and qualified paths
                 Key: HBASE-26605
                 URL: https://issues.apache.org/jira/browse/HBASE-26605
             Project: HBase
          Issue Type: Bug
          Components: test
    Affects Versions: 2.4.9
            Reporter: Josh Elser
            Assignee: Josh Elser


Was looking at a failures of this method where 
{noformat}
[ERROR] org.apache.hadoop.hbase.regionserver.TestHStore.testRefreshStoreFiles  Time elapsed: 4.2 s  <<< ERROR!
java.util.NoSuchElementException
    at org.apache.hbase.thirdparty.com.google.common.collect.AbstractIndexedListIterator.next(AbstractIndexedListIterator.java:75)
    at org.apache.hadoop.hbase.regionserver.TestHStore.closeCompactedFile(TestHStore.java:962)
    at org.apache.hadoop.hbase.regionserver.TestHStore.testRefreshStoreFiles(TestHStore.java:1000) {noformat}
This was on a branch where I had some HBASE-26067 changes backported, so I thought the problem _was_ those changes. After a bit of digging, I believe the test case itself is "broken" (the test passes, but for the wrong reasons).

This test methods adds some files to a Store (via memstore flush or direct addition of a file) and eventually tries to get the first file which is candidate to be removed. The test {*}never compacted any files{*}. This was the first sign that the test itself was wrong.

After lots of comparison with the HBASE-26067 logging to compare against, I found that the Store was listing a file which was created by the memstore flush as a file to retain AND a file to remove. Second warning. Upon closer inspection, I finally noticed that one of the files was qualified with the filesystem URI and the other was not.
{noformat}
2021-12-18 16:57:10,903 INFO  [Time-limited test] regionserver.HStore(675): toBeAddedFiles=[file:/Users/jelser/projects/cldr/hbase-copy.git/hbase-server/target/test-data/f4ed7913-e62a-8d5f-a968-bb4c94d5494a/TestStoretestRefreshStoreFiles/data/default/table/297ad8361c3326bfb1520dbc54b1c3bd/family/dd8a430b391546d8b9bdc39bb77d447b, file:/Users/jelser/projects/cldr/hbase-copy.git/hbase-server/target/test-data/f4ed7913-e62a-8d5f-a968-bb4c94d5494a/TestStoretestRefreshStoreFiles/data/default/table/297ad8361c3326bfb1520dbc54b1c3bd/family/d4c5442b772c43fd9ebdfed1a11c0e73], toBeRemovedFiles=[/Users/jelser/projects/cldr/hbase-copy.git/hbase-server/target/test-data/f4ed7913-e62a-8d5f-a968-bb4c94d5494a/TestStoretestRefreshStoreFiles/data/default/table/297ad8361c3326bfb1520dbc54b1c3bd/family/d4c5442b772c43fd9ebdfed1a11c0e73] {noformat}
{{d4c5442b772c43fd9ebdfed1a11c0e73}} how are we both adding and removing this file! Turns out, this is because one of them is "/..." and the other is "file:/...". Either the problem is in TestHStore in how it is creating/adding these files behind the scenes or we should be qualifying the Path inside of StoreFileInfo with the filesystem that we're using.

I remember too vividly the problems when trying to separate the rootdir and waldir from each other and am cautious against adding something to StoreFileInfo to {{{}fs.qualifyPath(p){}}}. Need to look some more, but will get a patch up to fix.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)