[GitHub] [hbase] wchevreuil commented on a diff in pull request #5014: HBASE-27619 Bulkload fails when trying to bulkload files with invalid names after HBASE-26707

["wchevreuil (via GitHub)" <gi...@apache.org>]

wchevreuil commented on code in PR #5014:
URL: https://github.com/apache/hbase/pull/5014#discussion_r1100224722


##########
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java:
##########
@@ -7280,7 +7280,7 @@ public Map<byte[], List<Path>> bulkLoadHFiles(Collection<Pair<byte[], String>> f
               reqTmp ? null : fs.getRegionDir().toString());
           }
           Pair<Path, Path> pair = null;
-          if (reqTmp) {
+          if (reqTmp || (!StoreFileInfo.isHFile(finalPath))) {

Review Comment:
   > do you have any stacktrace and example of the non-HFile ?
   
   For the case of split files, BulkLoadHFilesTool splits the file into two, naming the resulting files with " .bottom/.top" suffixes [here](https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/tool/BulkLoadHFilesTool.java#L594). In this case, when using SFT, we'll put these files into the store dir, then when StoreFileReader tries to load it, it will try to resolve it as a reference, giving the following errors:
   1) At the bulkload job side:
   
   `ERROR org.apache.hadoop.hbase.regionserver.HRegion: There was a partial failure due to IO when attempting to load CF : s3a://my-s3-bucket/my-datalake/hbase/data/default/my-table/2462397a71fa06cff0c5598434a02c5f/CF/e245ce23ae9b48ebac7713568ce3a569.top`
   
   2) At RS side:
   `FileNotFoundException: No such file or directory: s3a://my-s3-bucket/my-datalake/hbase/data/default/my-table/top/CF/e245ce23ae9b48ebac7713568ce3a569`
   
   Note in the path of error #2 that the RS assumed the file was a reference and tried tor resolve it as such.
   
   There's also the possibility clients pass files with invalid naming format. These would also be placed in the store dir as it is, then failing the validations performed by StoreFileReader during file loading. These would manifest such as below, on RS logs:
   
   `2023-02-08T10:27:26,447 ERROR [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=49900] regionserver.HRegion(7346): There was a partial failure due to IO when attempting to load myfam : s3a://my-s3-bucket/my-datalake/hbase/data/default/my-table/3c91abacbd1a6c41d7247bd3d5a6bb00/CF/hfile_0
   java.io.IOException: path=s3a://my-s3-bucket/my-datalake/hbase/data/default/my-table/3c91abacbd1a6c41d7247bd3d5a6bb00/CF/hfile_0 doesn't look like a valid StoreFile`
   
   These two are also easily reproducible by the UT added here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org