[jira] [Resolved] (HBASE-24021) Fail fast when bulkLoadHFiles method catch some IOException

["Guanghao Zhang (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HBASE-24021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Guanghao Zhang resolved HBASE-24021.
------------------------------------
    Fix Version/s: 2.2.5
                   2.4.0
                   2.3.0
                   3.0.0
       Resolution: Fixed

Pushed to branch-2.2+.

> Fail fast when bulkLoadHFiles method catch some IOException
> -----------------------------------------------------------
>
>                 Key: HBASE-24021
>                 URL: https://issues.apache.org/jira/browse/HBASE-24021
>             Project: HBase
>          Issue Type: Improvement
>          Components: HFile, regionserver
>            Reporter: niuyulin
>            Assignee: niuyulin
>            Priority: Major
>             Fix For: 3.0.0, 2.3.0, 2.4.0, 2.2.5
>
>
> In production environment, we usually do bulkload huge amount hfile . It reasonable  fail fast when any  IOException occur
>  
> hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
> {code:java}
> public Map<byte[], List<Path>> bulkLoadHFiles(Collection<Pair<byte[], String>> familyPaths,
>     boolean assignSeqId, BulkLoadListener bulkLoadListener,
>       boolean copyFile, List<String> clusterIds, boolean replicate) throws IOException {
>   ......
>   try {
>     this.writeRequestsCount.increment();
>     // There possibly was a split that happened between when the split keys
>     // were gathered and before the HRegion's write lock was taken.  We need
>     // to validate the HFile region before attempting to bulk load all of them
>     List<IOException> ioes = new ArrayList<>();
>     List<Pair<byte[], String>> failures = new ArrayList<>();
>     for (Pair<byte[], String> p : familyPaths) {
>       byte[] familyName = p.getFirst();
>       String path = p.getSecond();
>       HStore store = getStore(familyName);
>       if (store == null) {
>         IOException ioe = new org.apache.hadoop.hbase.DoNotRetryIOException(
>             "No such column family " + Bytes.toStringBinary(familyName));
>         ioes.add(ioe);
>       } else {
>         try {
>           store.assertBulkLoadHFileOk(new Path(path));
>         } catch (WrongRegionException wre) {
>           // recoverable (file doesn't fit in region)
>           failures.add(p);
>         } catch (IOException ioe) {
>           // unrecoverable (hdfs problem)
>           ioes.add(ioe);
>         }
>       }
>     }
>     // validation failed because of some sort of IO problem.
>     if (ioes.size() != 0) {
>       IOException e = MultipleIOException.createIOException(ioes);
>       LOG.error("There were one or more IO errors when checking if the bulk load is ok.", e);
>       throw e;
>     }
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)