[jira] [Updated] (HBASE-24021) Fail fast when bulkLoadHFiles method catch some IOException during invoke assertBulkLoadHFileOK

["niuyulin (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HBASE-24021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

niuyulin updated HBASE-24021:
-----------------------------
    Description: 
{code:java}
public Map<byte[], List<Path>> bulkLoadHFiles(Collection<Pair<byte[], String>> familyPaths,
    boolean assignSeqId, BulkLoadListener bulkLoadListener,
      boolean copyFile, List<String> clusterIds, boolean replicate) throws IOException {
  long seqId = -1;
  Map<byte[], List<Path>> storeFiles = new TreeMap<>(Bytes.BYTES_COMPARATOR);
  Map<String, Long> storeFilesSizes = new HashMap<>();
  Preconditions.checkNotNull(familyPaths);
  // we need writeLock for multi-family bulk load
  startBulkRegionOperation(hasMultipleColumnFamilies(familyPaths));
  boolean isSuccessful = false;
  try {
    this.writeRequestsCount.increment();

    // There possibly was a split that happened between when the split keys
    // were gathered and before the HRegion's write lock was taken.  We need
    // to validate the HFile region before attempting to bulk load all of them
    List<IOException> ioes = new ArrayList<>();
    List<Pair<byte[], String>> failures = new ArrayList<>();
    for (Pair<byte[], String> p : familyPaths) {
      byte[] familyName = p.getFirst();
      String path = p.getSecond();

      HStore store = getStore(familyName);
      if (store == null) {
        IOException ioe = new org.apache.hadoop.hbase.DoNotRetryIOException(
            "No such column family " + Bytes.toStringBinary(familyName));
        ioes.add(ioe);
      } else {
        try {
          store.assertBulkLoadHFileOk(new Path(path));
        } catch (WrongRegionException wre) {
          // recoverable (file doesn't fit in region)
          failures.add(p);
        } catch (IOException ioe) {
          // unrecoverable (hdfs problem)
          ioes.add(ioe);
        }
      }
    }

    // validation failed because of some sort of IO problem.
    if (ioes.size() != 0) {
      IOException e = MultipleIOException.createIOException(ioes);
      LOG.error("There were one or more IO errors when checking if the bulk load is ok.", e);
      throw e;
    }
{code}

> Fail fast when bulkLoadHFiles method catch some IOException during invoke assertBulkLoadHFileOK
> -----------------------------------------------------------------------------------------------
>
>                 Key: HBASE-24021
>                 URL: https://issues.apache.org/jira/browse/HBASE-24021
>             Project: HBase
>          Issue Type: Improvement
>          Components: HFile, regionserver
>            Reporter: niuyulin
>            Assignee: niuyulin
>            Priority: Major
>
> {code:java}
> public Map<byte[], List<Path>> bulkLoadHFiles(Collection<Pair<byte[], String>> familyPaths,
>     boolean assignSeqId, BulkLoadListener bulkLoadListener,
>       boolean copyFile, List<String> clusterIds, boolean replicate) throws IOException {
>   long seqId = -1;
>   Map<byte[], List<Path>> storeFiles = new TreeMap<>(Bytes.BYTES_COMPARATOR);
>   Map<String, Long> storeFilesSizes = new HashMap<>();
>   Preconditions.checkNotNull(familyPaths);
>   // we need writeLock for multi-family bulk load
>   startBulkRegionOperation(hasMultipleColumnFamilies(familyPaths));
>   boolean isSuccessful = false;
>   try {
>     this.writeRequestsCount.increment();
>     // There possibly was a split that happened between when the split keys
>     // were gathered and before the HRegion's write lock was taken.  We need
>     // to validate the HFile region before attempting to bulk load all of them
>     List<IOException> ioes = new ArrayList<>();
>     List<Pair<byte[], String>> failures = new ArrayList<>();
>     for (Pair<byte[], String> p : familyPaths) {
>       byte[] familyName = p.getFirst();
>       String path = p.getSecond();
>       HStore store = getStore(familyName);
>       if (store == null) {
>         IOException ioe = new org.apache.hadoop.hbase.DoNotRetryIOException(
>             "No such column family " + Bytes.toStringBinary(familyName));
>         ioes.add(ioe);
>       } else {
>         try {
>           store.assertBulkLoadHFileOk(new Path(path));
>         } catch (WrongRegionException wre) {
>           // recoverable (file doesn't fit in region)
>           failures.add(p);
>         } catch (IOException ioe) {
>           // unrecoverable (hdfs problem)
>           ioes.add(ioe);
>         }
>       }
>     }
>     // validation failed because of some sort of IO problem.
>     if (ioes.size() != 0) {
>       IOException e = MultipleIOException.createIOException(ioes);
>       LOG.error("There were one or more IO errors when checking if the bulk load is ok.", e);
>       throw e;
>     }
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)