[jira] [Created] (IMPALA-5431) Calling FileSystem.Exists() twice in a row for the same partition adds unnecessary latency to metadata loading

["Mostafa Mokhtar (JIRA)" <ji...@apache.org>]

Mostafa Mokhtar created IMPALA-5431:
---------------------------------------

             Summary: Calling FileSystem.Exists() twice in a row for the same partition adds unnecessary latency to metadata loading
                 Key: IMPALA-5431
                 URL: https://issues.apache.org/jira/browse/IMPALA-5431
             Project: IMPALA
          Issue Type: Bug
            Reporter: Mostafa Mokhtar
            Assignee: Alexander Behm
         Attachments: After removing redundant fs.exists().jfr, Baseline.jfr

FileSystem.exists() is called in loadPartitionFileMetadata then again in refreshFileMetadata which seems redundant. 

When dealing with a large number of partitions this can become a bottleneck.

{code}
 private void loadPartitionFileMetadata(StorageDescriptor storageDescriptor,
      HdfsPartition partition) throws Exception {
    Preconditions.checkNotNull(storageDescriptor);
    Preconditions.checkNotNull(partition);
    Path partDirPath = new Path(storageDescriptor.getLocation());
    FileSystem fs = partDirPath.getFileSystem(CONF);
   if (!fs.exists(partDirPath)) return;
    refreshFileMetadata(partition);
  }
{code}

{code}
private void refreshFileMetadata(HdfsPartition partition) throws CatalogException {
    Path partDir = partition.getLocationPath();
    Preconditions.checkNotNull(partDir);
    try {
      FileSystem fs = partDir.getFileSystem(CONF);
      if (!fs.exists(partDir)) {
        partition.setFileDescriptors(new ArrayList<FileDescriptor>());
        return;
      }
      if (!FileSystemUtil.supportsStorageIds(fs)) {
        synthesizeBlockMetadata(fs, partition);
        return;
      }
      // Index the partition file descriptors by their file names for O(1) look ups.
      ImmutableMap<String, FileDescriptor> fileDescsByName = Maps.uniqueIndex(
          partition.getFileDescriptors(), new Function<FileDescriptor, String>() {
            public String apply(FileDescriptor desc) {
              return desc.getFileName();
            }
          });
{code}

Before and after Java profiles attached, the number of socket reads goes down from 1,639 to 1,046.  For a table with 80 partitions and 250K files this gave a 15-20% speedup. 




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)