[jira] [Created] (IMPALA-12523) Avoid loading all partitions to get the file formats

["Quanlong Huang (Jira)" <ji...@apache.org>]

Quanlong Huang created IMPALA-12523:
---------------------------------------

             Summary: Avoid loading all partitions to get the file formats
                 Key: IMPALA-12523
                 URL: https://issues.apache.org/jira/browse/IMPALA-12523
             Project: IMPALA
          Issue Type: Improvement
          Components: Frontend
            Reporter: Quanlong Huang


There is a TODO in LocalFsTable that we should address:
{code}
  public Set<HdfsFileFormat> getFileFormats() {
    // TODO(todd): can we avoid loading all partitions here? this is called
    // for any INSERT query, even if the partition is specified.
    Collection<? extends FeFsPartition> parts;
    if (ref_ != null) {
      parts = FeCatalogUtils.loadAllPartitions(this);
    } else {
      // If this is a CTAS target, we don't want to try to load the partition list.
      parts = Collections.emptyList();
    }
    // In the case that we have no partitions added to the table yet, it's
    // important to add the "prototype" partition as a fallback.
    Iterable<FeFsPartition> partitionsToConsider = Iterables.concat(
        parts, Collections.singleton(createPrototypePartition()));
    return FeCatalogUtils.getFileFormats(partitionsToConsider);
  }
{code}
We can consider passing the set of file formats to coordinators (local-catalog mode) when loading the TableMetaRef, just like how we load the sql_constraints, etc.
* Add a field of 'want_file_formats' in TTableInfoSelector.
* Catalogd add the set of file formats in the response.
* Add the set of file formats to TableMetaRefImpl and use it in LocalFsTable.getFileFormats()



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org