You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by GitBox <gi...@apache.org> on 2020/03/13 20:00:01 UTC

[GitHub] [drill] vvysotskyi opened a new pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

vvysotskyi opened a new pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026
 
 
   # [DRILL-7330](https://issues.apache.org/jira/browse/DRILL-7330): Implement metadata usage for all format plugins
   
   ## Description
   This pull request adds support of all format plugins for Metastore.
   
   It contains two parts:
   - Making changes to collect metadata for easy group scan
   - Allowing usage Metastore metadata for easy group scan during querying data
   
   To reach the first goal was added AnalyzeFileInfoProviderImpl to handle correctly EasyGroupScan tables in MetastoreAnalyzeTableHandler and specify correct table type.
   Accommodated EVF scan framework to incorporate Metastore-specific additional implicit columns and handle the case of empty files when metadata should be returned.
   
   To reach the second goal, EasyGroupScan was updated to extend AbstractGroupScanWithMetadata.
   MetastoreFileTableMetadataProvider based on existing MetastoreParquetTableMetadataProvider was introduced to provide metastore metadata for file-based tables.
   Accommodated ParquetPushDownFilter rule to work with easy format plugins.
   
   Refactored some code and fixed additional issues.
   
   ## Documentation
   After this PR is merged, Drill Web Site docs should be updated to reflect this feature.
   
   ## Testing
   Added tests to verify the correctness of collected metadata and ensure that new optimizations work correctly.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392705403
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/easy/EasyFormatPlugin.java
 ##########
 @@ -387,15 +386,10 @@ protected void initScanBuilder(FileScanBuilder builder, EasySubScan scan) {
 
     // Additional error context to identify this plugin
     builder.errorContext(
-        new CustomErrorContext() {
-          @Override
-          public void addContext(UserException.Builder builder) {
-            builder.addContext("Format plugin:", easyConfig.defaultName);
-            builder.addContext("Format plugin:",
-                EasyFormatPlugin.this.getClass().getSimpleName());
-            builder.addContext("Plugin config name:", getName());
-          }
-        });
+        currentBuilder -> currentBuilder
+            .addContext("Format plugin:", easyConfig.defaultName)
+            .addContext("Format plugin:", EasyFormatPlugin.this.getClass().getSimpleName())
+            .addContext("Plugin config name:", getName()));
 
 Review comment:
   Thanks, removed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392608927
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/scan/framework/ShimBatchReader.java
 ##########
 @@ -131,6 +131,8 @@ public boolean next() {
       eof = ! reader.next();
     }
 
+    readerOrchestrator.setEof(eof);
 
 Review comment:
   Is there a need for this? The design is that EOF is indicated when:
   
   * `open()` returns false. (As in an empty file with no schema. Think blank CSV file.)
   * `next()` returns false with no data. (Simple EOF.)
   * `next()` returns false with data. EOF is on the next batch.
   
   Eof handling logic is built into the `ScanOperatorExec` and, in particular, the `ReaderState`. I worry about adding additional state that must be kept in sync.
   
   What is the intent here? Maybe I can help you find a solution that uses the existing mechanisms.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392680073
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/AbstractGroupScanWithMetadata.java
 ##########
 @@ -634,6 +642,62 @@ public NonInterestingColumnsMetadata getNonInterestingColumnsMetadata() {
     return nonInterestingColumnsMetadata;
   }
 
+  /**
+   * Returns {@link TableMetadataProviderBuilder} instance based on specified
+   * {@link MetadataProviderManager} source.
+   *
+   * @param source metadata provider manager
+   * @return {@link TableMetadataProviderBuilder} instance
+   */
+  protected abstract TableMetadataProviderBuilder tableMetadataProviderBuilder(MetadataProviderManager source);
+
+  /**
+   * Returns {@link TableMetadataProviderBuilder} instance which may provide metadata
+   * without using Drill Metastore.
+   *
+   * @param source metadata provider manager
+   * @return {@link TableMetadataProviderBuilder} instance
+   */
+  protected abstract TableMetadataProviderBuilder defaultTableMetadataProviderBuilder(MetadataProviderManager source);
+
+  /**
+   * Compares the last modified time of files obtained from specified selection with
+   * the Metastore last modified time to determine whether Metastore metadata
+   * is not outdated. If metadata is outdated, {@link MetadataException} will be thrown.
 
 Review comment:
   Thanks, reworded it.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r395145166
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/metastore/store/SimpleFileTableMetadataProvider.java
 ##########
 @@ -97,47 +100,38 @@ public FileMetadata getFileMetadata(Path location) {
 
   @Override
   public NonInterestingColumnsMetadata getNonInterestingColumnsMetadata() {
-    return null;
+    return new NonInterestingColumnsMetadata(Collections.emptyMap());
 
 Review comment:
   Yes, that's correct that we don't track stats for such columns. This naming came from parquet metadata cache improvements and left to be consistent with that code.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392594319
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/easy/EasyGroupScan.java
 ##########
 @@ -209,17 +214,12 @@ public boolean hasFiles() {
   }
 
   @JsonProperty("files")
+  @JsonIgnore(value = false)
 
 Review comment:
   It was ignored in the parent class and unignored in this one by specifying false value.
   Thanks for pointing this, removed ignoring in the parent class and just ignored in implementation classes were required.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392716665
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/metastore/store/MetastoreFileTableMetadataProvider.java
 ##########
 @@ -291,124 +250,74 @@ public boolean checkMetadataVersion() {
     }
   }
 
-  private ParquetTableMetadataProvider getFallbackTableMetadataProvider() throws IOException {
-    if (fallback == null) {
-      fallback = fallbackBuilder == null ? null : fallbackBuilder.build();
-    }
-    return fallback;
-  }
+  public static class Builder<T extends Builder<T>> implements FileTableMetadataProviderBuilder<T> {
+    protected final MetastoreMetadataProviderManager metadataProviderManager;
 
-  private void throwIfChanged() {
-    if (basicTablesRequests.hasMetastoreTableInfoChanged(metastoreTableInfo)) {
-      throw MetadataException.of(MetadataException.MetadataExceptionType.INCONSISTENT_METADATA);
-    }
-  }
+    // builder for fallback ParquetFileTableMetadataProvider
+    // for the case when required metadata is absent in Metastore
+    protected final TableMetadataProviderBuilder fallback;
 
-  public static class Builder implements ParquetFileTableMetadataProviderBuilder {
-    private final MetastoreMetadataProviderManager metadataProviderManager;
+    protected TupleMetadata schema;
 
-    private List<ReadEntryWithPath> entries;
-    private DrillFileSystem fs;
-    private TupleMetadata schema;
+    protected List<String> paths;
 
     private FileSelection selection;
 
-    // builder for fallback ParquetFileTableMetadataProvider
-    // for the case when required metadata is absent in Metastore
-    private final ParquetFileTableMetadataProviderBuilder fallback;
+    private DrillFileSystem fs;
 
     public Builder(MetastoreMetadataProviderManager source) {
-      this.metadataProviderManager = source;
-      this.fallback = new ParquetTableMetadataProviderImpl.Builder(FileSystemMetadataProviderManager.init());
+      this(source, new SimpleFileTableMetadataProvider.Builder(FileSystemMetadataProviderManager.init()));
     }
 
-    @Override
-    public ParquetFileTableMetadataProviderBuilder withEntries(List<ReadEntryWithPath> entries) {
-      this.entries = entries;
-      fallback.withEntries(entries);
-      return this;
+    protected Builder(MetastoreMetadataProviderManager source, TableMetadataProviderBuilder fallback) {
+      this.metadataProviderManager = source;
+      this.fallback = fallback;
     }
 
     @Override
-    public ParquetFileTableMetadataProviderBuilder withSelectionRoot(Path selectionRoot) {
-      fallback.withSelectionRoot(selectionRoot);
-      return this;
+    public T withSchema(TupleMetadata schema) {
+      this.schema = schema;
+      return self();
     }
 
-    @Override
-    public ParquetFileTableMetadataProviderBuilder withCacheFileRoot(Path cacheFileRoot) {
-      fallback.withCacheFileRoot(cacheFileRoot);
-      return this;
+    public T withSelection(FileSelection selection) {
+      this.selection = selection;
+      return self();
     }
 
-    @Override
-    public ParquetFileTableMetadataProviderBuilder withReaderConfig(ParquetReaderConfig readerConfig) {
-      fallback.withReaderConfig(readerConfig);
-      return this;
+    public T withFileSystem(DrillFileSystem fs) {
+      this.fs = fs;
+      return self();
     }
 
-    @Override
-    public ParquetFileTableMetadataProviderBuilder withFileSystem(DrillFileSystem fs) {
-      fallback.withFileSystem(fs);
-      this.fs = fs;
-      return this;
+    protected T self() {
+      return (T) this;
     }
 
-    @Override
-    public ParquetFileTableMetadataProviderBuilder withCorrectCorruptedDates(boolean autoCorrectCorruptedDates) {
-      fallback.withCorrectCorruptedDates(autoCorrectCorruptedDates);
-      return this;
+    public MetastoreMetadataProviderManager metadataProviderManager() {
+      return metadataProviderManager;
     }
 
-    @Override
-    public ParquetFileTableMetadataProviderBuilder withSelection(FileSelection selection) {
-      fallback.withSelection(selection);
-      this.selection = selection;
-      return this;
+    public FileSelection selection() {
+      return selection;
     }
 
-    @Override
-    public ParquetFileTableMetadataProviderBuilder withSchema(TupleMetadata schema) {
-      fallback.withSchema(schema);
-      this.schema = schema;
-      return this;
+    public DrillFileSystem fs() {
+      return fs;
     }
 
     @Override
-    public ParquetTableMetadataProvider build() throws IOException {
-      MetastoreParquetTableMetadataProvider provider;
-      SchemaProvider schemaProvider = metadataProviderManager.getSchemaProvider();
-      ParquetMetadataProvider source = (ParquetTableMetadataProvider) metadataProviderManager.getTableMetadataProvider();
-
-      DrillStatsTable statsProvider = metadataProviderManager.getStatsProvider();
-      // schema passed into the builder has greater priority
-      try {
-        if (this.schema == null) {
-          schema = schemaProvider != null ? schemaProvider.read().getSchema() : null;
-        }
-      } catch (IOException e) {
-        logger.debug("Unable to deserialize schema from schema file for table: {}", metadataProviderManager.getTableInfo().name(), e);
-      }
-      if (entries == null) {
-        if (!selection.isExpandedFully()) {
-          entries = DrillFileSystemUtil.listFiles(fs, selection.getSelectionRoot(), true).stream()
-              .map(fileStatus -> new ReadEntryWithPath(Path.getPathWithoutSchemeAndAuthority(fileStatus.getPath())))
-              .collect(Collectors.toList());
-        } else {
-          entries = selection.getFiles().stream()
-              .map(Path::getPathWithoutSchemeAndAuthority)
-              .map(ReadEntryWithPath::new)
-              .collect(Collectors.toList());
-        }
-      }
-      provider = new MetastoreParquetTableMetadataProvider(entries, metadataProviderManager.getMetastoreRegistry(),
-          metadataProviderManager.getTableInfo(), schema, fallback, metadataProviderManager.getConfig(), statsProvider);
-      // store results into metadataProviderManager to be able to use them when creating new instances
-      // for the case when source wasn't provided or it contains less row group metadata than the provider
-      if (source == null || source.getRowGroupsMeta().size() < provider.getRowGroupsMeta().size()) {
-        metadataProviderManager.setTableMetadataProvider(provider);
+    public TableMetadataProvider build() throws IOException {
+      if (!selection().isExpandedFully()) {
 
 Review comment:
   Thanks, inverted the condition and flipped `if else` branches.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392642282
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/scan/file/ImplicitColumnManager.java
 ##########
 @@ -199,6 +192,21 @@ public ImplicitColumnManager(OptionSet optionManager,
     }
   }
 
+  /**
+   * Specifies whether to plan based on the legacy meaning of "*". See
+   * <a href="https://issues.apache.org/jira/browse/DRILL-5542">DRILL-5542</a>.
+   * If true, then the star column <i>includes</i> implicit and partition
+   * columns. If false, then star matches <i>only</i> table columns.
+   *
+   * @param optionManager access to the options for this query; used
+   *                      too look up custom names for the metadata columns
+   * @param config        implicit column options
+   */
+  public ImplicitColumnManager(OptionSet optionManager,
 
 Review comment:
   Just noticed the comment makes no sense, it has nothing to do with the ctor. Looks like I removed a field but did not remove the comment.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392607989
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/scan/file/FileMetadata.java
 ##########
 @@ -74,4 +81,8 @@ public int dirPathLength() {
   }
 
   public boolean isSet() { return filePath != null; }
+
+  public String getImplicitColumnValue(ColumnExplorer.ImplicitFileColumn column) {
+    return ColumnExplorer.getImplicitColumnValue(column, filePath, fs);
 
 Review comment:
   The prior version returned values based on the static file name and path; no need for the file system. How has that changed here?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392685293
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/scan/file/ImplicitColumnManager.java
 ##########
 @@ -199,6 +192,21 @@ public ImplicitColumnManager(OptionSet optionManager,
     }
   }
 
+  /**
+   * Specifies whether to plan based on the legacy meaning of "*". See
+   * <a href="https://issues.apache.org/jira/browse/DRILL-5542">DRILL-5542</a>.
+   * If true, then the star column <i>includes</i> implicit and partition
+   * columns. If false, then star matches <i>only</i> table columns.
+   *
+   * @param optionManager access to the options for this query; used
+   *                      too look up custom names for the metadata columns
+   * @param config        implicit column options
+   */
+  public ImplicitColumnManager(OptionSet optionManager,
 
 Review comment:
   Thanks for pointing this, removed obsolete JavaDoc.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] arina-ielchiieva commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
arina-ielchiieva commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392590498
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/easy/EasyGroupScan.java
 ##########
 @@ -333,21 +338,73 @@ public boolean canPushdownProjects(List<SchemaPath> columns) {
   }
 
   @JsonProperty
+  @JsonIgnore(value = false)
   public TupleMetadata getSchema() {
     return getTableMetadata().getSchema();
   }
 
   @Override
-  @JsonIgnore
-  public TableMetadata getTableMetadata() {
-    if (tableMetadata == null) {
-      tableMetadata = metadataProvider.getTableMetadata();
+  public AnalyzeInfoProvider getAnalyzeInfoProvider() {
+    return new AnalyzeFileInfoProviderImpl(formatPlugin.getName());
+  }
+
+  @Override
+  protected GroupScanWithMetadataFilterer<?> getFilterer() {
+    return new EasyGroupScanFilterer(this);
+  }
+
+  @Override
+  protected FileTableMetadataProviderBuilder<?> tableMetadataProviderBuilder(MetadataProviderManager source) {
+    if (source.usesMetastore()) {
+      return new MetastoreFileTableMetadataProvider.Builder<>((MetastoreMetadataProviderManager) source);
+    } else {
+      return defaultTableMetadataProviderBuilder(source);
     }
-    return tableMetadata;
   }
 
   @Override
-  public TableMetadataProvider getMetadataProvider() {
-    return metadataProvider;
+  protected FileTableMetadataProviderBuilder<?> defaultTableMetadataProviderBuilder(MetadataProviderManager source) {
+    return new SimpleFileTableMetadataProvider.Builder(source);
+  }
+
+  /**
+   * Implementation of GroupScanWithMetadataFilterer which uses {@link EasyGroupScan} as source and
+   * builds {@link EasyGroupScan} instance with filtered metadata.
+   */
+  private static class EasyGroupScanFilterer extends GroupScanWithMetadataFilterer<EasyGroupScanFilterer> {
+
+    EasyGroupScanFilterer(EasyGroupScan source) {
+      super(source);
+    }
+
+    @Override
+    public AbstractGroupScanWithMetadata<?> build() {
+      EasyGroupScan newScan = new EasyGroupScan((EasyGroupScan) source);
+      newScan.tableMetadata = tableMetadata;
+      // updates common row count and nulls counts for every column
+      if (newScan.getTableMetadata() != null && files != null && newScan.getFilesMetadata().size() != files.size()) {
+        newScan.tableMetadata = TableMetadataUtils.updateRowCount(newScan.getTableMetadata(), files.values());
+      }
+      newScan.partitions = partitions;
+      newScan.segments = segments;
+      newScan.files = files;
+      newScan.matchAllMetadata = matchAllMetadata;
+      newScan.nonInterestingColumnsMetadata = nonInterestingColumnsMetadata;
+
+      newScan.fileSet = newScan.getFilesMetadata().keySet();
+      newScan.selection = FileSelection.create(null, new ArrayList<>(newScan.fileSet), newScan.selectionRoot);
+      try {
+        newScan.initFromSelection(newScan.selection, newScan.formatPlugin);
+      } catch (IOException e) {
+        throw new RuntimeException("Failed to initialize scan from the selection.", e);
 
 Review comment:
   Just runtime?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392605983
 
 

 ##########
 File path: exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/scan/TestFileScanFramework.java
 ##########
 @@ -25,6 +25,8 @@
 import java.util.Iterator;
 import java.util.List;
 
+import mockit.Mock;
+import mockit.MockUp;
 
 Review comment:
   I have removed mocks and used real files for these tests.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r394691383
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/ScanBatch.java
 ##########
 @@ -242,12 +242,12 @@ private IterOutcome internalNext() {
       if (implicitValues != null) {
         String projectMetadataColumn = context.getOptions().getOption(ExecConstants.IMPLICIT_PROJECT_METADATA_COLUMN_LABEL_VALIDATOR);
         if (recordCount > 0) {
-          // Sets the implicit value to false to signal that some results were returned and there is no need for creating an additional record.
-          implicitValues.replace(projectMetadataColumn, Boolean.FALSE.toString());
-        } else if (Boolean.parseBoolean(implicitValues.get(projectMetadataColumn))) {
+          // Sets the implicit value to null to signal that some results were returned and there is no need for creating an additional record.
+          implicitValues.replace(projectMetadataColumn, null);
+        } else if (Boolean.FALSE.toString().equals(implicitValues.get(projectMetadataColumn))) {
 
 Review comment:
   Any reason for use to be so specific here? The rule we've used in column properties is that `true` is if the value, converted to lower case, is "true", `false` otherwise. Else, we end up with values which are neither `true` nor `false` because they are some value like "FALSE".
   
   Maybe we need a utility conversion function: `isTrue(String value)` and `isFalse(String value)` where an invariant is that, for any string `s`, `isTrue(s) = !isFalse(s)`.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] arina-ielchiieva commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
arina-ielchiieva commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392590781
 
 

 ##########
 File path: exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/scan/TestFileScanFramework.java
 ##########
 @@ -25,6 +25,8 @@
 import java.util.Iterator;
 import java.util.List;
 
+import mockit.Mock;
+import mockit.MockUp;
 
 Review comment:
   Agree about mocks, please try to avoid them...

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r394693824
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/store/ColumnExplorer.java
 ##########
 @@ -292,39 +289,86 @@ public static int getPartitionDepth(FileSelection selection) {
    * @param includeFileImplicitColumns if file implicit columns should be included into the result
    * @param fs                         file system
    * @param index                      index of row group to populate
+   * @param start                      start of row group to populate
+   * @param length                     length of row group to populate
    * @return implicit columns map
    */
   public Map<String, String> populateImplicitAndInternalColumns(Path filePath,
       List<String> partitionValues, boolean includeFileImplicitColumns, FileSystem fs, int index, long start, long length) {
 
     Map<String, String> implicitValues =
-        new LinkedHashMap<>(populateImplicitColumns(filePath, partitionValues, includeFileImplicitColumns));
+        new LinkedHashMap<>(populateImplicitAndInternalColumns(filePath, partitionValues, includeFileImplicitColumns, fs));
 
-    selectedInternalColumns.forEach((key, value) -> {
-      switch (value) {
+    selectedInternalColumns.forEach(
+        (key, value) -> implicitValues.put(key, getImplicitColumnValue(value, filePath, fs, index, start, length)));
+
+    return implicitValues;
+  }
+
+  /**
+   * Returns implicit column value for specified implicit file column.
+   *
+   * @param column   implicit file column
+   * @param filePath file path, used to populate file implicit columns
+   * @param fs       file system
+   * @param index    row group index
+   * @param start    row group start
+   * @param length   row group length
+   * @return implicit column value for specified implicit file column
+   */
+  private static String getImplicitColumnValue(ImplicitFileColumn column, Path filePath,
+      FileSystem fs, Integer index, Long start, Long length) {
+    if (column instanceof ImplicitFileColumns) {
+      ImplicitFileColumns fileColumn = (ImplicitFileColumns) column;
+      return fileColumn.getValue(filePath);
+    } else if (column instanceof ImplicitInternalFileColumns) {
+      ImplicitInternalFileColumns fileColumn = (ImplicitInternalFileColumns) column;
+      switch (fileColumn) {
         case ROW_GROUP_INDEX:
-          implicitValues.put(key, String.valueOf(index));
-          break;
+          return index != null ? String.valueOf(index) : null;
         case ROW_GROUP_START:
-          implicitValues.put(key, String.valueOf(start));
-          break;
+          return start != null ? String.valueOf(start) : null;
         case ROW_GROUP_LENGTH:
-          implicitValues.put(key, String.valueOf(length));
-          break;
+          return length != null ? String.valueOf(length) : null;
         case PROJECT_METADATA:
-          implicitValues.put(key, Boolean.TRUE.toString());
-          break;
+          return Boolean.TRUE.toString();
         case LAST_MODIFIED_TIME:
           try {
-            implicitValues.put(key, String.valueOf(fs.getFileStatus(filePath).getModificationTime()));
+            return fs != null ? String.valueOf(fs.getFileStatus(filePath).getModificationTime()) : null;
 
 Review comment:
   Actually, the if the reader is "early schema", we might try to resolve the metadata columns before the file is opened. For example, for CSV without headers (and without provided schema), we know the one and only column will be `columns[]`, plus any implicit columns, so we can compute the final "output schema" without even opening the file.
   
   Since I'm reworking the whole scan framework, I'll clean up this corner case in that PR.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392708180
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/FilePushDownFilter.java
 ##########
 @@ -244,4 +247,15 @@ protected void doOnMatch(RelOptRuleCall call, FilterPrel filter, ProjectPrel pro
     final RelNode newFilter = filter.copy(filter.getTraitSet(), Collections.singletonList(newNode));
     call.transformTo(newFilter);
   }
+
+  /**
+   * Checks whether specified group scan supports filter push down.
+   *
+   * @param groupScan group scan to check
+   * @return {@code true} if specified group scan supports filter push down, {@code false} otherwise
+   */
+  private static boolean supportsFilterPushDown(GroupScan groupScan) {
+    return groupScan instanceof AbstractGroupScanWithMetadata
 
 Review comment:
   Good idea, thanks! Added such method into `GroupScan`.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392608927
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/scan/framework/ShimBatchReader.java
 ##########
 @@ -131,6 +131,8 @@ public boolean next() {
       eof = ! reader.next();
     }
 
+    readerOrchestrator.setEof(eof);
 
 Review comment:
   Is there a need for this? The design is that EOF is indicated when:
   
   * `open()` returns false. (As in an empty file with no schema. Think blank CSV file.)
   * `next()` returns false with no data. (Simple EOF.)
   * `next()` returns false with data. EOF is on the next batch.
   
   Eof handling logic is built into the `ScanOperatorExec` and, in particular, the `ReaderState`. I worry about adding additional state that must be kept in sync.
   
   What is the intent here? Maybe I can help you find a solution that uses the existing mechanisms.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r395142724
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/BaseParquetMetadataProvider.java
 ##########
 @@ -103,34 +110,39 @@
   // whether metadata for row groups should be collected to create files, partitions and table metadata
   private final boolean collectMetadata = false;
 
-  public BaseParquetMetadataProvider(List<ReadEntryWithPath> entries,
-                                     ParquetReaderConfig readerConfig,
-                                     String tableName,
-                                     Path tableLocation,
-                                     TupleMetadata schema,
-                                     DrillStatsTable statsTable) {
-    this(readerConfig, entries, tableName, tableLocation, schema, statsTable);
-  }
+  protected BaseParquetMetadataProvider(Builder<?> builder) {
+    if (builder.entries != null) {
+      // reuse previously stored metadata
+      this.entries = builder.entries;
+      this.tableName = builder.selectionRoot != null ? builder.selectionRoot.toUri().getPath() : "";
+      this.tableLocation = builder.selectionRoot;
+    } else if (builder.selection != null) {
+      this.entries = new ArrayList<>();
+      this.tableName = builder.selection.getSelectionRoot() != null ? builder.selection.getSelectionRoot().toUri().getPath() : "";
+      this.tableLocation = builder.selection.getSelectionRoot();
+    } else {
+      // case of hive parquet table
+      this.entries = new ArrayList<>();
+      this.tableName = null;
+      this.tableLocation = null;
+    }
 
-  public BaseParquetMetadataProvider(ParquetReaderConfig readerConfig,
-                                     List<ReadEntryWithPath> entries,
-                                     String tableName,
-                                     Path tableLocation,
-                                     TupleMetadata schema,
-                                     DrillStatsTable statsTable) {
-    this.entries = entries == null ? new ArrayList<>() : entries;
-    this.readerConfig = readerConfig == null ? ParquetReaderConfig.getDefaultInstance() : readerConfig;
-    this.tableName = tableName;
-    this.tableLocation = tableLocation;
-    this.schema = schema;
-    this.statsTable = statsTable;
-  }
+    SchemaProvider schemaProvider = builder.metadataProviderManager.getSchemaProvider();
+    TupleMetadata schema = builder.schema;
+    // schema passed into the builder has greater priority
+    if (schema == null && schemaProvider != null) {
+      try {
+        schema = schemaProvider.read().getSchema();
 
 Review comment:
   The highest priority has the inline schema. Also, the schema file may be provided in a table function, but it cannot be used with the inline schema for the same table within the same query. If the table function wasn't specified, but the schema file is present and `store.table.use_schema_file` was enabled, the schema file will be used.
   In the opposite case, metastore schema will be used. Only a single schema source may be used.
   
   Yes, `ANALYZE TABLE` will use the provided schema.
   
   In this builder, the `schema` field - is a target resolved schema after group scan deserialization. For the case when the schema is null - will be used `SchemaProvider` which provides schema either from table function or from schema file. If the schema is still null, metastore schema wouldn't be overridden and will be used.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392610581
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/easy/EasyFormatPlugin.java
 ##########
 @@ -315,16 +314,16 @@ private CloseableRecordBatch buildScanBatch(FragmentContext context,
       readers.add(recordReader);
       final List<String> partitionValues = ColumnExplorer.listPartitionValues(
           work.getPath(), scan.getSelectionRoot(), false);
-      final Map<String, String> implicitValues = columnExplorer.populateImplicitColumns(
-          work.getPath(), partitionValues, supportsFileImplicitColumns);
+      final Map<String, String> implicitValues = columnExplorer.populateImplicitAndInternalColumns(
 
 Review comment:
   The method name seems a bit long. Maybe just `populateColumns()`?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392705053
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/easy/EasyGroupScan.java
 ##########
 @@ -230,6 +230,16 @@ public void modifyFileSelection(FileSelection selection) {
     this.selection = selection;
   }
 
+  @Override
+  protected boolean supportsFileImplicitColumns() {
+    return true;
+  }
 
 Review comment:
   Thanks, added option to format plugin config and used it there.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on issue #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
vvysotskyi commented on issue #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#issuecomment-600661075
 
 
   @arina-ielchiieva, thanks a lot for the review!
   
   @paul-rogers, did you have a chance to take a look at new changes?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392706872
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/easy/EasyGroupScan.java
 ##########
 @@ -90,17 +95,14 @@ public EasyGroupScan(
       @JsonProperty("selectionRoot") Path selectionRoot,
       @JsonProperty("schema") TupleMetadata schema
       ) throws IOException {
-    super(ImpersonationUtil.resolveUserName(userName));
+    super(ImpersonationUtil.resolveUserName(userName), columns, ValueExpressions.BooleanExpression.TRUE);
     this.selection = FileSelection.create(null, files, selectionRoot);
     this.formatPlugin = engineRegistry.resolveFormat(storageConfig, formatConfig, EasyFormatPlugin.class);
     this.columns = columns == null ? ALL_COLUMNS : columns;
     this.selectionRoot = selectionRoot;
-    SimpleFileTableMetadataProviderBuilder builder =
-        (SimpleFileTableMetadataProviderBuilder)
-        new FileSystemMetadataProviderManager()
-        .builder(MetadataProviderManager.MetadataProviderKind.SCHEMA_STATS_ONLY);
 
-    this.metadataProvider = builder.withLocation(selection.getSelectionRoot())
+    this.metadataProvider = defaultTableMetadataProviderBuilder(new FileSystemMetadataProviderManager())
 
 Review comment:
   Caching here is done in the scope of the query - since it may be excessive to store metadata between queries submitting, sine even if several queries use the same tables, there is no guarantee that the same drillbit will be foreman for all these queries. Also, there may be a problem with cache actuality.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392607688
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/AbstractGroupScanWithMetadata.java
 ##########
 @@ -634,6 +642,62 @@ public NonInterestingColumnsMetadata getNonInterestingColumnsMetadata() {
     return nonInterestingColumnsMetadata;
   }
 
+  /**
+   * Returns {@link TableMetadataProviderBuilder} instance based on specified
+   * {@link MetadataProviderManager} source.
+   *
+   * @param source metadata provider manager
+   * @return {@link TableMetadataProviderBuilder} instance
+   */
+  protected abstract TableMetadataProviderBuilder tableMetadataProviderBuilder(MetadataProviderManager source);
+
+  /**
+   * Returns {@link TableMetadataProviderBuilder} instance which may provide metadata
+   * without using Drill Metastore.
+   *
+   * @param source metadata provider manager
+   * @return {@link TableMetadataProviderBuilder} instance
+   */
+  protected abstract TableMetadataProviderBuilder defaultTableMetadataProviderBuilder(MetadataProviderManager source);
+
+  /**
+   * Compares the last modified time of files obtained from specified selection with
+   * the Metastore last modified time to determine whether Metastore metadata
+   * is not outdated. If metadata is outdated, {@link MetadataException} will be thrown.
+   *
+   * @param selection the source of files to check
+   * @throws MetadataException if metadata is outdated
+   */
+  protected void checkMetadataConsistency(FileSelection selection, Configuration fsConf) throws IOException {
+    if (metadataProvider.checkMetadataVersion()) {
+      DrillFileSystem fileSystem =
+          ImpersonationUtil.createFileSystem(ImpersonationUtil.resolveUserName(getUserName()), fsConf);
+
+      List<FileStatus> fileStatuses = FileMetadataInfoCollector.getFileStatuses(selection, fileSystem);
+
+      long lastModifiedTime = metadataProvider.getTableMetadata().getLastModifiedTime();
+
+      Set<Path> removedFiles = new HashSet<>(metadataProvider.getFilesMetadataMap().keySet());
+      Set<Path> newFiles = new HashSet<>();
+
+      boolean isChanged = false;
+
+      for (FileStatus fileStatus : fileStatuses) {
+        if (!removedFiles.remove(Path.getPathWithoutSchemeAndAuthority(fileStatus.getPath()))) {
+          newFiles.add(fileStatus.getPath());
+        }
+        if (fileStatus.getModificationTime() > lastModifiedTime) {
+          isChanged = true;
+          break;
+        }
+      }
 
 Review comment:
   The above will be a very costly operation for millions of files. Do we have a way that an external system can ping us when a file is added? Or, that Drill can work with the files available until an external system tells us to refresh? And, can we do that refresh in parallel with the current metadata so that queries continue to run with the existing metadata as we gather the new set?
   
   Again, imagine the case that @dobesv recently explained: millions of files, data constantly arriving.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392708973
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/easy/EasyGroupScan.java
 ##########
 @@ -333,21 +338,73 @@ public boolean canPushdownProjects(List<SchemaPath> columns) {
   }
 
   @JsonProperty
+  @JsonIgnore(value = false)
   public TupleMetadata getSchema() {
     return getTableMetadata().getSchema();
   }
 
   @Override
-  @JsonIgnore
-  public TableMetadata getTableMetadata() {
-    if (tableMetadata == null) {
-      tableMetadata = metadataProvider.getTableMetadata();
+  public AnalyzeInfoProvider getAnalyzeInfoProvider() {
+    return new AnalyzeFileInfoProviderImpl(formatPlugin.getName());
+  }
+
+  @Override
+  protected GroupScanWithMetadataFilterer<?> getFilterer() {
+    return new EasyGroupScanFilterer(this);
+  }
+
+  @Override
+  protected FileTableMetadataProviderBuilder<?> tableMetadataProviderBuilder(MetadataProviderManager source) {
+    if (source.usesMetastore()) {
+      return new MetastoreFileTableMetadataProvider.Builder<>((MetastoreMetadataProviderManager) source);
+    } else {
+      return defaultTableMetadataProviderBuilder(source);
     }
-    return tableMetadata;
   }
 
   @Override
-  public TableMetadataProvider getMetadataProvider() {
-    return metadataProvider;
+  protected FileTableMetadataProviderBuilder<?> defaultTableMetadataProviderBuilder(MetadataProviderManager source) {
+    return new SimpleFileTableMetadataProvider.Builder(source);
+  }
+
+  /**
+   * Implementation of GroupScanWithMetadataFilterer which uses {@link EasyGroupScan} as source and
+   * builds {@link EasyGroupScan} instance with filtered metadata.
+   */
+  private static class EasyGroupScanFilterer extends GroupScanWithMetadataFilterer<EasyGroupScanFilterer> {
+
+    EasyGroupScanFilterer(EasyGroupScan source) {
+      super(source);
+    }
+
+    @Override
+    public AbstractGroupScanWithMetadata<?> build() {
+      EasyGroupScan newScan = new EasyGroupScan((EasyGroupScan) source);
+      newScan.tableMetadata = tableMetadata;
+      // updates common row count and nulls counts for every column
+      if (newScan.getTableMetadata() != null && files != null && newScan.getFilesMetadata().size() != files.size()) {
+        newScan.tableMetadata = TableMetadataUtils.updateRowCount(newScan.getTableMetadata(), files.values());
+      }
+      newScan.partitions = partitions;
+      newScan.segments = segments;
+      newScan.files = files;
+      newScan.matchAllMetadata = matchAllMetadata;
+      newScan.nonInterestingColumnsMetadata = nonInterestingColumnsMetadata;
+
+      newScan.fileSet = newScan.getFilesMetadata().keySet();
+      newScan.selection = FileSelection.create(null, new ArrayList<>(newScan.fileSet), newScan.selectionRoot);
+      try {
+        newScan.initFromSelection(newScan.selection, newScan.formatPlugin);
+      } catch (IOException e) {
+        throw new RuntimeException("Failed to initialize scan from the selection.", e);
+      }
+
+      return newScan;
+    }
+
+    @Override
+    protected EasyGroupScanFilterer self() {
+      return this;
+    }
 
 Review comment:
   It is required for the case of builders inheritance. With using generics in the base class and using this method, we can call methods from the base class and methods from the child class in the same chain.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r394696569
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/easy/EasyGroupScan.java
 ##########
 @@ -333,21 +338,73 @@ public boolean canPushdownProjects(List<SchemaPath> columns) {
   }
 
   @JsonProperty
+  @JsonIgnore(value = false)
   public TupleMetadata getSchema() {
     return getTableMetadata().getSchema();
   }
 
   @Override
-  @JsonIgnore
-  public TableMetadata getTableMetadata() {
-    if (tableMetadata == null) {
-      tableMetadata = metadataProvider.getTableMetadata();
+  public AnalyzeInfoProvider getAnalyzeInfoProvider() {
+    return new AnalyzeFileInfoProviderImpl(formatPlugin.getName());
+  }
+
+  @Override
+  protected GroupScanWithMetadataFilterer<?> getFilterer() {
+    return new EasyGroupScanFilterer(this);
+  }
+
+  @Override
+  protected FileTableMetadataProviderBuilder<?> tableMetadataProviderBuilder(MetadataProviderManager source) {
+    if (source.usesMetastore()) {
+      return new MetastoreFileTableMetadataProvider.Builder<>((MetastoreMetadataProviderManager) source);
+    } else {
+      return defaultTableMetadataProviderBuilder(source);
     }
-    return tableMetadata;
   }
 
   @Override
-  public TableMetadataProvider getMetadataProvider() {
-    return metadataProvider;
+  protected FileTableMetadataProviderBuilder<?> defaultTableMetadataProviderBuilder(MetadataProviderManager source) {
+    return new SimpleFileTableMetadataProvider.Builder(source);
+  }
+
+  /**
+   * Implementation of GroupScanWithMetadataFilterer which uses {@link EasyGroupScan} as source and
+   * builds {@link EasyGroupScan} instance with filtered metadata.
+   */
+  private static class EasyGroupScanFilterer extends GroupScanWithMetadataFilterer<EasyGroupScanFilterer> {
+
+    EasyGroupScanFilterer(EasyGroupScan source) {
+      super(source);
+    }
+
+    @Override
+    public AbstractGroupScanWithMetadata<?> build() {
+      EasyGroupScan newScan = new EasyGroupScan((EasyGroupScan) source);
+      newScan.tableMetadata = tableMetadata;
+      // updates common row count and nulls counts for every column
+      if (newScan.getTableMetadata() != null && files != null && newScan.getFilesMetadata().size() != files.size()) {
+        newScan.tableMetadata = TableMetadataUtils.updateRowCount(newScan.getTableMetadata(), files.values());
+      }
+      newScan.partitions = partitions;
+      newScan.segments = segments;
+      newScan.files = files;
+      newScan.matchAllMetadata = matchAllMetadata;
+      newScan.nonInterestingColumnsMetadata = nonInterestingColumnsMetadata;
 
 Review comment:
   Agreed. Elsewhere, we copy the group scan by passing the new value (filters, columns, etc.) into a constructor. Passing a half-dozen fields would be awkward. But, passing a single `MetadataSpec` (or whatever) would be clean:
   
   ```
   private MyGroupScan(MyGroupScan from, MetadataSpec mdSpec) {
     this.foo = from.foo;
     ...
     this.metadata = mdSpec;
   }
   
   public MyGroupScan newWithMetadata(MetadataSpec mdSpec) {
     return new MyGroupScan(this, mdSpec);
   }
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on issue #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
paul-rogers commented on issue #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#issuecomment-599018591
 
 
   Looks like a very cool feature. I've not been following the metadata implementation closely. Can you help get me up to speed by providing a bit more background information? What is the goal of this PR? Does it enable the format plugins to gather metadata if they choose, or does this PR actually add the metadata gathering itself?
   
   As I understand it, one of the things that the metadata framework does is to infer schema. Whether inferring schema for metadata, or inferring schema for a scan, we hit the same ambiguities. How does this code handle a schema conflict? Or, do we just assume the schema is whatever we get in the sample?
   
   How do we gather stats? Do we have the reader read all the data and have a downstream operator make sense of the data?
   
   For files that need a provided schema (CSV, say), do we apply stats to the columns after type conversion, or are stats gathered on the raw text values? That is, does this work use the provided schema if available? How does the provided schema relate to the metadata schema?
   
   What stats will we gather for non-Parquet files? How will we use them? Looks like there is code for partitions (have not looked in depth, so I may be wrong). Are we using stats for partition pruning? If so, how does that differ from the existing practice of just walking the directory tree?
   
   I think that if I understand some of this background I'll be able to do a more complete review. Thanks!
   
   Just so we're on the same page, I'm working on a revision to how we handle schema. Basically, the EVF-based operators will fully integrate the provided schema, and will be ready for a "defined" schema created by the planner (as in a classic query engine where the planner does all the schema calculations.) The idea is to use dynamic schema (what Drill has always done) when sampling the first row tells us all we need to know (as in Parquet), but to encourage a provided schema when sampling is not reliable (as in JSON.)
   
   This means that we have a flow something like this:
   
   ```
   User --> Provided Schema --> Scan <-- Reader <-- Input Source Schema
                                  |
                                  v
                        Scan output schema
   ```
   The scan output schema describes the data a scan will deliver. Hopefully, this is also the schema used by stats gathering.
   
   Do you see any potential conflicts between your metadata model and the above provided schema model?
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on issue #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
vvysotskyi commented on issue #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#issuecomment-599261516
 
 
   @paul-rogers, your comments as always highly useful - they either enforce making the code clearer and making the design better.
   
   Thanks a lot for the review and for revising the managed framework, I definitely need your help with the managed framework, I tried do not to break the logic a lot with my changes connected with handling metadata for empty files, but I agree that it looks somewhere slightly odd.
   Also, thanks for offering to use for now the design here. In this case, I'll address other CR comments ASAP and provide the intension for my changes in the managed framework classes.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392719079
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/scan/project/ReaderSchemaOrchestrator.java
 ##########
 @@ -118,9 +135,59 @@ public void endBatch() {
       // Fill in the null and metadata columns.
       populateNonDataColumns();
     }
+    if (projected) {
+      setProjectMetadata(null);
+    }
     rootTuple.setRowCount(tableContainer.getRecordCount());
   }
 
+  /**
+   * Updates {@code PROJECT_METADATA} implicit column value to {@code "FALSE"} to handle current batch as
+   * a batch with metadata information only for the case when this batch is first and empty.
+   */
 
 Review comment:
   I need to add this logic before implicit columns are resolved, so their values will be populated for a single row. I didn't find another place, where it may be done.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392730040
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetGroupScan.java
 ##########
 @@ -142,25 +141,25 @@ public ParquetGroupScan(String userName,
       metadataProviderManager = new FileSystemMetadataProviderManager();
     }
 
-    ParquetFileTableMetadataProviderBuilder builder =
-        (ParquetFileTableMetadataProviderBuilder) metadataProviderManager.builder(MetadataProviderManager.MetadataProviderKind.PARQUET_TABLE);
+    ParquetTableMetadataProviderBuilder<?> builder =
+        tableMetadataProviderBuilder(metadataProviderManager);
 
-    this.metadataProvider = builder
+    ParquetTableMetadataProvider metadataProvider = builder
         .withSelection(selection)
         .withReaderConfig(readerConfig)
         .withFileSystem(fs)
         .withCorrectCorruptedDates(formatConfig.areCorruptDatesAutoCorrected())
         .build();
 
-    ParquetTableMetadataProvider parquetTableMetadataProvider = (ParquetTableMetadataProvider) this.metadataProvider;
-    this.usedMetadataCache = parquetTableMetadataProvider.isUsedMetadataCache();
-    this.usedMetastore = parquetTableMetadataProvider instanceof MetastoreParquetTableMetadataProvider;
-    this.selectionRoot = parquetTableMetadataProvider.getSelectionRoot();
-    this.entries = parquetTableMetadataProvider.getEntries();
-    this.fileSet = parquetTableMetadataProvider.getFileSet();
+    this.usedMetadataCache = metadataProvider.isUsedMetadataCache();
+    this.usedMetastore = metadataProviderManager.usesMetastore();
+    this.selectionRoot = metadataProvider.getSelectionRoot();
+    this.entries = metadataProvider.getEntries();
+    this.fileSet = metadataProvider.getFileSet();
+    this.metadataProvider = metadataProvider;
 
 Review comment:
   There are only three common fields: `usedMetastore`, `fileSet` and `metadataProvider` which belong to the same parent class. `selectionRoot` is present in easy group scan and parquet one only, but not in their parent class.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392708620
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/BaseParquetMetadataProvider.java
 ##########
 @@ -103,34 +110,39 @@
   // whether metadata for row groups should be collected to create files, partitions and table metadata
   private final boolean collectMetadata = false;
 
-  public BaseParquetMetadataProvider(List<ReadEntryWithPath> entries,
-                                     ParquetReaderConfig readerConfig,
-                                     String tableName,
-                                     Path tableLocation,
-                                     TupleMetadata schema,
-                                     DrillStatsTable statsTable) {
-    this(readerConfig, entries, tableName, tableLocation, schema, statsTable);
-  }
+  protected BaseParquetMetadataProvider(Builder<?> builder) {
+    if (builder.entries != null) {
+      // reuse previously stored metadata
+      this.entries = builder.entries;
+      this.tableName = builder.selectionRoot != null ? builder.selectionRoot.toUri().getPath() : "";
+      this.tableLocation = builder.selectionRoot;
+    } else if (builder.selection != null) {
+      this.entries = new ArrayList<>();
+      this.tableName = builder.selection.getSelectionRoot() != null ? builder.selection.getSelectionRoot().toUri().getPath() : "";
+      this.tableLocation = builder.selection.getSelectionRoot();
+    } else {
+      // case of hive parquet table
+      this.entries = new ArrayList<>();
+      this.tableName = null;
+      this.tableLocation = null;
+    }
 
-  public BaseParquetMetadataProvider(ParquetReaderConfig readerConfig,
-                                     List<ReadEntryWithPath> entries,
-                                     String tableName,
-                                     Path tableLocation,
-                                     TupleMetadata schema,
-                                     DrillStatsTable statsTable) {
-    this.entries = entries == null ? new ArrayList<>() : entries;
-    this.readerConfig = readerConfig == null ? ParquetReaderConfig.getDefaultInstance() : readerConfig;
-    this.tableName = tableName;
-    this.tableLocation = tableLocation;
-    this.schema = schema;
-    this.statsTable = statsTable;
-  }
+    SchemaProvider schemaProvider = builder.metadataProviderManager.getSchemaProvider();
+    TupleMetadata schema = builder.schema;
+    // schema passed into the builder has greater priority
+    if (schema == null && schemaProvider != null) {
+      try {
+        schema = schemaProvider.read().getSchema();
 
 Review comment:
   This code is connected with schema provisioning. The schema will be used further in the scan using existing schema provisioning rules for formats that support it.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392609647
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/scan/project/ReaderSchemaOrchestrator.java
 ##########
 @@ -118,9 +135,59 @@ public void endBatch() {
       // Fill in the null and metadata columns.
       populateNonDataColumns();
     }
+    if (projected) {
+      setProjectMetadata(null);
+    }
     rootTuple.setRowCount(tableContainer.getRecordCount());
   }
 
+  /**
+   * Updates {@code PROJECT_METADATA} implicit column value to {@code "FALSE"} to handle current batch as
+   * a batch with metadata information only for the case when this batch is first and empty.
+   */
+  private boolean resolveProjectingMetadata() {
+    if (tableContainer.getRecordCount() == 0 && !hasSchema() && eof) {
+      if (setProjectMetadata(Boolean.FALSE.toString())) {
+        tableContainer.setValueCount(tableContainer.getRecordCount() + 1);
+        return true;
+      }
+    }
+    return false;
+  }
+
+  /**
+   * Sets {@code PROJECT_METADATA} implicit column value.
+   *
+   * @param value value to set
+   * @return {@code true} if {@code PROJECT_METADATA} implicit column
+   * is present in scan projection and its value is updated
+   */
+  private boolean setProjectMetadata(String value) {
 
 Review comment:
   I suspect I'll need to help with this. The design in this present mechanism is that you'd add a "metadata" (should be "implicit") column parser that will look for, or insert, the target column. You'd then add a mechanism to populate the resulting vector using the `ResultSetLoader`. See `MetadataManager` (which should be renamed to avoid confusion with your "real" metadata.) Then, a generic merge mechanism combines the reader data, file implicit columns, missing (null) columns, and your new implicit column.
   
   As it turns out, this is exactly the bit that I'm slightly revising at the moment. (The current design works, and did make explicit what the original code was trying to do, but ended up being a bit overly complex itself.) Still we can get your PR to work with the existing mechanism, then I can adjust this code as needed in my revision.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r395124840
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/easy/EasyGroupScan.java
 ##########
 @@ -333,21 +338,73 @@ public boolean canPushdownProjects(List<SchemaPath> columns) {
   }
 
   @JsonProperty
+  @JsonIgnore(value = false)
   public TupleMetadata getSchema() {
     return getTableMetadata().getSchema();
   }
 
   @Override
-  @JsonIgnore
-  public TableMetadata getTableMetadata() {
-    if (tableMetadata == null) {
-      tableMetadata = metadataProvider.getTableMetadata();
+  public AnalyzeInfoProvider getAnalyzeInfoProvider() {
+    return new AnalyzeFileInfoProviderImpl(formatPlugin.getName());
+  }
+
+  @Override
+  protected GroupScanWithMetadataFilterer<?> getFilterer() {
+    return new EasyGroupScanFilterer(this);
+  }
+
+  @Override
+  protected FileTableMetadataProviderBuilder<?> tableMetadataProviderBuilder(MetadataProviderManager source) {
+    if (source.usesMetastore()) {
+      return new MetastoreFileTableMetadataProvider.Builder<>((MetastoreMetadataProviderManager) source);
+    } else {
+      return defaultTableMetadataProviderBuilder(source);
     }
-    return tableMetadata;
   }
 
   @Override
-  public TableMetadataProvider getMetadataProvider() {
-    return metadataProvider;
+  protected FileTableMetadataProviderBuilder<?> defaultTableMetadataProviderBuilder(MetadataProviderManager source) {
+    return new SimpleFileTableMetadataProvider.Builder(source);
+  }
+
+  /**
+   * Implementation of GroupScanWithMetadataFilterer which uses {@link EasyGroupScan} as source and
+   * builds {@link EasyGroupScan} instance with filtered metadata.
+   */
+  private static class EasyGroupScanFilterer extends GroupScanWithMetadataFilterer<EasyGroupScanFilterer> {
+
+    EasyGroupScanFilterer(EasyGroupScan source) {
+      super(source);
+    }
+
+    @Override
+    public AbstractGroupScanWithMetadata<?> build() {
+      EasyGroupScan newScan = new EasyGroupScan((EasyGroupScan) source);
+      newScan.tableMetadata = tableMetadata;
+      // updates common row count and nulls counts for every column
+      if (newScan.getTableMetadata() != null && files != null && newScan.getFilesMetadata().size() != files.size()) {
+        newScan.tableMetadata = TableMetadataUtils.updateRowCount(newScan.getTableMetadata(), files.values());
+      }
+      newScan.partitions = partitions;
+      newScan.segments = segments;
+      newScan.files = files;
+      newScan.matchAllMetadata = matchAllMetadata;
+      newScan.nonInterestingColumnsMetadata = nonInterestingColumnsMetadata;
 
 Review comment:
   Yes, we have constructors with too many arguments, the most of the fields, set here aren't passed into the constructor, they either calculated when calling the corresponding method or assigned in the same way as in this code. Introducing a holder for them might make code more complicated.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392611244
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/easy/EasyGroupScan.java
 ##########
 @@ -230,6 +230,16 @@ public void modifyFileSelection(FileSelection selection) {
     this.selection = selection;
   }
 
+  @Override
+  protected boolean supportsFileImplicitColumns() {
+    return true;
+  }
 
 Review comment:
   This same info is passed to the EVF-based mechanism. Should this instead be specified on the format plugin and used both here and in the EVF builder so we get consistent results? Otherwise, how will folks like @cgivre know to specify the same thing twice?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392608747
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/scan/file/ImplicitColumnManager.java
 ##########
 @@ -199,6 +192,21 @@ public ImplicitColumnManager(OptionSet optionManager,
     }
   }
 
+  /**
+   * Specifies whether to plan based on the legacy meaning of "*". See
+   * <a href="https://issues.apache.org/jira/browse/DRILL-5542">DRILL-5542</a>.
+   * If true, then the star column <i>includes</i> implicit and partition
+   * columns. If false, then star matches <i>only</i> table columns.
+   *
+   * @param optionManager access to the options for this query; used
+   *                      too look up custom names for the metadata columns
+   * @param config        implicit column options
+   */
+  public ImplicitColumnManager(OptionSet optionManager,
 
 Review comment:
   For testing, would be much better to resolve options in the caller. I've tried to keep dependencies minimal in these classes to simplify testing and to separate responsibilities. This class cares about the flag, not its source.
   
   Also, the technical term for "star column" is "wildcard."

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392729363
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/easy/EasyGroupScan.java
 ##########
 @@ -333,21 +338,73 @@ public boolean canPushdownProjects(List<SchemaPath> columns) {
   }
 
   @JsonProperty
+  @JsonIgnore(value = false)
   public TupleMetadata getSchema() {
     return getTableMetadata().getSchema();
   }
 
   @Override
-  @JsonIgnore
-  public TableMetadata getTableMetadata() {
-    if (tableMetadata == null) {
-      tableMetadata = metadataProvider.getTableMetadata();
+  public AnalyzeInfoProvider getAnalyzeInfoProvider() {
+    return new AnalyzeFileInfoProviderImpl(formatPlugin.getName());
+  }
+
+  @Override
+  protected GroupScanWithMetadataFilterer<?> getFilterer() {
+    return new EasyGroupScanFilterer(this);
+  }
+
+  @Override
+  protected FileTableMetadataProviderBuilder<?> tableMetadataProviderBuilder(MetadataProviderManager source) {
+    if (source.usesMetastore()) {
+      return new MetastoreFileTableMetadataProvider.Builder<>((MetastoreMetadataProviderManager) source);
+    } else {
+      return defaultTableMetadataProviderBuilder(source);
     }
-    return tableMetadata;
   }
 
   @Override
-  public TableMetadataProvider getMetadataProvider() {
-    return metadataProvider;
+  protected FileTableMetadataProviderBuilder<?> defaultTableMetadataProviderBuilder(MetadataProviderManager source) {
+    return new SimpleFileTableMetadataProvider.Builder(source);
+  }
+
+  /**
+   * Implementation of GroupScanWithMetadataFilterer which uses {@link EasyGroupScan} as source and
+   * builds {@link EasyGroupScan} instance with filtered metadata.
+   */
+  private static class EasyGroupScanFilterer extends GroupScanWithMetadataFilterer<EasyGroupScanFilterer> {
+
+    EasyGroupScanFilterer(EasyGroupScan source) {
+      super(source);
+    }
+
+    @Override
+    public AbstractGroupScanWithMetadata<?> build() {
+      EasyGroupScan newScan = new EasyGroupScan((EasyGroupScan) source);
+      newScan.tableMetadata = tableMetadata;
+      // updates common row count and nulls counts for every column
+      if (newScan.getTableMetadata() != null && files != null && newScan.getFilesMetadata().size() != files.size()) {
+        newScan.tableMetadata = TableMetadataUtils.updateRowCount(newScan.getTableMetadata(), files.values());
+      }
+      newScan.partitions = partitions;
+      newScan.segments = segments;
+      newScan.files = files;
+      newScan.matchAllMetadata = matchAllMetadata;
+      newScan.nonInterestingColumnsMetadata = nonInterestingColumnsMetadata;
 
 Review comment:
   Thanks, it is a good idea for refactoring, but the issue here is that here we create a copy object of `EasyGroupScan` and reassigning some of its field values.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r395078523
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/store/ColumnExplorer.java
 ##########
 @@ -292,39 +289,86 @@ public static int getPartitionDepth(FileSelection selection) {
    * @param includeFileImplicitColumns if file implicit columns should be included into the result
    * @param fs                         file system
    * @param index                      index of row group to populate
+   * @param start                      start of row group to populate
+   * @param length                     length of row group to populate
    * @return implicit columns map
    */
   public Map<String, String> populateImplicitAndInternalColumns(Path filePath,
       List<String> partitionValues, boolean includeFileImplicitColumns, FileSystem fs, int index, long start, long length) {
 
     Map<String, String> implicitValues =
-        new LinkedHashMap<>(populateImplicitColumns(filePath, partitionValues, includeFileImplicitColumns));
+        new LinkedHashMap<>(populateImplicitAndInternalColumns(filePath, partitionValues, includeFileImplicitColumns, fs));
 
-    selectedInternalColumns.forEach((key, value) -> {
-      switch (value) {
+    selectedInternalColumns.forEach(
+        (key, value) -> implicitValues.put(key, getImplicitColumnValue(value, filePath, fs, index, start, length)));
+
+    return implicitValues;
+  }
+
+  /**
+   * Returns implicit column value for specified implicit file column.
+   *
+   * @param column   implicit file column
+   * @param filePath file path, used to populate file implicit columns
+   * @param fs       file system
+   * @param index    row group index
+   * @param start    row group start
+   * @param length   row group length
+   * @return implicit column value for specified implicit file column
+   */
+  private static String getImplicitColumnValue(ImplicitFileColumn column, Path filePath,
+      FileSystem fs, Integer index, Long start, Long length) {
+    if (column instanceof ImplicitFileColumns) {
+      ImplicitFileColumns fileColumn = (ImplicitFileColumns) column;
+      return fileColumn.getValue(filePath);
+    } else if (column instanceof ImplicitInternalFileColumns) {
+      ImplicitInternalFileColumns fileColumn = (ImplicitInternalFileColumns) column;
+      switch (fileColumn) {
         case ROW_GROUP_INDEX:
-          implicitValues.put(key, String.valueOf(index));
-          break;
+          return index != null ? String.valueOf(index) : null;
         case ROW_GROUP_START:
-          implicitValues.put(key, String.valueOf(start));
-          break;
+          return start != null ? String.valueOf(start) : null;
         case ROW_GROUP_LENGTH:
-          implicitValues.put(key, String.valueOf(length));
-          break;
+          return length != null ? String.valueOf(length) : null;
         case PROJECT_METADATA:
-          implicitValues.put(key, Boolean.TRUE.toString());
-          break;
+          return Boolean.TRUE.toString();
         case LAST_MODIFIED_TIME:
           try {
-            implicitValues.put(key, String.valueOf(fs.getFileStatus(filePath).getModificationTime()));
+            return fs != null ? String.valueOf(fs.getFileStatus(filePath).getModificationTime()) : null;
 
 Review comment:
   Ok, thanks!

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392612658
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/easy/EasyGroupScan.java
 ##########
 @@ -333,21 +338,73 @@ public boolean canPushdownProjects(List<SchemaPath> columns) {
   }
 
   @JsonProperty
+  @JsonIgnore(value = false)
   public TupleMetadata getSchema() {
     return getTableMetadata().getSchema();
   }
 
   @Override
-  @JsonIgnore
-  public TableMetadata getTableMetadata() {
-    if (tableMetadata == null) {
-      tableMetadata = metadataProvider.getTableMetadata();
+  public AnalyzeInfoProvider getAnalyzeInfoProvider() {
+    return new AnalyzeFileInfoProviderImpl(formatPlugin.getName());
+  }
+
+  @Override
+  protected GroupScanWithMetadataFilterer<?> getFilterer() {
+    return new EasyGroupScanFilterer(this);
+  }
+
+  @Override
+  protected FileTableMetadataProviderBuilder<?> tableMetadataProviderBuilder(MetadataProviderManager source) {
+    if (source.usesMetastore()) {
+      return new MetastoreFileTableMetadataProvider.Builder<>((MetastoreMetadataProviderManager) source);
+    } else {
+      return defaultTableMetadataProviderBuilder(source);
     }
-    return tableMetadata;
   }
 
   @Override
-  public TableMetadataProvider getMetadataProvider() {
-    return metadataProvider;
+  protected FileTableMetadataProviderBuilder<?> defaultTableMetadataProviderBuilder(MetadataProviderManager source) {
+    return new SimpleFileTableMetadataProvider.Builder(source);
+  }
+
+  /**
+   * Implementation of GroupScanWithMetadataFilterer which uses {@link EasyGroupScan} as source and
+   * builds {@link EasyGroupScan} instance with filtered metadata.
+   */
+  private static class EasyGroupScanFilterer extends GroupScanWithMetadataFilterer<EasyGroupScanFilterer> {
+
+    EasyGroupScanFilterer(EasyGroupScan source) {
+      super(source);
+    }
+
+    @Override
+    public AbstractGroupScanWithMetadata<?> build() {
+      EasyGroupScan newScan = new EasyGroupScan((EasyGroupScan) source);
+      newScan.tableMetadata = tableMetadata;
+      // updates common row count and nulls counts for every column
+      if (newScan.getTableMetadata() != null && files != null && newScan.getFilesMetadata().size() != files.size()) {
+        newScan.tableMetadata = TableMetadataUtils.updateRowCount(newScan.getTableMetadata(), files.values());
+      }
+      newScan.partitions = partitions;
+      newScan.segments = segments;
+      newScan.files = files;
+      newScan.matchAllMetadata = matchAllMetadata;
+      newScan.nonInterestingColumnsMetadata = nonInterestingColumnsMetadata;
 
 Review comment:
   Can you wrap the metadata items in a single object so you can pass it into the constructor or a new group scan instance, so the group scan itself is (mostly) immutable?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392571650
 
 

 ##########
 File path: exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/scan/TestFileScanFramework.java
 ##########
 @@ -25,6 +25,8 @@
 import java.util.Iterator;
 import java.util.List;
 
+import mockit.Mock;
+import mockit.MockUp;
 
 Review comment:
   These tests use non-existent data files for populating the implicit column values, but with new changes, the file system is used to obtain other implicit column values such as last modified time. So mock was used to avoid rewriting tests to use real files. 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392613250
 
 

 ##########
 File path: exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/scan/ScanTestUtils.java
 ##########
 @@ -52,6 +52,8 @@
   public static final String FILE_PATH_COL = "filepath";
   public static final String SUFFIX_COL = "suffix";
   public static final String PARTITION_COL = "dir";
+  public static final String LMT_COL = "lmt";
 
 Review comment:
   This one would benefit from documentation. "limit", "Last Metadata Time"? "Let Me Try"?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392683321
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/scan/file/FileMetadataColumn.java
 ##########
 @@ -65,6 +82,11 @@ public FileMetadataColumn(String name, FileMetadataColumnDefn defn) {
 
   @Override
   public MetadataColumn resolve(FileMetadata fileInfo, VectorSource source, int sourceIndex) {
-    return new FileMetadataColumn(name(), defn, fileInfo, source, sourceIndex);
+    if (value() == null) {
+      return new FileMetadataColumn(name(), defn, fileInfo, source, sourceIndex);
+    } else {
+      // case of metadata column, which should have null value independently of its column definition
+      return new FileMetadataColumn(name(), defn, (String) null, source, sourceIndex);
 
 Review comment:
   This change is required only for the case when we need to enforce the value of the implicit column to `null` before it is written to the vector.
   I have reworked changes in the `ReaderSchemaOrchestrator`, so this one is not required anymore.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392684331
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/scan/file/ImplicitColumnManager.java
 ##########
 @@ -81,40 +83,36 @@
     protected boolean useLegacyWildcardExpansion = true;
 
     /**
-      * Specify the selection root for a directory scan, if any.
-      * Used to populate partition columns. Also, specify the maximum
-      * partition depth.
-      *
-      * @param rootPath Hadoop file path for the directory
-      * @param partitionDepth maximum partition depth across all files
-      * within this logical scan operator (files in this scan may be
-      * shallower)
-      */
-
-     public void setSelectionRoot(Path rootPath) {
-       this.rootDir = rootPath;
-     }
-
-     public void setPartitionDepth(int partitionDepth) {
-       this.partitionCount = partitionDepth;
-     }
-
-     public void setFiles(List<Path> files) {
+     * Specify the selection root for a directory scan, if any.
+     * Used to populate partition columns. Also, specify the maximum
+     * partition depth.
+     *
+     * @param rootPath Hadoop file path for the directory
+     */
+    public void setSelectionRoot(Path rootPath) {
+      this.rootDir = rootPath;
+    }
+
+    public void setPartitionDepth(int partitionDepth) {
+      this.partitionCount = partitionDepth;
+    }
+
+    public void setFiles(List<Path> files) {
       this.files = files;
     }
 
-     /**
-      * Indicates whether to expand partition columns when the query contains a wildcard.
-      * Supports queries such as the following:<code><pre>
-      * select * from dfs.`partitioned-dir`</pre></code>
-      * In which the output columns will be (columns, dir0) if the partitioned directory
-      * has one level of nesting.
-      *
-      * See {@link TestImplicitFileColumns#testImplicitColumns}
-      */
-     public void useLegacyWildcardExpansion(boolean flag) {
-       useLegacyWildcardExpansion = flag;
-     }
+    /**
+     * Indicates whether to expand partition columns when the query contains a wildcard.
+     * Supports queries such as the following:<code><pre>
+     * select * from dfs.`partitioned-dir`</pre></code>
+     * In which the output columns will be (columns, dir0) if the partitioned directory
+     * has one level of nesting.
+     * <p>
+     * See {@code TestImplicitFileColumns#testImplicitColumns}
 
 Review comment:
   This class is available in test scope only, so just `@see` annotation will cause warnings for JavaDocs.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392610951
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/easy/EasyGroupScan.java
 ##########
 @@ -90,17 +95,14 @@ public EasyGroupScan(
       @JsonProperty("selectionRoot") Path selectionRoot,
       @JsonProperty("schema") TupleMetadata schema
       ) throws IOException {
-    super(ImpersonationUtil.resolveUserName(userName));
+    super(ImpersonationUtil.resolveUserName(userName), columns, ValueExpressions.BooleanExpression.TRUE);
     this.selection = FileSelection.create(null, files, selectionRoot);
     this.formatPlugin = engineRegistry.resolveFormat(storageConfig, formatConfig, EasyFormatPlugin.class);
     this.columns = columns == null ? ALL_COLUMNS : columns;
     this.selectionRoot = selectionRoot;
-    SimpleFileTableMetadataProviderBuilder builder =
-        (SimpleFileTableMetadataProviderBuilder)
-        new FileSystemMetadataProviderManager()
-        .builder(MetadataProviderManager.MetadataProviderKind.SCHEMA_STATS_ONLY);
 
-    this.metadataProvider = builder.withLocation(selection.getSelectionRoot())
+    this.metadataProvider = defaultTableMetadataProviderBuilder(new FileSystemMetadataProviderManager())
 
 Review comment:
   Will this load data from the metastore on each query? Or, is there some caching going on behind the scenes? If caching, is the cache global or tied to, say, the `DrillbitContext`?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392596477
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/easy/EasyGroupScan.java
 ##########
 @@ -333,21 +338,73 @@ public boolean canPushdownProjects(List<SchemaPath> columns) {
   }
 
   @JsonProperty
+  @JsonIgnore(value = false)
 
 Review comment:
   I have also removed `@JsonIgnore` from the parent class and removed unignoring the property.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392608554
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/scan/file/ImplicitColumnManager.java
 ##########
 @@ -81,40 +83,36 @@
     protected boolean useLegacyWildcardExpansion = true;
 
     /**
-      * Specify the selection root for a directory scan, if any.
-      * Used to populate partition columns. Also, specify the maximum
-      * partition depth.
-      *
-      * @param rootPath Hadoop file path for the directory
-      * @param partitionDepth maximum partition depth across all files
-      * within this logical scan operator (files in this scan may be
-      * shallower)
-      */
-
-     public void setSelectionRoot(Path rootPath) {
-       this.rootDir = rootPath;
-     }
-
-     public void setPartitionDepth(int partitionDepth) {
-       this.partitionCount = partitionDepth;
-     }
-
-     public void setFiles(List<Path> files) {
+     * Specify the selection root for a directory scan, if any.
+     * Used to populate partition columns. Also, specify the maximum
+     * partition depth.
+     *
+     * @param rootPath Hadoop file path for the directory
+     */
+    public void setSelectionRoot(Path rootPath) {
+      this.rootDir = rootPath;
+    }
+
+    public void setPartitionDepth(int partitionDepth) {
+      this.partitionCount = partitionDepth;
+    }
+
+    public void setFiles(List<Path> files) {
       this.files = files;
     }
 
-     /**
-      * Indicates whether to expand partition columns when the query contains a wildcard.
-      * Supports queries such as the following:<code><pre>
-      * select * from dfs.`partitioned-dir`</pre></code>
-      * In which the output columns will be (columns, dir0) if the partitioned directory
-      * has one level of nesting.
-      *
-      * See {@link TestImplicitFileColumns#testImplicitColumns}
-      */
-     public void useLegacyWildcardExpansion(boolean flag) {
-       useLegacyWildcardExpansion = flag;
-     }
+    /**
+     * Indicates whether to expand partition columns when the query contains a wildcard.
+     * Supports queries such as the following:<code><pre>
+     * select * from dfs.`partitioned-dir`</pre></code>
+     * In which the output columns will be (columns, dir0) if the partitioned directory
+     * has one level of nesting.
+     * <p>
+     * See {@code TestImplicitFileColumns#testImplicitColumns}
 
 Review comment:
   Maybe `@see`?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392607366
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/metastore/store/MetastoreFileTableMetadataProvider.java
 ##########
 @@ -291,124 +250,74 @@ public boolean checkMetadataVersion() {
     }
   }
 
-  private ParquetTableMetadataProvider getFallbackTableMetadataProvider() throws IOException {
-    if (fallback == null) {
-      fallback = fallbackBuilder == null ? null : fallbackBuilder.build();
-    }
-    return fallback;
-  }
+  public static class Builder<T extends Builder<T>> implements FileTableMetadataProviderBuilder<T> {
+    protected final MetastoreMetadataProviderManager metadataProviderManager;
 
-  private void throwIfChanged() {
-    if (basicTablesRequests.hasMetastoreTableInfoChanged(metastoreTableInfo)) {
-      throw MetadataException.of(MetadataException.MetadataExceptionType.INCONSISTENT_METADATA);
-    }
-  }
+    // builder for fallback ParquetFileTableMetadataProvider
+    // for the case when required metadata is absent in Metastore
+    protected final TableMetadataProviderBuilder fallback;
 
-  public static class Builder implements ParquetFileTableMetadataProviderBuilder {
-    private final MetastoreMetadataProviderManager metadataProviderManager;
+    protected TupleMetadata schema;
 
-    private List<ReadEntryWithPath> entries;
-    private DrillFileSystem fs;
-    private TupleMetadata schema;
+    protected List<String> paths;
 
     private FileSelection selection;
 
-    // builder for fallback ParquetFileTableMetadataProvider
-    // for the case when required metadata is absent in Metastore
-    private final ParquetFileTableMetadataProviderBuilder fallback;
+    private DrillFileSystem fs;
 
     public Builder(MetastoreMetadataProviderManager source) {
-      this.metadataProviderManager = source;
-      this.fallback = new ParquetTableMetadataProviderImpl.Builder(FileSystemMetadataProviderManager.init());
+      this(source, new SimpleFileTableMetadataProvider.Builder(FileSystemMetadataProviderManager.init()));
     }
 
-    @Override
-    public ParquetFileTableMetadataProviderBuilder withEntries(List<ReadEntryWithPath> entries) {
-      this.entries = entries;
-      fallback.withEntries(entries);
-      return this;
+    protected Builder(MetastoreMetadataProviderManager source, TableMetadataProviderBuilder fallback) {
+      this.metadataProviderManager = source;
+      this.fallback = fallback;
     }
 
     @Override
-    public ParquetFileTableMetadataProviderBuilder withSelectionRoot(Path selectionRoot) {
-      fallback.withSelectionRoot(selectionRoot);
-      return this;
+    public T withSchema(TupleMetadata schema) {
+      this.schema = schema;
+      return self();
     }
 
-    @Override
-    public ParquetFileTableMetadataProviderBuilder withCacheFileRoot(Path cacheFileRoot) {
-      fallback.withCacheFileRoot(cacheFileRoot);
-      return this;
+    public T withSelection(FileSelection selection) {
+      this.selection = selection;
+      return self();
     }
 
-    @Override
-    public ParquetFileTableMetadataProviderBuilder withReaderConfig(ParquetReaderConfig readerConfig) {
-      fallback.withReaderConfig(readerConfig);
-      return this;
+    public T withFileSystem(DrillFileSystem fs) {
+      this.fs = fs;
+      return self();
     }
 
-    @Override
-    public ParquetFileTableMetadataProviderBuilder withFileSystem(DrillFileSystem fs) {
-      fallback.withFileSystem(fs);
-      this.fs = fs;
-      return this;
+    protected T self() {
+      return (T) this;
     }
 
-    @Override
-    public ParquetFileTableMetadataProviderBuilder withCorrectCorruptedDates(boolean autoCorrectCorruptedDates) {
-      fallback.withCorrectCorruptedDates(autoCorrectCorruptedDates);
-      return this;
+    public MetastoreMetadataProviderManager metadataProviderManager() {
+      return metadataProviderManager;
     }
 
-    @Override
-    public ParquetFileTableMetadataProviderBuilder withSelection(FileSelection selection) {
-      fallback.withSelection(selection);
-      this.selection = selection;
-      return this;
+    public FileSelection selection() {
+      return selection;
     }
 
-    @Override
-    public ParquetFileTableMetadataProviderBuilder withSchema(TupleMetadata schema) {
-      fallback.withSchema(schema);
-      this.schema = schema;
-      return this;
+    public DrillFileSystem fs() {
+      return fs;
     }
 
     @Override
-    public ParquetTableMetadataProvider build() throws IOException {
-      MetastoreParquetTableMetadataProvider provider;
-      SchemaProvider schemaProvider = metadataProviderManager.getSchemaProvider();
-      ParquetMetadataProvider source = (ParquetTableMetadataProvider) metadataProviderManager.getTableMetadataProvider();
-
-      DrillStatsTable statsProvider = metadataProviderManager.getStatsProvider();
-      // schema passed into the builder has greater priority
-      try {
-        if (this.schema == null) {
-          schema = schemaProvider != null ? schemaProvider.read().getSchema() : null;
-        }
-      } catch (IOException e) {
-        logger.debug("Unable to deserialize schema from schema file for table: {}", metadataProviderManager.getTableInfo().name(), e);
-      }
-      if (entries == null) {
-        if (!selection.isExpandedFully()) {
-          entries = DrillFileSystemUtil.listFiles(fs, selection.getSelectionRoot(), true).stream()
-              .map(fileStatus -> new ReadEntryWithPath(Path.getPathWithoutSchemeAndAuthority(fileStatus.getPath())))
-              .collect(Collectors.toList());
-        } else {
-          entries = selection.getFiles().stream()
-              .map(Path::getPathWithoutSchemeAndAuthority)
-              .map(ReadEntryWithPath::new)
-              .collect(Collectors.toList());
-        }
-      }
-      provider = new MetastoreParquetTableMetadataProvider(entries, metadataProviderManager.getMetastoreRegistry(),
-          metadataProviderManager.getTableInfo(), schema, fallback, metadataProviderManager.getConfig(), statsProvider);
-      // store results into metadataProviderManager to be able to use them when creating new instances
-      // for the case when source wasn't provided or it contains less row group metadata than the provider
-      if (source == null || source.getRowGroupsMeta().size() < provider.getRowGroupsMeta().size()) {
-        metadataProviderManager.setTableMetadataProvider(provider);
+    public TableMetadataProvider build() throws IOException {
+      if (!selection().isExpandedFully()) {
+        paths = DrillFileSystemUtil.listFiles(fs, selection.getSelectionRoot(), true).stream()
+            .map(fileStatus -> Path.getPathWithoutSchemeAndAuthority(fileStatus.getPath()).toUri().getPath())
+            .collect(Collectors.toList());
+      } else {
+        paths = selection.getFiles().stream()
+            .map(path -> Path.getPathWithoutSchemeAndAuthority(path).toUri().getPath())
+            .collect(Collectors.toList());
 
 Review comment:
   Showing my ignorance here, but is this information cached between queries? Reading this info per file for very large data sets will be expensive (which is why Impala caches the metadata, which leads to the version issues discussed above.)

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392606338
 
 

 ##########
 File path: exec/java-exec/src/test/java/org/apache/drill/exec/metastore/TestMetastoreWithEasyFormatPlugin.java
 ##########
 @@ -0,0 +1,1111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.metastore;
+
+import org.apache.commons.io.FileUtils;
+import org.apache.drill.categories.MetastoreTest;
+import org.apache.drill.categories.SlowTest;
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.common.types.TypeProtos;
+import org.apache.drill.exec.ExecConstants;
+import org.apache.drill.exec.planner.physical.PlannerSettings;
+import org.apache.drill.exec.record.metadata.SchemaBuilder;
+import org.apache.drill.exec.record.metadata.TupleMetadata;
+import org.apache.drill.metastore.components.tables.MetastoreTableInfo;
+import org.apache.drill.metastore.metadata.BaseTableMetadata;
+import org.apache.drill.metastore.metadata.FileMetadata;
+import org.apache.drill.metastore.metadata.MetadataInfo;
+import org.apache.drill.metastore.metadata.MetadataType;
+import org.apache.drill.metastore.metadata.SegmentMetadata;
+import org.apache.drill.metastore.metadata.TableInfo;
+import org.apache.drill.metastore.statistics.ColumnStatistics;
+import org.apache.drill.metastore.statistics.ColumnStatisticsKind;
+import org.apache.drill.metastore.statistics.StatisticsHolder;
+import org.apache.drill.metastore.statistics.TableStatisticsKind;
+import org.apache.drill.shaded.guava.com.google.common.collect.ImmutableMap;
+import org.apache.drill.shaded.guava.com.google.common.collect.ImmutableSet;
+import org.apache.drill.test.ClusterFixture;
+import org.apache.drill.test.ClusterFixtureBuilder;
+import org.apache.drill.test.ClusterTest;
+import org.apache.hadoop.fs.Path;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.rules.ExpectedException;
+import org.junit.rules.TemporaryFolder;
+
+import java.io.File;
+import java.nio.file.Paths;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+import static org.apache.drill.exec.sql.TestMetastoreCommands.DIR0_1994_Q1_SEGMENT_COLUMN_STATISTICS;
+import static org.apache.drill.exec.sql.TestMetastoreCommands.DIR0_1994_SEGMENT_COLUMN_STATISTICS;
+import static org.apache.drill.exec.sql.TestMetastoreCommands.TABLE_COLUMN_STATISTICS;
+import static org.apache.drill.exec.sql.TestMetastoreCommands.TABLE_META_INFO;
+import static org.apache.drill.exec.sql.TestMetastoreCommands.getBaseTableMetadata;
+import static org.apache.drill.exec.sql.TestMetastoreCommands.getColumnStatistics;
+import static org.apache.drill.exec.sql.TestMetastoreCommands.getMaxLastModified;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+@Category({SlowTest.class, MetastoreTest.class})
+public class TestMetastoreWithEasyFormatPlugin extends ClusterTest {
+
+  private static String SCHEMA_STRING = "'inline=(" +
+      "`o_orderkey` INT not null, " +
+      "`o_custkey` INT not null, " +
+      "`o_orderstatus` VARCHAR not null, " +
+      "`o_totalprice` DOUBLE not null, " +
+      "`o_orderdate` DATE not null format \"yyyy-MM-dd''T''HH:mm:ss.SSSZZ\", " +
+      "`o_orderpriority` VARCHAR not null, " +
+      "`o_clerk` VARCHAR not null, " +
+      "`o_shippriority` INT not null, " +
+      "`o_comment` VARCHAR not null)'";
+
+  private static final TupleMetadata SCHEMA = new SchemaBuilder()
+      .add("o_orderkey", TypeProtos.MinorType.INT)
+      .add("o_custkey", TypeProtos.MinorType.INT)
+      .add("o_orderstatus", TypeProtos.MinorType.VARCHAR)
+      .add("o_totalprice", TypeProtos.MinorType.FLOAT8)
+      .add("o_orderdate", TypeProtos.MinorType.DATE)
+      .add("o_orderpriority", TypeProtos.MinorType.VARCHAR)
+      .add("o_clerk", TypeProtos.MinorType.VARCHAR)
+      .add("o_shippriority", TypeProtos.MinorType.INT)
+      .add("o_comment", TypeProtos.MinorType.VARCHAR)
+      .addNullable("dir0", TypeProtos.MinorType.VARCHAR)
+      .addNullable("dir1", TypeProtos.MinorType.VARCHAR)
+      .build();
+
+  @ClassRule
+  public static TemporaryFolder defaultFolder = new TemporaryFolder();
 
 Review comment:
   Thanks, removed it and replaced its usage with `dirTestWatcher`.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392684132
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/scan/file/FileMetadataColumnDefn.java
 ##########
 @@ -53,7 +53,7 @@ public String toString() {
   public MajorType dataType() {
     return MajorType.newBuilder()
         .setMinorType(MinorType.VARCHAR)
-        .setMode(DataMode.REQUIRED)
+        .setMode(defn.isOptional() ? DataMode.OPTIONAL : DataMode.REQUIRED)
 
 Review comment:
   This is a per-file value and it is required for new implicit column required for collecting metadata. This column may have a null value if file is empty. It is already handled using the "missing columns" mechanism, but this change is required to point that column is nullable.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392612998
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/BaseParquetMetadataProvider.java
 ##########
 @@ -103,34 +110,39 @@
   // whether metadata for row groups should be collected to create files, partitions and table metadata
   private final boolean collectMetadata = false;
 
-  public BaseParquetMetadataProvider(List<ReadEntryWithPath> entries,
-                                     ParquetReaderConfig readerConfig,
-                                     String tableName,
-                                     Path tableLocation,
-                                     TupleMetadata schema,
-                                     DrillStatsTable statsTable) {
-    this(readerConfig, entries, tableName, tableLocation, schema, statsTable);
-  }
+  protected BaseParquetMetadataProvider(Builder<?> builder) {
+    if (builder.entries != null) {
+      // reuse previously stored metadata
+      this.entries = builder.entries;
+      this.tableName = builder.selectionRoot != null ? builder.selectionRoot.toUri().getPath() : "";
+      this.tableLocation = builder.selectionRoot;
+    } else if (builder.selection != null) {
+      this.entries = new ArrayList<>();
+      this.tableName = builder.selection.getSelectionRoot() != null ? builder.selection.getSelectionRoot().toUri().getPath() : "";
+      this.tableLocation = builder.selection.getSelectionRoot();
+    } else {
+      // case of hive parquet table
+      this.entries = new ArrayList<>();
+      this.tableName = null;
+      this.tableLocation = null;
+    }
 
-  public BaseParquetMetadataProvider(ParquetReaderConfig readerConfig,
-                                     List<ReadEntryWithPath> entries,
-                                     String tableName,
-                                     Path tableLocation,
-                                     TupleMetadata schema,
-                                     DrillStatsTable statsTable) {
-    this.entries = entries == null ? new ArrayList<>() : entries;
-    this.readerConfig = readerConfig == null ? ParquetReaderConfig.getDefaultInstance() : readerConfig;
-    this.tableName = tableName;
-    this.tableLocation = tableLocation;
-    this.schema = schema;
-    this.statsTable = statsTable;
-  }
+    SchemaProvider schemaProvider = builder.metadataProviderManager.getSchemaProvider();
+    TupleMetadata schema = builder.schema;
+    // schema passed into the builder has greater priority
+    if (schema == null && schemaProvider != null) {
+      try {
+        schema = schemaProvider.read().getSchema();
 
 Review comment:
   What is the intent of the schema here? We want to plan a query. Historically, the schema is entirely dynamic: provided at run time. With the provided schema, the user can constrain that dynamic schema to use the provided schema definition (which can be strict or lenient.)
   
   With metadata, we are feeding the scan's "exhaust" (dynamically-discovered schema) back to itself as part of the plan.
   
   What do we expect the can do do with this schema? Ignore and continue to produce a dynamic schema (that is, discover any new columns, omit any old columns which have disappeared as the last of old files were deleted, etc.)
   
   Or, do we expect the scan to adjust its new scans to use the schema from prior scans? There is nothing in this PR to enforce such as schema. (As it turns out, that is what I'm currently doing.)
   
   So, if this PR has no way to use the schema, do we even care about it? Do we need to even fetch it? Or, is the schema here needed to understand the available metadata for partition pruning?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392719225
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/scan/project/ReaderSchemaOrchestrator.java
 ##########
 @@ -118,9 +135,59 @@ public void endBatch() {
       // Fill in the null and metadata columns.
       populateNonDataColumns();
     }
+    if (projected) {
+      setProjectMetadata(null);
+    }
     rootTuple.setRowCount(tableContainer.getRecordCount());
   }
 
+  /**
+   * Updates {@code PROJECT_METADATA} implicit column value to {@code "FALSE"} to handle current batch as
+   * a batch with metadata information only for the case when this batch is first and empty.
+   */
+  private boolean resolveProjectingMetadata() {
+    if (tableContainer.getRecordCount() == 0 && !hasSchema() && eof) {
+      if (setProjectMetadata(Boolean.FALSE.toString())) {
+        tableContainer.setValueCount(tableContainer.getRecordCount() + 1);
+        return true;
+      }
+    }
+    return false;
+  }
+
+  /**
+   * Sets {@code PROJECT_METADATA} implicit column value.
+   *
+   * @param value value to set
+   * @return {@code true} if {@code PROJECT_METADATA} implicit column
+   * is present in scan projection and its value is updated
+   */
+  private boolean setProjectMetadata(String value) {
 
 Review comment:
   It would be great if you can help with this part.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392716591
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/metastore/store/MetastoreFileTableMetadataProvider.java
 ##########
 @@ -291,124 +250,74 @@ public boolean checkMetadataVersion() {
     }
   }
 
-  private ParquetTableMetadataProvider getFallbackTableMetadataProvider() throws IOException {
-    if (fallback == null) {
-      fallback = fallbackBuilder == null ? null : fallbackBuilder.build();
-    }
-    return fallback;
-  }
+  public static class Builder<T extends Builder<T>> implements FileTableMetadataProviderBuilder<T> {
+    protected final MetastoreMetadataProviderManager metadataProviderManager;
 
-  private void throwIfChanged() {
-    if (basicTablesRequests.hasMetastoreTableInfoChanged(metastoreTableInfo)) {
-      throw MetadataException.of(MetadataException.MetadataExceptionType.INCONSISTENT_METADATA);
-    }
-  }
+    // builder for fallback ParquetFileTableMetadataProvider
+    // for the case when required metadata is absent in Metastore
+    protected final TableMetadataProviderBuilder fallback;
 
-  public static class Builder implements ParquetFileTableMetadataProviderBuilder {
-    private final MetastoreMetadataProviderManager metadataProviderManager;
+    protected TupleMetadata schema;
 
-    private List<ReadEntryWithPath> entries;
-    private DrillFileSystem fs;
-    private TupleMetadata schema;
+    protected List<String> paths;
 
     private FileSelection selection;
 
-    // builder for fallback ParquetFileTableMetadataProvider
-    // for the case when required metadata is absent in Metastore
-    private final ParquetFileTableMetadataProviderBuilder fallback;
+    private DrillFileSystem fs;
 
     public Builder(MetastoreMetadataProviderManager source) {
-      this.metadataProviderManager = source;
-      this.fallback = new ParquetTableMetadataProviderImpl.Builder(FileSystemMetadataProviderManager.init());
+      this(source, new SimpleFileTableMetadataProvider.Builder(FileSystemMetadataProviderManager.init()));
     }
 
-    @Override
-    public ParquetFileTableMetadataProviderBuilder withEntries(List<ReadEntryWithPath> entries) {
-      this.entries = entries;
-      fallback.withEntries(entries);
-      return this;
+    protected Builder(MetastoreMetadataProviderManager source, TableMetadataProviderBuilder fallback) {
+      this.metadataProviderManager = source;
+      this.fallback = fallback;
     }
 
     @Override
-    public ParquetFileTableMetadataProviderBuilder withSelectionRoot(Path selectionRoot) {
-      fallback.withSelectionRoot(selectionRoot);
-      return this;
+    public T withSchema(TupleMetadata schema) {
 
 Review comment:
   Good question. Drill has only two kinds of storage formats: Parquet and "easy". All formats except Parquet, including Avro, use `EasyGroupScan` and current `MetastoreFileTableMetadataProvider` class. But since there is a lot of common code for Parquet and "easy" format kinds, it was reasonable to inherit parquet class from this one and use the common code.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392610791
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/easy/EasyFormatPlugin.java
 ##########
 @@ -387,15 +386,10 @@ protected void initScanBuilder(FileScanBuilder builder, EasySubScan scan) {
 
     // Additional error context to identify this plugin
     builder.errorContext(
-        new CustomErrorContext() {
-          @Override
-          public void addContext(UserException.Builder builder) {
-            builder.addContext("Format plugin:", easyConfig.defaultName);
-            builder.addContext("Format plugin:",
-                EasyFormatPlugin.this.getClass().getSimpleName());
-            builder.addContext("Plugin config name:", getName());
-          }
-        });
+        currentBuilder -> currentBuilder
+            .addContext("Format plugin:", easyConfig.defaultName)
+            .addContext("Format plugin:", EasyFormatPlugin.this.getClass().getSimpleName())
+            .addContext("Plugin config name:", getName()));
 
 Review comment:
   Nice. If you are changing this, please remove the colons. `UserException` will now insert them automatically.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392613377
 
 

 ##########
 File path: exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/scan/TestFileScanFramework.java
 ##########
 @@ -25,6 +25,8 @@
 import java.util.Iterator;
 import java.util.List;
 
+import mockit.Mock;
+import mockit.MockUp;
 
 Review comment:
   What we want is to pass into the implicit column provider an interface that will answer file-specific questions. That is the thing that holds the file system. For testing, implement the interface to pretend to read files. For DFS, implement a real interface.
   
   This will also allow someone to implement their own version for systems not based on DFS.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392631705
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/scan/file/ImplicitColumnManager.java
 ##########
 @@ -199,6 +192,21 @@ public ImplicitColumnManager(OptionSet optionManager,
     }
   }
 
+  /**
+   * Specifies whether to plan based on the legacy meaning of "*". See
+   * <a href="https://issues.apache.org/jira/browse/DRILL-5542">DRILL-5542</a>.
+   * If true, then the star column <i>includes</i> implicit and partition
+   * columns. If false, then star matches <i>only</i> table columns.
+   *
+   * @param optionManager access to the options for this query; used
+   *                      too look up custom names for the metadata columns
+   * @param config        implicit column options
+   */
+  public ImplicitColumnManager(OptionSet optionManager,
 
 Review comment:
   Help me understand your intent. Older Drill versions always expanded implicit columns in wildcard queries. Then the project threw away those columns. We changed this so that the user asks for the columns they want. We could do this because Calcite changed (as I understand it) so that it will allow a projection list of the form `SELECT *, filename`.
   
   Does metadata need the implicit columns expanded for a wildcard query? Is this the behavior for the other, non-EVF readers?
   
   If so, we'll need a way to identify a metadata query (which needs those columns) vs. other queries (do not need them and don't want to pay the cost.)

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392608506
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/scan/file/FileMetadataColumnDefn.java
 ##########
 @@ -29,10 +29,10 @@
  */
 
 public class FileMetadataColumnDefn {
-  public final ImplicitFileColumns defn;
+  public final ImplicitFileColumn defn;
 
 Review comment:
   Thanks for cleaning up the implicit column definition! Always seemed odd for it to be plural.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392722720
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/easy/EasyGroupScan.java
 ##########
 @@ -124,13 +127,16 @@ public EasyGroupScan(
       // use file system metadata provider without specified schema and statistics
       metadataProviderManager = new FileSystemMetadataProviderManager();
     }
-    SimpleFileTableMetadataProviderBuilder builder =
-        (SimpleFileTableMetadataProviderBuilder) metadataProviderManager.builder(
-            MetadataProviderManager.MetadataProviderKind.SCHEMA_STATS_ONLY);
+    DrillFileSystem fs =
+        ImpersonationUtil.createFileSystem(ImpersonationUtil.resolveUserName(userName), formatPlugin.getFsConf());
 
-    this.metadataProvider = builder.withLocation(selection.getSelectionRoot())
+    this.metadataProvider = tableMetadataProviderBuilder(metadataProviderManager)
+        .withSelection(selection)
+        .withFileSystem(fs)
         .build();
+    this.usedMetastore = metadataProviderManager.usesMetastore();
     initFromSelection(selection, formatPlugin);
+    checkMetadataConsistency(selection, formatPlugin.getFsConf());
 
 Review comment:
   Using the metadata, we can prune files that shouldn't be read, or even drop the filters which always true, or drop unnecessary files for limit queries. This check is one of the fees for these optimizations, but I believe with them we can significantly improve query runtime.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] arina-ielchiieva commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
arina-ielchiieva commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392590806
 
 

 ##########
 File path: exec/java-exec/src/test/java/org/apache/drill/exec/metastore/TestMetastoreWithEasyFormatPlugin.java
 ##########
 @@ -0,0 +1,1111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.metastore;
+
+import org.apache.commons.io.FileUtils;
+import org.apache.drill.categories.MetastoreTest;
+import org.apache.drill.categories.SlowTest;
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.common.types.TypeProtos;
+import org.apache.drill.exec.ExecConstants;
+import org.apache.drill.exec.planner.physical.PlannerSettings;
+import org.apache.drill.exec.record.metadata.SchemaBuilder;
+import org.apache.drill.exec.record.metadata.TupleMetadata;
+import org.apache.drill.metastore.components.tables.MetastoreTableInfo;
+import org.apache.drill.metastore.metadata.BaseTableMetadata;
+import org.apache.drill.metastore.metadata.FileMetadata;
+import org.apache.drill.metastore.metadata.MetadataInfo;
+import org.apache.drill.metastore.metadata.MetadataType;
+import org.apache.drill.metastore.metadata.SegmentMetadata;
+import org.apache.drill.metastore.metadata.TableInfo;
+import org.apache.drill.metastore.statistics.ColumnStatistics;
+import org.apache.drill.metastore.statistics.ColumnStatisticsKind;
+import org.apache.drill.metastore.statistics.StatisticsHolder;
+import org.apache.drill.metastore.statistics.TableStatisticsKind;
+import org.apache.drill.shaded.guava.com.google.common.collect.ImmutableMap;
+import org.apache.drill.shaded.guava.com.google.common.collect.ImmutableSet;
+import org.apache.drill.test.ClusterFixture;
+import org.apache.drill.test.ClusterFixtureBuilder;
+import org.apache.drill.test.ClusterTest;
+import org.apache.hadoop.fs.Path;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.rules.ExpectedException;
+import org.junit.rules.TemporaryFolder;
+
+import java.io.File;
+import java.nio.file.Paths;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+import static org.apache.drill.exec.sql.TestMetastoreCommands.DIR0_1994_Q1_SEGMENT_COLUMN_STATISTICS;
+import static org.apache.drill.exec.sql.TestMetastoreCommands.DIR0_1994_SEGMENT_COLUMN_STATISTICS;
+import static org.apache.drill.exec.sql.TestMetastoreCommands.TABLE_COLUMN_STATISTICS;
+import static org.apache.drill.exec.sql.TestMetastoreCommands.TABLE_META_INFO;
+import static org.apache.drill.exec.sql.TestMetastoreCommands.getBaseTableMetadata;
+import static org.apache.drill.exec.sql.TestMetastoreCommands.getColumnStatistics;
+import static org.apache.drill.exec.sql.TestMetastoreCommands.getMaxLastModified;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+@Category({SlowTest.class, MetastoreTest.class})
+public class TestMetastoreWithEasyFormatPlugin extends ClusterTest {
+
+  private static String SCHEMA_STRING = "'inline=(" +
+      "`o_orderkey` INT not null, " +
+      "`o_custkey` INT not null, " +
+      "`o_orderstatus` VARCHAR not null, " +
+      "`o_totalprice` DOUBLE not null, " +
+      "`o_orderdate` DATE not null format \"yyyy-MM-dd''T''HH:mm:ss.SSSZZ\", " +
+      "`o_orderpriority` VARCHAR not null, " +
+      "`o_clerk` VARCHAR not null, " +
+      "`o_shippriority` INT not null, " +
+      "`o_comment` VARCHAR not null)'";
+
+  private static final TupleMetadata SCHEMA = new SchemaBuilder()
+      .add("o_orderkey", TypeProtos.MinorType.INT)
+      .add("o_custkey", TypeProtos.MinorType.INT)
+      .add("o_orderstatus", TypeProtos.MinorType.VARCHAR)
+      .add("o_totalprice", TypeProtos.MinorType.FLOAT8)
+      .add("o_orderdate", TypeProtos.MinorType.DATE)
+      .add("o_orderpriority", TypeProtos.MinorType.VARCHAR)
+      .add("o_clerk", TypeProtos.MinorType.VARCHAR)
+      .add("o_shippriority", TypeProtos.MinorType.INT)
+      .add("o_comment", TypeProtos.MinorType.VARCHAR)
+      .addNullable("dir0", TypeProtos.MinorType.VARCHAR)
+      .addNullable("dir1", TypeProtos.MinorType.VARCHAR)
+      .build();
+
+  @ClassRule
+  public static TemporaryFolder defaultFolder = new TemporaryFolder();
 
 Review comment:
   You already have `dirTestWatcher` from `ClusterTest`, please use it.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392613191
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/ParquetGroupScan.java
 ##########
 @@ -142,25 +141,25 @@ public ParquetGroupScan(String userName,
       metadataProviderManager = new FileSystemMetadataProviderManager();
     }
 
-    ParquetFileTableMetadataProviderBuilder builder =
-        (ParquetFileTableMetadataProviderBuilder) metadataProviderManager.builder(MetadataProviderManager.MetadataProviderKind.PARQUET_TABLE);
+    ParquetTableMetadataProviderBuilder<?> builder =
+        tableMetadataProviderBuilder(metadataProviderManager);
 
-    this.metadataProvider = builder
+    ParquetTableMetadataProvider metadataProvider = builder
         .withSelection(selection)
         .withReaderConfig(readerConfig)
         .withFileSystem(fs)
         .withCorrectCorruptedDates(formatConfig.areCorruptDatesAutoCorrected())
         .build();
 
-    ParquetTableMetadataProvider parquetTableMetadataProvider = (ParquetTableMetadataProvider) this.metadataProvider;
-    this.usedMetadataCache = parquetTableMetadataProvider.isUsedMetadataCache();
-    this.usedMetastore = parquetTableMetadataProvider instanceof MetastoreParquetTableMetadataProvider;
-    this.selectionRoot = parquetTableMetadataProvider.getSelectionRoot();
-    this.entries = parquetTableMetadataProvider.getEntries();
-    this.fileSet = parquetTableMetadataProvider.getFileSet();
+    this.usedMetadataCache = metadataProvider.isUsedMetadataCache();
+    this.usedMetastore = metadataProviderManager.usesMetastore();
+    this.selectionRoot = metadataProvider.getSelectionRoot();
+    this.entries = metadataProvider.getEntries();
+    this.fileSet = metadataProvider.getFileSet();
+    this.metadataProvider = metadataProvider;
 
 Review comment:
   These are the same six attributes mentioned for the generic Easy group scan. Rather than duplicating them, can we encapsulate them in a single `MetadataResult` class? (You pick the name.)

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] arina-ielchiieva commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
arina-ielchiieva commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392590666
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/BaseParquetMetadataProvider.java
 ##########
 @@ -103,34 +110,39 @@
   // whether metadata for row groups should be collected to create files, partitions and table metadata
   private final boolean collectMetadata = false;
 
-  public BaseParquetMetadataProvider(List<ReadEntryWithPath> entries,
-                                     ParquetReaderConfig readerConfig,
-                                     String tableName,
-                                     Path tableLocation,
-                                     TupleMetadata schema,
-                                     DrillStatsTable statsTable) {
-    this(readerConfig, entries, tableName, tableLocation, schema, statsTable);
-  }
+  protected BaseParquetMetadataProvider(Builder<?> builder) {
+    if (builder.entries != null) {
+      // reuse previously stored metadata
+      this.entries = builder.entries;
+      this.tableName = builder.selectionRoot != null ? builder.selectionRoot.toUri().getPath() : "";
+      this.tableLocation = builder.selectionRoot;
+    } else if (builder.selection != null) {
+      this.entries = new ArrayList<>();
+      this.tableName = builder.selection.getSelectionRoot() != null ? builder.selection.getSelectionRoot().toUri().getPath() : "";
+      this.tableLocation = builder.selection.getSelectionRoot();
+    } else {
+      // case of hive parquet table
+      this.entries = new ArrayList<>();
+      this.tableName = null;
+      this.tableLocation = null;
+    }
 
-  public BaseParquetMetadataProvider(ParquetReaderConfig readerConfig,
-                                     List<ReadEntryWithPath> entries,
-                                     String tableName,
-                                     Path tableLocation,
-                                     TupleMetadata schema,
-                                     DrillStatsTable statsTable) {
-    this.entries = entries == null ? new ArrayList<>() : entries;
-    this.readerConfig = readerConfig == null ? ParquetReaderConfig.getDefaultInstance() : readerConfig;
-    this.tableName = tableName;
-    this.tableLocation = tableLocation;
-    this.schema = schema;
-    this.statsTable = statsTable;
-  }
+    SchemaProvider schemaProvider = builder.metadataProviderManager.getSchemaProvider();
+    TupleMetadata schema = builder.schema;
+    // schema passed into the builder has greater priority
+    if (schema == null && schemaProvider != null) {
+      try {
+        schema = schemaProvider.read().getSchema();
+      } catch (IOException e) {
+        logger.debug("Unable to read schema from schema provider [{}]: {}",
 
 Review comment:
   I guess this should be a warning actually, you also should consider adding explanation of the next steps that if provided schema is ignored, you are moving without schema.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392561161
 
 

 ##########
 File path: exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/scan/TestFileScanFramework.java
 ##########
 @@ -25,6 +25,8 @@
 import java.util.Iterator;
 import java.util.List;
 
+import mockit.Mock;
+import mockit.MockUp;
 
 Review comment:
   Please, if possible, try to avoid the use of Mocks here. These tend to work poorly in Eclipse. These tests try to use classes and interfaces designed to allow proper text fixture classes rather than dynamic mocks.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r395153171
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/scan/file/FileMetadata.java
 ##########
 @@ -31,9 +32,15 @@
 
   private final Path filePath;
   private final String[] dirPath;
+  private final FileSystem fs;
 
   public FileMetadata(Path filePath, Path selectionRoot) {
+    this(filePath, selectionRoot, null);
+  }
+
+  public FileMetadata(Path filePath, Path selectionRoot, FileSystem fs) {
 
 Review comment:
   This class has good JavaDoc which explains the relation between `filePath` and `selectionRoot`. The file system should accept paths with or without file system prefix, so no additional relation is introduced.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392610708
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/easy/EasyFormatPlugin.java
 ##########
 @@ -315,16 +314,16 @@ private CloseableRecordBatch buildScanBatch(FragmentContext context,
       readers.add(recordReader);
       final List<String> partitionValues = ColumnExplorer.listPartitionValues(
           work.getPath(), scan.getSelectionRoot(), false);
-      final Map<String, String> implicitValues = columnExplorer.populateImplicitColumns(
-          work.getPath(), partitionValues, supportsFileImplicitColumns);
+      final Map<String, String> implicitValues = columnExplorer.populateImplicitAndInternalColumns(
+          work.getPath(), partitionValues, supportsFileImplicitColumns, dfs);
 
 Review comment:
   As we discussed, this can fail with the present design. We should catch the exception and translate it to something useful for the user.
   
   BTW: how might you test the missing file case so we can see what error is returned? Maybe we need a way to inject an error when checking the modification date?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r395113673
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/easy/EasyGroupScan.java
 ##########
 @@ -333,21 +338,73 @@ public boolean canPushdownProjects(List<SchemaPath> columns) {
   }
 
   @JsonProperty
+  @JsonIgnore(value = false)
   public TupleMetadata getSchema() {
     return getTableMetadata().getSchema();
   }
 
   @Override
-  @JsonIgnore
-  public TableMetadata getTableMetadata() {
-    if (tableMetadata == null) {
-      tableMetadata = metadataProvider.getTableMetadata();
+  public AnalyzeInfoProvider getAnalyzeInfoProvider() {
+    return new AnalyzeFileInfoProviderImpl(formatPlugin.getName());
+  }
+
+  @Override
+  protected GroupScanWithMetadataFilterer<?> getFilterer() {
+    return new EasyGroupScanFilterer(this);
+  }
+
+  @Override
+  protected FileTableMetadataProviderBuilder<?> tableMetadataProviderBuilder(MetadataProviderManager source) {
+    if (source.usesMetastore()) {
+      return new MetastoreFileTableMetadataProvider.Builder<>((MetastoreMetadataProviderManager) source);
+    } else {
+      return defaultTableMetadataProviderBuilder(source);
     }
-    return tableMetadata;
   }
 
   @Override
-  public TableMetadataProvider getMetadataProvider() {
-    return metadataProvider;
+  protected FileTableMetadataProviderBuilder<?> defaultTableMetadataProviderBuilder(MetadataProviderManager source) {
+    return new SimpleFileTableMetadataProvider.Builder(source);
+  }
+
+  /**
+   * Implementation of GroupScanWithMetadataFilterer which uses {@link EasyGroupScan} as source and
+   * builds {@link EasyGroupScan} instance with filtered metadata.
+   */
+  private static class EasyGroupScanFilterer extends GroupScanWithMetadataFilterer<EasyGroupScanFilterer> {
+
+    EasyGroupScanFilterer(EasyGroupScan source) {
+      super(source);
+    }
+
+    @Override
+    public AbstractGroupScanWithMetadata<?> build() {
+      EasyGroupScan newScan = new EasyGroupScan((EasyGroupScan) source);
+      newScan.tableMetadata = tableMetadata;
+      // updates common row count and nulls counts for every column
+      if (newScan.getTableMetadata() != null && files != null && newScan.getFilesMetadata().size() != files.size()) {
+        newScan.tableMetadata = TableMetadataUtils.updateRowCount(newScan.getTableMetadata(), files.values());
+      }
+      newScan.partitions = partitions;
+      newScan.segments = segments;
+      newScan.files = files;
+      newScan.matchAllMetadata = matchAllMetadata;
+      newScan.nonInterestingColumnsMetadata = nonInterestingColumnsMetadata;
+
+      newScan.fileSet = newScan.getFilesMetadata().keySet();
+      newScan.selection = FileSelection.create(null, new ArrayList<>(newScan.fileSet), newScan.selectionRoot);
+      try {
+        newScan.initFromSelection(newScan.selection, newScan.formatPlugin);
+      } catch (IOException e) {
+        throw new RuntimeException("Failed to initialize scan from the selection.", e);
+      }
+
+      return newScan;
+    }
+
+    @Override
+    protected EasyGroupScanFilterer self() {
+      return this;
+    }
 
 Review comment:
   Its parent class is `AbstractGroupScanWithMetadata.GroupScanWithMetadataFilterer`, it is used in `AbstractGroupScanWithMetadata.applyFilter()` method. This class is extended by `AbstractParquetGroupScan.RowGroupScanFilterer` where additional methods provided. `AbstractParquetGroupScan.RowGroupScanFilterer` is used in `AbstractParquetGroupScan.applyFilter()`.
   
   Another example of such an approach is `BaseMetadataBuilder` class.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] arina-ielchiieva commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
arina-ielchiieva commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392590481
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/easy/EasyGroupScan.java
 ##########
 @@ -333,21 +338,73 @@ public boolean canPushdownProjects(List<SchemaPath> columns) {
   }
 
   @JsonProperty
+  @JsonIgnore(value = false)
 
 Review comment:
   I see this for the second time, I guess we might need som explanation behind this change in PR and in code...

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392608447
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/scan/file/FileMetadataColumnDefn.java
 ##########
 @@ -53,7 +53,7 @@ public String toString() {
   public MajorType dataType() {
     return MajorType.newBuilder()
         .setMinorType(MinorType.VARCHAR)
-        .setMode(DataMode.REQUIRED)
+        .setMode(defn.isOptional() ? DataMode.OPTIONAL : DataMode.REQUIRED)
 
 Review comment:
   Does the `isOptional()` apply per-file or per-query? That is, if you've added a new implicit column `foo`, will `foo` be set on some files but not others? Or, is it either available if using metadata, but not available without metadata?
   
   If the value is per-file optional, then using an `OPTIONAL` mode makes sense. Though, it is hard to see how we have file attributes available for some files but not others. If per-query (query planned with metadata or without), then the column could be handled using the "missing columns" mechanism: if the implicit value is not available, just treat it as missing and let the "null column manager" fill it with nulls.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392610254
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/store/ColumnExplorer.java
 ##########
 @@ -292,39 +289,86 @@ public static int getPartitionDepth(FileSelection selection) {
    * @param includeFileImplicitColumns if file implicit columns should be included into the result
    * @param fs                         file system
    * @param index                      index of row group to populate
+   * @param start                      start of row group to populate
+   * @param length                     length of row group to populate
    * @return implicit columns map
    */
   public Map<String, String> populateImplicitAndInternalColumns(Path filePath,
       List<String> partitionValues, boolean includeFileImplicitColumns, FileSystem fs, int index, long start, long length) {
 
     Map<String, String> implicitValues =
-        new LinkedHashMap<>(populateImplicitColumns(filePath, partitionValues, includeFileImplicitColumns));
+        new LinkedHashMap<>(populateImplicitAndInternalColumns(filePath, partitionValues, includeFileImplicitColumns, fs));
 
-    selectedInternalColumns.forEach((key, value) -> {
-      switch (value) {
+    selectedInternalColumns.forEach(
+        (key, value) -> implicitValues.put(key, getImplicitColumnValue(value, filePath, fs, index, start, length)));
+
+    return implicitValues;
+  }
+
+  /**
+   * Returns implicit column value for specified implicit file column.
+   *
+   * @param column   implicit file column
+   * @param filePath file path, used to populate file implicit columns
+   * @param fs       file system
+   * @param index    row group index
+   * @param start    row group start
+   * @param length   row group length
+   * @return implicit column value for specified implicit file column
+   */
+  private static String getImplicitColumnValue(ImplicitFileColumn column, Path filePath,
+      FileSystem fs, Integer index, Long start, Long length) {
+    if (column instanceof ImplicitFileColumns) {
+      ImplicitFileColumns fileColumn = (ImplicitFileColumns) column;
+      return fileColumn.getValue(filePath);
+    } else if (column instanceof ImplicitInternalFileColumns) {
+      ImplicitInternalFileColumns fileColumn = (ImplicitInternalFileColumns) column;
+      switch (fileColumn) {
         case ROW_GROUP_INDEX:
-          implicitValues.put(key, String.valueOf(index));
-          break;
+          return index != null ? String.valueOf(index) : null;
         case ROW_GROUP_START:
-          implicitValues.put(key, String.valueOf(start));
-          break;
+          return start != null ? String.valueOf(start) : null;
         case ROW_GROUP_LENGTH:
-          implicitValues.put(key, String.valueOf(length));
-          break;
+          return length != null ? String.valueOf(length) : null;
         case PROJECT_METADATA:
-          implicitValues.put(key, Boolean.TRUE.toString());
-          break;
+          return Boolean.TRUE.toString();
         case LAST_MODIFIED_TIME:
           try {
-            implicitValues.put(key, String.valueOf(fs.getFileStatus(filePath).getModificationTime()));
+            return fs != null ? String.valueOf(fs.getFileStatus(filePath).getModificationTime()) : null;
 
 Review comment:
   Here is exactly the issue that concerned me above. If the file disappeared, it will now be this code that detects it. Notice the exception is very generic: it will provide the user no information about the problem.
   
   The `ScanOperatorExec` is designed to handle this case: if a file does not exist, the batch reader's `open()` should return `false`. The `ScanOperatorExec` then skips over that file and moves to the next. There is some tricky code to deal with schemas in that case.
   
   Perhaps a better solution here is to return 0 if the file is not found, realizing that something else will deal with the missing file before any harm can be done with the false modification time.
   
   Why do we handle missing files this way? On the desktop, a missing file might want to fail the query. (I had only one file and it was delete. That is bad.) But, on a system such as that described by @dobesv, data will arrive and expire all the time. Failing an hour-long query because one file of millions is missing might fall more on the "bug" than "feature" side of the spectrum. 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392606629
 
 

 ##########
 File path: exec/java-exec/src/test/java/org/apache/drill/exec/metastore/TestMetastoreWithEasyFormatPlugin.java
 ##########
 @@ -0,0 +1,1111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.metastore;
+
+import org.apache.commons.io.FileUtils;
+import org.apache.drill.categories.MetastoreTest;
+import org.apache.drill.categories.SlowTest;
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.common.types.TypeProtos;
+import org.apache.drill.exec.ExecConstants;
+import org.apache.drill.exec.planner.physical.PlannerSettings;
+import org.apache.drill.exec.record.metadata.SchemaBuilder;
+import org.apache.drill.exec.record.metadata.TupleMetadata;
+import org.apache.drill.metastore.components.tables.MetastoreTableInfo;
+import org.apache.drill.metastore.metadata.BaseTableMetadata;
+import org.apache.drill.metastore.metadata.FileMetadata;
+import org.apache.drill.metastore.metadata.MetadataInfo;
+import org.apache.drill.metastore.metadata.MetadataType;
+import org.apache.drill.metastore.metadata.SegmentMetadata;
+import org.apache.drill.metastore.metadata.TableInfo;
+import org.apache.drill.metastore.statistics.ColumnStatistics;
+import org.apache.drill.metastore.statistics.ColumnStatisticsKind;
+import org.apache.drill.metastore.statistics.StatisticsHolder;
+import org.apache.drill.metastore.statistics.TableStatisticsKind;
+import org.apache.drill.shaded.guava.com.google.common.collect.ImmutableMap;
+import org.apache.drill.shaded.guava.com.google.common.collect.ImmutableSet;
+import org.apache.drill.test.ClusterFixture;
+import org.apache.drill.test.ClusterFixtureBuilder;
+import org.apache.drill.test.ClusterTest;
+import org.apache.hadoop.fs.Path;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.rules.ExpectedException;
+import org.junit.rules.TemporaryFolder;
+
+import java.io.File;
+import java.nio.file.Paths;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+import static org.apache.drill.exec.sql.TestMetastoreCommands.DIR0_1994_Q1_SEGMENT_COLUMN_STATISTICS;
+import static org.apache.drill.exec.sql.TestMetastoreCommands.DIR0_1994_SEGMENT_COLUMN_STATISTICS;
+import static org.apache.drill.exec.sql.TestMetastoreCommands.TABLE_COLUMN_STATISTICS;
+import static org.apache.drill.exec.sql.TestMetastoreCommands.TABLE_META_INFO;
+import static org.apache.drill.exec.sql.TestMetastoreCommands.getBaseTableMetadata;
+import static org.apache.drill.exec.sql.TestMetastoreCommands.getColumnStatistics;
+import static org.apache.drill.exec.sql.TestMetastoreCommands.getMaxLastModified;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+@Category({SlowTest.class, MetastoreTest.class})
+public class TestMetastoreWithEasyFormatPlugin extends ClusterTest {
+
+  private static String SCHEMA_STRING = "'inline=(" +
+      "`o_orderkey` INT not null, " +
+      "`o_custkey` INT not null, " +
+      "`o_orderstatus` VARCHAR not null, " +
+      "`o_totalprice` DOUBLE not null, " +
+      "`o_orderdate` DATE not null format \"yyyy-MM-dd''T''HH:mm:ss.SSSZZ\", " +
+      "`o_orderpriority` VARCHAR not null, " +
+      "`o_clerk` VARCHAR not null, " +
+      "`o_shippriority` INT not null, " +
+      "`o_comment` VARCHAR not null)'";
+
+  private static final TupleMetadata SCHEMA = new SchemaBuilder()
+      .add("o_orderkey", TypeProtos.MinorType.INT)
+      .add("o_custkey", TypeProtos.MinorType.INT)
+      .add("o_orderstatus", TypeProtos.MinorType.VARCHAR)
+      .add("o_totalprice", TypeProtos.MinorType.FLOAT8)
+      .add("o_orderdate", TypeProtos.MinorType.DATE)
+      .add("o_orderpriority", TypeProtos.MinorType.VARCHAR)
+      .add("o_clerk", TypeProtos.MinorType.VARCHAR)
+      .add("o_shippriority", TypeProtos.MinorType.INT)
+      .add("o_comment", TypeProtos.MinorType.VARCHAR)
+      .addNullable("dir0", TypeProtos.MinorType.VARCHAR)
+      .addNullable("dir1", TypeProtos.MinorType.VARCHAR)
+      .build();
+
+  @ClassRule
+  public static TemporaryFolder defaultFolder = new TemporaryFolder();
+
+  @Rule
+  public ExpectedException thrown = ExpectedException.none();
+
+  @BeforeClass
+  public static void setUp() throws Exception {
+    ClusterFixtureBuilder builder = ClusterFixture.builder(dirTestWatcher);
+    builder.configProperty(ExecConstants.ZK_ROOT, defaultFolder.getRoot().getPath());
 
 Review comment:
   We need to override its value to store iceberg tables created by metastore in a temporary folder instead of the default `drill/happy` folder.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] arina-ielchiieva commented on issue #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
arina-ielchiieva commented on issue #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#issuecomment-599465366
 
 
   LGTM, +1

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392595095
 
 

 ##########
 File path: contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveDrillNativeParquetScan.java
 ##########
 @@ -213,11 +231,25 @@ protected boolean supportsFileImplicitColumns() {
     return hivePartitionHolder.get(locationProvider.getPath());
   }
 
+  @Override
+  protected ParquetMetadataProviderBuilder<?> tableMetadataProviderBuilder(MetadataProviderManager source) {
+    if (source.usesMetastore()) {
 
 Review comment:
   Currently, this case cannot happen since this group scan uses `FileSystemMetadataProviderManager` only.
   Thanks, I'll remove this check to avoid confusion.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r394696805
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/easy/EasyGroupScan.java
 ##########
 @@ -333,21 +338,73 @@ public boolean canPushdownProjects(List<SchemaPath> columns) {
   }
 
   @JsonProperty
+  @JsonIgnore(value = false)
   public TupleMetadata getSchema() {
     return getTableMetadata().getSchema();
   }
 
   @Override
-  @JsonIgnore
-  public TableMetadata getTableMetadata() {
-    if (tableMetadata == null) {
-      tableMetadata = metadataProvider.getTableMetadata();
+  public AnalyzeInfoProvider getAnalyzeInfoProvider() {
+    return new AnalyzeFileInfoProviderImpl(formatPlugin.getName());
+  }
+
+  @Override
+  protected GroupScanWithMetadataFilterer<?> getFilterer() {
+    return new EasyGroupScanFilterer(this);
+  }
+
+  @Override
+  protected FileTableMetadataProviderBuilder<?> tableMetadataProviderBuilder(MetadataProviderManager source) {
+    if (source.usesMetastore()) {
+      return new MetastoreFileTableMetadataProvider.Builder<>((MetastoreMetadataProviderManager) source);
+    } else {
+      return defaultTableMetadataProviderBuilder(source);
     }
-    return tableMetadata;
   }
 
   @Override
-  public TableMetadataProvider getMetadataProvider() {
-    return metadataProvider;
+  protected FileTableMetadataProviderBuilder<?> defaultTableMetadataProviderBuilder(MetadataProviderManager source) {
+    return new SimpleFileTableMetadataProvider.Builder(source);
+  }
+
+  /**
+   * Implementation of GroupScanWithMetadataFilterer which uses {@link EasyGroupScan} as source and
+   * builds {@link EasyGroupScan} instance with filtered metadata.
+   */
+  private static class EasyGroupScanFilterer extends GroupScanWithMetadataFilterer<EasyGroupScanFilterer> {
+
+    EasyGroupScanFilterer(EasyGroupScan source) {
+      super(source);
+    }
+
+    @Override
+    public AbstractGroupScanWithMetadata<?> build() {
+      EasyGroupScan newScan = new EasyGroupScan((EasyGroupScan) source);
+      newScan.tableMetadata = tableMetadata;
+      // updates common row count and nulls counts for every column
+      if (newScan.getTableMetadata() != null && files != null && newScan.getFilesMetadata().size() != files.size()) {
+        newScan.tableMetadata = TableMetadataUtils.updateRowCount(newScan.getTableMetadata(), files.values());
+      }
+      newScan.partitions = partitions;
+      newScan.segments = segments;
+      newScan.files = files;
+      newScan.matchAllMetadata = matchAllMetadata;
+      newScan.nonInterestingColumnsMetadata = nonInterestingColumnsMetadata;
+
+      newScan.fileSet = newScan.getFilesMetadata().keySet();
+      newScan.selection = FileSelection.create(null, new ArrayList<>(newScan.fileSet), newScan.selectionRoot);
+      try {
+        newScan.initFromSelection(newScan.selection, newScan.formatPlugin);
+      } catch (IOException e) {
+        throw new RuntimeException("Failed to initialize scan from the selection.", e);
+      }
+
+      return newScan;
+    }
+
+    @Override
+    protected EasyGroupScanFilterer self() {
+      return this;
+    }
 
 Review comment:
   Cool. I could really use this feature. Can you point me to an example of where you've used it?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi merged pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
vvysotskyi merged pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026
 
 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392611388
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/easy/EasyGroupScan.java
 ##########
 @@ -333,21 +338,73 @@ public boolean canPushdownProjects(List<SchemaPath> columns) {
   }
 
   @JsonProperty
+  @JsonIgnore(value = false)
 
 Review comment:
   Odd. The [Javadoc](https://fasterxml.github.io/jackson-annotations/javadoc/2.5/com/fasterxml/jackson/annotation/JsonIgnore.html) says that `value = false` means that this `@JsonIgnore` is not active, meaning it acts as if it is not present. My guess is that the intent of the flag was to allow compile time constants that conditionally include/remove certain attributes.
   
   What are we trying to do here?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392707001
 
 

 ##########
 File path: exec/java-exec/src/test/java/org/apache/drill/exec/physical/impl/scan/ScanTestUtils.java
 ##########
 @@ -52,6 +52,8 @@
   public static final String FILE_PATH_COL = "filepath";
   public static final String SUFFIX_COL = "suffix";
   public static final String PARTITION_COL = "dir";
+  public static final String LMT_COL = "lmt";
 
 Review comment:
   Thanks for pointing this, I have renamed it to `LAST_MODIFIED_TIME_COL`.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392681793
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/scan/file/FileMetadata.java
 ##########
 @@ -74,4 +81,8 @@ public int dirPathLength() {
   }
 
   public boolean isSet() { return filePath != null; }
+
+  public String getImplicitColumnValue(ColumnExplorer.ImplicitFileColumn column) {
+    return ColumnExplorer.getImplicitColumnValue(column, filePath, fs);
 
 Review comment:
   Except for the information which may be obtained from the file name, we now require such things as last modified time, which may be obtained only with file system usage.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392705189
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/easy/EasyFormatPlugin.java
 ##########
 @@ -315,16 +314,16 @@ private CloseableRecordBatch buildScanBatch(FragmentContext context,
       readers.add(recordReader);
       final List<String> partitionValues = ColumnExplorer.listPartitionValues(
           work.getPath(), scan.getSelectionRoot(), false);
-      final Map<String, String> implicitValues = columnExplorer.populateImplicitColumns(
-          work.getPath(), partitionValues, supportsFileImplicitColumns);
+      final Map<String, String> implicitValues = columnExplorer.populateImplicitAndInternalColumns(
 
 Review comment:
   Thanks, renamed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392613050
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/FilePushDownFilter.java
 ##########
 @@ -244,4 +247,15 @@ protected void doOnMatch(RelOptRuleCall call, FilterPrel filter, ProjectPrel pro
     final RelNode newFilter = filter.copy(filter.getTraitSet(), Collections.singletonList(newNode));
     call.transformTo(newFilter);
   }
+
+  /**
+   * Checks whether specified group scan supports filter push down.
+   *
+   * @param groupScan group scan to check
+   * @return {@code true} if specified group scan supports filter push down, {@code false} otherwise
+   */
+  private static boolean supportsFilterPushDown(GroupScan groupScan) {
+    return groupScan instanceof AbstractGroupScanWithMetadata
 
 Review comment:
   This kind of logic is fragile. Can we just add a method on the group scan that will answer this question, then let each group scan figure out the answer or itself?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392596751
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/easy/EasyGroupScan.java
 ##########
 @@ -333,21 +338,73 @@ public boolean canPushdownProjects(List<SchemaPath> columns) {
   }
 
   @JsonProperty
+  @JsonIgnore(value = false)
   public TupleMetadata getSchema() {
     return getTableMetadata().getSchema();
   }
 
   @Override
-  @JsonIgnore
-  public TableMetadata getTableMetadata() {
-    if (tableMetadata == null) {
-      tableMetadata = metadataProvider.getTableMetadata();
+  public AnalyzeInfoProvider getAnalyzeInfoProvider() {
+    return new AnalyzeFileInfoProviderImpl(formatPlugin.getName());
+  }
+
+  @Override
+  protected GroupScanWithMetadataFilterer<?> getFilterer() {
+    return new EasyGroupScanFilterer(this);
+  }
+
+  @Override
+  protected FileTableMetadataProviderBuilder<?> tableMetadataProviderBuilder(MetadataProviderManager source) {
+    if (source.usesMetastore()) {
+      return new MetastoreFileTableMetadataProvider.Builder<>((MetastoreMetadataProviderManager) source);
+    } else {
+      return defaultTableMetadataProviderBuilder(source);
     }
-    return tableMetadata;
   }
 
   @Override
-  public TableMetadataProvider getMetadataProvider() {
-    return metadataProvider;
+  protected FileTableMetadataProviderBuilder<?> defaultTableMetadataProviderBuilder(MetadataProviderManager source) {
+    return new SimpleFileTableMetadataProvider.Builder(source);
+  }
+
+  /**
+   * Implementation of GroupScanWithMetadataFilterer which uses {@link EasyGroupScan} as source and
+   * builds {@link EasyGroupScan} instance with filtered metadata.
+   */
+  private static class EasyGroupScanFilterer extends GroupScanWithMetadataFilterer<EasyGroupScanFilterer> {
+
+    EasyGroupScanFilterer(EasyGroupScan source) {
+      super(source);
+    }
+
+    @Override
+    public AbstractGroupScanWithMetadata<?> build() {
+      EasyGroupScan newScan = new EasyGroupScan((EasyGroupScan) source);
+      newScan.tableMetadata = tableMetadata;
+      // updates common row count and nulls counts for every column
+      if (newScan.getTableMetadata() != null && files != null && newScan.getFilesMetadata().size() != files.size()) {
+        newScan.tableMetadata = TableMetadataUtils.updateRowCount(newScan.getTableMetadata(), files.values());
+      }
+      newScan.partitions = partitions;
+      newScan.segments = segments;
+      newScan.files = files;
+      newScan.matchAllMetadata = matchAllMetadata;
+      newScan.nonInterestingColumnsMetadata = nonInterestingColumnsMetadata;
+
+      newScan.fileSet = newScan.getFilesMetadata().keySet();
+      newScan.selection = FileSelection.create(null, new ArrayList<>(newScan.fileSet), newScan.selectionRoot);
+      try {
+        newScan.initFromSelection(newScan.selection, newScan.formatPlugin);
+      } catch (IOException e) {
+        throw new RuntimeException("Failed to initialize scan from the selection.", e);
 
 Review comment:
   Thanks, replaced it with `DrillRuntimeException`.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r394695247
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/easy/EasyGroupScan.java
 ##########
 @@ -90,17 +95,14 @@ public EasyGroupScan(
       @JsonProperty("selectionRoot") Path selectionRoot,
       @JsonProperty("schema") TupleMetadata schema
       ) throws IOException {
-    super(ImpersonationUtil.resolveUserName(userName));
+    super(ImpersonationUtil.resolveUserName(userName), columns, ValueExpressions.BooleanExpression.TRUE);
     this.selection = FileSelection.create(null, files, selectionRoot);
     this.formatPlugin = engineRegistry.resolveFormat(storageConfig, formatConfig, EasyFormatPlugin.class);
     this.columns = columns == null ? ALL_COLUMNS : columns;
     this.selectionRoot = selectionRoot;
-    SimpleFileTableMetadataProviderBuilder builder =
-        (SimpleFileTableMetadataProviderBuilder)
-        new FileSystemMetadataProviderManager()
-        .builder(MetadataProviderManager.MetadataProviderKind.SCHEMA_STATS_ONLY);
 
-    this.metadataProvider = builder.withLocation(selection.getSelectionRoot())
+    this.metadataProvider = defaultTableMetadataProviderBuilder(new FileSystemMetadataProviderManager())
 
 Review comment:
   This will be a huge problem in an actual production system. Fixing it is beyond the scope of this PR. I would suggest that the team think a bit about how this can work longer term. As noted, Impala struggled with this issue for years, so it is not simple.
   
   One answer is to know when directories change. Use cached metadata for unchanged directories (which will be most of data history) and expand only those that are "live" (i.e. partitions for the last day or two.)
   
   Caching is essential for good performance. In the past, Drill was s-l-o-w when reading the cached Parquet metadata.
   
   But, again, let's leave this issue for another project.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392721861
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/store/ColumnExplorer.java
 ##########
 @@ -292,39 +289,86 @@ public static int getPartitionDepth(FileSelection selection) {
    * @param includeFileImplicitColumns if file implicit columns should be included into the result
    * @param fs                         file system
    * @param index                      index of row group to populate
+   * @param start                      start of row group to populate
+   * @param length                     length of row group to populate
    * @return implicit columns map
    */
   public Map<String, String> populateImplicitAndInternalColumns(Path filePath,
       List<String> partitionValues, boolean includeFileImplicitColumns, FileSystem fs, int index, long start, long length) {
 
     Map<String, String> implicitValues =
-        new LinkedHashMap<>(populateImplicitColumns(filePath, partitionValues, includeFileImplicitColumns));
+        new LinkedHashMap<>(populateImplicitAndInternalColumns(filePath, partitionValues, includeFileImplicitColumns, fs));
 
-    selectedInternalColumns.forEach((key, value) -> {
-      switch (value) {
+    selectedInternalColumns.forEach(
+        (key, value) -> implicitValues.put(key, getImplicitColumnValue(value, filePath, fs, index, start, length)));
+
+    return implicitValues;
+  }
+
+  /**
+   * Returns implicit column value for specified implicit file column.
+   *
+   * @param column   implicit file column
+   * @param filePath file path, used to populate file implicit columns
+   * @param fs       file system
+   * @param index    row group index
+   * @param start    row group start
+   * @param length   row group length
+   * @return implicit column value for specified implicit file column
+   */
+  private static String getImplicitColumnValue(ImplicitFileColumn column, Path filePath,
+      FileSystem fs, Integer index, Long start, Long length) {
+    if (column instanceof ImplicitFileColumns) {
+      ImplicitFileColumns fileColumn = (ImplicitFileColumns) column;
+      return fileColumn.getValue(filePath);
+    } else if (column instanceof ImplicitInternalFileColumns) {
+      ImplicitInternalFileColumns fileColumn = (ImplicitInternalFileColumns) column;
+      switch (fileColumn) {
         case ROW_GROUP_INDEX:
-          implicitValues.put(key, String.valueOf(index));
-          break;
+          return index != null ? String.valueOf(index) : null;
         case ROW_GROUP_START:
-          implicitValues.put(key, String.valueOf(start));
-          break;
+          return start != null ? String.valueOf(start) : null;
         case ROW_GROUP_LENGTH:
-          implicitValues.put(key, String.valueOf(length));
-          break;
+          return length != null ? String.valueOf(length) : null;
         case PROJECT_METADATA:
-          implicitValues.put(key, Boolean.TRUE.toString());
-          break;
+          return Boolean.TRUE.toString();
         case LAST_MODIFIED_TIME:
           try {
-            implicitValues.put(key, String.valueOf(fs.getFileStatus(filePath).getModificationTime()));
+            return fs != null ? String.valueOf(fs.getFileStatus(filePath).getModificationTime()) : null;
           } catch (IOException e) {
             throw new DrillRuntimeException(e);
           }
-          break;
       }
-    });
+    }
+    return null;
+  }
 
-    return implicitValues;
+  /**
+   * Returns implicit column value for specified implicit file column.
+   *
+   * @param column   implicit file column
+   * @param filePath file path
+   * @param fs       file system
+   * @return implicit column value for specified implicit file column
+   */
+  public static String getImplicitColumnValue(ImplicitFileColumn column, Path filePath, FileSystem fs) {
+    return getImplicitColumnValue(column, filePath, fs, null, null, null);
+  }
+
+  /**
+   * Returns list of implicit file columns which includes all elements from {@link ImplicitFileColumns},
+   * {@link ImplicitInternalFileColumns#LAST_MODIFIED_TIME} and {@link ImplicitInternalFileColumns#PROJECT_METADATA}
+   * columns.
+   *
+   * @return list of implicit file columns
+   */
+  public static List<ImplicitFileColumn> getImplicitFileColumns() {
+    List<ImplicitFileColumn> implicitColumns = new ArrayList<>();
+    Collections.addAll(implicitColumns, ImplicitFileColumns.values());
+
+    implicitColumns.add(ImplicitInternalFileColumns.LAST_MODIFIED_TIME);
+    implicitColumns.add(ImplicitInternalFileColumns.PROJECT_METADATA);
 
 Review comment:
   Yes, that's correct that reader will set this column value only for the case of EOF.
   I have made changes to populate the non-null column value only for the case of EOF for the old scan, so this value wouldn't be copied.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392608220
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/scan/file/FileMetadataColumn.java
 ##########
 @@ -65,6 +82,11 @@ public FileMetadataColumn(String name, FileMetadataColumnDefn defn) {
 
   @Override
   public MetadataColumn resolve(FileMetadata fileInfo, VectorSource source, int sourceIndex) {
-    return new FileMetadataColumn(name(), defn, fileInfo, source, sourceIndex);
+    if (value() == null) {
+      return new FileMetadataColumn(name(), defn, fileInfo, source, sourceIndex);
+    } else {
+      // case of metadata column, which should have null value independently of its column definition
+      return new FileMetadataColumn(name(), defn, (String) null, source, sourceIndex);
 
 Review comment:
   Please explain this a bit more. In the original implementation, the reader is given a path to a file to read. The path and file name are broken apart and passed to these implicit column objects. (I should rename them.) In what case would the reader not know the name or path of the file it should read, and must look the data up in the metastore? Did we add new columns that require going out to the file system to gather a value? (I'm changing this area, so I want to make sure I know the intent here.)
   
   If this class will check the fs, then there is a chance the file has been deleted. It will now be this class, rather than the reader, that detects missing files. I would think we'd want to handle that further up the call stack, then pass the resolved info into this class.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392609028
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/scan/project/ReaderSchemaOrchestrator.java
 ##########
 @@ -36,10 +41,16 @@
 public class ReaderSchemaOrchestrator implements VectorSource {
 
   private final ScanSchemaOrchestrator scanOrchestrator;
-  private int readerBatchSize = ScanSchemaOrchestrator.MAX_BATCH_ROW_COUNT;
+  private int readerBatchSize;
   private ResultSetLoaderImpl tableLoader;
   private int prevTableSchemaVersion = -1;
 
+  /**
+   * True once the reader reports EOF. This shim may keep going for another
+   * batch to handle any look-ahead row on the last batch.
+   */
+  private boolean eof;
 
 Review comment:
   See note above. The methods here should give you what you need. 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r395148096
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/ScanBatch.java
 ##########
 @@ -242,12 +242,12 @@ private IterOutcome internalNext() {
       if (implicitValues != null) {
         String projectMetadataColumn = context.getOptions().getOption(ExecConstants.IMPLICIT_PROJECT_METADATA_COLUMN_LABEL_VALIDATOR);
         if (recordCount > 0) {
-          // Sets the implicit value to false to signal that some results were returned and there is no need for creating an additional record.
-          implicitValues.replace(projectMetadataColumn, Boolean.FALSE.toString());
-        } else if (Boolean.parseBoolean(implicitValues.get(projectMetadataColumn))) {
+          // Sets the implicit value to null to signal that some results were returned and there is no need for creating an additional record.
+          implicitValues.replace(projectMetadataColumn, null);
+        } else if (Boolean.FALSE.toString().equals(implicitValues.get(projectMetadataColumn))) {
 
 Review comment:
   Values here are written using either using `Boolean.FALSE.toString()` or `Boolean.TRUE.toString()` or unset.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392610460
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/store/ColumnExplorer.java
 ##########
 @@ -292,39 +289,86 @@ public static int getPartitionDepth(FileSelection selection) {
    * @param includeFileImplicitColumns if file implicit columns should be included into the result
    * @param fs                         file system
    * @param index                      index of row group to populate
+   * @param start                      start of row group to populate
+   * @param length                     length of row group to populate
    * @return implicit columns map
    */
   public Map<String, String> populateImplicitAndInternalColumns(Path filePath,
       List<String> partitionValues, boolean includeFileImplicitColumns, FileSystem fs, int index, long start, long length) {
 
     Map<String, String> implicitValues =
-        new LinkedHashMap<>(populateImplicitColumns(filePath, partitionValues, includeFileImplicitColumns));
+        new LinkedHashMap<>(populateImplicitAndInternalColumns(filePath, partitionValues, includeFileImplicitColumns, fs));
 
-    selectedInternalColumns.forEach((key, value) -> {
-      switch (value) {
+    selectedInternalColumns.forEach(
+        (key, value) -> implicitValues.put(key, getImplicitColumnValue(value, filePath, fs, index, start, length)));
+
+    return implicitValues;
+  }
+
+  /**
+   * Returns implicit column value for specified implicit file column.
+   *
+   * @param column   implicit file column
+   * @param filePath file path, used to populate file implicit columns
+   * @param fs       file system
+   * @param index    row group index
+   * @param start    row group start
+   * @param length   row group length
+   * @return implicit column value for specified implicit file column
+   */
+  private static String getImplicitColumnValue(ImplicitFileColumn column, Path filePath,
+      FileSystem fs, Integer index, Long start, Long length) {
+    if (column instanceof ImplicitFileColumns) {
+      ImplicitFileColumns fileColumn = (ImplicitFileColumns) column;
+      return fileColumn.getValue(filePath);
+    } else if (column instanceof ImplicitInternalFileColumns) {
+      ImplicitInternalFileColumns fileColumn = (ImplicitInternalFileColumns) column;
+      switch (fileColumn) {
         case ROW_GROUP_INDEX:
-          implicitValues.put(key, String.valueOf(index));
-          break;
+          return index != null ? String.valueOf(index) : null;
         case ROW_GROUP_START:
-          implicitValues.put(key, String.valueOf(start));
-          break;
+          return start != null ? String.valueOf(start) : null;
         case ROW_GROUP_LENGTH:
-          implicitValues.put(key, String.valueOf(length));
-          break;
+          return length != null ? String.valueOf(length) : null;
         case PROJECT_METADATA:
-          implicitValues.put(key, Boolean.TRUE.toString());
-          break;
+          return Boolean.TRUE.toString();
         case LAST_MODIFIED_TIME:
           try {
-            implicitValues.put(key, String.valueOf(fs.getFileStatus(filePath).getModificationTime()));
+            return fs != null ? String.valueOf(fs.getFileStatus(filePath).getModificationTime()) : null;
           } catch (IOException e) {
             throw new DrillRuntimeException(e);
           }
-          break;
       }
-    });
+    }
+    return null;
+  }
 
-    return implicitValues;
+  /**
+   * Returns implicit column value for specified implicit file column.
+   *
+   * @param column   implicit file column
+   * @param filePath file path
+   * @param fs       file system
+   * @return implicit column value for specified implicit file column
+   */
+  public static String getImplicitColumnValue(ImplicitFileColumn column, Path filePath, FileSystem fs) {
+    return getImplicitColumnValue(column, filePath, fs, null, null, null);
+  }
+
+  /**
+   * Returns list of implicit file columns which includes all elements from {@link ImplicitFileColumns},
+   * {@link ImplicitInternalFileColumns#LAST_MODIFIED_TIME} and {@link ImplicitInternalFileColumns#PROJECT_METADATA}
+   * columns.
+   *
+   * @return list of implicit file columns
+   */
+  public static List<ImplicitFileColumn> getImplicitFileColumns() {
+    List<ImplicitFileColumn> implicitColumns = new ArrayList<>();
+    Collections.addAll(implicitColumns, ImplicitFileColumns.values());
+
+    implicitColumns.add(ImplicitInternalFileColumns.LAST_MODIFIED_TIME);
+    implicitColumns.add(ImplicitInternalFileColumns.PROJECT_METADATA);
 
 Review comment:
   Please explain this one. What does it mean and how is it used? I'm a bit confused because the reader does not know how its data will be used. When would the reader/scan set the value for this field? Only in that EOF case discussed above? If so, then may not be needed for the "new" scan. (And, perhaps we should fix the case in the old one since it seems silly to include billions of copies of this value so we can mark one or two empty batches, if I understand the meaning correctly.)

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392598431
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/BaseParquetMetadataProvider.java
 ##########
 @@ -103,34 +110,39 @@
   // whether metadata for row groups should be collected to create files, partitions and table metadata
   private final boolean collectMetadata = false;
 
-  public BaseParquetMetadataProvider(List<ReadEntryWithPath> entries,
-                                     ParquetReaderConfig readerConfig,
-                                     String tableName,
-                                     Path tableLocation,
-                                     TupleMetadata schema,
-                                     DrillStatsTable statsTable) {
-    this(readerConfig, entries, tableName, tableLocation, schema, statsTable);
-  }
+  protected BaseParquetMetadataProvider(Builder<?> builder) {
+    if (builder.entries != null) {
+      // reuse previously stored metadata
+      this.entries = builder.entries;
+      this.tableName = builder.selectionRoot != null ? builder.selectionRoot.toUri().getPath() : "";
+      this.tableLocation = builder.selectionRoot;
+    } else if (builder.selection != null) {
+      this.entries = new ArrayList<>();
+      this.tableName = builder.selection.getSelectionRoot() != null ? builder.selection.getSelectionRoot().toUri().getPath() : "";
+      this.tableLocation = builder.selection.getSelectionRoot();
+    } else {
+      // case of hive parquet table
+      this.entries = new ArrayList<>();
+      this.tableName = null;
+      this.tableLocation = null;
+    }
 
-  public BaseParquetMetadataProvider(ParquetReaderConfig readerConfig,
-                                     List<ReadEntryWithPath> entries,
-                                     String tableName,
-                                     Path tableLocation,
-                                     TupleMetadata schema,
-                                     DrillStatsTable statsTable) {
-    this.entries = entries == null ? new ArrayList<>() : entries;
-    this.readerConfig = readerConfig == null ? ParquetReaderConfig.getDefaultInstance() : readerConfig;
-    this.tableName = tableName;
-    this.tableLocation = tableLocation;
-    this.schema = schema;
-    this.statsTable = statsTable;
-  }
+    SchemaProvider schemaProvider = builder.metadataProviderManager.getSchemaProvider();
+    TupleMetadata schema = builder.schema;
+    // schema passed into the builder has greater priority
+    if (schema == null && schemaProvider != null) {
+      try {
+        schema = schemaProvider.read().getSchema();
+      } catch (IOException e) {
+        logger.debug("Unable to read schema from schema provider [{}]: {}",
 
 Review comment:
   Thanks, changed to log it as warn and updated message to point that execution will continue without using schema here and in other similar places.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392681623
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/AbstractGroupScanWithMetadata.java
 ##########
 @@ -634,6 +642,62 @@ public NonInterestingColumnsMetadata getNonInterestingColumnsMetadata() {
     return nonInterestingColumnsMetadata;
   }
 
+  /**
+   * Returns {@link TableMetadataProviderBuilder} instance based on specified
+   * {@link MetadataProviderManager} source.
+   *
+   * @param source metadata provider manager
+   * @return {@link TableMetadataProviderBuilder} instance
+   */
+  protected abstract TableMetadataProviderBuilder tableMetadataProviderBuilder(MetadataProviderManager source);
+
+  /**
+   * Returns {@link TableMetadataProviderBuilder} instance which may provide metadata
+   * without using Drill Metastore.
+   *
+   * @param source metadata provider manager
+   * @return {@link TableMetadataProviderBuilder} instance
+   */
+  protected abstract TableMetadataProviderBuilder defaultTableMetadataProviderBuilder(MetadataProviderManager source);
+
+  /**
+   * Compares the last modified time of files obtained from specified selection with
+   * the Metastore last modified time to determine whether Metastore metadata
+   * is not outdated. If metadata is outdated, {@link MetadataException} will be thrown.
+   *
+   * @param selection the source of files to check
+   * @throws MetadataException if metadata is outdated
+   */
+  protected void checkMetadataConsistency(FileSelection selection, Configuration fsConf) throws IOException {
+    if (metadataProvider.checkMetadataVersion()) {
+      DrillFileSystem fileSystem =
+          ImpersonationUtil.createFileSystem(ImpersonationUtil.resolveUserName(getUserName()), fsConf);
+
+      List<FileStatus> fileStatuses = FileMetadataInfoCollector.getFileStatuses(selection, fileSystem);
+
+      long lastModifiedTime = metadataProvider.getTableMetadata().getLastModifiedTime();
+
+      Set<Path> removedFiles = new HashSet<>(metadataProvider.getFilesMetadataMap().keySet());
+      Set<Path> newFiles = new HashSet<>();
+
+      boolean isChanged = false;
+
+      for (FileStatus fileStatus : fileStatuses) {
+        if (!removedFiles.remove(Path.getPathWithoutSchemeAndAuthority(fileStatus.getPath()))) {
+          newFiles.add(fileStatus.getPath());
+        }
+        if (fileStatus.getModificationTime() > lastModifiedTime) {
+          isChanged = true;
+          break;
+        }
+      }
 
 Review comment:
   I agree that it may be costly. But in the case when we wouldn't do this check now, we can obtain incorrect results. Regarding integrations with some external systems which may do this, it is a good idea, but I don't know about such systems. Currently, we either use actual metadata for queries or do not use it at all.
   
   Regarding making auto-refresh, there is another Jira https://issues.apache.org/jira/browse/DRILL-7430 which is a holder for further improvements for metastore, so we can discuss it there.
   
   Regarding the case where data continuously arriving, I don't think that this is the right case for using metastore, since refreshing the metadata, even with our incremental update is too costly to do it so often.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392607989
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/scan/file/FileMetadata.java
 ##########
 @@ -74,4 +81,8 @@ public int dirPathLength() {
   }
 
   public boolean isSet() { return filePath != null; }
+
+  public String getImplicitColumnValue(ColumnExplorer.ImplicitFileColumn column) {
+    return ColumnExplorer.getImplicitColumnValue(column, filePath, fs);
 
 Review comment:
   The prior version returned values based on the static file name and path; no need for the file system. How has that changed here?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] arina-ielchiieva commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
arina-ielchiieva commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392590429
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/easy/EasyGroupScan.java
 ##########
 @@ -209,17 +214,12 @@ public boolean hasFiles() {
   }
 
   @JsonProperty("files")
+  @JsonIgnore(value = false)
 
 Review comment:
   Why now we are ignoring this property?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on issue #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
vvysotskyi commented on issue #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#issuecomment-599032547
 
 
   @paul-rogers, this pull request enables the format plugin to gather metadata. Metadata gathering logic was added in DRILL-7273.
   
   Regarding the schema, when metadata is collecting, rules are the same as for regular select queries - Drill tries to infer the table schema or uses user-provided schema.
   
   Collecting metadata logic may become clearer after reading this section of docs: https://github.com/apache/drill/blob/master/docs/dev/MetastoreAnalyze.md#analyze-operators-description or this design doc: https://docs.google.com/document/d/14pSIzKqDltjLEEpEebwmKnsDPxyS_6jGrPOjXu6M_NM/edit?usp=sharing
   In short, yes, we use a reader that reads all the data and downstream operators for transforming and storing its statistics.
   
   > For files that need a provided schema (CSV, say), do we apply stats to the columns after type conversion, or are stats gathered on the raw text values? That is, does this work use the provided schema if available?
   
   Yes, we apply stats to the columns after schema conversion, so such stats as min/max would have correct values in the scope of natural ordering.
   
   > How does the provided schema relate to the metadata schema?
   
   After the provided schema is used in the scan, Drill will use the resolved schema for columns and store it to the metastore.
   
   > What stats will we gather for non-Parquet files? How will we use them? Looks like there is code for partitions (have not looked in depth, so I may be wrong). Are we using stats for partition pruning? If so, how does that differ from the existing practice of just walking the directory tree?
   
   We collect exactly the same stats for non-parquet files. We may use them in the same way as it is used in parquet - prune files when filter for specific columns is specified, prune unneeded files for limit queries. Dirs pruning would still work in the same way as it worked before changes (it also works for parquet).
   I think some tests in `TestMetastoreWithEasyFormatPlugin` will help to understand which optimizations are added.
   
   > Do you see any potential conflicts between your metadata model and the above provided schema model?
   
   Looks like there shouldn't be any conflicts.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on issue #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
paul-rogers commented on issue #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#issuecomment-599258635
 
 
   @vvysotskyi, I hope the comments more-or-less made sense; I was relearning this code, and learning the metadata stuff, as I went.
   
   An earlier note mentioned that I'm revising the managed framework. It also suggested a different way to parse out the implicit columns. To avoid blocking you, we can use the design here. Then, I'll adjust it as needed to fit the new classes. (In fact, I've already made some changes based on what I learned here.) 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392716963
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/metastore/store/MetastoreFileTableMetadataProvider.java
 ##########
 @@ -291,124 +250,74 @@ public boolean checkMetadataVersion() {
     }
   }
 
-  private ParquetTableMetadataProvider getFallbackTableMetadataProvider() throws IOException {
-    if (fallback == null) {
-      fallback = fallbackBuilder == null ? null : fallbackBuilder.build();
-    }
-    return fallback;
-  }
+  public static class Builder<T extends Builder<T>> implements FileTableMetadataProviderBuilder<T> {
+    protected final MetastoreMetadataProviderManager metadataProviderManager;
 
-  private void throwIfChanged() {
-    if (basicTablesRequests.hasMetastoreTableInfoChanged(metastoreTableInfo)) {
-      throw MetadataException.of(MetadataException.MetadataExceptionType.INCONSISTENT_METADATA);
-    }
-  }
+    // builder for fallback ParquetFileTableMetadataProvider
+    // for the case when required metadata is absent in Metastore
+    protected final TableMetadataProviderBuilder fallback;
 
-  public static class Builder implements ParquetFileTableMetadataProviderBuilder {
-    private final MetastoreMetadataProviderManager metadataProviderManager;
+    protected TupleMetadata schema;
 
-    private List<ReadEntryWithPath> entries;
-    private DrillFileSystem fs;
-    private TupleMetadata schema;
+    protected List<String> paths;
 
     private FileSelection selection;
 
-    // builder for fallback ParquetFileTableMetadataProvider
-    // for the case when required metadata is absent in Metastore
-    private final ParquetFileTableMetadataProviderBuilder fallback;
+    private DrillFileSystem fs;
 
     public Builder(MetastoreMetadataProviderManager source) {
-      this.metadataProviderManager = source;
-      this.fallback = new ParquetTableMetadataProviderImpl.Builder(FileSystemMetadataProviderManager.init());
+      this(source, new SimpleFileTableMetadataProvider.Builder(FileSystemMetadataProviderManager.init()));
     }
 
-    @Override
-    public ParquetFileTableMetadataProviderBuilder withEntries(List<ReadEntryWithPath> entries) {
-      this.entries = entries;
-      fallback.withEntries(entries);
-      return this;
+    protected Builder(MetastoreMetadataProviderManager source, TableMetadataProviderBuilder fallback) {
+      this.metadataProviderManager = source;
+      this.fallback = fallback;
     }
 
     @Override
-    public ParquetFileTableMetadataProviderBuilder withSelectionRoot(Path selectionRoot) {
-      fallback.withSelectionRoot(selectionRoot);
-      return this;
+    public T withSchema(TupleMetadata schema) {
+      this.schema = schema;
+      return self();
     }
 
-    @Override
-    public ParquetFileTableMetadataProviderBuilder withCacheFileRoot(Path cacheFileRoot) {
-      fallback.withCacheFileRoot(cacheFileRoot);
-      return this;
+    public T withSelection(FileSelection selection) {
+      this.selection = selection;
+      return self();
     }
 
-    @Override
-    public ParquetFileTableMetadataProviderBuilder withReaderConfig(ParquetReaderConfig readerConfig) {
-      fallback.withReaderConfig(readerConfig);
-      return this;
+    public T withFileSystem(DrillFileSystem fs) {
+      this.fs = fs;
+      return self();
     }
 
-    @Override
-    public ParquetFileTableMetadataProviderBuilder withFileSystem(DrillFileSystem fs) {
-      fallback.withFileSystem(fs);
-      this.fs = fs;
-      return this;
+    protected T self() {
+      return (T) this;
     }
 
-    @Override
-    public ParquetFileTableMetadataProviderBuilder withCorrectCorruptedDates(boolean autoCorrectCorruptedDates) {
-      fallback.withCorrectCorruptedDates(autoCorrectCorruptedDates);
-      return this;
+    public MetastoreMetadataProviderManager metadataProviderManager() {
+      return metadataProviderManager;
     }
 
-    @Override
-    public ParquetFileTableMetadataProviderBuilder withSelection(FileSelection selection) {
-      fallback.withSelection(selection);
-      this.selection = selection;
-      return this;
+    public FileSelection selection() {
+      return selection;
     }
 
-    @Override
-    public ParquetFileTableMetadataProviderBuilder withSchema(TupleMetadata schema) {
-      fallback.withSchema(schema);
-      this.schema = schema;
-      return this;
+    public DrillFileSystem fs() {
+      return fs;
     }
 
     @Override
-    public ParquetTableMetadataProvider build() throws IOException {
-      MetastoreParquetTableMetadataProvider provider;
-      SchemaProvider schemaProvider = metadataProviderManager.getSchemaProvider();
-      ParquetMetadataProvider source = (ParquetTableMetadataProvider) metadataProviderManager.getTableMetadataProvider();
-
-      DrillStatsTable statsProvider = metadataProviderManager.getStatsProvider();
-      // schema passed into the builder has greater priority
-      try {
-        if (this.schema == null) {
-          schema = schemaProvider != null ? schemaProvider.read().getSchema() : null;
-        }
-      } catch (IOException e) {
-        logger.debug("Unable to deserialize schema from schema file for table: {}", metadataProviderManager.getTableInfo().name(), e);
-      }
-      if (entries == null) {
-        if (!selection.isExpandedFully()) {
-          entries = DrillFileSystemUtil.listFiles(fs, selection.getSelectionRoot(), true).stream()
-              .map(fileStatus -> new ReadEntryWithPath(Path.getPathWithoutSchemeAndAuthority(fileStatus.getPath())))
-              .collect(Collectors.toList());
-        } else {
-          entries = selection.getFiles().stream()
-              .map(Path::getPathWithoutSchemeAndAuthority)
-              .map(ReadEntryWithPath::new)
-              .collect(Collectors.toList());
-        }
-      }
-      provider = new MetastoreParquetTableMetadataProvider(entries, metadataProviderManager.getMetastoreRegistry(),
-          metadataProviderManager.getTableInfo(), schema, fallback, metadataProviderManager.getConfig(), statsProvider);
-      // store results into metadataProviderManager to be able to use them when creating new instances
-      // for the case when source wasn't provided or it contains less row group metadata than the provider
-      if (source == null || source.getRowGroupsMeta().size() < provider.getRowGroupsMeta().size()) {
-        metadataProviderManager.setTableMetadataProvider(provider);
+    public TableMetadataProvider build() throws IOException {
+      if (!selection().isExpandedFully()) {
+        paths = DrillFileSystemUtil.listFiles(fs, selection.getSelectionRoot(), true).stream()
+            .map(fileStatus -> Path.getPathWithoutSchemeAndAuthority(fileStatus.getPath()).toUri().getPath())
+            .collect(Collectors.toList());
+      } else {
+        paths = selection.getFiles().stream()
+            .map(path -> Path.getPathWithoutSchemeAndAuthority(path).toUri().getPath())
+            .collect(Collectors.toList());
 
 Review comment:
   No, it is not cached between the queries, since the file may be deleted, or new files may be added, so it would cause wrong results.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] arina-ielchiieva commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
arina-ielchiieva commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392590826
 
 

 ##########
 File path: exec/java-exec/src/test/java/org/apache/drill/exec/metastore/TestMetastoreWithEasyFormatPlugin.java
 ##########
 @@ -0,0 +1,1111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.metastore;
+
+import org.apache.commons.io.FileUtils;
+import org.apache.drill.categories.MetastoreTest;
+import org.apache.drill.categories.SlowTest;
+import org.apache.drill.common.expression.SchemaPath;
+import org.apache.drill.common.types.TypeProtos;
+import org.apache.drill.exec.ExecConstants;
+import org.apache.drill.exec.planner.physical.PlannerSettings;
+import org.apache.drill.exec.record.metadata.SchemaBuilder;
+import org.apache.drill.exec.record.metadata.TupleMetadata;
+import org.apache.drill.metastore.components.tables.MetastoreTableInfo;
+import org.apache.drill.metastore.metadata.BaseTableMetadata;
+import org.apache.drill.metastore.metadata.FileMetadata;
+import org.apache.drill.metastore.metadata.MetadataInfo;
+import org.apache.drill.metastore.metadata.MetadataType;
+import org.apache.drill.metastore.metadata.SegmentMetadata;
+import org.apache.drill.metastore.metadata.TableInfo;
+import org.apache.drill.metastore.statistics.ColumnStatistics;
+import org.apache.drill.metastore.statistics.ColumnStatisticsKind;
+import org.apache.drill.metastore.statistics.StatisticsHolder;
+import org.apache.drill.metastore.statistics.TableStatisticsKind;
+import org.apache.drill.shaded.guava.com.google.common.collect.ImmutableMap;
+import org.apache.drill.shaded.guava.com.google.common.collect.ImmutableSet;
+import org.apache.drill.test.ClusterFixture;
+import org.apache.drill.test.ClusterFixtureBuilder;
+import org.apache.drill.test.ClusterTest;
+import org.apache.hadoop.fs.Path;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.rules.ExpectedException;
+import org.junit.rules.TemporaryFolder;
+
+import java.io.File;
+import java.nio.file.Paths;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+import static org.apache.drill.exec.sql.TestMetastoreCommands.DIR0_1994_Q1_SEGMENT_COLUMN_STATISTICS;
+import static org.apache.drill.exec.sql.TestMetastoreCommands.DIR0_1994_SEGMENT_COLUMN_STATISTICS;
+import static org.apache.drill.exec.sql.TestMetastoreCommands.TABLE_COLUMN_STATISTICS;
+import static org.apache.drill.exec.sql.TestMetastoreCommands.TABLE_META_INFO;
+import static org.apache.drill.exec.sql.TestMetastoreCommands.getBaseTableMetadata;
+import static org.apache.drill.exec.sql.TestMetastoreCommands.getColumnStatistics;
+import static org.apache.drill.exec.sql.TestMetastoreCommands.getMaxLastModified;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+@Category({SlowTest.class, MetastoreTest.class})
+public class TestMetastoreWithEasyFormatPlugin extends ClusterTest {
+
+  private static String SCHEMA_STRING = "'inline=(" +
+      "`o_orderkey` INT not null, " +
+      "`o_custkey` INT not null, " +
+      "`o_orderstatus` VARCHAR not null, " +
+      "`o_totalprice` DOUBLE not null, " +
+      "`o_orderdate` DATE not null format \"yyyy-MM-dd''T''HH:mm:ss.SSSZZ\", " +
+      "`o_orderpriority` VARCHAR not null, " +
+      "`o_clerk` VARCHAR not null, " +
+      "`o_shippriority` INT not null, " +
+      "`o_comment` VARCHAR not null)'";
+
+  private static final TupleMetadata SCHEMA = new SchemaBuilder()
+      .add("o_orderkey", TypeProtos.MinorType.INT)
+      .add("o_custkey", TypeProtos.MinorType.INT)
+      .add("o_orderstatus", TypeProtos.MinorType.VARCHAR)
+      .add("o_totalprice", TypeProtos.MinorType.FLOAT8)
+      .add("o_orderdate", TypeProtos.MinorType.DATE)
+      .add("o_orderpriority", TypeProtos.MinorType.VARCHAR)
+      .add("o_clerk", TypeProtos.MinorType.VARCHAR)
+      .add("o_shippriority", TypeProtos.MinorType.INT)
+      .add("o_comment", TypeProtos.MinorType.VARCHAR)
+      .addNullable("dir0", TypeProtos.MinorType.VARCHAR)
+      .addNullable("dir1", TypeProtos.MinorType.VARCHAR)
+      .build();
+
+  @ClassRule
+  public static TemporaryFolder defaultFolder = new TemporaryFolder();
+
+  @Rule
+  public ExpectedException thrown = ExpectedException.none();
+
+  @BeforeClass
+  public static void setUp() throws Exception {
+    ClusterFixtureBuilder builder = ClusterFixture.builder(dirTestWatcher);
+    builder.configProperty(ExecConstants.ZK_ROOT, defaultFolder.getRoot().getPath());
 
 Review comment:
   Why we need to define zk root, in other tests we don't...

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r394698856
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/BaseParquetMetadataProvider.java
 ##########
 @@ -103,34 +110,39 @@
   // whether metadata for row groups should be collected to create files, partitions and table metadata
   private final boolean collectMetadata = false;
 
-  public BaseParquetMetadataProvider(List<ReadEntryWithPath> entries,
-                                     ParquetReaderConfig readerConfig,
-                                     String tableName,
-                                     Path tableLocation,
-                                     TupleMetadata schema,
-                                     DrillStatsTable statsTable) {
-    this(readerConfig, entries, tableName, tableLocation, schema, statsTable);
-  }
+  protected BaseParquetMetadataProvider(Builder<?> builder) {
+    if (builder.entries != null) {
+      // reuse previously stored metadata
+      this.entries = builder.entries;
+      this.tableName = builder.selectionRoot != null ? builder.selectionRoot.toUri().getPath() : "";
+      this.tableLocation = builder.selectionRoot;
+    } else if (builder.selection != null) {
+      this.entries = new ArrayList<>();
+      this.tableName = builder.selection.getSelectionRoot() != null ? builder.selection.getSelectionRoot().toUri().getPath() : "";
+      this.tableLocation = builder.selection.getSelectionRoot();
+    } else {
+      // case of hive parquet table
+      this.entries = new ArrayList<>();
+      this.tableName = null;
+      this.tableLocation = null;
+    }
 
-  public BaseParquetMetadataProvider(ParquetReaderConfig readerConfig,
-                                     List<ReadEntryWithPath> entries,
-                                     String tableName,
-                                     Path tableLocation,
-                                     TupleMetadata schema,
-                                     DrillStatsTable statsTable) {
-    this.entries = entries == null ? new ArrayList<>() : entries;
-    this.readerConfig = readerConfig == null ? ParquetReaderConfig.getDefaultInstance() : readerConfig;
-    this.tableName = tableName;
-    this.tableLocation = tableLocation;
-    this.schema = schema;
-    this.statsTable = statsTable;
-  }
+    SchemaProvider schemaProvider = builder.metadataProviderManager.getSchemaProvider();
+    TupleMetadata schema = builder.schema;
+    // schema passed into the builder has greater priority
+    if (schema == null && schemaProvider != null) {
+      try {
+        schema = schemaProvider.read().getSchema();
 
 Review comment:
   Please explain a bit more. At present, the `EasySubScan` has a `schema` attribute that contains a provided schema. Does the code here choose which schema appears in that field? Or, does the code here change that value?
   
   What I'm trying to understand is this: we now have three ways to specify schema: 1) Metastore, 2) Provided schema file, 3) Table function. (Just reviewed the documentation PR that describe this - very helpful.) What is the precedence? I'd think table functions are first. But, does that schema have to agree with the other schemas? Does the planner check?
   
   Can I have both a provided and metadata schema? If I have a provided schema, will the metadata system (`ANALYZE TABLE`) use it?
   
   How do those precedence rules affect the builder precedence here?
   
   I'm working on cleaning up scan-time schema usage, so I need to understand how you want all this to work.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392606953
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/metastore/store/MetastoreFileTableMetadataProvider.java
 ##########
 @@ -15,149 +15,108 @@
  * See the License for the specific language governing permissions and
  * limitations under the License.
  */
-package org.apache.drill.exec.metastore;
+package org.apache.drill.exec.metastore.store;
 
 import org.apache.drill.common.expression.SchemaPath;
 import org.apache.drill.exec.exception.MetadataException;
-import org.apache.drill.exec.metastore.MetastoreMetadataProviderManager.MetastoreMetadataProviderConfig;
+import org.apache.drill.exec.metastore.MetastoreMetadataProviderManager;
 import org.apache.drill.exec.planner.common.DrillStatsTable;
 import org.apache.drill.exec.record.SchemaUtil;
 import org.apache.drill.exec.record.metadata.TupleMetadata;
 import org.apache.drill.exec.record.metadata.schema.SchemaProvider;
 import org.apache.drill.exec.store.dfs.DrillFileSystem;
 import org.apache.drill.exec.store.dfs.FileSelection;
-import org.apache.drill.exec.store.dfs.ReadEntryWithPath;
-import org.apache.drill.exec.store.parquet.ParquetFileTableMetadataProviderBuilder;
-import org.apache.drill.exec.store.parquet.ParquetReaderConfig;
-import org.apache.drill.exec.store.parquet.ParquetTableMetadataProviderImpl;
 import org.apache.drill.exec.store.parquet.ParquetTableMetadataUtils;
 import org.apache.drill.exec.util.DrillFileSystemUtil;
-import org.apache.drill.metastore.MetastoreRegistry;
 import org.apache.drill.metastore.components.tables.BasicTablesRequests;
 import org.apache.drill.metastore.components.tables.MetastoreTableInfo;
 import org.apache.drill.metastore.metadata.BaseTableMetadata;
 import org.apache.drill.metastore.metadata.FileMetadata;
 import org.apache.drill.metastore.metadata.NonInterestingColumnsMetadata;
 import org.apache.drill.metastore.metadata.PartitionMetadata;
-import org.apache.drill.metastore.metadata.RowGroupMetadata;
 import org.apache.drill.metastore.metadata.SegmentMetadata;
 import org.apache.drill.metastore.metadata.TableInfo;
 import org.apache.drill.metastore.metadata.TableMetadata;
+import org.apache.drill.metastore.metadata.TableMetadataProvider;
+import org.apache.drill.metastore.metadata.TableMetadataProviderBuilder;
 import org.apache.drill.metastore.statistics.ColumnStatistics;
 import org.apache.drill.metastore.statistics.ColumnStatisticsKind;
 import org.apache.drill.metastore.statistics.Statistic;
 import org.apache.drill.metastore.statistics.StatisticsHolder;
 import org.apache.drill.metastore.util.SchemaPathUtils;
-import org.apache.drill.shaded.guava.com.google.common.collect.LinkedListMultimap;
-import org.apache.drill.shaded.guava.com.google.common.collect.Multimap;
 import org.apache.hadoop.fs.Path;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
 import java.io.IOException;
-import java.util.ArrayList;
 import java.util.Collections;
 import java.util.List;
 import java.util.Map;
-import java.util.Set;
 import java.util.function.Function;
 import java.util.stream.Collectors;
 
-public class MetastoreParquetTableMetadataProvider implements ParquetTableMetadataProvider {
-  private static final Logger logger = LoggerFactory.getLogger(MetastoreParquetTableMetadataProvider.class);
-
-  private final BasicTablesRequests basicTablesRequests;
-  private final TableInfo tableInfo;
-  private final MetastoreTableInfo metastoreTableInfo;
-  private final TupleMetadata schema;
-  private final List<ReadEntryWithPath> entries;
-  private final List<String> paths;
-  private final DrillStatsTable statsProvider;
-
-  private final boolean useSchema;
-  private final boolean useStatistics;
-  private final boolean fallbackToFileMetadata;
-
-  private BaseTableMetadata tableMetadata;
-  private Map<Path, SegmentMetadata> segmentsMetadata;
-  private List<PartitionMetadata> partitions;
-  private Map<Path, FileMetadata> files;
-  private Multimap<Path, RowGroupMetadata> rowGroups;
-  private NonInterestingColumnsMetadata nonInterestingColumnsMetadata;
-  // stores builder to provide lazy init for fallback ParquetTableMetadataProvider
-  private final ParquetFileTableMetadataProviderBuilder fallbackBuilder;
-  private ParquetTableMetadataProvider fallback;
-
-  private MetastoreParquetTableMetadataProvider(List<ReadEntryWithPath> entries,
-      MetastoreRegistry metastoreRegistry, TableInfo tableInfo, TupleMetadata schema,
-      ParquetFileTableMetadataProviderBuilder fallbackBuilder, MetastoreMetadataProviderConfig config, DrillStatsTable statsProvider) {
-    this.basicTablesRequests = metastoreRegistry.get().tables().basicRequests();
-    this.tableInfo = tableInfo;
-    this.metastoreTableInfo = basicTablesRequests.metastoreTableInfo(tableInfo);
-    this.useSchema = config.useSchema();
-    this.useStatistics = config.useStatistics();
-    this.fallbackToFileMetadata = config.fallbackToFileMetadata();
-    this.schema = schema;
-    this.entries = entries == null ? new ArrayList<>() : entries;
-    this.fallbackBuilder = fallbackBuilder;
-    this.statsProvider = statsProvider;
-    this.paths = this.entries.stream()
-        .map(readEntryWithPath -> readEntryWithPath.getPath().toUri().getPath())
-        .collect(Collectors.toList());
-  }
-
-  @Override
-  public boolean isUsedMetadataCache() {
-    return false;
-  }
-
-  @Override
-  public Path getSelectionRoot() {
-    return getTableMetadata().getLocation();
-  }
-
-  @Override
-  public List<ReadEntryWithPath> getEntries() {
-    return entries;
-  }
-
-  @Override
-  public List<RowGroupMetadata> getRowGroupsMeta() {
-    return new ArrayList<>(getRowGroupsMetadataMap().values());
-  }
+/**
+ * Implementation of {@link TableMetadataProvider} which uses Drill Metastore for providing table metadata
+ * for file-based tables.
+ */
+public class MetastoreFileTableMetadataProvider implements TableMetadataProvider {
+  private static final Logger logger = LoggerFactory.getLogger(MetastoreFileTableMetadataProvider.class);
+
+  protected final BasicTablesRequests basicTablesRequests;
+  protected final TableInfo tableInfo;
+  protected final MetastoreTableInfo metastoreTableInfo;
+  protected final TupleMetadata schema;
+  protected final List<String> paths;
+  protected final DrillStatsTable statsProvider;
+  protected final TableMetadataProviderBuilder fallbackBuilder;
+
+  protected final boolean useSchema;
+  protected final boolean useStatistics;
+  protected final boolean fallbackToFileMetadata;
+
+  protected BaseTableMetadata tableMetadata;
+  protected Map<Path, SegmentMetadata> segmentsMetadata;
+  protected List<PartitionMetadata> partitions;
+  protected Map<Path, FileMetadata> files;
 
 Review comment:
   One area were we struggled in Impala is the sheer number of partitions and files. Impala caches these in its "state store", replicated in each "coordinator" (like Drill's foreman.) The idea works well -- until the system scales. At that point, it takes a huge amount of memory to store the information. Impala went to great lengths to compress the information to control memory use.
   
   I don't think Drill has the scale of users as Impala (sadly), but if it did, how would this structure scale? @dobesv just discussed a case on the user list where he might have millions of files. Will this hash map work then?
   
   Also, the files form a hierarchy of directories (partitions). Does the same file metadata appear down the partition tree, or do we have to do a hash lookup from the partition to the files map for each of our million files?
   
   Another issue encountered in Impala is the cost of refresh when new files arrive. Reloading all the metadata was slow. Impala had no versioning so that a query could start planning with version 1 of metadata, which could be replaced during planning. Once the query realizes it is now working with version 2 objects, it throws an exception and starts planning over from scratch.
   
   How does Drill handle this case?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392718208
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/scan/framework/ShimBatchReader.java
 ##########
 @@ -131,6 +131,8 @@ public boolean next() {
       eof = ! reader.next();
     }
 
+    readerOrchestrator.setEof(eof);
 
 Review comment:
   This flag is only required for `readerOrchestrator.endBatch()` method to decide whether the file is empty and only its metadata row should be returned. I have removed the field and introduced the new `readerOrchestrator.endBatch(boolean)` method to make the code clearer.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392715737
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/metastore/store/MetastoreFileTableMetadataProvider.java
 ##########
 @@ -15,149 +15,108 @@
  * See the License for the specific language governing permissions and
  * limitations under the License.
  */
-package org.apache.drill.exec.metastore;
+package org.apache.drill.exec.metastore.store;
 
 import org.apache.drill.common.expression.SchemaPath;
 import org.apache.drill.exec.exception.MetadataException;
-import org.apache.drill.exec.metastore.MetastoreMetadataProviderManager.MetastoreMetadataProviderConfig;
+import org.apache.drill.exec.metastore.MetastoreMetadataProviderManager;
 import org.apache.drill.exec.planner.common.DrillStatsTable;
 import org.apache.drill.exec.record.SchemaUtil;
 import org.apache.drill.exec.record.metadata.TupleMetadata;
 import org.apache.drill.exec.record.metadata.schema.SchemaProvider;
 import org.apache.drill.exec.store.dfs.DrillFileSystem;
 import org.apache.drill.exec.store.dfs.FileSelection;
-import org.apache.drill.exec.store.dfs.ReadEntryWithPath;
-import org.apache.drill.exec.store.parquet.ParquetFileTableMetadataProviderBuilder;
-import org.apache.drill.exec.store.parquet.ParquetReaderConfig;
-import org.apache.drill.exec.store.parquet.ParquetTableMetadataProviderImpl;
 import org.apache.drill.exec.store.parquet.ParquetTableMetadataUtils;
 import org.apache.drill.exec.util.DrillFileSystemUtil;
-import org.apache.drill.metastore.MetastoreRegistry;
 import org.apache.drill.metastore.components.tables.BasicTablesRequests;
 import org.apache.drill.metastore.components.tables.MetastoreTableInfo;
 import org.apache.drill.metastore.metadata.BaseTableMetadata;
 import org.apache.drill.metastore.metadata.FileMetadata;
 import org.apache.drill.metastore.metadata.NonInterestingColumnsMetadata;
 import org.apache.drill.metastore.metadata.PartitionMetadata;
-import org.apache.drill.metastore.metadata.RowGroupMetadata;
 import org.apache.drill.metastore.metadata.SegmentMetadata;
 import org.apache.drill.metastore.metadata.TableInfo;
 import org.apache.drill.metastore.metadata.TableMetadata;
+import org.apache.drill.metastore.metadata.TableMetadataProvider;
+import org.apache.drill.metastore.metadata.TableMetadataProviderBuilder;
 import org.apache.drill.metastore.statistics.ColumnStatistics;
 import org.apache.drill.metastore.statistics.ColumnStatisticsKind;
 import org.apache.drill.metastore.statistics.Statistic;
 import org.apache.drill.metastore.statistics.StatisticsHolder;
 import org.apache.drill.metastore.util.SchemaPathUtils;
-import org.apache.drill.shaded.guava.com.google.common.collect.LinkedListMultimap;
-import org.apache.drill.shaded.guava.com.google.common.collect.Multimap;
 import org.apache.hadoop.fs.Path;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
 import java.io.IOException;
-import java.util.ArrayList;
 import java.util.Collections;
 import java.util.List;
 import java.util.Map;
-import java.util.Set;
 import java.util.function.Function;
 import java.util.stream.Collectors;
 
-public class MetastoreParquetTableMetadataProvider implements ParquetTableMetadataProvider {
-  private static final Logger logger = LoggerFactory.getLogger(MetastoreParquetTableMetadataProvider.class);
-
-  private final BasicTablesRequests basicTablesRequests;
-  private final TableInfo tableInfo;
-  private final MetastoreTableInfo metastoreTableInfo;
-  private final TupleMetadata schema;
-  private final List<ReadEntryWithPath> entries;
-  private final List<String> paths;
-  private final DrillStatsTable statsProvider;
-
-  private final boolean useSchema;
-  private final boolean useStatistics;
-  private final boolean fallbackToFileMetadata;
-
-  private BaseTableMetadata tableMetadata;
-  private Map<Path, SegmentMetadata> segmentsMetadata;
-  private List<PartitionMetadata> partitions;
-  private Map<Path, FileMetadata> files;
-  private Multimap<Path, RowGroupMetadata> rowGroups;
-  private NonInterestingColumnsMetadata nonInterestingColumnsMetadata;
-  // stores builder to provide lazy init for fallback ParquetTableMetadataProvider
-  private final ParquetFileTableMetadataProviderBuilder fallbackBuilder;
-  private ParquetTableMetadataProvider fallback;
-
-  private MetastoreParquetTableMetadataProvider(List<ReadEntryWithPath> entries,
-      MetastoreRegistry metastoreRegistry, TableInfo tableInfo, TupleMetadata schema,
-      ParquetFileTableMetadataProviderBuilder fallbackBuilder, MetastoreMetadataProviderConfig config, DrillStatsTable statsProvider) {
-    this.basicTablesRequests = metastoreRegistry.get().tables().basicRequests();
-    this.tableInfo = tableInfo;
-    this.metastoreTableInfo = basicTablesRequests.metastoreTableInfo(tableInfo);
-    this.useSchema = config.useSchema();
-    this.useStatistics = config.useStatistics();
-    this.fallbackToFileMetadata = config.fallbackToFileMetadata();
-    this.schema = schema;
-    this.entries = entries == null ? new ArrayList<>() : entries;
-    this.fallbackBuilder = fallbackBuilder;
-    this.statsProvider = statsProvider;
-    this.paths = this.entries.stream()
-        .map(readEntryWithPath -> readEntryWithPath.getPath().toUri().getPath())
-        .collect(Collectors.toList());
-  }
-
-  @Override
-  public boolean isUsedMetadataCache() {
-    return false;
-  }
-
-  @Override
-  public Path getSelectionRoot() {
-    return getTableMetadata().getLocation();
-  }
-
-  @Override
-  public List<ReadEntryWithPath> getEntries() {
-    return entries;
-  }
-
-  @Override
-  public List<RowGroupMetadata> getRowGroupsMeta() {
-    return new ArrayList<>(getRowGroupsMetadataMap().values());
-  }
+/**
+ * Implementation of {@link TableMetadataProvider} which uses Drill Metastore for providing table metadata
+ * for file-based tables.
+ */
+public class MetastoreFileTableMetadataProvider implements TableMetadataProvider {
+  private static final Logger logger = LoggerFactory.getLogger(MetastoreFileTableMetadataProvider.class);
+
+  protected final BasicTablesRequests basicTablesRequests;
+  protected final TableInfo tableInfo;
+  protected final MetastoreTableInfo metastoreTableInfo;
+  protected final TupleMetadata schema;
+  protected final List<String> paths;
+  protected final DrillStatsTable statsProvider;
+  protected final TableMetadataProviderBuilder fallbackBuilder;
+
+  protected final boolean useSchema;
+  protected final boolean useStatistics;
+  protected final boolean fallbackToFileMetadata;
+
+  protected BaseTableMetadata tableMetadata;
+  protected Map<Path, SegmentMetadata> segmentsMetadata;
+  protected List<PartitionMetadata> partitions;
+  protected Map<Path, FileMetadata> files;
 
 Review comment:
   For the case of many files, the user can store metadata for partitions only, so Drill will do the pruning at partitions level.
   
   Drill is also able to discover that metadata was changed, and for this case, it starts planning the query with a newer version of metadata. Number of attempts is configured using `metastore.retrieval.retry_attempts` option. For the case when the number of attempts is exceeded, the query will be planned without metastore usage.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r394692519
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/scan/file/FileMetadata.java
 ##########
 @@ -31,9 +32,15 @@
 
   private final Path filePath;
   private final String[] dirPath;
+  private final FileSystem fs;
 
   public FileMetadata(Path filePath, Path selectionRoot) {
+    this(filePath, selectionRoot, null);
+  }
+
+  public FileMetadata(Path filePath, Path selectionRoot, FileSystem fs) {
 
 Review comment:
   Can we add a comment to relate these? The `filePath` must be fully qualified for the file system `fs`. The `selectionRoot`, if provided, must be a prefix of `filePath`. `filePath` should not include the file system prefix. (I *think* those are the relationships, but your understanding is fresh, is this correct?)

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392623969
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/easy/EasyGroupScan.java
 ##########
 @@ -124,13 +127,16 @@ public EasyGroupScan(
       // use file system metadata provider without specified schema and statistics
       metadataProviderManager = new FileSystemMetadataProviderManager();
     }
-    SimpleFileTableMetadataProviderBuilder builder =
-        (SimpleFileTableMetadataProviderBuilder) metadataProviderManager.builder(
-            MetadataProviderManager.MetadataProviderKind.SCHEMA_STATS_ONLY);
+    DrillFileSystem fs =
+        ImpersonationUtil.createFileSystem(ImpersonationUtil.resolveUserName(userName), formatPlugin.getFsConf());
 
-    this.metadataProvider = builder.withLocation(selection.getSelectionRoot())
+    this.metadataProvider = tableMetadataProviderBuilder(metadataProviderManager)
+        .withSelection(selection)
+        .withFileSystem(fs)
         .build();
+    this.usedMetastore = metadataProviderManager.usesMetastore();
     initFromSelection(selection, formatPlugin);
+    checkMetadataConsistency(selection, formatPlugin.getFsConf());
 
 Review comment:
   This has been nagging at me. For Parquet, metadata includes both partition information and information about the insides of files (row groups, etc.) But, for files other than Parquet, there is no useful information in metadata about file contents. As a result, all of the benefit of metadata is to assist with partition pruning. Metadata avoids the need to walk the directory tree.
   
   However, in order to ensure that the metadata is consistent we... walk the directory tree.
   
   So, for files other than Parquet, are we gaining anything (other than more complexity) by using metadata if we must check the tree on each query?
   
   There *is* a gain if we can trust the metadata and avoid the walk of the directory tree. (See comments elsewhere which no longer appear in this code view since my comments overlapped with your next round of changes.)

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392607246
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/metastore/store/MetastoreFileTableMetadataProvider.java
 ##########
 @@ -291,124 +250,74 @@ public boolean checkMetadataVersion() {
     }
   }
 
-  private ParquetTableMetadataProvider getFallbackTableMetadataProvider() throws IOException {
-    if (fallback == null) {
-      fallback = fallbackBuilder == null ? null : fallbackBuilder.build();
-    }
-    return fallback;
-  }
+  public static class Builder<T extends Builder<T>> implements FileTableMetadataProviderBuilder<T> {
+    protected final MetastoreMetadataProviderManager metadataProviderManager;
 
-  private void throwIfChanged() {
-    if (basicTablesRequests.hasMetastoreTableInfoChanged(metastoreTableInfo)) {
-      throw MetadataException.of(MetadataException.MetadataExceptionType.INCONSISTENT_METADATA);
-    }
-  }
+    // builder for fallback ParquetFileTableMetadataProvider
+    // for the case when required metadata is absent in Metastore
+    protected final TableMetadataProviderBuilder fallback;
 
-  public static class Builder implements ParquetFileTableMetadataProviderBuilder {
-    private final MetastoreMetadataProviderManager metadataProviderManager;
+    protected TupleMetadata schema;
 
-    private List<ReadEntryWithPath> entries;
-    private DrillFileSystem fs;
-    private TupleMetadata schema;
+    protected List<String> paths;
 
     private FileSelection selection;
 
-    // builder for fallback ParquetFileTableMetadataProvider
-    // for the case when required metadata is absent in Metastore
-    private final ParquetFileTableMetadataProviderBuilder fallback;
+    private DrillFileSystem fs;
 
     public Builder(MetastoreMetadataProviderManager source) {
-      this.metadataProviderManager = source;
-      this.fallback = new ParquetTableMetadataProviderImpl.Builder(FileSystemMetadataProviderManager.init());
+      this(source, new SimpleFileTableMetadataProvider.Builder(FileSystemMetadataProviderManager.init()));
     }
 
-    @Override
-    public ParquetFileTableMetadataProviderBuilder withEntries(List<ReadEntryWithPath> entries) {
-      this.entries = entries;
-      fallback.withEntries(entries);
-      return this;
+    protected Builder(MetastoreMetadataProviderManager source, TableMetadataProviderBuilder fallback) {
+      this.metadataProviderManager = source;
+      this.fallback = fallback;
     }
 
     @Override
-    public ParquetFileTableMetadataProviderBuilder withSelectionRoot(Path selectionRoot) {
-      fallback.withSelectionRoot(selectionRoot);
-      return this;
+    public T withSchema(TupleMetadata schema) {
+      this.schema = schema;
+      return self();
     }
 
-    @Override
-    public ParquetFileTableMetadataProviderBuilder withCacheFileRoot(Path cacheFileRoot) {
-      fallback.withCacheFileRoot(cacheFileRoot);
-      return this;
+    public T withSelection(FileSelection selection) {
+      this.selection = selection;
+      return self();
     }
 
-    @Override
-    public ParquetFileTableMetadataProviderBuilder withReaderConfig(ParquetReaderConfig readerConfig) {
-      fallback.withReaderConfig(readerConfig);
-      return this;
+    public T withFileSystem(DrillFileSystem fs) {
+      this.fs = fs;
+      return self();
     }
 
-    @Override
-    public ParquetFileTableMetadataProviderBuilder withFileSystem(DrillFileSystem fs) {
-      fallback.withFileSystem(fs);
-      this.fs = fs;
-      return this;
+    protected T self() {
+      return (T) this;
     }
 
-    @Override
-    public ParquetFileTableMetadataProviderBuilder withCorrectCorruptedDates(boolean autoCorrectCorruptedDates) {
-      fallback.withCorrectCorruptedDates(autoCorrectCorruptedDates);
-      return this;
+    public MetastoreMetadataProviderManager metadataProviderManager() {
+      return metadataProviderManager;
     }
 
-    @Override
-    public ParquetFileTableMetadataProviderBuilder withSelection(FileSelection selection) {
-      fallback.withSelection(selection);
-      this.selection = selection;
-      return this;
+    public FileSelection selection() {
+      return selection;
     }
 
-    @Override
-    public ParquetFileTableMetadataProviderBuilder withSchema(TupleMetadata schema) {
-      fallback.withSchema(schema);
-      this.schema = schema;
-      return this;
+    public DrillFileSystem fs() {
+      return fs;
     }
 
     @Override
-    public ParquetTableMetadataProvider build() throws IOException {
-      MetastoreParquetTableMetadataProvider provider;
-      SchemaProvider schemaProvider = metadataProviderManager.getSchemaProvider();
-      ParquetMetadataProvider source = (ParquetTableMetadataProvider) metadataProviderManager.getTableMetadataProvider();
-
-      DrillStatsTable statsProvider = metadataProviderManager.getStatsProvider();
-      // schema passed into the builder has greater priority
-      try {
-        if (this.schema == null) {
-          schema = schemaProvider != null ? schemaProvider.read().getSchema() : null;
-        }
-      } catch (IOException e) {
-        logger.debug("Unable to deserialize schema from schema file for table: {}", metadataProviderManager.getTableInfo().name(), e);
-      }
-      if (entries == null) {
-        if (!selection.isExpandedFully()) {
-          entries = DrillFileSystemUtil.listFiles(fs, selection.getSelectionRoot(), true).stream()
-              .map(fileStatus -> new ReadEntryWithPath(Path.getPathWithoutSchemeAndAuthority(fileStatus.getPath())))
-              .collect(Collectors.toList());
-        } else {
-          entries = selection.getFiles().stream()
-              .map(Path::getPathWithoutSchemeAndAuthority)
-              .map(ReadEntryWithPath::new)
-              .collect(Collectors.toList());
-        }
-      }
-      provider = new MetastoreParquetTableMetadataProvider(entries, metadataProviderManager.getMetastoreRegistry(),
-          metadataProviderManager.getTableInfo(), schema, fallback, metadataProviderManager.getConfig(), statsProvider);
-      // store results into metadataProviderManager to be able to use them when creating new instances
-      // for the case when source wasn't provided or it contains less row group metadata than the provider
-      if (source == null || source.getRowGroupsMeta().size() < provider.getRowGroupsMeta().size()) {
-        metadataProviderManager.setTableMetadataProvider(provider);
+    public TableMetadataProvider build() throws IOException {
+      if (!selection().isExpandedFully()) {
 
 Review comment:
   Nit: maybe reverse polarity if the if/then clauses and remove the `!`.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392631705
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/scan/file/ImplicitColumnManager.java
 ##########
 @@ -199,6 +192,21 @@ public ImplicitColumnManager(OptionSet optionManager,
     }
   }
 
+  /**
+   * Specifies whether to plan based on the legacy meaning of "*". See
+   * <a href="https://issues.apache.org/jira/browse/DRILL-5542">DRILL-5542</a>.
+   * If true, then the star column <i>includes</i> implicit and partition
+   * columns. If false, then star matches <i>only</i> table columns.
+   *
+   * @param optionManager access to the options for this query; used
+   *                      too look up custom names for the metadata columns
+   * @param config        implicit column options
+   */
+  public ImplicitColumnManager(OptionSet optionManager,
 
 Review comment:
   Help me understand your intent. Older Drill versions always expanded implicit columns in wildcard queries. Then the project threw away those columns. We changed this so that the user asks for the columns they want. We could do this because Calcite changed (as I understand it) so that it will allow a projection list of the form `SELECT *, filename`.
   
   Does metadata need the implicit columns expanded for a wildcard query? Is this the behavior for the other, non-EVF readers?
   
   If so, we'll want to balance the needs of metadata (needs those columns) vs. other queries (do not need them and don't want to pay the cost.)

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] arina-ielchiieva edited a comment on issue #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
arina-ielchiieva edited a comment on issue #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#issuecomment-599465366
 
 
   LGTM, +1. Let's wait for Paul's approval as well.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392609028
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/scan/project/ReaderSchemaOrchestrator.java
 ##########
 @@ -36,10 +41,16 @@
 public class ReaderSchemaOrchestrator implements VectorSource {
 
   private final ScanSchemaOrchestrator scanOrchestrator;
-  private int readerBatchSize = ScanSchemaOrchestrator.MAX_BATCH_ROW_COUNT;
+  private int readerBatchSize;
   private ResultSetLoaderImpl tableLoader;
   private int prevTableSchemaVersion = -1;
 
+  /**
+   * True once the reader reports EOF. This shim may keep going for another
+   * batch to handle any look-ahead row on the last batch.
+   */
+  private boolean eof;
 
 Review comment:
   See note above. The methods here should give you what you need. 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392607198
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/metastore/store/MetastoreFileTableMetadataProvider.java
 ##########
 @@ -291,124 +250,74 @@ public boolean checkMetadataVersion() {
     }
   }
 
-  private ParquetTableMetadataProvider getFallbackTableMetadataProvider() throws IOException {
-    if (fallback == null) {
-      fallback = fallbackBuilder == null ? null : fallbackBuilder.build();
-    }
-    return fallback;
-  }
+  public static class Builder<T extends Builder<T>> implements FileTableMetadataProviderBuilder<T> {
+    protected final MetastoreMetadataProviderManager metadataProviderManager;
 
-  private void throwIfChanged() {
-    if (basicTablesRequests.hasMetastoreTableInfoChanged(metastoreTableInfo)) {
-      throw MetadataException.of(MetadataException.MetadataExceptionType.INCONSISTENT_METADATA);
-    }
-  }
+    // builder for fallback ParquetFileTableMetadataProvider
+    // for the case when required metadata is absent in Metastore
+    protected final TableMetadataProviderBuilder fallback;
 
-  public static class Builder implements ParquetFileTableMetadataProviderBuilder {
-    private final MetastoreMetadataProviderManager metadataProviderManager;
+    protected TupleMetadata schema;
 
-    private List<ReadEntryWithPath> entries;
-    private DrillFileSystem fs;
-    private TupleMetadata schema;
+    protected List<String> paths;
 
     private FileSelection selection;
 
-    // builder for fallback ParquetFileTableMetadataProvider
-    // for the case when required metadata is absent in Metastore
-    private final ParquetFileTableMetadataProviderBuilder fallback;
+    private DrillFileSystem fs;
 
     public Builder(MetastoreMetadataProviderManager source) {
-      this.metadataProviderManager = source;
-      this.fallback = new ParquetTableMetadataProviderImpl.Builder(FileSystemMetadataProviderManager.init());
+      this(source, new SimpleFileTableMetadataProvider.Builder(FileSystemMetadataProviderManager.init()));
     }
 
-    @Override
-    public ParquetFileTableMetadataProviderBuilder withEntries(List<ReadEntryWithPath> entries) {
-      this.entries = entries;
-      fallback.withEntries(entries);
-      return this;
+    protected Builder(MetastoreMetadataProviderManager source, TableMetadataProviderBuilder fallback) {
+      this.metadataProviderManager = source;
+      this.fallback = fallback;
     }
 
     @Override
-    public ParquetFileTableMetadataProviderBuilder withSelectionRoot(Path selectionRoot) {
-      fallback.withSelectionRoot(selectionRoot);
-      return this;
+    public T withSchema(TupleMetadata schema) {
 
 Review comment:
   Nice job generalizing this for any kind of builder. I wonder, however, will we need to create a builder for each format? Or, is there a generic non-Parquet builder which all other file types can use (at least until it becomes clear that some type would benefit from its own builder. Avro might be an example since it provides its own schema.)

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] arina-ielchiieva commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
arina-ielchiieva commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392591157
 
 

 ##########
 File path: contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/HiveDrillNativeParquetScan.java
 ##########
 @@ -213,11 +231,25 @@ protected boolean supportsFileImplicitColumns() {
     return hivePartitionHolder.get(locationProvider.getPath());
   }
 
+  @Override
+  protected ParquetMetadataProviderBuilder<?> tableMetadataProviderBuilder(MetadataProviderManager source) {
+    if (source.usesMetastore()) {
 
 Review comment:
   When this case is possible?
   When wee try to run analyze over Hive table using native parquet scan? Or when metastore is enabled and we try to query Hive table with native parquet scan? I really hope this is not the second case.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r394689649
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/metastore/store/SimpleFileTableMetadataProvider.java
 ##########
 @@ -97,47 +100,38 @@ public FileMetadata getFileMetadata(Path location) {
 
   @Override
   public NonInterestingColumnsMetadata getNonInterestingColumnsMetadata() {
-    return null;
+    return new NonInterestingColumnsMetadata(Collections.emptyMap());
 
 Review comment:
   Better name? `NonInteresting` --> `Uninteresting`?
   
   But, why are they not interesting? Is it because they we don't track their statistics? Is so, maybe we can define a term "tracked? untracked?" to describe those columns which do (or do not) participate in statistics. I did a quick check or Oracle, Postgres and SqlServer, but I didn't find a good term. Maybe "excluded columns"?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392609285
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/scan/project/ReaderSchemaOrchestrator.java
 ##########
 @@ -118,9 +135,59 @@ public void endBatch() {
       // Fill in the null and metadata columns.
       populateNonDataColumns();
     }
+    if (projected) {
+      setProjectMetadata(null);
+    }
     rootTuple.setRowCount(tableContainer.getRecordCount());
   }
 
+  /**
+   * Updates {@code PROJECT_METADATA} implicit column value to {@code "FALSE"} to handle current batch as
+   * a batch with metadata information only for the case when this batch is first and empty.
+   */
 
 Review comment:
   Why? As it turns out, this kind of logic is handled in the `ScanOperatorExec` and `ReaderState`. See https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/scan/ReaderState.java#L444
   
   I have to apologize, that reader code is quite tricky. It was hard to handle all the strange things that readers can do such as hitting EOF on open (a CSV file declared with headers but which is empty), on the first batch (a CVS file without headers which is empty), and so on.
   
   Can we insert this logic there somehow? Even better, does the existing handling provide the behavior you are trying to achieve? Maybe this logic was needed for the old scan which was less graceful at handling odd cases.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392642125
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/easy/EasyGroupScan.java
 ##########
 @@ -333,21 +338,73 @@ public boolean canPushdownProjects(List<SchemaPath> columns) {
   }
 
   @JsonProperty
+  @JsonIgnore(value = false)
   public TupleMetadata getSchema() {
     return getTableMetadata().getSchema();
   }
 
   @Override
-  @JsonIgnore
-  public TableMetadata getTableMetadata() {
-    if (tableMetadata == null) {
-      tableMetadata = metadataProvider.getTableMetadata();
+  public AnalyzeInfoProvider getAnalyzeInfoProvider() {
+    return new AnalyzeFileInfoProviderImpl(formatPlugin.getName());
+  }
+
+  @Override
+  protected GroupScanWithMetadataFilterer<?> getFilterer() {
+    return new EasyGroupScanFilterer(this);
+  }
+
+  @Override
+  protected FileTableMetadataProviderBuilder<?> tableMetadataProviderBuilder(MetadataProviderManager source) {
+    if (source.usesMetastore()) {
+      return new MetastoreFileTableMetadataProvider.Builder<>((MetastoreMetadataProviderManager) source);
+    } else {
+      return defaultTableMetadataProviderBuilder(source);
     }
-    return tableMetadata;
   }
 
   @Override
-  public TableMetadataProvider getMetadataProvider() {
-    return metadataProvider;
+  protected FileTableMetadataProviderBuilder<?> defaultTableMetadataProviderBuilder(MetadataProviderManager source) {
+    return new SimpleFileTableMetadataProvider.Builder(source);
+  }
+
+  /**
+   * Implementation of GroupScanWithMetadataFilterer which uses {@link EasyGroupScan} as source and
+   * builds {@link EasyGroupScan} instance with filtered metadata.
+   */
+  private static class EasyGroupScanFilterer extends GroupScanWithMetadataFilterer<EasyGroupScanFilterer> {
+
+    EasyGroupScanFilterer(EasyGroupScan source) {
+      super(source);
+    }
+
+    @Override
+    public AbstractGroupScanWithMetadata<?> build() {
+      EasyGroupScan newScan = new EasyGroupScan((EasyGroupScan) source);
+      newScan.tableMetadata = tableMetadata;
+      // updates common row count and nulls counts for every column
+      if (newScan.getTableMetadata() != null && files != null && newScan.getFilesMetadata().size() != files.size()) {
+        newScan.tableMetadata = TableMetadataUtils.updateRowCount(newScan.getTableMetadata(), files.values());
+      }
+      newScan.partitions = partitions;
+      newScan.segments = segments;
+      newScan.files = files;
+      newScan.matchAllMetadata = matchAllMetadata;
+      newScan.nonInterestingColumnsMetadata = nonInterestingColumnsMetadata;
+
+      newScan.fileSet = newScan.getFilesMetadata().keySet();
+      newScan.selection = FileSelection.create(null, new ArrayList<>(newScan.fileSet), newScan.selectionRoot);
+      try {
+        newScan.initFromSelection(newScan.selection, newScan.formatPlugin);
+      } catch (IOException e) {
+        throw new RuntimeException("Failed to initialize scan from the selection.", e);
 
 Review comment:
   I wonder, should this be a `UserException` so we can give the user some context about what went wrong? Or, is this a "should never occur unless the code is wrong" exception, in which case `IllegalStateException` would be better.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
vvysotskyi commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392719881
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/store/ColumnExplorer.java
 ##########
 @@ -292,39 +289,86 @@ public static int getPartitionDepth(FileSelection selection) {
    * @param includeFileImplicitColumns if file implicit columns should be included into the result
    * @param fs                         file system
    * @param index                      index of row group to populate
+   * @param start                      start of row group to populate
+   * @param length                     length of row group to populate
    * @return implicit columns map
    */
   public Map<String, String> populateImplicitAndInternalColumns(Path filePath,
       List<String> partitionValues, boolean includeFileImplicitColumns, FileSystem fs, int index, long start, long length) {
 
     Map<String, String> implicitValues =
-        new LinkedHashMap<>(populateImplicitColumns(filePath, partitionValues, includeFileImplicitColumns));
+        new LinkedHashMap<>(populateImplicitAndInternalColumns(filePath, partitionValues, includeFileImplicitColumns, fs));
 
-    selectedInternalColumns.forEach((key, value) -> {
-      switch (value) {
+    selectedInternalColumns.forEach(
+        (key, value) -> implicitValues.put(key, getImplicitColumnValue(value, filePath, fs, index, start, length)));
+
+    return implicitValues;
+  }
+
+  /**
+   * Returns implicit column value for specified implicit file column.
+   *
+   * @param column   implicit file column
+   * @param filePath file path, used to populate file implicit columns
+   * @param fs       file system
+   * @param index    row group index
+   * @param start    row group start
+   * @param length   row group length
+   * @return implicit column value for specified implicit file column
+   */
+  private static String getImplicitColumnValue(ImplicitFileColumn column, Path filePath,
+      FileSystem fs, Integer index, Long start, Long length) {
+    if (column instanceof ImplicitFileColumns) {
+      ImplicitFileColumns fileColumn = (ImplicitFileColumns) column;
+      return fileColumn.getValue(filePath);
+    } else if (column instanceof ImplicitInternalFileColumns) {
+      ImplicitInternalFileColumns fileColumn = (ImplicitInternalFileColumns) column;
+      switch (fileColumn) {
         case ROW_GROUP_INDEX:
-          implicitValues.put(key, String.valueOf(index));
-          break;
+          return index != null ? String.valueOf(index) : null;
         case ROW_GROUP_START:
-          implicitValues.put(key, String.valueOf(start));
-          break;
+          return start != null ? String.valueOf(start) : null;
         case ROW_GROUP_LENGTH:
-          implicitValues.put(key, String.valueOf(length));
-          break;
+          return length != null ? String.valueOf(length) : null;
         case PROJECT_METADATA:
-          implicitValues.put(key, Boolean.TRUE.toString());
-          break;
+          return Boolean.TRUE.toString();
         case LAST_MODIFIED_TIME:
           try {
-            implicitValues.put(key, String.valueOf(fs.getFileStatus(filePath).getModificationTime()));
+            return fs != null ? String.valueOf(fs.getFileStatus(filePath).getModificationTime()) : null;
 
 Review comment:
   Yes, that's correct that it will fail if the file will be deleted before this value is obtained. But if I understand the flow correctly, at first, the batch reader will open the file and try to read. After the batch is full or all the data is read, the framework will populate the implicit columns. So theoretically, it may fail here only after its data reading is finished and the file is deleted before implicit column values are populated.
   
   Also, this implicit column is for internal usage and used only when collecting table metadata during `ANALYZE` queries.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392611438
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/easy/EasyGroupScan.java
 ##########
 @@ -333,21 +338,73 @@ public boolean canPushdownProjects(List<SchemaPath> columns) {
   }
 
   @JsonProperty
+  @JsonIgnore(value = false)
   public TupleMetadata getSchema() {
     return getTableMetadata().getSchema();
   }
 
   @Override
-  @JsonIgnore
-  public TableMetadata getTableMetadata() {
-    if (tableMetadata == null) {
-      tableMetadata = metadataProvider.getTableMetadata();
+  public AnalyzeInfoProvider getAnalyzeInfoProvider() {
+    return new AnalyzeFileInfoProviderImpl(formatPlugin.getName());
+  }
+
+  @Override
+  protected GroupScanWithMetadataFilterer<?> getFilterer() {
+    return new EasyGroupScanFilterer(this);
+  }
+
+  @Override
+  protected FileTableMetadataProviderBuilder<?> tableMetadataProviderBuilder(MetadataProviderManager source) {
+    if (source.usesMetastore()) {
+      return new MetastoreFileTableMetadataProvider.Builder<>((MetastoreMetadataProviderManager) source);
+    } else {
+      return defaultTableMetadataProviderBuilder(source);
     }
-    return tableMetadata;
   }
 
   @Override
-  public TableMetadataProvider getMetadataProvider() {
-    return metadataProvider;
+  protected FileTableMetadataProviderBuilder<?> defaultTableMetadataProviderBuilder(MetadataProviderManager source) {
+    return new SimpleFileTableMetadataProvider.Builder(source);
+  }
+
+  /**
+   * Implementation of GroupScanWithMetadataFilterer which uses {@link EasyGroupScan} as source and
+   * builds {@link EasyGroupScan} instance with filtered metadata.
+   */
+  private static class EasyGroupScanFilterer extends GroupScanWithMetadataFilterer<EasyGroupScanFilterer> {
+
+    EasyGroupScanFilterer(EasyGroupScan source) {
+      super(source);
+    }
+
+    @Override
+    public AbstractGroupScanWithMetadata<?> build() {
+      EasyGroupScan newScan = new EasyGroupScan((EasyGroupScan) source);
+      newScan.tableMetadata = tableMetadata;
+      // updates common row count and nulls counts for every column
+      if (newScan.getTableMetadata() != null && files != null && newScan.getFilesMetadata().size() != files.size()) {
+        newScan.tableMetadata = TableMetadataUtils.updateRowCount(newScan.getTableMetadata(), files.values());
+      }
+      newScan.partitions = partitions;
+      newScan.segments = segments;
+      newScan.files = files;
+      newScan.matchAllMetadata = matchAllMetadata;
+      newScan.nonInterestingColumnsMetadata = nonInterestingColumnsMetadata;
+
+      newScan.fileSet = newScan.getFilesMetadata().keySet();
+      newScan.selection = FileSelection.create(null, new ArrayList<>(newScan.fileSet), newScan.selectionRoot);
+      try {
+        newScan.initFromSelection(newScan.selection, newScan.formatPlugin);
+      } catch (IOException e) {
+        throw new RuntimeException("Failed to initialize scan from the selection.", e);
+      }
+
+      return newScan;
+    }
+
+    @Override
+    protected EasyGroupScanFilterer self() {
+      return this;
+    }
 
 Review comment:
   Seems odd. What does this do?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392607688
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/AbstractGroupScanWithMetadata.java
 ##########
 @@ -634,6 +642,62 @@ public NonInterestingColumnsMetadata getNonInterestingColumnsMetadata() {
     return nonInterestingColumnsMetadata;
   }
 
+  /**
+   * Returns {@link TableMetadataProviderBuilder} instance based on specified
+   * {@link MetadataProviderManager} source.
+   *
+   * @param source metadata provider manager
+   * @return {@link TableMetadataProviderBuilder} instance
+   */
+  protected abstract TableMetadataProviderBuilder tableMetadataProviderBuilder(MetadataProviderManager source);
+
+  /**
+   * Returns {@link TableMetadataProviderBuilder} instance which may provide metadata
+   * without using Drill Metastore.
+   *
+   * @param source metadata provider manager
+   * @return {@link TableMetadataProviderBuilder} instance
+   */
+  protected abstract TableMetadataProviderBuilder defaultTableMetadataProviderBuilder(MetadataProviderManager source);
+
+  /**
+   * Compares the last modified time of files obtained from specified selection with
+   * the Metastore last modified time to determine whether Metastore metadata
+   * is not outdated. If metadata is outdated, {@link MetadataException} will be thrown.
+   *
+   * @param selection the source of files to check
+   * @throws MetadataException if metadata is outdated
+   */
+  protected void checkMetadataConsistency(FileSelection selection, Configuration fsConf) throws IOException {
+    if (metadataProvider.checkMetadataVersion()) {
+      DrillFileSystem fileSystem =
+          ImpersonationUtil.createFileSystem(ImpersonationUtil.resolveUserName(getUserName()), fsConf);
+
+      List<FileStatus> fileStatuses = FileMetadataInfoCollector.getFileStatuses(selection, fileSystem);
+
+      long lastModifiedTime = metadataProvider.getTableMetadata().getLastModifiedTime();
+
+      Set<Path> removedFiles = new HashSet<>(metadataProvider.getFilesMetadataMap().keySet());
+      Set<Path> newFiles = new HashSet<>();
+
+      boolean isChanged = false;
+
+      for (FileStatus fileStatus : fileStatuses) {
+        if (!removedFiles.remove(Path.getPathWithoutSchemeAndAuthority(fileStatus.getPath()))) {
+          newFiles.add(fileStatus.getPath());
+        }
+        if (fileStatus.getModificationTime() > lastModifiedTime) {
+          isChanged = true;
+          break;
+        }
+      }
 
 Review comment:
   The above may be costly for millions of files. Do we have a way that an external system can ping us when a file is added? (HDFS has some partially-completed feature to send an event when the file system changes.) Or, can Drill with the files available until an external system (or the user) tells us to refresh? Impala had a `REFRESH METADATA` command for this purpose. One of the big problems was that people would run it before every query; as a result Impala spent more time refreshing metadata than doing actual work. Would be nice if Drill didn't have to learn that same lesson.
   
   Can we do that refresh in parallel with the current metadata so that queries continue to run with the existing metadata as we gather the new set?
   
   Again, imagine the case that @dobesv recently explained: millions of files, data constantly arriving.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

[GitHub] [drill] paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins

Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #2026: DRILL-7330: Implement metadata usage for all format plugins
URL: https://github.com/apache/drill/pull/2026#discussion_r392607531
 
 

 ##########
 File path: exec/java-exec/src/main/java/org/apache/drill/exec/physical/base/AbstractGroupScanWithMetadata.java
 ##########
 @@ -634,6 +642,62 @@ public NonInterestingColumnsMetadata getNonInterestingColumnsMetadata() {
     return nonInterestingColumnsMetadata;
   }
 
+  /**
+   * Returns {@link TableMetadataProviderBuilder} instance based on specified
+   * {@link MetadataProviderManager} source.
+   *
+   * @param source metadata provider manager
+   * @return {@link TableMetadataProviderBuilder} instance
+   */
+  protected abstract TableMetadataProviderBuilder tableMetadataProviderBuilder(MetadataProviderManager source);
+
+  /**
+   * Returns {@link TableMetadataProviderBuilder} instance which may provide metadata
+   * without using Drill Metastore.
+   *
+   * @param source metadata provider manager
+   * @return {@link TableMetadataProviderBuilder} instance
+   */
+  protected abstract TableMetadataProviderBuilder defaultTableMetadataProviderBuilder(MetadataProviderManager source);
+
+  /**
+   * Compares the last modified time of files obtained from specified selection with
+   * the Metastore last modified time to determine whether Metastore metadata
+   * is not outdated. If metadata is outdated, {@link MetadataException} will be thrown.
 
 Review comment:
   "is not outdated" --> "is current" or "is outdated"

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services