[jira] [Commented] (DRILL-5106) Refactor SkipRecordsInspector to exclude check for predefined file formats

["Chunhui Shi (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/DRILL-5106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15722892#comment-15722892 ] 

Chunhui Shi commented on DRILL-5106:
------------------------------------

And one more improvement can be made: bufAdd keep pushing and popping if footerCount is 0, this is unnecessary cost. 

The same when headerCount is zero, we don't have to do doSkipHeader all the time. Even when headerCount is not zero, Ideally we should not need to do doSkipHeader all the time if there is a way we could skip enough records then we go to another stage/function there we don't need to check skip or not again and again.


> Refactor SkipRecordsInspector to exclude check for predefined file formats
> --------------------------------------------------------------------------
>
>                 Key: DRILL-5106
>                 URL: https://issues.apache.org/jira/browse/DRILL-5106
>             Project: Apache Drill
>          Issue Type: Improvement
>          Components: Storage - Hive
>    Affects Versions: 1.9.0
>            Reporter: Arina Ielchiieva
>            Priority: Minor
>
> After changes introduced in DRILL-4982, SkipRecordInspector is used only for predefined formats (using hasHeaderFooter: false / true). But SkipRecordInspector has its own check for formats where skip strategy can be applied. Acceptable file formats are stored in private final Set<Object> fileFormats and initialized in constructor, currently it contains only one format - TextInputFormat. Now this check is redundant and may lead to ignoring hasHeaderFooter setting to true for any other format except of Text.
> To do:
> 1. remove private final Set<Object> fileFormats
> 2. remove if block from SkipRecordsInspector.retrievePositiveIntProperty:
> {code}
>  if (!fileFormats.contains(tableProperties.get(hive_metastoreConstants.FILE_INPUT_FORMAT))) {
> return propertyIntValue;
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)