You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@drill.apache.org by "Arina Ielchiieva (JIRA)" <ji...@apache.org> on 2016/12/05 12:20:58 UTC
[jira] [Updated] (DRILL-5106) Refactor SkipRecordsInspector to
exclude check for predefined file formats
[ https://issues.apache.org/jira/browse/DRILL-5106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arina Ielchiieva updated DRILL-5106:
------------------------------------
Description:
After changes introduced in DRILL-4982, SkipRecordInspector is used only for predefined formats (using hasHeaderFooter: false / true). But SkipRecordInspector has its own check for formats where skip strategy can be applied. Acceptable file formats are stored in private final Set<Object> fileFormats and initialized in constructor, currently it contains only one format - TextInputFormat. Now this check is redundant and may lead to ignoring hasHeaderFooter setting to true for any other format except of Text.
To do:
1. remove private final Set<Object> fileFormats
2. remove if block from SkipRecordsInspector.retrievePositiveIntProperty:
{code}
if (!fileFormats.contains(tableProperties.get(hive_metastoreConstants.FILE_INPUT_FORMAT))) {
return propertyIntValue;
}
{code}
was:
After changes introduced in DRILL-4982, SkipRecordInspector is used only for predefined formats (using hasHeaderFooter: false / true). But SkipRecordInspector has its own check for formats where skip strategy can be applied. Acceptable file formats are stored in private final Set<Object> fileFormats and initialized in constructor, currently it contains only one format - TextInputFormat. Now this check is redundant and may lead to ignoring hasHeaderFooter setting true for any other format except of Text.
To do:
1. remove private final Set<Object> fileFormats
2. remove if block from SkipRecordsInspector.retrievePositiveIntProperty:
{code}
if (!fileFormats.contains(tableProperties.get(hive_metastoreConstants.FILE_INPUT_FORMAT))) {
return propertyIntValue;
}
{code}
> Refactor SkipRecordsInspector to exclude check for predefined file formats
> --------------------------------------------------------------------------
>
> Key: DRILL-5106
> URL: https://issues.apache.org/jira/browse/DRILL-5106
> Project: Apache Drill
> Issue Type: Improvement
> Components: Storage - Hive
> Affects Versions: 1.9.0
> Reporter: Arina Ielchiieva
> Priority: Minor
>
> After changes introduced in DRILL-4982, SkipRecordInspector is used only for predefined formats (using hasHeaderFooter: false / true). But SkipRecordInspector has its own check for formats where skip strategy can be applied. Acceptable file formats are stored in private final Set<Object> fileFormats and initialized in constructor, currently it contains only one format - TextInputFormat. Now this check is redundant and may lead to ignoring hasHeaderFooter setting to true for any other format except of Text.
> To do:
> 1. remove private final Set<Object> fileFormats
> 2. remove if block from SkipRecordsInspector.retrievePositiveIntProperty:
> {code}
> if (!fileFormats.contains(tableProperties.get(hive_metastoreConstants.FILE_INPUT_FORMAT))) {
> return propertyIntValue;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)