You are viewing a plain text version of this content. The canonical link for it is here.

Posted to common-dev@hadoop.apache.org by "Steven Phillips (JIRA)" <ji...@apache.org> on 2015/03/26 02:31:52 UTC

[jira] [Created] (HADOOP-11751) Parquet scan fails when directory contains _SUCCESS or _logs

Steven Phillips created HADOOP-11751:
----------------------------------------

             Summary: Parquet scan fails when directory contains _SUCCESS or _logs
                 Key: HADOOP-11751
                 URL: https://issues.apache.org/jira/browse/HADOOP-11751
             Project: Hadoop Common
          Issue Type: Bug
            Reporter: Steven Phillips
            Assignee: Steven Phillips


_SUCCESS and _logs are often created from map reduce jobs, and typically ignored. This is commonly done using the OutputFilesFilter in hadoop.

The new FooterGatherer class, which is used to read parquet footers in parallel, does not use this filter. So it attempts to read these as parquet files and fails.

The fix is to use the DrillPathFilter, which extends OutputFilesFilter, in the FooterGather code.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)