You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Gavin (JIRA)" <ji...@apache.org> on 2018/08/01 06:21:49 UTC
[jira] [Issue Comment Deleted] (HBASE-18885) HFileOutputFormat2
hardcodes default FileOutputCommitter
[ https://issues.apache.org/jira/browse/HBASE-18885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Gavin updated HBASE-18885:
--------------------------
Comment: was deleted
(was: A comment with security level 'jira-users' was removed.)
> HFileOutputFormat2 hardcodes default FileOutputCommitter
> --------------------------------------------------------
>
> Key: HBASE-18885
> URL: https://issues.apache.org/jira/browse/HBASE-18885
> Project: HBase
> Issue Type: Bug
> Components: mapreduce
> Reporter: Shaofeng SHI
> Assignee: Shaofeng SHI
> Priority: Major
> Fix For: 1.4.0, 1.3.2, 1.2.7, 2.0.0-alpha-4, 2.0.0
>
> Attachments: HBASE-18885.branch-1.001.patch, HBASE-18885.master.001.patch
>
>
> Apache Kylin uses HBase's HFileOutputFormat2.java to configure the MR job. The original reporting is in KYLIN-2788[1]. After some investigation, we found this class always uses the default "FileOutputCommitter", see [2], regardless of the job's configuration; so it always writing to "_temporary" folder. Since AWS EMR configured to use DirectOutputCommitter for S3, then this problem occurs: Hadoop expects to see the file directly under output path, while the RecordWriter generates them in "_temporary" folder. This caused no data be loaded to HTable.
> Seems this problem exists in all versions so far.
> [1] https://issues.apache.org/jira/browse/KYLIN-2788
> [2] https://github.com/apache/hbase/blob/master/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java#L193
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)