You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "Vladimir Klimontovich (JIRA)" <ji...@apache.org> on 2010/03/03 12:06:27 UTC
[jira] Created: (HBASE-2280) HFileOutputFormat writes output to
"unsafe" directory
HFileOutputFormat writes output to "unsafe" directory
-----------------------------------------------------
Key: HBASE-2280
URL: https://issues.apache.org/jira/browse/HBASE-2280
Project: Hadoop HBase
Issue Type: Bug
Affects Versions: 0.20.3
Reporter: Vladimir Klimontovich
Priority: Critical
Fix For: 0.20.5, 0.21.0, 0.22.0
HFileOutputFormat writes data direct to output folder. It's incorrect as failed (or killed, or interrupted) reducers leaves inconsistent files in output folder.
The convinient way to ouput data from OutputFormat is to use "working directory". The content of this directory is being moved to output directory at the end of reducer process if only reducer succeeded (this process is called "output commit" or "atomic commit").
If means that instead of
final Path outputdir = FileOutputFormat.getOutputPath(context);
hbase should use
final Path outputdir = FileOutputFormat.getWorkOutputPath(context);
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2280) HFileOutputFormat writes output to
"unsafe" directory
Posted by "Ruslan Salyakhov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12840641#action_12840641 ]
Ruslan Salyakhov commented on HBASE-2280:
-----------------------------------------
Patch is available in scope HBASE-2063
> HFileOutputFormat writes output to "unsafe" directory
> -----------------------------------------------------
>
> Key: HBASE-2280
> URL: https://issues.apache.org/jira/browse/HBASE-2280
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.3
> Reporter: Vladimir Klimontovich
> Priority: Critical
> Fix For: 0.20.5, 0.21.0, 0.22.0
>
>
> HFileOutputFormat writes data direct to output folder. It's incorrect as failed (or killed, or interrupted) reducers leaves inconsistent files in output folder.
> The convinient way to ouput data from OutputFormat is to use "working directory". The content of this directory is being moved to output directory at the end of reducer process if only reducer succeeded (this process is called "output commit" or "atomic commit").
> If means that instead of
> final Path outputdir = FileOutputFormat.getOutputPath(context);
> hbase should use
> final Path outputdir = FileOutputFormat.getWorkOutputPath(context);
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HBASE-2280) HFileOutputFormat writes output to
"unsafe" directory
Posted by "Vladimir Klimontovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vladimir Klimontovich resolved HBASE-2280.
------------------------------------------
Resolution: Duplicate
Duplicate of HBASE-2063
> HFileOutputFormat writes output to "unsafe" directory
> -----------------------------------------------------
>
> Key: HBASE-2280
> URL: https://issues.apache.org/jira/browse/HBASE-2280
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.3
> Reporter: Vladimir Klimontovich
> Priority: Critical
> Fix For: 0.20.5, 0.21.0, 0.22.0
>
>
> HFileOutputFormat writes data direct to output folder. It's incorrect as failed (or killed, or interrupted) reducers leaves inconsistent files in output folder.
> The convinient way to ouput data from OutputFormat is to use "working directory". The content of this directory is being moved to output directory at the end of reducer process if only reducer succeeded (this process is called "output commit" or "atomic commit").
> If means that instead of
> final Path outputdir = FileOutputFormat.getOutputPath(context);
> hbase should use
> final Path outputdir = FileOutputFormat.getWorkOutputPath(context);
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.