You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Doug Cutting (JIRA)" <ji...@apache.org> on 2007/04/09 20:01:32 UTC
[jira] Created: (HADOOP-1229) want InputFormat and OutputFormat for
zip archives
want InputFormat and OutputFormat for zip archives
--------------------------------------------------
Key: HADOOP-1229
URL: https://issues.apache.org/jira/browse/HADOOP-1229
Project: Hadoop
Issue Type: New Feature
Components: mapred
Reporter: Doug Cutting
An InputFormat and OutputFormat for zip-format archives would be useful. It would provide a standard file format, friendly to foreign applications (e.g., streaming), that is also optimized for Hadoop.
The OutputFormat could generate archives containing a series of small (~1MB) files, each individually compressed. The InputFormat could then generate a split for each file in the archive. (Zip archives contain a directory at the end of the file that may be efficiently read when constructing splits.) Input splits could thus be sent to a node where they are stored locally for the map phase, providing good performance.
This would thus permit applications to (1) use a standard file format; (2) keep data compressed; and (3) efficiently split input into chunks substantially smaller than HDFS blocks. This is not available today.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1229) want InputFormat and OutputFormat
for zip archives
Posted by "arkady borkovsky (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12524180 ]
arkady borkovsky commented on HADOOP-1229:
------------------------------------------
It would be nice to automatically detect that the input is compressed and inserts this the corresponding conversion in from of actual application's InputFormat .
There should also be an option for Streaming to bypass the decompression.
> want InputFormat and OutputFormat for zip archives
> --------------------------------------------------
>
> Key: HADOOP-1229
> URL: https://issues.apache.org/jira/browse/HADOOP-1229
> Project: Hadoop
> Issue Type: New Feature
> Components: mapred
> Reporter: Doug Cutting
>
> An InputFormat and OutputFormat for zip-format archives would be useful. It would provide a standard file format, friendly to foreign applications (e.g., streaming), that is also optimized for Hadoop.
> The OutputFormat could generate archives containing a series of small (~1MB) files, each individually compressed. The InputFormat could then generate a split for each file in the archive. (Zip archives contain a directory at the end of the file that may be efficiently read when constructing splits.) Input splits could thus be sent to a node where they are stored locally for the map phase, providing good performance.
> This would thus permit applications to (1) use a standard file format; (2) keep data compressed; and (3) efficiently split input into chunks substantially smaller than HDFS blocks. This is not available today.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.