You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Chris Douglas (JIRA)" <ji...@apache.org> on 2007/12/11 03:04:43 UTC
[jira] Created: (HADOOP-2402) Lzo compression compresses each write
from TextOutputFormat
Lzo compression compresses each write from TextOutputFormat
-----------------------------------------------------------
Key: HADOOP-2402
URL: https://issues.apache.org/jira/browse/HADOOP-2402
Project: Hadoop
Issue Type: Bug
Components: io, mapred, native
Reporter: Chris Douglas
Fix For: 0.16.0
Outputting with TextOutputFormat and Lzo compression generates a file such that each key, tab delimiter, and value are compressed separately.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2402) Lzo compression compresses each write
from TextOutputFormat
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-2402:
----------------------------------
Status: Patch Available (was: Open)
I don't think the timeout in contrib is related; trying again.
> Lzo compression compresses each write from TextOutputFormat
> -----------------------------------------------------------
>
> Key: HADOOP-2402
> URL: https://issues.apache.org/jira/browse/HADOOP-2402
> Project: Hadoop
> Issue Type: Bug
> Components: io, mapred, native
> Reporter: Chris Douglas
> Assignee: Chris Douglas
> Fix For: 0.16.0
>
> Attachments: 2402-0.patch, 2402-1.patch
>
>
> Outputting with TextOutputFormat and Lzo compression generates a file such that each key, tab delimiter, and value are compressed separately.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2402) Lzo compression compresses each
write from TextOutputFormat
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12556157#action_12556157 ]
Hadoop QA commented on HADOOP-2402:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12372526/2402-1.patch
against trunk revision .
@author +1. The patch does not contain any @author tags.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new compiler warnings.
findbugs +1. The patch does not introduce any new Findbugs warnings.
core tests +1. The patch passed core unit tests.
contrib tests -1. The patch failed contrib unit tests.
Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1471/testReport/
Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1471/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1471/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1471/console
This message is automatically generated.
> Lzo compression compresses each write from TextOutputFormat
> -----------------------------------------------------------
>
> Key: HADOOP-2402
> URL: https://issues.apache.org/jira/browse/HADOOP-2402
> Project: Hadoop
> Issue Type: Bug
> Components: io, mapred, native
> Reporter: Chris Douglas
> Assignee: Chris Douglas
> Fix For: 0.16.0
>
> Attachments: 2402-0.patch, 2402-1.patch
>
>
> Outputting with TextOutputFormat and Lzo compression generates a file such that each key, tab delimiter, and value are compressed separately.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2402) Lzo compression compresses each write
from TextOutputFormat
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-2402:
----------------------------------
Comment: was deleted
> Lzo compression compresses each write from TextOutputFormat
> -----------------------------------------------------------
>
> Key: HADOOP-2402
> URL: https://issues.apache.org/jira/browse/HADOOP-2402
> Project: Hadoop
> Issue Type: Bug
> Components: io, mapred, native
> Reporter: Chris Douglas
> Assignee: Chris Douglas
> Fix For: 0.16.0
>
> Attachments: 2402-0.patch, 2402-1.patch
>
>
> Outputting with TextOutputFormat and Lzo compression generates a file such that each key, tab delimiter, and value are compressed separately.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2402) Lzo compression compresses each write
from TextOutputFormat
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-2402:
----------------------------------
Status: Open (was: Patch Available)
> Lzo compression compresses each write from TextOutputFormat
> -----------------------------------------------------------
>
> Key: HADOOP-2402
> URL: https://issues.apache.org/jira/browse/HADOOP-2402
> Project: Hadoop
> Issue Type: Bug
> Components: io, mapred, native
> Reporter: Chris Douglas
> Assignee: Chris Douglas
> Fix For: 0.16.0
>
> Attachments: 2402-0.patch, 2402-1.patch
>
>
> Outputting with TextOutputFormat and Lzo compression generates a file such that each key, tab delimiter, and value are compressed separately.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2402) Lzo compression compresses each write
from TextOutputFormat
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-2402:
----------------------------------
Status: Patch Available (was: Open)
> Lzo compression compresses each write from TextOutputFormat
> -----------------------------------------------------------
>
> Key: HADOOP-2402
> URL: https://issues.apache.org/jira/browse/HADOOP-2402
> Project: Hadoop
> Issue Type: Bug
> Components: io, mapred, native
> Reporter: Chris Douglas
> Assignee: Chris Douglas
> Fix For: 0.16.0
>
> Attachments: 2402-0.patch, 2402-1.patch
>
>
> Outputting with TextOutputFormat and Lzo compression generates a file such that each key, tab delimiter, and value are compressed separately.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2402) Lzo compression compresses each
write from TextOutputFormat
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12551584 ]
Chris Douglas commented on HADOOP-2402:
---------------------------------------
In the native libs, it looks like io.file.buffer.size determines the maximum size of the copy to the OutputStream from the buffer containing the compressed data. The GzipCodec and LzoCodec define their own properties and defaults for the size of this native buffer (both 64k). The reasoning went, if the buffer is larger than the native lib's buffer, it's still going to be blocked until that buffer's been flushed to the OutputStream. If the buffer is io.file.buffer.size (defaulting to 4k), then it's going to be giving the compression codec data 4k at a time. For Lzo, this means it will compress no more than 4k at a time, yielding even less than 20% compression.
We could introduce a new property that sets the size of this buffer- or use the property given to Gzip/Lzo- but that's not very attractive, either.
LzoCodec returns a stream wrapped in a BlockCompressorStream, but it doesn't provide any buffering. It ensures that no more than MAX_INPUT_SIZE (defaulting to 64k less the compression overhead) is compressed at once. This might be a better place to add some buffering, but then the codec will be returning a buffered stream.
> Lzo compression compresses each write from TextOutputFormat
> -----------------------------------------------------------
>
> Key: HADOOP-2402
> URL: https://issues.apache.org/jira/browse/HADOOP-2402
> Project: Hadoop
> Issue Type: Bug
> Components: io, mapred, native
> Reporter: Chris Douglas
> Fix For: 0.16.0
>
> Attachments: 2402-0.patch
>
>
> Outputting with TextOutputFormat and Lzo compression generates a file such that each key, tab delimiter, and value are compressed separately.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2402) Lzo compression compresses each
write from TextOutputFormat
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12551587 ]
Doug Cutting commented on HADOOP-2402:
--------------------------------------
> For Lzo, this means it will compress no more than 4k at a time, yielding even less than 20% compression.
We can't compress multiple buffers together with the lzo codec? It only compresses buffer-at-a-time? If so, then it should do the buffering & set the buffer size, since this is an lzo-specific issue.
> This might be a better place to add some buffering, but then the codec will be returning a buffered stream.
I don't think it's a bug for a codec to return a buffered stream if a particular buffer size is required to get good performance from that codec. If it's impossible to get lzo to compress data across buffers, and 64k or larger is required to get good compression, then it should mandate that buffer size, perhaps adding a new configuration parameter.
Separately, we should consider whether to (a) unilaterally add an io.file.buffer.size buffer in TextOutputFormat, since it helps other codecs, or (b) assume that all codecs return appropriately buffered streams, and add a buffer in the Zip codec if it improves performance. If a io.file.buffer.size buffer=4k gives somewhat improved Zip performance, and a 64k buffer gives even better performance, I think that's okay. Performance should improve a bit by increasing io.file.buffer.size, at the expense of chewing up more memory per open file. The default setting should be for decent performance with minimal memory use.
> Lzo compression compresses each write from TextOutputFormat
> -----------------------------------------------------------
>
> Key: HADOOP-2402
> URL: https://issues.apache.org/jira/browse/HADOOP-2402
> Project: Hadoop
> Issue Type: Bug
> Components: io, mapred, native
> Reporter: Chris Douglas
> Fix For: 0.16.0
>
> Attachments: 2402-0.patch
>
>
> Outputting with TextOutputFormat and Lzo compression generates a file such that each key, tab delimiter, and value are compressed separately.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2402) Lzo compression compresses each write
from TextOutputFormat
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-2402:
----------------------------------
Attachment: 2402-1.patch
This patch reworks the lzo codec to fill its nio buffer before spilling to the native code. It effects the following results for 5GB of random text:
Write:
|| Format || Type || Time (s from trunk, pcnt change) || Compression (% from uncompressed) ||
| SEQ | BLK | -51 (-31.1%) | 54.9% (23.7% using trunk) |
| SEQ | REC | +5 (+2.7%) | -3.6% (-3.6% using trunk) |
| TXT | | -96 (-63.9%) | 54.8% (-12.3% using trunk) |
Read:
|| Format || Type || Time (s from trunk, pcnt change) ||
| SEQ | BLK | +7 (+8.6%) |
| SEQ | REC | +8 (+5.6%) |
| TXT | | -83 (-39.1%) |
The default extension should probably be something other than .lzo, since we're not compatible with lzop.
> Lzo compression compresses each write from TextOutputFormat
> -----------------------------------------------------------
>
> Key: HADOOP-2402
> URL: https://issues.apache.org/jira/browse/HADOOP-2402
> Project: Hadoop
> Issue Type: Bug
> Components: io, mapred, native
> Reporter: Chris Douglas
> Fix For: 0.16.0
>
> Attachments: 2402-0.patch, 2402-1.patch
>
>
> Outputting with TextOutputFormat and Lzo compression generates a file such that each key, tab delimiter, and value are compressed separately.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2402) Lzo compression compresses each write
from TextOutputFormat
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated HADOOP-2402:
----------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this. Thanks, Chris!
> Lzo compression compresses each write from TextOutputFormat
> -----------------------------------------------------------
>
> Key: HADOOP-2402
> URL: https://issues.apache.org/jira/browse/HADOOP-2402
> Project: Hadoop Core
> Issue Type: Bug
> Components: io, mapred, native
> Reporter: Chris Douglas
> Assignee: Chris Douglas
> Fix For: 0.16.0
>
> Attachments: 2402-0.patch, 2402-1.patch, 2402-2.patch
>
>
> Outputting with TextOutputFormat and Lzo compression generates a file such that each key, tab delimiter, and value are compressed separately.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2402) Lzo compression compresses each write
from TextOutputFormat
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-2402:
----------------------------------
Attachment: 2402-2.patch
Removed most whitespace changes, except those reformatting sections with lines greater than 80 chars
> Lzo compression compresses each write from TextOutputFormat
> -----------------------------------------------------------
>
> Key: HADOOP-2402
> URL: https://issues.apache.org/jira/browse/HADOOP-2402
> Project: Hadoop
> Issue Type: Bug
> Components: io, mapred, native
> Reporter: Chris Douglas
> Assignee: Chris Douglas
> Fix For: 0.16.0
>
> Attachments: 2402-0.patch, 2402-1.patch, 2402-2.patch
>
>
> Outputting with TextOutputFormat and Lzo compression generates a file such that each key, tab delimiter, and value are compressed separately.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2402) Lzo compression compresses each write
from TextOutputFormat
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-2402:
----------------------------------
Status: Patch Available (was: Open)
> Lzo compression compresses each write from TextOutputFormat
> -----------------------------------------------------------
>
> Key: HADOOP-2402
> URL: https://issues.apache.org/jira/browse/HADOOP-2402
> Project: Hadoop
> Issue Type: Bug
> Components: io, mapred, native
> Reporter: Chris Douglas
> Assignee: Chris Douglas
> Fix For: 0.16.0
>
> Attachments: 2402-0.patch, 2402-1.patch
>
>
> Outputting with TextOutputFormat and Lzo compression generates a file such that each key, tab delimiter, and value are compressed separately.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2402) Lzo compression compresses each write
from TextOutputFormat
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-2402:
----------------------------------
Status: Patch Available (was: Open)
> Lzo compression compresses each write from TextOutputFormat
> -----------------------------------------------------------
>
> Key: HADOOP-2402
> URL: https://issues.apache.org/jira/browse/HADOOP-2402
> Project: Hadoop
> Issue Type: Bug
> Components: io, mapred, native
> Reporter: Chris Douglas
> Fix For: 0.16.0
>
> Attachments: 2402-0.patch
>
>
> Outputting with TextOutputFormat and Lzo compression generates a file such that each key, tab delimiter, and value are compressed separately.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-2402) Lzo compression compresses each
write from TextOutputFormat
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas reassigned HADOOP-2402:
-------------------------------------
Assignee: Chris Douglas
> Lzo compression compresses each write from TextOutputFormat
> -----------------------------------------------------------
>
> Key: HADOOP-2402
> URL: https://issues.apache.org/jira/browse/HADOOP-2402
> Project: Hadoop
> Issue Type: Bug
> Components: io, mapred, native
> Reporter: Chris Douglas
> Assignee: Chris Douglas
> Fix For: 0.16.0
>
> Attachments: 2402-0.patch, 2402-1.patch
>
>
> Outputting with TextOutputFormat and Lzo compression generates a file such that each key, tab delimiter, and value are compressed separately.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2402) Lzo compression compresses each
write from TextOutputFormat
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12551195 ]
Hadoop QA commented on HADOOP-2402:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12371567/2402-0.patch
against trunk revision r603795.
@author +1. The patch does not contain any @author tags.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new compiler warnings.
findbugs +1. The patch does not introduce any new Findbugs warnings.
core tests +1. The patch passed core unit tests.
contrib tests +1. The patch passed contrib unit tests.
Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1333/testReport/
Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1333/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1333/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1333/console
This message is automatically generated.
> Lzo compression compresses each write from TextOutputFormat
> -----------------------------------------------------------
>
> Key: HADOOP-2402
> URL: https://issues.apache.org/jira/browse/HADOOP-2402
> Project: Hadoop
> Issue Type: Bug
> Components: io, mapred, native
> Reporter: Chris Douglas
> Fix For: 0.16.0
>
> Attachments: 2402-0.patch
>
>
> Outputting with TextOutputFormat and Lzo compression generates a file such that each key, tab delimiter, and value are compressed separately.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2402) Lzo compression compresses each write
from TextOutputFormat
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-2402:
----------------------------------
Status: Patch Available (was: Open)
> Lzo compression compresses each write from TextOutputFormat
> -----------------------------------------------------------
>
> Key: HADOOP-2402
> URL: https://issues.apache.org/jira/browse/HADOOP-2402
> Project: Hadoop
> Issue Type: Bug
> Components: io, mapred, native
> Reporter: Chris Douglas
> Assignee: Chris Douglas
> Fix For: 0.16.0
>
> Attachments: 2402-0.patch, 2402-1.patch, 2402-2.patch
>
>
> Outputting with TextOutputFormat and Lzo compression generates a file such that each key, tab delimiter, and value are compressed separately.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2402) Lzo compression compresses each
write from TextOutputFormat
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12556218#action_12556218 ]
Hadoop QA commented on HADOOP-2402:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12372526/2402-1.patch
against trunk revision .
@author +1. The patch does not contain any @author tags.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new compiler warnings.
findbugs +1. The patch does not introduce any new Findbugs warnings.
core tests +1. The patch passed core unit tests.
contrib tests -1. The patch failed contrib unit tests.
Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1476/testReport/
Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1476/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1476/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1476/console
This message is automatically generated.
> Lzo compression compresses each write from TextOutputFormat
> -----------------------------------------------------------
>
> Key: HADOOP-2402
> URL: https://issues.apache.org/jira/browse/HADOOP-2402
> Project: Hadoop
> Issue Type: Bug
> Components: io, mapred, native
> Reporter: Chris Douglas
> Assignee: Chris Douglas
> Fix For: 0.16.0
>
> Attachments: 2402-0.patch, 2402-1.patch
>
>
> Outputting with TextOutputFormat and Lzo compression generates a file such that each key, tab delimiter, and value are compressed separately.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2402) Lzo compression compresses each write
from TextOutputFormat
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-2402:
----------------------------------
Attachment: 2402-0.patch
Inserted a 64k BufferedOutputStream to TextOutputFormat::getRecordWriter. Effects compression similar to block, lzo compressed SequenceFile (~20%). For comparison, lzop (command line compression utility backed by lzo lib) uses 256k blocks (and a different file format) and compresses the same 100M sample by 60%.
As an aside, with this applied, Zip compressed text files are written approximately 10% faster; uncompressed text about 4% faster.
> Lzo compression compresses each write from TextOutputFormat
> -----------------------------------------------------------
>
> Key: HADOOP-2402
> URL: https://issues.apache.org/jira/browse/HADOOP-2402
> Project: Hadoop
> Issue Type: Bug
> Components: io, mapred, native
> Reporter: Chris Douglas
> Fix For: 0.16.0
>
> Attachments: 2402-0.patch
>
>
> Outputting with TextOutputFormat and Lzo compression generates a file such that each key, tab delimiter, and value are compressed separately.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2402) Lzo compression compresses each
write from TextOutputFormat
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12562507#action_12562507 ]
Hudson commented on HADOOP-2402:
--------------------------------
Integrated in Hadoop-trunk #379 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/379/])
> Lzo compression compresses each write from TextOutputFormat
> -----------------------------------------------------------
>
> Key: HADOOP-2402
> URL: https://issues.apache.org/jira/browse/HADOOP-2402
> Project: Hadoop Core
> Issue Type: Bug
> Components: io, mapred, native
> Reporter: Chris Douglas
> Assignee: Chris Douglas
> Fix For: 0.16.0
>
> Attachments: 2402-0.patch, 2402-1.patch, 2402-2.patch
>
>
> Outputting with TextOutputFormat and Lzo compression generates a file such that each key, tab delimiter, and value are compressed separately.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2402) Lzo compression compresses each write
from TextOutputFormat
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated HADOOP-2402:
----------------------------------
Status: Open (was: Patch Available)
Mostly looks ok, but there are too many unrelated white-space changes - hence, I'm cancelling this patch.
> Lzo compression compresses each write from TextOutputFormat
> -----------------------------------------------------------
>
> Key: HADOOP-2402
> URL: https://issues.apache.org/jira/browse/HADOOP-2402
> Project: Hadoop
> Issue Type: Bug
> Components: io, mapred, native
> Reporter: Chris Douglas
> Assignee: Chris Douglas
> Fix For: 0.16.0
>
> Attachments: 2402-0.patch, 2402-1.patch
>
>
> Outputting with TextOutputFormat and Lzo compression generates a file such that each key, tab delimiter, and value are compressed separately.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2402) Lzo compression compresses each
write from TextOutputFormat
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12557407#action_12557407 ]
Hadoop QA commented on HADOOP-2402:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12372526/2402-1.patch
against trunk revision .
@author +1. The patch does not contain any @author tags.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new compiler warnings.
findbugs +1. The patch does not introduce any new Findbugs warnings.
core tests +1. The patch passed core unit tests.
contrib tests -1. The patch failed contrib unit tests.
Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1525/testReport/
Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1525/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1525/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1525/console
This message is automatically generated.
> Lzo compression compresses each write from TextOutputFormat
> -----------------------------------------------------------
>
> Key: HADOOP-2402
> URL: https://issues.apache.org/jira/browse/HADOOP-2402
> Project: Hadoop
> Issue Type: Bug
> Components: io, mapred, native
> Reporter: Chris Douglas
> Assignee: Chris Douglas
> Fix For: 0.16.0
>
> Attachments: 2402-0.patch, 2402-1.patch
>
>
> Outputting with TextOutputFormat and Lzo compression generates a file such that each key, tab delimiter, and value are compressed separately.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2402) Lzo compression compresses each write
from TextOutputFormat
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated HADOOP-2402:
---------------------------------
Status: Open (was: Patch Available)
I think this is probably the right approach, to not require the codecs themselves to return buffered streams. But the size of the buffer should be io.file.buffer.size, rather than a fixed 64k, no?
> Lzo compression compresses each write from TextOutputFormat
> -----------------------------------------------------------
>
> Key: HADOOP-2402
> URL: https://issues.apache.org/jira/browse/HADOOP-2402
> Project: Hadoop
> Issue Type: Bug
> Components: io, mapred, native
> Reporter: Chris Douglas
> Fix For: 0.16.0
>
> Attachments: 2402-0.patch
>
>
> Outputting with TextOutputFormat and Lzo compression generates a file such that each key, tab delimiter, and value are compressed separately.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2402) Lzo compression compresses each write
from TextOutputFormat
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-2402:
----------------------------------
Status: Open (was: Patch Available)
> Lzo compression compresses each write from TextOutputFormat
> -----------------------------------------------------------
>
> Key: HADOOP-2402
> URL: https://issues.apache.org/jira/browse/HADOOP-2402
> Project: Hadoop
> Issue Type: Bug
> Components: io, mapred, native
> Reporter: Chris Douglas
> Assignee: Chris Douglas
> Fix For: 0.16.0
>
> Attachments: 2402-0.patch, 2402-1.patch
>
>
> Outputting with TextOutputFormat and Lzo compression generates a file such that each key, tab delimiter, and value are compressed separately.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2402) Lzo compression compresses each
write from TextOutputFormat
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12560750#action_12560750 ]
Hadoop QA commented on HADOOP-2402:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12373566/2402-2.patch
against trunk revision r613446.
@author +1. The patch does not contain any @author tags.
javadoc +1. The javadoc tool did not generate any warning messages.
javac +1. The applied patch does not generate any new compiler warnings.
findbugs +1. The patch does not introduce any new Findbugs warnings.
core tests -1. The patch failed core unit tests.
contrib tests +1. The patch passed contrib unit tests.
Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1656/testReport/
Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1656/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1656/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1656/console
This message is automatically generated.
> Lzo compression compresses each write from TextOutputFormat
> -----------------------------------------------------------
>
> Key: HADOOP-2402
> URL: https://issues.apache.org/jira/browse/HADOOP-2402
> Project: Hadoop
> Issue Type: Bug
> Components: io, mapred, native
> Reporter: Chris Douglas
> Assignee: Chris Douglas
> Fix For: 0.16.0
>
> Attachments: 2402-0.patch, 2402-1.patch, 2402-2.patch
>
>
> Outputting with TextOutputFormat and Lzo compression generates a file such that each key, tab delimiter, and value are compressed separately.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.