You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by "Steve Hoffman (Updated) (JIRA)" <ji...@apache.org> on 2012/02/21 17:28:34 UTC
[jira] [Updated] (FLUME-983) snappy compression via
AvroDataFileOutputFormat suboptimal
[ https://issues.apache.org/jira/browse/FLUME-983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Hoffman updated FLUME-983:
--------------------------------
Description:
I used the AvroDataFileOutputFormat with the snappy compression option to write compressed avro files to HDFS via flume.
The original file was 106,514,936 bytes of json. The output is written to HDFS as raw (no flume wrapper).
The file size I got using the snappy compression option was 47,520,735 bytes which is about 1/2 the size. Looking at the file directly it didn't look like it had been compressed too much.
So I used avro-tools tojson to convert my final flume-written output back to json which resulted in a file size of 79,773,371 bytes (so this is basically the starting size of the data being compressed). Then I used the avro-tools fromjson, giving it the same schema as getschema returned, and the snappy compression option. The resulting file was 11,904,857 bytes (which seemed much better).
So I asked myself, why the data written via flume record by record wasn't compressed as much? Looking at the raw file written to HDFS clearly showed 'snappy' in the header and the data looked minimally encoded/compressed.
I looked at the source and was struck by a call to sink.flush() after the sink.append() in AvroDataFileOutputFormat.format().
It appears that calling sink() was the root cause of the not so great compression.
To test this theory, I recompiled the sink with the flush() line commented out. The resulting test wrote a file for the same sample data to be 11,870,573 bytes (pretty much matching the command-line avro-tools created version).
I'm filing this 'cause I think this may be a bug wasting lots of space by users trying to use snappy compression (or any compression for that matter). Not really sure what the impact of removing this flush() call is either (since the file doesn't really exist in HDFS until it is closed).
was:
I used the AvroDataFileOutputFormat with the snappy compression option to write compressed avro files to HDFS via flume.
The original file was 106514936 bytes of json. The output is written to HDFS as raw (no flume wrapper).
The file size I got using the snappy compression option was 47520735 bytes which is about 1/2 the size. Looking at the file directly it didn't look like it had been compressed too much.
So I used avro-tools tojson to convert my final flume-written output back to json which resulted in a file size of 79773371 bytes (so this is basically the starting size of the data being compressed). Then I used the avro-tools fromjson, giving it the same schema as getschema returned, and the snappy compression option. The resulting file was 11904857 bytes (which seemed much better).
So I asked myself, why the data written via flume record by record wasn't compressed as much? Looking at the raw file written to HDFS clearly showed 'snappy' in the header and the data looked minimally encoded/compressed.
I looked at the source and was struck by a call to sink.flush() after the sink.append() in AvroDataFileOutputFormat.format().
It appears that calling sink() was the root cause of the not so great compression.
To test this theory, I recompiled the sink with the flush() line commented out. The resulting test wrote a file for the same sample data to be 11870573 bytes (pretty much matching the command-line avro-tools created version).
I'm filing this 'cause I think this may be a bug wasting lots of space by users trying to use snappy compression (or any compression for that matter). Not really sure what the impact of removing this flush() call is either (since the file doesn't really exist in HDFS until it is closed).
> snappy compression via AvroDataFileOutputFormat suboptimal
> ----------------------------------------------------------
>
> Key: FLUME-983
> URL: https://issues.apache.org/jira/browse/FLUME-983
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Affects Versions: v0.9.4
> Environment: Cloudera CDH3u2 flume + hadoop
> Reporter: Steve Hoffman
> Priority: Critical
>
> I used the AvroDataFileOutputFormat with the snappy compression option to write compressed avro files to HDFS via flume.
> The original file was 106,514,936 bytes of json. The output is written to HDFS as raw (no flume wrapper).
> The file size I got using the snappy compression option was 47,520,735 bytes which is about 1/2 the size. Looking at the file directly it didn't look like it had been compressed too much.
> So I used avro-tools tojson to convert my final flume-written output back to json which resulted in a file size of 79,773,371 bytes (so this is basically the starting size of the data being compressed). Then I used the avro-tools fromjson, giving it the same schema as getschema returned, and the snappy compression option. The resulting file was 11,904,857 bytes (which seemed much better).
> So I asked myself, why the data written via flume record by record wasn't compressed as much? Looking at the raw file written to HDFS clearly showed 'snappy' in the header and the data looked minimally encoded/compressed.
> I looked at the source and was struck by a call to sink.flush() after the sink.append() in AvroDataFileOutputFormat.format().
> It appears that calling sink() was the root cause of the not so great compression.
> To test this theory, I recompiled the sink with the flush() line commented out. The resulting test wrote a file for the same sample data to be 11,870,573 bytes (pretty much matching the command-line avro-tools created version).
> I'm filing this 'cause I think this may be a bug wasting lots of space by users trying to use snappy compression (or any compression for that matter). Not really sure what the impact of removing this flush() call is either (since the file doesn't really exist in HDFS until it is closed).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira