You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by "Leslin (Hong Xiang Lin) (JIRA)" <ji...@apache.org> on 2012/06/18 16:07:45 UTC

[jira] [Comment Edited] (FLUME-1200) HDFSEventSink causes *.snappy file to be created in HDFS even when snappy isn't used (due to missing lib)

    [ https://issues.apache.org/jira/browse/FLUME-1200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13395899#comment-13395899 ] 

Leslin (Hong Xiang Lin) edited comment on FLUME-1200 at 6/18/12 2:06 PM:
-------------------------------------------------------------------------

Posted code for review at https://reviews.apache.org/r/5378/
                
      was (Author: leslin123):
    Posted for code review at https://reviews.apache.org/r/5378/
                  
> HDFSEventSink causes *.snappy file to be created in HDFS even when snappy isn't used (due to missing lib)
> ---------------------------------------------------------------------------------------------------------
>
>                 Key: FLUME-1200
>                 URL: https://issues.apache.org/jira/browse/FLUME-1200
>             Project: Flume
>          Issue Type: Bug
>          Components: Sinks+Sources
>    Affects Versions: v1.2.0
>         Environment: RHEL 6.2 64-bit
>            Reporter: Will McQueen
>            Assignee: Leslin (Hong Xiang Lin)
>             Fix For: v1.2.0
>
>         Attachments: FLUME-1200.patch
>
>
> If I use HDFSEventSink and specify the codec to be snappy, then the sink writes data to HDFS with the ".snappy" extension... but the content of those HDFS files is not in snappy format when the snappy libs aren't found. The log files mention this:
>      2012-05-11 19:38:49,868 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
>      2012-05-11 19:38:49,868 WARN snappy.LoadSnappy: Snappy native library not loaded
> ...and I think it should be an error rather than a warning... the sink shouldn't write data at all to HDFS if it's not in the format expected by the config file (ie, not compressed with snappy). The config file I used is:
> agent.channels = c1
> agent.sources = r1
> agent.sinks = k1
> #
> agent.channels.c1.type = MEMORY
> #
> agent.sources.r1.channels = c1
> agent.sources.r1.type = SEQ
> #
> agent.sinks.k1.channel = c1
> agent.sinks.k1.type = LOGGER
> #
> agent.sinks.k1.channel = c1
> agent.sinks.k1.type = HDFS
> agent.sinks.k1.hdfs.path = hdfs://<host>:<port>:<path>
> agent.sinks.k1.hdfs.fileType = DataStream
> agent.sinks.k1.hdfs.codeC = SnappyCodec

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira