You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by "Jarek Jarcec Cecho (Updated) (JIRA)" <ji...@apache.org> on 2011/11/25 08:14:40 UTC
[jira] [Updated] (FLUME-860) Combination of ExecSource and HDFS
DataStream is removing end line characters
[ https://issues.apache.org/jira/browse/FLUME-860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jarek Jarcec Cecho updated FLUME-860:
-------------------------------------
Attachment: input
> Combination of ExecSource and HDFS DataStream is removing end line characters
> -----------------------------------------------------------------------------
>
> Key: FLUME-860
> URL: https://issues.apache.org/jira/browse/FLUME-860
> Project: Flume
> Issue Type: Bug
> Affects Versions: NG alpha 2
> Reporter: Jarek Jarcec Cecho
> Attachments: hddev01.log, hddev01.properties, hddev02.log, hddev02.properties, input, output
>
>
> I've noticed that combination of ExecSource and HDFS Sink configured to use DataStream is removing end line characters and thus is creating one line output file.
> I've used two centos boxes where first was acting as an agent, reading local log file using ExecSource. Second machine was acting as a collector, waiting for input events and storing them on HDFS. Both machines were connected using AVRO sink+source combination. You can find configuration files for both machines with their logs as well attached to this JIRA bug.
> I've executed both flume-ng instances using following commands:
> ./bin/flume-ng node --conf conf/ --classpath flume-ng.jar --f conf/configuration.properties -n hddev01 > hddev01.log 2>&1
> ./bin/flume-ng node --conf conf/ --classpath flume-ng.jar --f conf/configuration.properties -n hddev02 > hddev02.log 2>&1
> Input file was created using following small bash script (it was executed after flume-ng was successfully started):
> for i in `seq -w 01 10`; do echo $i; echo Yoda-$i >> /var/log/jarcec; sleep 1s; done
> Please note that I had to apply patch from FLUME-858 in order to get DataStream file type working.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira