You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@flume.apache.org by Joshua Sumali <js...@kobo.com> on 2012/06/28 05:57:36 UTC

flume 0.9.4: customdfs working, but escapedCustomDfs not?

Hi All,

I'm stuck trying to get escapedCustomDfs() working on flume 0.9.4

This collector configuration works:
Collector : rpcSource(11463) | customdfs("hdfs://localhost:54310/flumed-test/file-1")

Sucessful logs:
2012-06-27 23:52:21,888 [logicalNode collector-457] INFO hdfs.CustomDfsSink: Creating HDFS file: hdfs://localhost:54310/flumed-test/file-1

This collector configuration does not work:
Collector: rpcSource(11463) | escapedCustomDfs("hdfs://localhost:54310/flumed-test/","%{rolltag}-",raw)

All I get is:
2012-06-27 23:54:26,894 [logicalNode collector-465] INFO thrift.ThriftEventSource: Starting blocking thread pool server on port 11463...

I've tried collectorSink() and escapedFormatDfs() with no success. It feels like using %{escapes} just doesn't work.

I'm attempting this with flume-distribution-0.9.4-bin.tar.gz and hadoop-0.20.2-cdh3u4.tar.gz.

Any ideas?

Thanks,
Josh

RE: flume 0.9.4: customdfs working, but escapedCustomDfs not?

Posted by Joshua Sumali <js...@kobo.com>.

Sorry for the noise - turns out I had to push data through for the file to be created in HDFS when using escapedCustomDfs.
The behavior for customdfs seems to be different in the sense that the file gets created on collector initialization.

From: Joshua Sumali [mailto:jsumali@kobo.com]
Sent: Wednesday, June 27, 2012 11:58 PM
To: flume-user@incubator.apache.org
Subject: flume 0.9.4: customdfs working, but escapedCustomDfs not?

Hi All,

I'm stuck trying to get escapedCustomDfs() working on flume 0.9.4

This collector configuration works:
Collector : rpcSource(11463) | customdfs("hdfs://localhost:54310/flumed-test/file-1")

Sucessful logs:
2012-06-27 23:52:21,888 [logicalNode collector-457] INFO hdfs.CustomDfsSink: Creating HDFS file: hdfs://localhost:54310/flumed-test/file-1

This collector configuration does not work:
Collector: rpcSource(11463) | escapedCustomDfs("hdfs://localhost:54310/flumed-test/","%{rolltag}-",raw)

All I get is:
2012-06-27 23:54:26,894 [logicalNode collector-465] INFO thrift.ThriftEventSource: Starting blocking thread pool server on port 11463...

I've tried collectorSink() and escapedFormatDfs() with no success. It feels like using %{escapes} just doesn't work.

I'm attempting this with flume-distribution-0.9.4-bin.tar.gz and hadoop-0.20.2-cdh3u4.tar.gz.

Any ideas?

Thanks,
Josh