You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flume.apache.org by mp...@apache.org on 2012/07/09 04:02:05 UTC
svn commit: r1358893 - /flume/trunk/flume-ng-doc/sphinx/FlumeUserGuide.rst
Author: mpercy
Date: Mon Jul 9 02:02:05 2012
New Revision: 1358893
URL: http://svn.apache.org/viewvc?rev=1358893&view=rev
Log:
FLUME-1352. Add documentation for HDFS path rounddown.
(Hari Shreedharan via Mike Percy)
Modified:
flume/trunk/flume-ng-doc/sphinx/FlumeUserGuide.rst
Modified: flume/trunk/flume-ng-doc/sphinx/FlumeUserGuide.rst
URL: http://svn.apache.org/viewvc/flume/trunk/flume-ng-doc/sphinx/FlumeUserGuide.rst?rev=1358893&r1=1358892&r2=1358893&view=diff
==============================================================================
--- flume/trunk/flume-ng-doc/sphinx/FlumeUserGuide.rst (original)
+++ flume/trunk/flume-ng-doc/sphinx/FlumeUserGuide.rst Mon Jul 9 02:02:05 2012
@@ -954,6 +954,9 @@ hdfs.threadsPoolSize 10 Nu
hdfs.rollTimerPoolSize 1 Number of threads per HDFS sink for scheduling timed file rolling
hdfs.kerberosPrincipal -- Kerberos user principal for accessing secure HDFS
hdfs.kerberosKeytab -- Kerberos keytab for accessing secure HDFS
+hdfs.round false Should the timestamp be rounded down (if true, affects all time based escape sequences except %t)
+hdfs.roundValue 1 Rounded down to the highest multiple of this (in the unit configured using ``hdfs.roundUnit``), less than current time.
+hdfs.roundUnit second The unit of the round down value - ``second``, ``minute`` or ``hour``.
serializer ``TEXT`` Other possible options include ``AVRO_EVENT`` or the
fully-qualified class name of an implementation of the
``EventSerializer.Builder`` interface.
@@ -968,8 +971,13 @@ Example for agent named **agent_foo**:
agent_foo.sinks = hdfsSink-1
agent_foo.sinks.hdfsSink-1.type = hdfs
agent_foo.sinks.hdfsSink-1.channels = memoryChannel-1
- agent_foo.sinks.hdfsSink-1.hdfs.path = /flume/events/%y-%m-%d
- agent_foo.sinks.hdfsSink-1.hdfs.filePrevix = events-
+ agent_foo.sinks.hdfsSink-1.hdfs.path = /flume/events/%y-%m-%d/%H%M/%S
+ agent_foo.sinks.hdfsSink-1.hdfs.filePrefix = events-
+ agent_foo.sinks.hdfsSink-1.hdfs.round = true
+ agent_foo.sinks.hdfsSink-1.hdfs.roundValue = 10
+ agent_foo.sinks.hdfsSink-1.hdfs.roundUnit = minute
+
+The above configuration will round down the timestamp to the last 10th minute. For example, an event with timestamp 11:54:34 AM, June 12, 2012 will cause the hdfs path to become ``/flume/events/2012-06-12/1150/00``.
Logger Sink