You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@flume.apache.org by sz...@apache.org on 2018/11/23 10:50:08 UTC
flume git commit: FLUME-1342 adding jmx metrics tables to docs
Repository: flume
Updated Branches:
refs/heads/trunk 23473e1d5 -> 84b6263cb
FLUME-1342 adding jmx metrics tables to docs
This PR adds a few tables to the User Guide that describe the metrics
published by sorurces, sinks and channels.
I used simple unix tools to gather the data then I wrote a small utility to
convert it to csv.
Then I used an online converter https://www.tablesgenerator.com/ to generate
the rst tables and then a little manual editing.
I discovered some rst formatting problems in the FlumeUserGuide.rst,
corrected them, too.
It was rather painful process to gather the data and find a decent
representation.
So far this PR only contains the end result. I would be happy to share the
utilities, just don't know what would be the best way.
This closes #242
Reviewers: Denes Arvay, Ferenc Szabo
(Endre Major via Ferenc Szabo)
Project: http://git-wip-us.apache.org/repos/asf/flume/repo
Commit: http://git-wip-us.apache.org/repos/asf/flume/commit/84b6263c
Tree: http://git-wip-us.apache.org/repos/asf/flume/tree/84b6263c
Diff: http://git-wip-us.apache.org/repos/asf/flume/diff/84b6263c
Branch: refs/heads/trunk
Commit: 84b6263cbd4c771144186d4c561e12ff569e1887
Parents: 23473e1
Author: Endre Major <em...@cloudera.com>
Authored: Fri Nov 23 11:47:15 2018 +0100
Committer: Ferenc Szabo <sz...@apache.org>
Committed: Fri Nov 23 11:47:15 2018 +0100
----------------------------------------------------------------------
flume-ng-doc/sphinx/FlumeUserGuide.rst | 182 +++++++++++++++++++++++++++-
1 file changed, 178 insertions(+), 4 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/flume/blob/84b6263c/flume-ng-doc/sphinx/FlumeUserGuide.rst
----------------------------------------------------------------------
diff --git a/flume-ng-doc/sphinx/FlumeUserGuide.rst b/flume-ng-doc/sphinx/FlumeUserGuide.rst
index 492d90e..df4d5c2 100644
--- a/flume-ng-doc/sphinx/FlumeUserGuide.rst
+++ b/flume-ng-doc/sphinx/FlumeUserGuide.rst
@@ -1993,7 +1993,7 @@ selector.type replicating or multiplexing
selector.* replicating Depends on the selector.type value
interceptors -- Space-separated list of interceptors
interceptors.*
-==================== =========== =================================================================
+==================== =========== ================================================================
For example, a syslog UDP source for agent named a1:
@@ -2387,9 +2387,7 @@ hdfs.filePrefix FlumeData Name prefixed to files created by Flume in
hdfs.fileSuffix -- Suffix to append to file (eg ``.avro`` - *NOTE: period is not automatically added*)
hdfs.inUsePrefix -- Prefix that is used for temporal files that flume actively writes into
hdfs.inUseSuffix ``.tmp`` Suffix that is used for temporal files that flume actively writes into
-hdfs.emptyInUseSuffix false If ``false`` an ``hdfs.inUseSuffix`` is used while writing the output. After closing the output
- ``hdfs.inUseSuffix`` is removed from the output file name.
- If ``true`` the ``hdfs.inUseSuffix`` parameter is ignored an empty string is used instead.
+hdfs.emptyInUseSuffix false If ``false`` an ``hdfs.inUseSuffix`` is used while writing the output. After closing the output ``hdfs.inUseSuffix`` is removed from the output file name. If ``true`` the ``hdfs.inUseSuffix`` parameter is ignored an empty string is used instead.
hdfs.rollInterval 30 Number of seconds to wait before rolling current file
(0 = never roll based on time interval)
hdfs.rollSize 1024 File size to trigger roll, in bytes (0: never roll based on file size)
@@ -4962,6 +4960,182 @@ Monitoring in Flume is still a work in progress. Changes can happen very often.
Several Flume components report metrics to the JMX platform MBean server. These
metrics can be queried using Jconsole.
+Available Component Metrics
+---------------------------
+
+The following tables show what metrics are available for components. Each component only maintains a
+set of metrics, indicated by an 'x', the unmaintained ones show default values, that is 0.
+These tables tell you where you can expect meaningful data.
+The name of the metrics should be descriptive enough, for more information you have to dig into the
+source code of the components.
+
+Sources 1
+~~~~~~~~~
+
++--------------------------+------+------+------+-----+-------+--------------------+--------+
+| | Avro | Exec | HTTP | JMS | Kafka | MultiportSyslogTCP | Scribe |
++--------------------------+------+------+------+-----+-------+--------------------+--------+
+| AppendAcceptedCount | x | | | | | | |
++--------------------------+------+------+------+-----+-------+--------------------+--------+
+| AppendBatchAcceptedCount | x | | x | x | | | |
++--------------------------+------+------+------+-----+-------+--------------------+--------+
+| AppendBatchReceivedCount | x | | x | x | | | |
++--------------------------+------+------+------+-----+-------+--------------------+--------+
+| AppendReceivedCount | x | | | | | | |
++--------------------------+------+------+------+-----+-------+--------------------+--------+
+| ChannelWriteFail | x | | x | x | x | x | x |
++--------------------------+------+------+------+-----+-------+--------------------+--------+
+| EventAcceptedCount | x | x | x | x | x | x | x |
++--------------------------+------+------+------+-----+-------+--------------------+--------+
+| EventReadFail | | | x | x | x | x | x |
++--------------------------+------+------+------+-----+-------+--------------------+--------+
+| EventReceivedCount | x | x | x | x | x | x | x |
++--------------------------+------+------+------+-----+-------+--------------------+--------+
+| GenericProcessingFail | | | x | | | x | |
++--------------------------+------+------+------+-----+-------+--------------------+--------+
+| KafkaCommitTimer | | | | | x | | |
++--------------------------+------+------+------+-----+-------+--------------------+--------+
+| KafkaEmptyCount | | | | | x | | |
++--------------------------+------+------+------+-----+-------+--------------------+--------+
+| KafkaEventGetTimer | | | | | x | | |
++--------------------------+------+------+------+-----+-------+--------------------+--------+
+| OpenConnectionCount | x | | | | | | |
++--------------------------+------+------+------+-----+-------+--------------------+--------+
+
+Sources 2
+~~~~~~~~~
+
++--------------------------+-------------------+----------------+-----------+-----------+---------+--------+
+| | SequenceGenerator | SpoolDirectory | SyslogTcp | SyslogUDP | Taildir | Thrift |
++--------------------------+-------------------+----------------+-----------+-----------+---------+--------+
+| AppendAcceptedCount | | | | | | x |
++--------------------------+-------------------+----------------+-----------+-----------+---------+--------+
+| AppendBatchAcceptedCount | x | x | | | x | x |
++--------------------------+-------------------+----------------+-----------+-----------+---------+--------+
+| AppendBatchReceivedCount | | x | | | x | x |
++--------------------------+-------------------+----------------+-----------+-----------+---------+--------+
+| AppendReceivedCount | | | | | | x |
++--------------------------+-------------------+----------------+-----------+-----------+---------+--------+
+| ChannelWriteFail | x | x | x | x | x | x |
++--------------------------+-------------------+----------------+-----------+-----------+---------+--------+
+| EventAcceptedCount | x | x | x | x | x | x |
++--------------------------+-------------------+----------------+-----------+-----------+---------+--------+
+| EventReadFail | | x | x | x | x | |
++--------------------------+-------------------+----------------+-----------+-----------+---------+--------+
+| EventReceivedCount | | x | x | x | x | x |
++--------------------------+-------------------+----------------+-----------+-----------+---------+--------+
+| GenericProcessingFail | | x | | | x | |
++--------------------------+-------------------+----------------+-----------+-----------+---------+--------+
+| KafkaCommitTimer | | | | | | |
++--------------------------+-------------------+----------------+-----------+-----------+---------+--------+
+| KafkaEmptyCount | | | | | | |
++--------------------------+-------------------+----------------+-----------+-----------+---------+--------+
+| KafkaEventGetTimer | | | | | | |
++--------------------------+-------------------+----------------+-----------+-----------+---------+--------+
+| OpenConnectionCount | | | | | | |
++--------------------------+-------------------+----------------+-----------+-----------+---------+--------+
+
+Sinks 1
+~~~~~~~
+
++------------------------+-------------+------------+---------------+-------+--------+
+| | Avro/Thrift | AsyncHBase | ElasticSearch | HBase | HBase2 |
++------------------------+-------------+------------+---------------+-------+--------+
+| BatchCompleteCount | x | x | x | x | x |
++------------------------+-------------+------------+---------------+-------+--------+
+| BatchEmptyCount | x | x | x | x | x |
++------------------------+-------------+------------+---------------+-------+--------+
+| BatchUnderflowCount | x | x | x | x | x |
++------------------------+-------------+------------+---------------+-------+--------+
+| ChannelReadFail | x | | | | x |
++------------------------+-------------+------------+---------------+-------+--------+
+| ConnectionClosedCount | x | x | x | x | x |
++------------------------+-------------+------------+---------------+-------+--------+
+| ConnectionCreatedCount | x | x | x | x | x |
++------------------------+-------------+------------+---------------+-------+--------+
+| ConnectionFailedCount | x | x | x | x | x |
++------------------------+-------------+------------+---------------+-------+--------+
+| EventDrainAttemptCount | x | x | x | x | x |
++------------------------+-------------+------------+---------------+-------+--------+
+| EventDrainSuccessCount | x | x | x | x | x |
++------------------------+-------------+------------+---------------+-------+--------+
+| EventWriteFail | x | | | | x |
++------------------------+-------------+------------+---------------+-------+--------+
+| KafkaEventSendTimer | | | | | |
++------------------------+-------------+------------+---------------+-------+--------+
+| RollbackCount | | | | | |
++------------------------+-------------+------------+---------------+-------+--------+
+
+Sinks 2
+~~~~~~~
+
++------------------------+-----------+------+------+-------+-----------+-------------+
+| | HDFSEvent | Hive | Http | Kafka | Morphline | RollingFile |
++------------------------+-----------+------+------+-------+-----------+-------------+
+| BatchCompleteCount | x | x | | | x | |
++------------------------+-----------+------+------+-------+-----------+-------------+
+| BatchEmptyCount | x | x | | x | x | |
++------------------------+-----------+------+------+-------+-----------+-------------+
+| BatchUnderflowCount | x | x | | x | x | |
++------------------------+-----------+------+------+-------+-----------+-------------+
+| ChannelReadFail | x | x | x | x | x | x |
++------------------------+-----------+------+------+-------+-----------+-------------+
+| ConnectionClosedCount | x | x | | | | x |
++------------------------+-----------+------+------+-------+-----------+-------------+
+| ConnectionCreatedCount | x | x | | | | x |
++------------------------+-----------+------+------+-------+-----------+-------------+
+| ConnectionFailedCount | x | x | | | | x |
++------------------------+-----------+------+------+-------+-----------+-------------+
+| EventDrainAttemptCount | x | x | x | | x | x |
++------------------------+-----------+------+------+-------+-----------+-------------+
+| EventDrainSuccessCount | x | x | x | x | x | x |
++------------------------+-----------+------+------+-------+-----------+-------------+
+| EventWriteFail | x | x | x | x | x | x |
++------------------------+-----------+------+------+-------+-----------+-------------+
+| KafkaEventSendTimer | | | | x | | |
++------------------------+-----------+------+------+-------+-----------+-------------+
+| RollbackCount | | | | x | | |
++------------------------+-----------+------+------+-------+-----------+-------------+
+
+Channels
+~~~~~~~~
+
++---------------------------------+------+-------+--------+-----------------+-----------------+
+| | File | Kafka | Memory | PseudoTxnMemory | SpillableMemory |
++---------------------------------+------+-------+--------+-----------------+-----------------+
+| ChannelCapacity | x | | x | | x |
++---------------------------------+------+-------+--------+-----------------+-----------------+
+| ChannelSize | x | | x | x | x |
++---------------------------------+------+-------+--------+-----------------+-----------------+
+| CheckpointBackupWriteErrorCount | x | | | | |
++---------------------------------+------+-------+--------+-----------------+-----------------+
+| CheckpointWriteErrorCount | x | | | | |
++---------------------------------+------+-------+--------+-----------------+-----------------+
+| EventPutAttemptCount | x | x | x | x | x |
++---------------------------------+------+-------+--------+-----------------+-----------------+
+| EventPutErrorCount | x | | | | |
++---------------------------------+------+-------+--------+-----------------+-----------------+
+| EventPutSuccessCount | x | x | x | x | x |
++---------------------------------+------+-------+--------+-----------------+-----------------+
+| EventTakeAttemptCount | x | x | x | x | x |
++---------------------------------+------+-------+--------+-----------------+-----------------+
+| EventTakeErrorCount | x | | | | |
++---------------------------------+------+-------+--------+-----------------+-----------------+
+| EventTakeSuccessCount | x | x | x | x | x |
++---------------------------------+------+-------+--------+-----------------+-----------------+
+| KafkaCommitTimer | | x | | | |
++---------------------------------+------+-------+--------+-----------------+-----------------+
+| KafkaEventGetTimer | | x | | | |
++---------------------------------+------+-------+--------+-----------------+-----------------+
+| KafkaEventSendTimer | | x | | | |
++---------------------------------+------+-------+--------+-----------------+-----------------+
+| Open | x | | | | |
++---------------------------------+------+-------+--------+-----------------+-----------------+
+| RollbackCounter | | x | | | |
++---------------------------------+------+-------+--------+-----------------+-----------------+
+| Unhealthy | x | | | | |
++---------------------------------+------+-------+--------+-----------------+-----------------+
+
JMX Reporting
-------------