You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org> on 2012/03/01 00:43:59 UTC
[jira] [Commented] (FLUME-992) Create configuration stubs for
sources, channels, sinks etc
[ https://issues.apache.org/jira/browse/FLUME-992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13219662#comment-13219662 ]
jiraposter@reviews.apache.org commented on FLUME-992:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4115/
-----------------------------------------------------------
Review request for Flume.
Summary
-------
Currently the configuration of each component sits deep inside the component themselves. There is no way to verify if a configuration is valid before run time. In most systems like Flume, it is likely that these confs will be deployed from one single host, on to the machines where flume agents are running. Only when each agent starts, invalid confs are identified because the Agents would terminate by throwing exceptions. This is a first attempt to make a component-aware configuration system which is independent of the Flume, and does not require the Flume jar to be installed. Each component needs a configuration manager, which configures the components.
I have not completed the validation part yet, but have completed the configuration managers for all the components that come in the flume jar. I will add support for the other components, such as HDFS sink, JDBC channel etc soon. I will also add the validation support to the classes, but would like a review of the model being proposed here.
This addresses bug FLUME-992.
https://issues.apache.org/jira/browse/FLUME-992
Diffs
-----
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannel.java bca0c50
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProvider.java e445d61
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProviderFactory.java 6fbd6ef
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java 1cf1c0c
flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestJdbcChannelProvider.java 7710d46
flume-ng-clients/flume-ng-log4jappender/src/test/java/org/apache/flume/clients/log4jappender/TestLog4jAppender.java 68d95fb
flume-ng-configuration/pom.xml PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/AbstractComponentConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ConfigurationException.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/Context.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationError.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationErrorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MemoryChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MultiplexingChannelSelectorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/PseudoTxnMemoryChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/AvroSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/FailoverSinkProcessorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/RollingFileSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkGroupConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/AvroSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/ExecSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/NetcatSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/test/resources/flume-ng-error-test PRE-CREATION
flume-ng-configuration/src/test/resources/flume-ng-test.conf PRE-CREATION
flume-ng-core/pom.xml d753fa1
flume-ng-core/src/main/java/org/apache/flume/Context.java f1c8f85
flume-ng-core/src/main/java/org/apache/flume/channel/AbstractChannel.java 352bf08
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorFactory.java a7d5f94
flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java e79490e
flume-ng-core/src/main/java/org/apache/flume/channel/MultiplexingChannelSelector.java c63d0a1
flume-ng-core/src/main/java/org/apache/flume/channel/PseudoTxnMemoryChannel.java 1df580e
flume-ng-core/src/main/java/org/apache/flume/channel/ReplicatingChannelSelector.java 8f22746
flume-ng-core/src/main/java/org/apache/flume/conf/Configurable.java 0fa4839
flume-ng-core/src/main/java/org/apache/flume/conf/Configurables.java 84492e5
flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 5440631
flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkProcessor.java 257bab3
flume-ng-core/src/main/java/org/apache/flume/sink/FailoverSinkProcessor.java 9f5b856
flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java 45c031d
flume-ng-core/src/main/java/org/apache/flume/sink/SinkGroup.java 1adc5ff
flume-ng-core/src/main/java/org/apache/flume/sink/SinkProcessorFactory.java 859f4fd
flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java 7b079f9
flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java a96016c
flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java d205bbc
flume-ng-core/src/test/java/org/apache/flume/TestContext.java a5e6aa8
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannel.java 3392dff
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelConcurrency.java 8dad0b2
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelTransaction.java bc81f26
flume-ng-core/src/test/java/org/apache/flume/channel/TestMultiplexingChannelSelector.java 2626b20
flume-ng-core/src/test/java/org/apache/flume/sink/TestAvroSink.java 467785f
flume-ng-core/src/test/java/org/apache/flume/sink/TestFailoverSinkProcessor.java 93ad3bf
flume-ng-core/src/test/java/org/apache/flume/sink/TestLoggerSink.java 92ff6fe
flume-ng-core/src/test/java/org/apache/flume/sink/TestRollingFileSink.java 7e26e2a
flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java c5c3f2f
flume-ng-core/src/test/java/org/apache/flume/source/TestExecSource.java 6035270
flume-ng-core/src/test/java/org/apache/flume/source/TestPollableSourceRunner.java c27f82c
flume-ng-core/src/test/java/org/apache/flume/source/TestSequenceGeneratorSource.java 579b257
flume-ng-legacy-sources/flume-avro-source/src/main/java/org/apache/flume/source/avroLegacy/AvroLegacySource.java b1e67f7
flume-ng-legacy-sources/flume-avro-source/src/test/java/org/apache/flume/source/avroLegacy/TestLegacyAvroSource.java 6e3eb53
flume-ng-legacy-sources/flume-thrift-source/src/main/java/org/apache/flume/source/thriftLegacy/ThriftLegacySource.java 5fe270a
flume-ng-legacy-sources/flume-thrift-source/src/test/java/org/apache/flume/source/thriftLegacy/TestThriftLegacySource.java ddd9478
flume-ng-node/pom.xml b9b062e
flume-ng-node/src/main/java/org/apache/flume/conf/properties/FlumeConfiguration.java d66f6d1
flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java 97f72e1
flume-ng-node/src/test/java/org/apache/flume/node/TestAbstractLogicalNodeManager.java 521b586
flume-ng-node/src/test/java/org/apache/flume/source/TestNetcatSource.java fb2a960
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 3da90a5
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 7d8ee8a
flume-ng-sinks/flume-irc-sink/src/main/java/org/apache/flume/sink/irc/IRCSink.java 0a5498f
pom.xml d785762
Diff: https://reviews.apache.org/r/4115/diff
Testing
-------
All existing unit tests for the components whose configuration has now been moved to these stubs pass.
Thanks,
Hari
> Create configuration stubs for sources, channels, sinks etc
> -----------------------------------------------------------
>
> Key: FLUME-992
> URL: https://issues.apache.org/jira/browse/FLUME-992
> Project: Flume
> Issue Type: Sub-task
> Affects Versions: v1.0.0
> Reporter: Hari Shreedharan
> Fix For: v1.1.0
>
>
> Currently the configuration of each component sits deep inside the component themselves. There is no way to verify if a configuration is valid before run time. In most systems like Flume, it is likely that these confs will be deployed from one single host, on to the machines where flume agents are running. Only when each agent starts, invalid confs are identified because the Agents would terminate by throwing exceptions. This is a first attempt to make a component-aware configuration system which is independent of the Flume, and does not require the Flume jar to be installed. Each component needs a configuration manager, which configures the components.
> Provide abstract Configuration stubs for each component type, sources, channels, sinks, selectors etc, which are in the new package, independent on ng-core. Now for each of the channels extend the abstract class and check the config properties for each of the required parameters for that component, for example: MultiplexingChannelSelectorConfigurator would look for default channel etc. If a particular component does not have a configuration class then let the current mechanism continue.
> This will require implementation for each component, but it should not be too complex. One additional advantage we get from this is that we can separate out the config validation from the components into these stubs, but we will still need to read the values out of the Context as we do currently(else we end up making the configuration dependent on flume-ng-core itself which we wanted to avoid).
> A problem with this is making a change to the configuration would require changes in the configuration classes and in the components also(where the configuration is read and the component is actually configured). I could not figure out a way of avoiding this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira