You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by Hari Shreedharan <hs...@cloudera.com> on 2012/03/01 00:42:19 UTC
Review Request: Component aware configuration managers for flume components
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4115/
-----------------------------------------------------------
Review request for Flume.
Summary
-------
Currently the configuration of each component sits deep inside the component themselves. There is no way to verify if a configuration is valid before run time. In most systems like Flume, it is likely that these confs will be deployed from one single host, on to the machines where flume agents are running. Only when each agent starts, invalid confs are identified because the Agents would terminate by throwing exceptions. This is a first attempt to make a component-aware configuration system which is independent of the Flume, and does not require the Flume jar to be installed. Each component needs a configuration manager, which configures the components.
I have not completed the validation part yet, but have completed the configuration managers for all the components that come in the flume jar. I will add support for the other components, such as HDFS sink, JDBC channel etc soon. I will also add the validation support to the classes, but would like a review of the model being proposed here.
This addresses bug FLUME-992.
https://issues.apache.org/jira/browse/FLUME-992
Diffs
-----
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannel.java bca0c50
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProvider.java e445d61
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProviderFactory.java 6fbd6ef
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java 1cf1c0c
flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestJdbcChannelProvider.java 7710d46
flume-ng-clients/flume-ng-log4jappender/src/test/java/org/apache/flume/clients/log4jappender/TestLog4jAppender.java 68d95fb
flume-ng-configuration/pom.xml PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/AbstractComponentConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ConfigurationException.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/Context.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationError.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationErrorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MemoryChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MultiplexingChannelSelectorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/PseudoTxnMemoryChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/AvroSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/FailoverSinkProcessorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/RollingFileSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkGroupConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/AvroSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/ExecSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/NetcatSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/test/resources/flume-ng-error-test PRE-CREATION
flume-ng-configuration/src/test/resources/flume-ng-test.conf PRE-CREATION
flume-ng-core/pom.xml d753fa1
flume-ng-core/src/main/java/org/apache/flume/Context.java f1c8f85
flume-ng-core/src/main/java/org/apache/flume/channel/AbstractChannel.java 352bf08
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorFactory.java a7d5f94
flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java e79490e
flume-ng-core/src/main/java/org/apache/flume/channel/MultiplexingChannelSelector.java c63d0a1
flume-ng-core/src/main/java/org/apache/flume/channel/PseudoTxnMemoryChannel.java 1df580e
flume-ng-core/src/main/java/org/apache/flume/channel/ReplicatingChannelSelector.java 8f22746
flume-ng-core/src/main/java/org/apache/flume/conf/Configurable.java 0fa4839
flume-ng-core/src/main/java/org/apache/flume/conf/Configurables.java 84492e5
flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 5440631
flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkProcessor.java 257bab3
flume-ng-core/src/main/java/org/apache/flume/sink/FailoverSinkProcessor.java 9f5b856
flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java 45c031d
flume-ng-core/src/main/java/org/apache/flume/sink/SinkGroup.java 1adc5ff
flume-ng-core/src/main/java/org/apache/flume/sink/SinkProcessorFactory.java 859f4fd
flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java 7b079f9
flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java a96016c
flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java d205bbc
flume-ng-core/src/test/java/org/apache/flume/TestContext.java a5e6aa8
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannel.java 3392dff
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelConcurrency.java 8dad0b2
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelTransaction.java bc81f26
flume-ng-core/src/test/java/org/apache/flume/channel/TestMultiplexingChannelSelector.java 2626b20
flume-ng-core/src/test/java/org/apache/flume/sink/TestAvroSink.java 467785f
flume-ng-core/src/test/java/org/apache/flume/sink/TestFailoverSinkProcessor.java 93ad3bf
flume-ng-core/src/test/java/org/apache/flume/sink/TestLoggerSink.java 92ff6fe
flume-ng-core/src/test/java/org/apache/flume/sink/TestRollingFileSink.java 7e26e2a
flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java c5c3f2f
flume-ng-core/src/test/java/org/apache/flume/source/TestExecSource.java 6035270
flume-ng-core/src/test/java/org/apache/flume/source/TestPollableSourceRunner.java c27f82c
flume-ng-core/src/test/java/org/apache/flume/source/TestSequenceGeneratorSource.java 579b257
flume-ng-legacy-sources/flume-avro-source/src/main/java/org/apache/flume/source/avroLegacy/AvroLegacySource.java b1e67f7
flume-ng-legacy-sources/flume-avro-source/src/test/java/org/apache/flume/source/avroLegacy/TestLegacyAvroSource.java 6e3eb53
flume-ng-legacy-sources/flume-thrift-source/src/main/java/org/apache/flume/source/thriftLegacy/ThriftLegacySource.java 5fe270a
flume-ng-legacy-sources/flume-thrift-source/src/test/java/org/apache/flume/source/thriftLegacy/TestThriftLegacySource.java ddd9478
flume-ng-node/pom.xml b9b062e
flume-ng-node/src/main/java/org/apache/flume/conf/properties/FlumeConfiguration.java d66f6d1
flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java 97f72e1
flume-ng-node/src/test/java/org/apache/flume/node/TestAbstractLogicalNodeManager.java 521b586
flume-ng-node/src/test/java/org/apache/flume/source/TestNetcatSource.java fb2a960
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 3da90a5
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 7d8ee8a
flume-ng-sinks/flume-irc-sink/src/main/java/org/apache/flume/sink/irc/IRCSink.java 0a5498f
pom.xml d785762
Diff: https://reviews.apache.org/r/4115/diff
Testing
-------
All existing unit tests for the components whose configuration has now been moved to these stubs pass.
Thanks,
Hari
Re: Review Request: Component aware configuration managers for flume
components
Posted by Arvind Prabhakar <ar...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4115/#review5783
-----------------------------------------------------------
Thanks for the interim patch Hari. I did a quick walk through of these changes and have some feedback that follows. Some general issues:
* The component configuration stubs do not report any configuration errors at this stage. While we cannot expect these stubs to do runtime validation, we should at least check all the static rules that may be being violated. For example a non-numeric port value is something that does not get caught/reported by the current implementation.
* As implemented, the configuration validation mechanism only reports errors. It should also report warnings where default values override the specified values etc.
More specific feedback below:
flume-ng-configuration/pom.xml
<https://reviews.apache.org/r/4115/#comment12598>
Better to call it Flume NG Configuration only.
flume-ng-configuration/pom.xml
<https://reviews.apache.org/r/4115/#comment12600>
Duplicate
flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfiguration.java
<https://reviews.apache.org/r/4115/#comment12623>
should be private
flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfiguration.java
<https://reviews.apache.org/r/4115/#comment12624>
Does not look like its being used, in which case it should be removed considering the risk it introduces.
flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfiguration.java
<https://reviews.apache.org/r/4115/#comment12625>
This method is not returning any configuration values. Assuming it is work in progress - you should have a way by which the specialized component classes can use this without having to copy/rewrite this method.
flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfigurationFactory.java
<https://reviews.apache.org/r/4115/#comment12614>
Please make a ComponentTypeEnum that has SOURCE, CHANNEL, SINKPROCESSOR, CHANNELSELECTOR etc in it.
flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfigurationFactory.java
<https://reviews.apache.org/r/4115/#comment12615>
Shouldn't this be Class<SinkConfiguration> ?
Ideally, you want to have an enum that can return the actual class instance directly rather than doing a Class.forName() invocation.
flume-ng-configuration/src/main/java/org/apache/flume/conf/ConfigurationException.java
<https://reviews.apache.org/r/4115/#comment12597>
ConfigurationException should extend FlumeException.
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfiguration.java
<https://reviews.apache.org/r/4115/#comment12602>
Actually creates a fully populated configuration.
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfiguration.java
<https://reviews.apache.org/r/4115/#comment12607>
Need a few test cases to make sure this is working as expected.
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfiguration.java
<https://reviews.apache.org/r/4115/#comment12603>
Misleading error. When value is null, the error should not be agent name missing.
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfiguration.java
<https://reviews.apache.org/r/4115/#comment12621>
This exception is currently raised only if the instantiation fails. Ideally though, you want the conf object to be created and invoke a .validate() method on it, and then be able to extract any errors for reporting upwards.
Same comment applies to validateSources etc.
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfiguration.java
<https://reviews.apache.org/r/4115/#comment12612>
Please use generics to ensure type compliance instead of casts.
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationError.java
<https://reviews.apache.org/r/4115/#comment12604>
This should have provision for value that is in question (since some errors are value specific), and also the raw property that triggered this. Otherwise the error could be ambiguous
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationErrorType.java
<https://reviews.apache.org/r/4115/#comment12605>
These enums should have a descriptive message that tell the user what this problem is. Something like:
PROPERTY_NAME_NULL("A null or empty value was specified for the given property")
etc.
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkConfiguration.java
<https://reviews.apache.org/r/4115/#comment12617>
I don't think this is enum should have sink group and sink processor information. They belong to separate category.
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkConfiguration.java
<https://reviews.apache.org/r/4115/#comment12616>
Please define an interface that this enum implements to provide the class name instead of overloading the toString() method. Same for other enums too.
flume-ng-core/pom.xml
<https://reviews.apache.org/r/4115/#comment12599>
Indentation.
- Arvind
On 2012-03-09 08:47:00, Hari Shreedharan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/4115/
> -----------------------------------------------------------
>
> (Updated 2012-03-09 08:47:00)
>
>
> Review request for Flume.
>
>
> Summary
> -------
>
> Currently the configuration of each component sits deep inside the component themselves. There is no way to verify if a configuration is valid before run time. In most systems like Flume, it is likely that these confs will be deployed from one single host, on to the machines where flume agents are running. Only when each agent starts, invalid confs are identified because the Agents would terminate by throwing exceptions. This is a first attempt to make a component-aware configuration system which is independent of the Flume, and does not require the Flume jar to be installed. Each component needs a configuration manager, which configures the components.
>
> I have not completed the validation part yet, but have completed the configuration managers for all the components that come in the flume jar. I will add support for the other components, such as HDFS sink, JDBC channel etc soon. I will also add the validation support to the classes, but would like a review of the model being proposed here.
>
> Note that I do know of a couple of bugs in the existing diff(like that if the name of a config key starts with an Upper case letter, an exception is thrown) - I will fix and upload the review here soon.
>
>
> This addresses bug FLUME-992.
> https://issues.apache.org/jira/browse/FLUME-992
>
>
> Diffs
> -----
>
> flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/source/NetcatSourceConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/source/ExecSourceConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/source/AvroSourceConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkProcessorType.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkType.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkProcessorConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkGroupConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/RollingFileSinkConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationError.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationErrorType.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorType.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelType.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MemoryChannelConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MultiplexingChannelSelectorConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/PseudoTxnMemoryChannelConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/AvroSinkConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/FailoverSinkProcessorConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/Context.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/ConfigurationException.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfigurationFactory.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfiguration.java PRE-CREATION
> flume-ng-clients/flume-ng-log4jappender/src/test/java/org/apache/flume/clients/log4jappender/TestLog4jAppender.java 68d95fb
> flume-ng-configuration/pom.xml PRE-CREATION
> flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestJdbcChannelProvider.java 7710d46
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453
> flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannel.java bca0c50
> flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProvider.java e445d61
> flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProviderFactory.java 6fbd6ef
> flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java 307ae89
> flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceType.java PRE-CREATION
> flume-ng-configuration/src/test/resources/flume-ng-error-test PRE-CREATION
> flume-ng-configuration/src/test/resources/flume-ng-test.conf PRE-CREATION
> flume-ng-core/pom.xml d753fa1
> flume-ng-core/src/main/java/org/apache/flume/ChannelSelector.java fba2dcb
> flume-ng-core/src/main/java/org/apache/flume/Context.java a6341a5
> flume-ng-core/src/main/java/org/apache/flume/SinkProcessor.java 11651ad
> flume-ng-core/src/main/java/org/apache/flume/SinkProcessorType.java be1891b
> flume-ng-core/src/main/java/org/apache/flume/channel/AbstractChannel.java 352bf08
> flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorFactory.java 800f471
> flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorType.java 511fc65
> flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8
> flume-ng-core/src/main/java/org/apache/flume/channel/DefaultChannelFactory.java 963a6a3
> flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java bfa1fde
> flume-ng-core/src/main/java/org/apache/flume/channel/MultiplexingChannelSelector.java 83928b7
> flume-ng-core/src/main/java/org/apache/flume/channel/PseudoTxnMemoryChannel.java 489d3e5
> flume-ng-core/src/main/java/org/apache/flume/channel/ReplicatingChannelSelector.java 8f22746
> flume-ng-core/src/main/java/org/apache/flume/conf/Configurable.java 0fa4839
> flume-ng-core/src/main/java/org/apache/flume/conf/ConfigurableComponent.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/conf/Configurables.java 84492e5
> flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 7386d06
> flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkFactory.java b89dfa0
> flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkProcessor.java 257bab3
> flume-ng-core/src/main/java/org/apache/flume/sink/FailoverSinkProcessor.java 9f5b856
> flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java 7f1d3b3
> flume-ng-core/src/main/java/org/apache/flume/sink/SinkGroup.java 2e80a56
> flume-ng-core/src/main/java/org/apache/flume/sink/SinkProcessorFactory.java 10f9f4e
> flume-ng-core/src/main/java/org/apache/flume/sink/SinkType.java 6b08c09
> flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java a3f6640
> flume-ng-core/src/main/java/org/apache/flume/source/DefaultSourceFactory.java a610e6f
> flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java b6b1181
> flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java 94245ac
> flume-ng-core/src/main/java/org/apache/flume/source/SourceType.java 9082470
> flume-ng-core/src/test/java/org/apache/flume/TestContext.java 51c350f
> flume-ng-core/src/test/java/org/apache/flume/channel/AbstractBasicChannelSemanticsTest.java 6e71e46
> flume-ng-core/src/test/java/org/apache/flume/channel/MockChannel.java 24b01e2
> flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannel.java e070864
> flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelConcurrency.java 8dad0b2
> flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelTransaction.java bc81f26
> flume-ng-core/src/test/java/org/apache/flume/sink/TestAvroSink.java 467785f
> flume-ng-core/src/test/java/org/apache/flume/sink/TestFailoverSinkProcessor.java 195c121
> flume-ng-core/src/test/java/org/apache/flume/sink/TestLoggerSink.java 92ff6fe
> flume-ng-core/src/test/java/org/apache/flume/sink/TestRollingFileSink.java 7e26e2a
> flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java c5c3f2f
> flume-ng-core/src/test/java/org/apache/flume/source/TestExecSource.java 6035270
> flume-ng-core/src/test/java/org/apache/flume/source/TestPollableSourceRunner.java c27f82c
> flume-ng-core/src/test/java/org/apache/flume/source/TestSequenceGeneratorSource.java 579b257
> flume-ng-legacy-sources/flume-avro-source/src/main/java/org/apache/flume/source/avroLegacy/AvroLegacySource.java 65d9142
> flume-ng-legacy-sources/flume-avro-source/src/test/java/org/apache/flume/source/avroLegacy/TestLegacyAvroSource.java 6e3eb53
> flume-ng-legacy-sources/flume-thrift-source/src/main/java/org/apache/flume/source/thriftLegacy/ThriftLegacySource.java fbf7362
> flume-ng-legacy-sources/flume-thrift-source/src/test/java/org/apache/flume/source/thriftLegacy/TestThriftLegacySource.java ddd9478
> flume-ng-node/src/main/java/org/apache/flume/conf/properties/FlumeConfiguration.java d66f6d1
> flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java 97f72e1
> flume-ng-node/src/test/java/org/apache/flume/node/TestAbstractLogicalNodeManager.java 521b586
> flume-ng-node/src/test/java/org/apache/flume/source/TestNetcatSource.java fb2a960
> flume-ng-node/src/test/resources/flume-conf.properties 848caca
> flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 524b69c
> flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 7d8ee8a
> flume-ng-sinks/flume-irc-sink/src/main/java/org/apache/flume/sink/irc/IRCSink.java 8e77218
> pom.xml d785762
>
> Diff: https://reviews.apache.org/r/4115/diff
>
>
> Testing
> -------
>
> All existing unit tests for the components whose configuration has now been moved to these stubs pass.
>
>
> Thanks,
>
> Hari
>
>
Re: Review Request: Component aware configuration managers for flume
components
Posted by Will McQueen <wi...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4115/#review6535
-----------------------------------------------------------
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceType.java
<https://reviews.apache.org/r/4115/#comment14197>
Missing:
AVRO_LEGACY("org.apache.flume.source.avroLegacy.AvroLegacySource"),
THRIFT_LEGACY("org.apache.flume.source.thriftLegacy.ThriftLegacySource");
- Will
On 2012-03-27 01:05:49, Hari Shreedharan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/4115/
> -----------------------------------------------------------
>
> (Updated 2012-03-27 01:05:49)
>
>
> Review request for Flume.
>
>
> Summary
> -------
>
> Currently the configuration of each component sits deep inside the component themselves. There is no way to verify if a configuration is valid before run time. In most systems like Flume, it is likely that these confs will be deployed from one single host, on to the machines where flume agents are running. Only when each agent starts, invalid confs are identified because the Agents would terminate by throwing exceptions. This is a first attempt to make a component-aware configuration system which is independent of the Flume, and does not require the Flume jar to be installed. Each component needs a configuration manager, which configures the components.
>
> I have not completed the validation part yet, but have completed the configuration managers for all the components that come in the flume jar. I will add support for the other components, such as HDFS sink, JDBC channel etc soon. I will also add the validation support to the classes, but would like a review of the model being proposed here.
>
> Note that I do know of a couple of bugs in the existing diff(like that if the name of a config key starts with an Upper case letter, an exception is thrown) - I will fix and upload the review here soon.
>
>
> This addresses bug FLUME-992.
> https://issues.apache.org/jira/browse/FLUME-992
>
>
> Diffs
> -----
>
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453
> flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannel.java bca0c50
> flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProvider.java e445d61
> flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProviderFactory.java 6fbd6ef
> flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java 307ae89
> flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestJdbcChannelProvider.java 7710d46
> flume-ng-clients/flume-ng-log4jappender/src/test/java/org/apache/flume/clients/log4jappender/TestLog4jAppender.java 68d95fb
> flume-ng-configuration/pom.xml PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfigurationFactory.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/ConfigurationException.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/Context.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationError.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationErrorType.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorType.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelType.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/JdbcChannelConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MemoryChannelConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MultiplexingChannelSelectorConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/PseudoTxnMemoryChannelConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/AvroSinkConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/FailoverSinkProcessorConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/HDFSSinkConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/RollingFileSinkConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkGroupConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkProcessorConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkProcessorType.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkType.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/source/AvroSourceConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/source/ExecSourceConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/source/NetcatSourceConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceType.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SyslogTcpSourceConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SyslogUDPSourceConfiguration.java PRE-CREATION
> flume-ng-configuration/src/test/resources/flume-ng-error-test PRE-CREATION
> flume-ng-configuration/src/test/resources/flume-ng-test.conf PRE-CREATION
> flume-ng-core/pom.xml 37fb112
> flume-ng-core/src/main/java/org/apache/flume/ChannelSelector.java fba2dcb
> flume-ng-core/src/main/java/org/apache/flume/Context.java 5294e31
> flume-ng-core/src/main/java/org/apache/flume/SinkProcessor.java d863ed0
> flume-ng-core/src/main/java/org/apache/flume/SinkProcessorType.java be1891b
> flume-ng-core/src/main/java/org/apache/flume/channel/AbstractChannel.java 352bf08
> flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorFactory.java 800f471
> flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorType.java 511fc65
> flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8
> flume-ng-core/src/main/java/org/apache/flume/channel/DefaultChannelFactory.java 963a6a3
> flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java bfa1fde
> flume-ng-core/src/main/java/org/apache/flume/channel/MultiplexingChannelSelector.java 83928b7
> flume-ng-core/src/main/java/org/apache/flume/channel/PseudoTxnMemoryChannel.java 489d3e5
> flume-ng-core/src/main/java/org/apache/flume/channel/ReplicatingChannelSelector.java 8f22746
> flume-ng-core/src/main/java/org/apache/flume/conf/Configurable.java 1d69112
> flume-ng-core/src/main/java/org/apache/flume/conf/ConfigurableComponent.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/conf/Configurables.java 84492e5
> flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 3edc563
> flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkFactory.java b89dfa0
> flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkProcessor.java 257bab3
> flume-ng-core/src/main/java/org/apache/flume/sink/FailoverSinkProcessor.java 066ee37
> flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java 7f1d3b3
> flume-ng-core/src/main/java/org/apache/flume/sink/SinkGroup.java 0dffd69
> flume-ng-core/src/main/java/org/apache/flume/sink/SinkProcessorFactory.java 6160a17
> flume-ng-core/src/main/java/org/apache/flume/sink/SinkType.java 6b08c09
> flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java 903889d
> flume-ng-core/src/main/java/org/apache/flume/source/DefaultSourceFactory.java a610e6f
> flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java dbf79e0
> flume-ng-core/src/main/java/org/apache/flume/source/ExecSourceConfigurationConstants.java 73c985e
> flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java a841b0e
> flume-ng-core/src/main/java/org/apache/flume/source/SequenceGeneratorSource.java 440c5a9
> flume-ng-core/src/main/java/org/apache/flume/source/SourceType.java 9082470
> flume-ng-core/src/main/java/org/apache/flume/source/SyslogTcpSource.java d78d27f
> flume-ng-core/src/main/java/org/apache/flume/source/SyslogUDPSource.java 34818f0
> flume-ng-core/src/test/java/org/apache/flume/TestContext.java 51c350f
> flume-ng-core/src/test/java/org/apache/flume/channel/AbstractBasicChannelSemanticsTest.java 6e71e46
> flume-ng-core/src/test/java/org/apache/flume/channel/MockChannel.java 24b01e2
> flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannel.java e070864
> flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelConcurrency.java 8dad0b2
> flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelTransaction.java bc81f26
> flume-ng-core/src/test/java/org/apache/flume/sink/TestAvroSink.java 3765924
> flume-ng-core/src/test/java/org/apache/flume/sink/TestFailoverSinkProcessor.java 3358cf4
> flume-ng-core/src/test/java/org/apache/flume/sink/TestLoggerSink.java 92ff6fe
> flume-ng-core/src/test/java/org/apache/flume/sink/TestRollingFileSink.java 7e26e2a
> flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java c5c3f2f
> flume-ng-core/src/test/java/org/apache/flume/source/TestExecSource.java 615f2a3
> flume-ng-core/src/test/java/org/apache/flume/source/TestPollableSourceRunner.java c27f82c
> flume-ng-core/src/test/java/org/apache/flume/source/TestSequenceGeneratorSource.java 579b257
> flume-ng-core/src/test/java/org/apache/flume/source/TestSyslogUdpSource.java d1db49d
> flume-ng-legacy-sources/flume-avro-source/src/main/java/org/apache/flume/source/avroLegacy/AvroLegacySource.java dde8f28
> flume-ng-legacy-sources/flume-avro-source/src/test/java/org/apache/flume/source/avroLegacy/TestLegacyAvroSource.java 6e3eb53
> flume-ng-legacy-sources/flume-thrift-source/src/main/java/org/apache/flume/source/thriftLegacy/ThriftLegacySource.java 00df7e4
> flume-ng-legacy-sources/flume-thrift-source/src/test/java/org/apache/flume/source/thriftLegacy/TestThriftLegacySource.java ddd9478
> flume-ng-node/src/main/java/org/apache/flume/conf/properties/FlumeConfiguration.java d66f6d1
> flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java 50b9f0c
> flume-ng-node/src/test/java/org/apache/flume/node/TestAbstractLogicalNodeManager.java 521b586
> flume-ng-node/src/test/java/org/apache/flume/source/TestNetcatSource.java fb2a960
> flume-ng-node/src/test/resources/flume-conf.properties 848caca
> flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java eee9221
> flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 6ff3737
> flume-ng-sinks/flume-irc-sink/src/main/java/org/apache/flume/sink/irc/IRCSink.java 8e77218
> pom.xml c91222f
>
> Diff: https://reviews.apache.org/r/4115/diff
>
>
> Testing
> -------
>
> All existing unit tests for the components whose configuration has now been moved to these stubs pass.
>
>
> Thanks,
>
> Hari
>
>
Re: Review Request: Component aware configuration managers for flume
components
Posted by Hari Shreedharan <hs...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4115/
-----------------------------------------------------------
(Updated 2012-03-27 01:05:49.485603)
Review request for Flume.
Changes
-------
<<---No longer for review - for keeping track of diffs only. I am splitting this up into smaller, easier to review patches. --->>
Summary
-------
Currently the configuration of each component sits deep inside the component themselves. There is no way to verify if a configuration is valid before run time. In most systems like Flume, it is likely that these confs will be deployed from one single host, on to the machines where flume agents are running. Only when each agent starts, invalid confs are identified because the Agents would terminate by throwing exceptions. This is a first attempt to make a component-aware configuration system which is independent of the Flume, and does not require the Flume jar to be installed. Each component needs a configuration manager, which configures the components.
I have not completed the validation part yet, but have completed the configuration managers for all the components that come in the flume jar. I will add support for the other components, such as HDFS sink, JDBC channel etc soon. I will also add the validation support to the classes, but would like a review of the model being proposed here.
Note that I do know of a couple of bugs in the existing diff(like that if the name of a config key starts with an Upper case letter, an exception is thrown) - I will fix and upload the review here soon.
This addresses bug FLUME-992.
https://issues.apache.org/jira/browse/FLUME-992
Diffs (updated)
-----
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannel.java bca0c50
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProvider.java e445d61
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProviderFactory.java 6fbd6ef
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java 307ae89
flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestJdbcChannelProvider.java 7710d46
flume-ng-clients/flume-ng-log4jappender/src/test/java/org/apache/flume/clients/log4jappender/TestLog4jAppender.java 68d95fb
flume-ng-configuration/pom.xml PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfigurationFactory.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ConfigurationException.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/Context.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationError.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationErrorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/JdbcChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MemoryChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MultiplexingChannelSelectorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/PseudoTxnMemoryChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/AvroSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/FailoverSinkProcessorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/HDFSSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/RollingFileSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkGroupConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkProcessorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkProcessorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/AvroSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/ExecSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/NetcatSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SyslogTcpSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SyslogUDPSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/test/resources/flume-ng-error-test PRE-CREATION
flume-ng-configuration/src/test/resources/flume-ng-test.conf PRE-CREATION
flume-ng-core/pom.xml 37fb112
flume-ng-core/src/main/java/org/apache/flume/ChannelSelector.java fba2dcb
flume-ng-core/src/main/java/org/apache/flume/Context.java 5294e31
flume-ng-core/src/main/java/org/apache/flume/SinkProcessor.java d863ed0
flume-ng-core/src/main/java/org/apache/flume/SinkProcessorType.java be1891b
flume-ng-core/src/main/java/org/apache/flume/channel/AbstractChannel.java 352bf08
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorFactory.java 800f471
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorType.java 511fc65
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8
flume-ng-core/src/main/java/org/apache/flume/channel/DefaultChannelFactory.java 963a6a3
flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java bfa1fde
flume-ng-core/src/main/java/org/apache/flume/channel/MultiplexingChannelSelector.java 83928b7
flume-ng-core/src/main/java/org/apache/flume/channel/PseudoTxnMemoryChannel.java 489d3e5
flume-ng-core/src/main/java/org/apache/flume/channel/ReplicatingChannelSelector.java 8f22746
flume-ng-core/src/main/java/org/apache/flume/conf/Configurable.java 1d69112
flume-ng-core/src/main/java/org/apache/flume/conf/ConfigurableComponent.java PRE-CREATION
flume-ng-core/src/main/java/org/apache/flume/conf/Configurables.java 84492e5
flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 3edc563
flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkFactory.java b89dfa0
flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkProcessor.java 257bab3
flume-ng-core/src/main/java/org/apache/flume/sink/FailoverSinkProcessor.java 066ee37
flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java 7f1d3b3
flume-ng-core/src/main/java/org/apache/flume/sink/SinkGroup.java 0dffd69
flume-ng-core/src/main/java/org/apache/flume/sink/SinkProcessorFactory.java 6160a17
flume-ng-core/src/main/java/org/apache/flume/sink/SinkType.java 6b08c09
flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java 903889d
flume-ng-core/src/main/java/org/apache/flume/source/DefaultSourceFactory.java a610e6f
flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java dbf79e0
flume-ng-core/src/main/java/org/apache/flume/source/ExecSourceConfigurationConstants.java 73c985e
flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java a841b0e
flume-ng-core/src/main/java/org/apache/flume/source/SequenceGeneratorSource.java 440c5a9
flume-ng-core/src/main/java/org/apache/flume/source/SourceType.java 9082470
flume-ng-core/src/main/java/org/apache/flume/source/SyslogTcpSource.java d78d27f
flume-ng-core/src/main/java/org/apache/flume/source/SyslogUDPSource.java 34818f0
flume-ng-core/src/test/java/org/apache/flume/TestContext.java 51c350f
flume-ng-core/src/test/java/org/apache/flume/channel/AbstractBasicChannelSemanticsTest.java 6e71e46
flume-ng-core/src/test/java/org/apache/flume/channel/MockChannel.java 24b01e2
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannel.java e070864
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelConcurrency.java 8dad0b2
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelTransaction.java bc81f26
flume-ng-core/src/test/java/org/apache/flume/sink/TestAvroSink.java 3765924
flume-ng-core/src/test/java/org/apache/flume/sink/TestFailoverSinkProcessor.java 3358cf4
flume-ng-core/src/test/java/org/apache/flume/sink/TestLoggerSink.java 92ff6fe
flume-ng-core/src/test/java/org/apache/flume/sink/TestRollingFileSink.java 7e26e2a
flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java c5c3f2f
flume-ng-core/src/test/java/org/apache/flume/source/TestExecSource.java 615f2a3
flume-ng-core/src/test/java/org/apache/flume/source/TestPollableSourceRunner.java c27f82c
flume-ng-core/src/test/java/org/apache/flume/source/TestSequenceGeneratorSource.java 579b257
flume-ng-core/src/test/java/org/apache/flume/source/TestSyslogUdpSource.java d1db49d
flume-ng-legacy-sources/flume-avro-source/src/main/java/org/apache/flume/source/avroLegacy/AvroLegacySource.java dde8f28
flume-ng-legacy-sources/flume-avro-source/src/test/java/org/apache/flume/source/avroLegacy/TestLegacyAvroSource.java 6e3eb53
flume-ng-legacy-sources/flume-thrift-source/src/main/java/org/apache/flume/source/thriftLegacy/ThriftLegacySource.java 00df7e4
flume-ng-legacy-sources/flume-thrift-source/src/test/java/org/apache/flume/source/thriftLegacy/TestThriftLegacySource.java ddd9478
flume-ng-node/src/main/java/org/apache/flume/conf/properties/FlumeConfiguration.java d66f6d1
flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java 50b9f0c
flume-ng-node/src/test/java/org/apache/flume/node/TestAbstractLogicalNodeManager.java 521b586
flume-ng-node/src/test/java/org/apache/flume/source/TestNetcatSource.java fb2a960
flume-ng-node/src/test/resources/flume-conf.properties 848caca
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java eee9221
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 6ff3737
flume-ng-sinks/flume-irc-sink/src/main/java/org/apache/flume/sink/irc/IRCSink.java 8e77218
pom.xml c91222f
Diff: https://reviews.apache.org/r/4115/diff
Testing
-------
All existing unit tests for the components whose configuration has now been moved to these stubs pass.
Thanks,
Hari
Re: Review Request: Component aware configuration managers for flume
components
Posted by Hari Shreedharan <hs...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4115/
-----------------------------------------------------------
(Updated 2012-03-26 22:50:32.784108)
Review request for Flume.
Changes
-------
missed a couple of files.
Summary
-------
Currently the configuration of each component sits deep inside the component themselves. There is no way to verify if a configuration is valid before run time. In most systems like Flume, it is likely that these confs will be deployed from one single host, on to the machines where flume agents are running. Only when each agent starts, invalid confs are identified because the Agents would terminate by throwing exceptions. This is a first attempt to make a component-aware configuration system which is independent of the Flume, and does not require the Flume jar to be installed. Each component needs a configuration manager, which configures the components.
I have not completed the validation part yet, but have completed the configuration managers for all the components that come in the flume jar. I will add support for the other components, such as HDFS sink, JDBC channel etc soon. I will also add the validation support to the classes, but would like a review of the model being proposed here.
Note that I do know of a couple of bugs in the existing diff(like that if the name of a config key starts with an Upper case letter, an exception is thrown) - I will fix and upload the review here soon.
This addresses bug FLUME-992.
https://issues.apache.org/jira/browse/FLUME-992
Diffs (updated)
-----
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannel.java bca0c50
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProvider.java e445d61
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProviderFactory.java 6fbd6ef
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java 307ae89
flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestJdbcChannelProvider.java 7710d46
flume-ng-clients/flume-ng-log4jappender/src/test/java/org/apache/flume/clients/log4jappender/TestLog4jAppender.java 68d95fb
flume-ng-configuration/pom.xml PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfigurationFactory.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ConfigurationException.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/Context.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationError.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationErrorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/JdbcChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MemoryChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MultiplexingChannelSelectorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/PseudoTxnMemoryChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/AvroSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/FailoverSinkProcessorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/HDFSSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/RollingFileSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkGroupConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkProcessorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkProcessorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/AvroSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/ExecSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/NetcatSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SyslogTcpSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SyslogUDPSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/test/resources/flume-ng-error-test PRE-CREATION
flume-ng-configuration/src/test/resources/flume-ng-test.conf PRE-CREATION
flume-ng-core/pom.xml 37fb112
flume-ng-core/src/main/java/org/apache/flume/ChannelSelector.java fba2dcb
flume-ng-core/src/main/java/org/apache/flume/Context.java 5294e31
flume-ng-core/src/main/java/org/apache/flume/SinkProcessor.java d863ed0
flume-ng-core/src/main/java/org/apache/flume/SinkProcessorType.java be1891b
flume-ng-core/src/main/java/org/apache/flume/channel/AbstractChannel.java 352bf08
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorFactory.java 800f471
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorType.java 511fc65
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8
flume-ng-core/src/main/java/org/apache/flume/channel/DefaultChannelFactory.java 963a6a3
flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java bfa1fde
flume-ng-core/src/main/java/org/apache/flume/channel/MultiplexingChannelSelector.java 83928b7
flume-ng-core/src/main/java/org/apache/flume/channel/PseudoTxnMemoryChannel.java 489d3e5
flume-ng-core/src/main/java/org/apache/flume/channel/ReplicatingChannelSelector.java 8f22746
flume-ng-core/src/main/java/org/apache/flume/conf/Configurable.java 1d69112
flume-ng-core/src/main/java/org/apache/flume/conf/ConfigurableComponent.java PRE-CREATION
flume-ng-core/src/main/java/org/apache/flume/conf/Configurables.java 84492e5
flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 3edc563
flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkFactory.java b89dfa0
flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkProcessor.java 257bab3
flume-ng-core/src/main/java/org/apache/flume/sink/FailoverSinkProcessor.java 066ee37
flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java 7f1d3b3
flume-ng-core/src/main/java/org/apache/flume/sink/SinkGroup.java 0dffd69
flume-ng-core/src/main/java/org/apache/flume/sink/SinkProcessorFactory.java 6160a17
flume-ng-core/src/main/java/org/apache/flume/sink/SinkType.java 6b08c09
flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java 903889d
flume-ng-core/src/main/java/org/apache/flume/source/DefaultSourceFactory.java a610e6f
flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java dbf79e0
flume-ng-core/src/main/java/org/apache/flume/source/ExecSourceConfigurationConstants.java 73c985e
flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java a841b0e
flume-ng-core/src/main/java/org/apache/flume/source/SequenceGeneratorSource.java 440c5a9
flume-ng-core/src/main/java/org/apache/flume/source/SourceType.java 9082470
flume-ng-core/src/main/java/org/apache/flume/source/SyslogTcpSource.java d78d27f
flume-ng-core/src/main/java/org/apache/flume/source/SyslogUDPSource.java 34818f0
flume-ng-core/src/test/java/org/apache/flume/TestContext.java 51c350f
flume-ng-core/src/test/java/org/apache/flume/channel/AbstractBasicChannelSemanticsTest.java 6e71e46
flume-ng-core/src/test/java/org/apache/flume/channel/MockChannel.java 24b01e2
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannel.java e070864
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelConcurrency.java 8dad0b2
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelTransaction.java bc81f26
flume-ng-core/src/test/java/org/apache/flume/sink/TestAvroSink.java 3765924
flume-ng-core/src/test/java/org/apache/flume/sink/TestFailoverSinkProcessor.java 3358cf4
flume-ng-core/src/test/java/org/apache/flume/sink/TestLoggerSink.java 92ff6fe
flume-ng-core/src/test/java/org/apache/flume/sink/TestRollingFileSink.java 7e26e2a
flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java c5c3f2f
flume-ng-core/src/test/java/org/apache/flume/source/TestExecSource.java 615f2a3
flume-ng-core/src/test/java/org/apache/flume/source/TestPollableSourceRunner.java c27f82c
flume-ng-core/src/test/java/org/apache/flume/source/TestSequenceGeneratorSource.java 579b257
flume-ng-core/src/test/java/org/apache/flume/source/TestSyslogUdpSource.java d1db49d
flume-ng-legacy-sources/flume-avro-source/src/main/java/org/apache/flume/source/avroLegacy/AvroLegacySource.java dde8f28
flume-ng-legacy-sources/flume-avro-source/src/test/java/org/apache/flume/source/avroLegacy/TestLegacyAvroSource.java 6e3eb53
flume-ng-legacy-sources/flume-thrift-source/src/main/java/org/apache/flume/source/thriftLegacy/ThriftLegacySource.java 00df7e4
flume-ng-legacy-sources/flume-thrift-source/src/test/java/org/apache/flume/source/thriftLegacy/TestThriftLegacySource.java ddd9478
flume-ng-node/src/main/java/org/apache/flume/conf/properties/FlumeConfiguration.java d66f6d1
flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java 50b9f0c
flume-ng-node/src/test/java/org/apache/flume/node/TestAbstractLogicalNodeManager.java 521b586
flume-ng-node/src/test/java/org/apache/flume/source/TestNetcatSource.java fb2a960
flume-ng-node/src/test/resources/flume-conf.properties 848caca
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java eee9221
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 6ff3737
flume-ng-sinks/flume-irc-sink/src/main/java/org/apache/flume/sink/irc/IRCSink.java 8e77218
pom.xml c91222f
Diff: https://reviews.apache.org/r/4115/diff
Testing
-------
All existing unit tests for the components whose configuration has now been moved to these stubs pass.
Thanks,
Hari
Re: Review Request: Component aware configuration managers for flume
components
Posted by Hari Shreedharan <hs...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4115/
-----------------------------------------------------------
(Updated 2012-03-26 07:34:11.739683)
Review request for Flume.
Changes
-------
Removed most whitespace changes. I will split this up soon, for now, just trying to make sure all whitespaces changes were reverted.
Summary
-------
Currently the configuration of each component sits deep inside the component themselves. There is no way to verify if a configuration is valid before run time. In most systems like Flume, it is likely that these confs will be deployed from one single host, on to the machines where flume agents are running. Only when each agent starts, invalid confs are identified because the Agents would terminate by throwing exceptions. This is a first attempt to make a component-aware configuration system which is independent of the Flume, and does not require the Flume jar to be installed. Each component needs a configuration manager, which configures the components.
I have not completed the validation part yet, but have completed the configuration managers for all the components that come in the flume jar. I will add support for the other components, such as HDFS sink, JDBC channel etc soon. I will also add the validation support to the classes, but would like a review of the model being proposed here.
Note that I do know of a couple of bugs in the existing diff(like that if the name of a config key starts with an Upper case letter, an exception is thrown) - I will fix and upload the review here soon.
This addresses bug FLUME-992.
https://issues.apache.org/jira/browse/FLUME-992
Diffs (updated)
-----
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannel.java bca0c50
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProvider.java e445d61
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProviderFactory.java 6fbd6ef
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java 307ae89
flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestJdbcChannelProvider.java 7710d46
flume-ng-clients/flume-ng-log4jappender/src/test/java/org/apache/flume/clients/log4jappender/TestLog4jAppender.java 68d95fb
flume-ng-configuration/pom.xml PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfigurationFactory.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ConfigurationException.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/Context.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationError.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationErrorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/JdbcChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MemoryChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MultiplexingChannelSelectorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/PseudoTxnMemoryChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/AvroSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/FailoverSinkProcessorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/HDFSSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/RollingFileSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkGroupConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkProcessorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkProcessorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/AvroSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/ExecSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/ExecSourceConfigurationConstants.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/NetcatSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceType.java PRE-CREATION
flume-ng-configuration/src/test/resources/flume-ng-error-test PRE-CREATION
flume-ng-configuration/src/test/resources/flume-ng-test.conf PRE-CREATION
flume-ng-core/pom.xml 37fb112
flume-ng-core/src/main/java/org/apache/flume/ChannelSelector.java fba2dcb
flume-ng-core/src/main/java/org/apache/flume/Context.java 5294e31
flume-ng-core/src/main/java/org/apache/flume/SinkProcessor.java d863ed0
flume-ng-core/src/main/java/org/apache/flume/SinkProcessorType.java be1891b
flume-ng-core/src/main/java/org/apache/flume/channel/AbstractChannel.java 352bf08
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorFactory.java 800f471
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorType.java 511fc65
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8
flume-ng-core/src/main/java/org/apache/flume/channel/DefaultChannelFactory.java 963a6a3
flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java bfa1fde
flume-ng-core/src/main/java/org/apache/flume/channel/MultiplexingChannelSelector.java 83928b7
flume-ng-core/src/main/java/org/apache/flume/channel/PseudoTxnMemoryChannel.java 489d3e5
flume-ng-core/src/main/java/org/apache/flume/channel/ReplicatingChannelSelector.java 8f22746
flume-ng-core/src/main/java/org/apache/flume/conf/Configurable.java 1d69112
flume-ng-core/src/main/java/org/apache/flume/conf/ConfigurableComponent.java PRE-CREATION
flume-ng-core/src/main/java/org/apache/flume/conf/Configurables.java 84492e5
flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 3edc563
flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkFactory.java b89dfa0
flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkProcessor.java 257bab3
flume-ng-core/src/main/java/org/apache/flume/sink/FailoverSinkProcessor.java 066ee37
flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java 7f1d3b3
flume-ng-core/src/main/java/org/apache/flume/sink/SinkGroup.java 0dffd69
flume-ng-core/src/main/java/org/apache/flume/sink/SinkProcessorFactory.java 6160a17
flume-ng-core/src/main/java/org/apache/flume/sink/SinkType.java 6b08c09
flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java 903889d
flume-ng-core/src/main/java/org/apache/flume/source/DefaultSourceFactory.java a610e6f
flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java dbf79e0
flume-ng-core/src/main/java/org/apache/flume/source/ExecSourceConfigurationConstants.java 73c985e
flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java a841b0e
flume-ng-core/src/main/java/org/apache/flume/source/SequenceGeneratorSource.java 440c5a9
flume-ng-core/src/main/java/org/apache/flume/source/SourceType.java 9082470
flume-ng-core/src/main/java/org/apache/flume/source/SyslogTcpSource.java d78d27f
flume-ng-core/src/main/java/org/apache/flume/source/SyslogUDPSource.java 34818f0
flume-ng-core/src/test/java/org/apache/flume/TestContext.java 51c350f
flume-ng-core/src/test/java/org/apache/flume/channel/AbstractBasicChannelSemanticsTest.java 6e71e46
flume-ng-core/src/test/java/org/apache/flume/channel/MockChannel.java 24b01e2
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannel.java e070864
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelConcurrency.java 8dad0b2
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelTransaction.java bc81f26
flume-ng-core/src/test/java/org/apache/flume/sink/TestAvroSink.java 3765924
flume-ng-core/src/test/java/org/apache/flume/sink/TestFailoverSinkProcessor.java 3358cf4
flume-ng-core/src/test/java/org/apache/flume/sink/TestLoggerSink.java 92ff6fe
flume-ng-core/src/test/java/org/apache/flume/sink/TestRollingFileSink.java 7e26e2a
flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java c5c3f2f
flume-ng-core/src/test/java/org/apache/flume/source/TestExecSource.java 615f2a3
flume-ng-core/src/test/java/org/apache/flume/source/TestPollableSourceRunner.java c27f82c
flume-ng-core/src/test/java/org/apache/flume/source/TestSequenceGeneratorSource.java 579b257
flume-ng-core/src/test/java/org/apache/flume/source/TestSyslogUdpSource.java d1db49d
flume-ng-legacy-sources/flume-avro-source/src/main/java/org/apache/flume/source/avroLegacy/AvroLegacySource.java dde8f28
flume-ng-legacy-sources/flume-avro-source/src/test/java/org/apache/flume/source/avroLegacy/TestLegacyAvroSource.java 6e3eb53
flume-ng-legacy-sources/flume-thrift-source/src/main/java/org/apache/flume/source/thriftLegacy/ThriftLegacySource.java 00df7e4
flume-ng-legacy-sources/flume-thrift-source/src/test/java/org/apache/flume/source/thriftLegacy/TestThriftLegacySource.java ddd9478
flume-ng-node/src/main/java/org/apache/flume/conf/properties/FlumeConfiguration.java d66f6d1
flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java 50b9f0c
flume-ng-node/src/test/java/org/apache/flume/node/TestAbstractLogicalNodeManager.java 521b586
flume-ng-node/src/test/java/org/apache/flume/source/TestNetcatSource.java fb2a960
flume-ng-node/src/test/resources/flume-conf.properties 848caca
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 203886e
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 6ff3737
flume-ng-sinks/flume-irc-sink/src/main/java/org/apache/flume/sink/irc/IRCSink.java 8e77218
pom.xml 19798b2
Diff: https://reviews.apache.org/r/4115/diff
Testing
-------
All existing unit tests for the components whose configuration has now been moved to these stubs pass.
Thanks,
Hari
Re: Review Request: Component aware configuration managers for flume
components
Posted by Arvind Prabhakar <ar...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4115/#review6199
-----------------------------------------------------------
Thanks for the patch Hari. The patch needs to be rebased on the latest sources and seems to have accidentally overlaid a lot of older code on top of the latest. For example, all the pom.xml files have their versions overwritten to 1.1.0-incubating. Not only the patch is very big, it actually has substantial formatting changes which are making it very hard to keep an eye on the actual change.
Therefore, for the benefit of the reviewers and for minimizing the risk, I suggest you update the patch to only have the configuration subsystem changes. This means that all components will get configured like custom components that do not have stubs exposed. Once we get that part reviewed and checked in, we can revive the rest of the changes component by component and get them committed.
I understand that you have gone through many painstaking revisions of the patch so far, and if you you so prefer - I will be happy to help out in breaking this patch into phased changes.
- Arvind
On 2012-03-19 23:01:52, Hari Shreedharan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/4115/
> -----------------------------------------------------------
>
> (Updated 2012-03-19 23:01:52)
>
>
> Review request for Flume.
>
>
> Summary
> -------
>
> Currently the configuration of each component sits deep inside the component themselves. There is no way to verify if a configuration is valid before run time. In most systems like Flume, it is likely that these confs will be deployed from one single host, on to the machines where flume agents are running. Only when each agent starts, invalid confs are identified because the Agents would terminate by throwing exceptions. This is a first attempt to make a component-aware configuration system which is independent of the Flume, and does not require the Flume jar to be installed. Each component needs a configuration manager, which configures the components.
>
> I have not completed the validation part yet, but have completed the configuration managers for all the components that come in the flume jar. I will add support for the other components, such as HDFS sink, JDBC channel etc soon. I will also add the validation support to the classes, but would like a review of the model being proposed here.
>
> Note that I do know of a couple of bugs in the existing diff(like that if the name of a config key starts with an Upper case letter, an exception is thrown) - I will fix and upload the review here soon.
>
>
> This addresses bug FLUME-992.
> https://issues.apache.org/jira/browse/FLUME-992
>
>
> Diffs
> -----
>
> flume-ng-channels/flume-file-channel/pom.xml 48d1481
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453
> flume-ng-channels/flume-jdbc-channel/pom.xml 8e82583
> flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannel.java bca0c50
> flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProvider.java e445d61
> flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProviderFactory.java 6fbd6ef
> flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java 307ae89
> flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestJdbcChannelProvider.java 7710d46
> flume-ng-channels/pom.xml 0bd8633
> flume-ng-clients/flume-ng-log4jappender/pom.xml d1c443d
> flume-ng-clients/flume-ng-log4jappender/src/test/java/org/apache/flume/clients/log4jappender/TestLog4jAppender.java 68d95fb
> flume-ng-clients/pom.xml 0a0dc04
> flume-ng-configuration/pom.xml PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfigurationFactory.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/ConfigurationException.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/Context.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationError.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationErrorType.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorType.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelType.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/JdbcChannelConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MemoryChannelConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MultiplexingChannelSelectorConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/PseudoTxnMemoryChannelConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/AvroSinkConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/FailoverSinkProcessorConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/HDFSSinkConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/RollingFileSinkConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkGroupConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkProcessorConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkProcessorType.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkType.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/source/AvroSourceConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/source/ExecSourceConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/source/NetcatSourceConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceType.java PRE-CREATION
> flume-ng-configuration/src/test/resources/flume-ng-error-test PRE-CREATION
> flume-ng-configuration/src/test/resources/flume-ng-test.conf PRE-CREATION
> flume-ng-core/pom.xml 37fb112
> flume-ng-core/src/main/java/org/apache/flume/ChannelSelector.java fba2dcb
> flume-ng-core/src/main/java/org/apache/flume/Context.java 5294e31
> flume-ng-core/src/main/java/org/apache/flume/SinkProcessor.java ea6000b
> flume-ng-core/src/main/java/org/apache/flume/SinkProcessorType.java be1891b
> flume-ng-core/src/main/java/org/apache/flume/channel/AbstractChannel.java 352bf08
> flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorFactory.java 800f471
> flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorType.java 511fc65
> flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8
> flume-ng-core/src/main/java/org/apache/flume/channel/DefaultChannelFactory.java 963a6a3
> flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java bfa1fde
> flume-ng-core/src/main/java/org/apache/flume/channel/MultiplexingChannelSelector.java 83928b7
> flume-ng-core/src/main/java/org/apache/flume/channel/PseudoTxnMemoryChannel.java 489d3e5
> flume-ng-core/src/main/java/org/apache/flume/channel/ReplicatingChannelSelector.java 8f22746
> flume-ng-core/src/main/java/org/apache/flume/conf/Configurable.java 0fa4839
> flume-ng-core/src/main/java/org/apache/flume/conf/ConfigurableComponent.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/conf/Configurables.java 84492e5
> flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 3edc563
> flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkFactory.java b89dfa0
> flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkProcessor.java 257bab3
> flume-ng-core/src/main/java/org/apache/flume/sink/FailoverSinkProcessor.java 9f5b856
> flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java 7f1d3b3
> flume-ng-core/src/main/java/org/apache/flume/sink/SinkGroup.java 0dffd69
> flume-ng-core/src/main/java/org/apache/flume/sink/SinkProcessorFactory.java 6160a17
> flume-ng-core/src/main/java/org/apache/flume/sink/SinkType.java 6b08c09
> flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java 903889d
> flume-ng-core/src/main/java/org/apache/flume/source/DefaultSourceFactory.java a610e6f
> flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java b6b1181
> flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java 94245ac
> flume-ng-core/src/main/java/org/apache/flume/source/SequenceGeneratorSource.java 440c5a9
> flume-ng-core/src/main/java/org/apache/flume/source/SourceType.java 9082470
> flume-ng-core/src/main/java/org/apache/flume/source/SyslogTcpSource.java d78d27f
> flume-ng-core/src/main/java/org/apache/flume/source/SyslogUDPSource.java 34818f0
> flume-ng-core/src/test/java/org/apache/flume/TestContext.java 51c350f
> flume-ng-core/src/test/java/org/apache/flume/channel/AbstractBasicChannelSemanticsTest.java 6e71e46
> flume-ng-core/src/test/java/org/apache/flume/channel/MockChannel.java 24b01e2
> flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannel.java e070864
> flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelConcurrency.java 8dad0b2
> flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelTransaction.java bc81f26
> flume-ng-core/src/test/java/org/apache/flume/sink/TestAvroSink.java 3765924
> flume-ng-core/src/test/java/org/apache/flume/sink/TestFailoverSinkProcessor.java 195c121
> flume-ng-core/src/test/java/org/apache/flume/sink/TestLoggerSink.java 92ff6fe
> flume-ng-core/src/test/java/org/apache/flume/sink/TestRollingFileSink.java 7e26e2a
> flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java c5c3f2f
> flume-ng-core/src/test/java/org/apache/flume/source/TestExecSource.java 6035270
> flume-ng-core/src/test/java/org/apache/flume/source/TestPollableSourceRunner.java c27f82c
> flume-ng-core/src/test/java/org/apache/flume/source/TestSequenceGeneratorSource.java 579b257
> flume-ng-core/src/test/java/org/apache/flume/source/TestSyslogUdpSource.java d1db49d
> flume-ng-dist/pom.xml f7e0dd9
> flume-ng-legacy-sources/flume-avro-source/pom.xml 216e479
> flume-ng-legacy-sources/flume-avro-source/src/main/java/org/apache/flume/source/avroLegacy/AvroLegacySource.java dde8f28
> flume-ng-legacy-sources/flume-avro-source/src/test/java/org/apache/flume/source/avroLegacy/TestLegacyAvroSource.java 6e3eb53
> flume-ng-legacy-sources/flume-thrift-source/pom.xml 70b67a6
> flume-ng-legacy-sources/flume-thrift-source/src/main/java/org/apache/flume/source/thriftLegacy/ThriftLegacySource.java fbf7362
> flume-ng-legacy-sources/flume-thrift-source/src/test/java/org/apache/flume/source/thriftLegacy/TestThriftLegacySource.java ddd9478
> flume-ng-legacy-sources/pom.xml 2e5b8dc
> flume-ng-node/pom.xml ffa89fa
> flume-ng-node/src/main/java/org/apache/flume/conf/properties/FlumeConfiguration.java d66f6d1
> flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java 97f72e1
> flume-ng-node/src/test/java/org/apache/flume/node/TestAbstractLogicalNodeManager.java 521b586
> flume-ng-node/src/test/java/org/apache/flume/source/TestNetcatSource.java fb2a960
> flume-ng-node/src/test/resources/flume-conf.properties 848caca
> flume-ng-sdk/pom.xml d1c6adf
> flume-ng-sinks/flume-hdfs-sink/pom.xml 76d69a3
> flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java da82f7e
> flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 8fa72a1
> flume-ng-sinks/flume-irc-sink/pom.xml d935faf
> flume-ng-sinks/flume-irc-sink/src/main/java/org/apache/flume/sink/irc/IRCSink.java 8e77218
> flume-ng-sinks/pom.xml acb3087
> pom.xml cf10727
>
> Diff: https://reviews.apache.org/r/4115/diff
>
>
> Testing
> -------
>
> All existing unit tests for the components whose configuration has now been moved to these stubs pass.
>
>
> Thanks,
>
> Hari
>
>
Re: Review Request: Component aware configuration managers for flume
components
Posted by Hari Shreedharan <hs...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4115/
-----------------------------------------------------------
(Updated 2012-03-19 23:01:52.087904)
Review request for Flume.
Changes
-------
bug fixes
Summary
-------
Currently the configuration of each component sits deep inside the component themselves. There is no way to verify if a configuration is valid before run time. In most systems like Flume, it is likely that these confs will be deployed from one single host, on to the machines where flume agents are running. Only when each agent starts, invalid confs are identified because the Agents would terminate by throwing exceptions. This is a first attempt to make a component-aware configuration system which is independent of the Flume, and does not require the Flume jar to be installed. Each component needs a configuration manager, which configures the components.
I have not completed the validation part yet, but have completed the configuration managers for all the components that come in the flume jar. I will add support for the other components, such as HDFS sink, JDBC channel etc soon. I will also add the validation support to the classes, but would like a review of the model being proposed here.
Note that I do know of a couple of bugs in the existing diff(like that if the name of a config key starts with an Upper case letter, an exception is thrown) - I will fix and upload the review here soon.
This addresses bug FLUME-992.
https://issues.apache.org/jira/browse/FLUME-992
Diffs (updated)
-----
flume-ng-channels/flume-file-channel/pom.xml 48d1481
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453
flume-ng-channels/flume-jdbc-channel/pom.xml 8e82583
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannel.java bca0c50
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProvider.java e445d61
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProviderFactory.java 6fbd6ef
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java 307ae89
flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestJdbcChannelProvider.java 7710d46
flume-ng-channels/pom.xml 0bd8633
flume-ng-clients/flume-ng-log4jappender/pom.xml d1c443d
flume-ng-clients/flume-ng-log4jappender/src/test/java/org/apache/flume/clients/log4jappender/TestLog4jAppender.java 68d95fb
flume-ng-clients/pom.xml 0a0dc04
flume-ng-configuration/pom.xml PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfigurationFactory.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ConfigurationException.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/Context.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationError.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationErrorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/JdbcChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MemoryChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MultiplexingChannelSelectorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/PseudoTxnMemoryChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/AvroSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/FailoverSinkProcessorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/HDFSSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/RollingFileSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkGroupConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkProcessorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkProcessorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/AvroSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/ExecSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/NetcatSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceType.java PRE-CREATION
flume-ng-configuration/src/test/resources/flume-ng-error-test PRE-CREATION
flume-ng-configuration/src/test/resources/flume-ng-test.conf PRE-CREATION
flume-ng-core/pom.xml 37fb112
flume-ng-core/src/main/java/org/apache/flume/ChannelSelector.java fba2dcb
flume-ng-core/src/main/java/org/apache/flume/Context.java 5294e31
flume-ng-core/src/main/java/org/apache/flume/SinkProcessor.java ea6000b
flume-ng-core/src/main/java/org/apache/flume/SinkProcessorType.java be1891b
flume-ng-core/src/main/java/org/apache/flume/channel/AbstractChannel.java 352bf08
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorFactory.java 800f471
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorType.java 511fc65
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8
flume-ng-core/src/main/java/org/apache/flume/channel/DefaultChannelFactory.java 963a6a3
flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java bfa1fde
flume-ng-core/src/main/java/org/apache/flume/channel/MultiplexingChannelSelector.java 83928b7
flume-ng-core/src/main/java/org/apache/flume/channel/PseudoTxnMemoryChannel.java 489d3e5
flume-ng-core/src/main/java/org/apache/flume/channel/ReplicatingChannelSelector.java 8f22746
flume-ng-core/src/main/java/org/apache/flume/conf/Configurable.java 0fa4839
flume-ng-core/src/main/java/org/apache/flume/conf/ConfigurableComponent.java PRE-CREATION
flume-ng-core/src/main/java/org/apache/flume/conf/Configurables.java 84492e5
flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 3edc563
flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkFactory.java b89dfa0
flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkProcessor.java 257bab3
flume-ng-core/src/main/java/org/apache/flume/sink/FailoverSinkProcessor.java 9f5b856
flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java 7f1d3b3
flume-ng-core/src/main/java/org/apache/flume/sink/SinkGroup.java 0dffd69
flume-ng-core/src/main/java/org/apache/flume/sink/SinkProcessorFactory.java 6160a17
flume-ng-core/src/main/java/org/apache/flume/sink/SinkType.java 6b08c09
flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java 903889d
flume-ng-core/src/main/java/org/apache/flume/source/DefaultSourceFactory.java a610e6f
flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java b6b1181
flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java 94245ac
flume-ng-core/src/main/java/org/apache/flume/source/SequenceGeneratorSource.java 440c5a9
flume-ng-core/src/main/java/org/apache/flume/source/SourceType.java 9082470
flume-ng-core/src/main/java/org/apache/flume/source/SyslogTcpSource.java d78d27f
flume-ng-core/src/main/java/org/apache/flume/source/SyslogUDPSource.java 34818f0
flume-ng-core/src/test/java/org/apache/flume/TestContext.java 51c350f
flume-ng-core/src/test/java/org/apache/flume/channel/AbstractBasicChannelSemanticsTest.java 6e71e46
flume-ng-core/src/test/java/org/apache/flume/channel/MockChannel.java 24b01e2
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannel.java e070864
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelConcurrency.java 8dad0b2
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelTransaction.java bc81f26
flume-ng-core/src/test/java/org/apache/flume/sink/TestAvroSink.java 3765924
flume-ng-core/src/test/java/org/apache/flume/sink/TestFailoverSinkProcessor.java 195c121
flume-ng-core/src/test/java/org/apache/flume/sink/TestLoggerSink.java 92ff6fe
flume-ng-core/src/test/java/org/apache/flume/sink/TestRollingFileSink.java 7e26e2a
flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java c5c3f2f
flume-ng-core/src/test/java/org/apache/flume/source/TestExecSource.java 6035270
flume-ng-core/src/test/java/org/apache/flume/source/TestPollableSourceRunner.java c27f82c
flume-ng-core/src/test/java/org/apache/flume/source/TestSequenceGeneratorSource.java 579b257
flume-ng-core/src/test/java/org/apache/flume/source/TestSyslogUdpSource.java d1db49d
flume-ng-dist/pom.xml f7e0dd9
flume-ng-legacy-sources/flume-avro-source/pom.xml 216e479
flume-ng-legacy-sources/flume-avro-source/src/main/java/org/apache/flume/source/avroLegacy/AvroLegacySource.java dde8f28
flume-ng-legacy-sources/flume-avro-source/src/test/java/org/apache/flume/source/avroLegacy/TestLegacyAvroSource.java 6e3eb53
flume-ng-legacy-sources/flume-thrift-source/pom.xml 70b67a6
flume-ng-legacy-sources/flume-thrift-source/src/main/java/org/apache/flume/source/thriftLegacy/ThriftLegacySource.java fbf7362
flume-ng-legacy-sources/flume-thrift-source/src/test/java/org/apache/flume/source/thriftLegacy/TestThriftLegacySource.java ddd9478
flume-ng-legacy-sources/pom.xml 2e5b8dc
flume-ng-node/pom.xml ffa89fa
flume-ng-node/src/main/java/org/apache/flume/conf/properties/FlumeConfiguration.java d66f6d1
flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java 97f72e1
flume-ng-node/src/test/java/org/apache/flume/node/TestAbstractLogicalNodeManager.java 521b586
flume-ng-node/src/test/java/org/apache/flume/source/TestNetcatSource.java fb2a960
flume-ng-node/src/test/resources/flume-conf.properties 848caca
flume-ng-sdk/pom.xml d1c6adf
flume-ng-sinks/flume-hdfs-sink/pom.xml 76d69a3
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java da82f7e
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 8fa72a1
flume-ng-sinks/flume-irc-sink/pom.xml d935faf
flume-ng-sinks/flume-irc-sink/src/main/java/org/apache/flume/sink/irc/IRCSink.java 8e77218
flume-ng-sinks/pom.xml acb3087
pom.xml cf10727
Diff: https://reviews.apache.org/r/4115/diff
Testing
-------
All existing unit tests for the components whose configuration has now been moved to these stubs pass.
Thanks,
Hari
Re: Review Request: Component aware configuration managers for flume
components
Posted by Hari Shreedharan <hs...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4115/
-----------------------------------------------------------
(Updated 2012-03-15 07:12:25.261211)
Review request for Flume.
Changes
-------
Fixing a bug which I found.
Summary
-------
Currently the configuration of each component sits deep inside the component themselves. There is no way to verify if a configuration is valid before run time. In most systems like Flume, it is likely that these confs will be deployed from one single host, on to the machines where flume agents are running. Only when each agent starts, invalid confs are identified because the Agents would terminate by throwing exceptions. This is a first attempt to make a component-aware configuration system which is independent of the Flume, and does not require the Flume jar to be installed. Each component needs a configuration manager, which configures the components.
I have not completed the validation part yet, but have completed the configuration managers for all the components that come in the flume jar. I will add support for the other components, such as HDFS sink, JDBC channel etc soon. I will also add the validation support to the classes, but would like a review of the model being proposed here.
Note that I do know of a couple of bugs in the existing diff(like that if the name of a config key starts with an Upper case letter, an exception is thrown) - I will fix and upload the review here soon.
This addresses bug FLUME-992.
https://issues.apache.org/jira/browse/FLUME-992
Diffs (updated)
-----
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannel.java bca0c50
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProvider.java e445d61
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProviderFactory.java 6fbd6ef
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java 307ae89
flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestJdbcChannelProvider.java 7710d46
flume-ng-clients/flume-ng-log4jappender/src/test/java/org/apache/flume/clients/log4jappender/TestLog4jAppender.java 68d95fb
flume-ng-configuration/pom.xml PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfigurationFactory.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ConfigurationException.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/Context.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationError.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationErrorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MemoryChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MultiplexingChannelSelectorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/PseudoTxnMemoryChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/AvroSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/FailoverSinkProcessorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/RollingFileSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkGroupConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkProcessorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkProcessorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/AvroSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/ExecSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/NetcatSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceType.java PRE-CREATION
flume-ng-configuration/src/test/resources/flume-ng-error-test PRE-CREATION
flume-ng-configuration/src/test/resources/flume-ng-test.conf PRE-CREATION
flume-ng-core/pom.xml 247df37
flume-ng-core/src/main/java/org/apache/flume/ChannelSelector.java fba2dcb
flume-ng-core/src/main/java/org/apache/flume/Context.java 5294e31
flume-ng-core/src/main/java/org/apache/flume/SinkProcessor.java ea6000b
flume-ng-core/src/main/java/org/apache/flume/SinkProcessorType.java be1891b
flume-ng-core/src/main/java/org/apache/flume/channel/AbstractChannel.java 352bf08
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorFactory.java 800f471
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorType.java 511fc65
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8
flume-ng-core/src/main/java/org/apache/flume/channel/DefaultChannelFactory.java 963a6a3
flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java bfa1fde
flume-ng-core/src/main/java/org/apache/flume/channel/MultiplexingChannelSelector.java 83928b7
flume-ng-core/src/main/java/org/apache/flume/channel/PseudoTxnMemoryChannel.java 489d3e5
flume-ng-core/src/main/java/org/apache/flume/channel/ReplicatingChannelSelector.java 8f22746
flume-ng-core/src/main/java/org/apache/flume/conf/Configurable.java 0fa4839
flume-ng-core/src/main/java/org/apache/flume/conf/ConfigurableComponent.java PRE-CREATION
flume-ng-core/src/main/java/org/apache/flume/conf/Configurables.java 84492e5
flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 3edc563
flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkFactory.java b89dfa0
flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkProcessor.java 257bab3
flume-ng-core/src/main/java/org/apache/flume/sink/FailoverSinkProcessor.java 9f5b856
flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java 7f1d3b3
flume-ng-core/src/main/java/org/apache/flume/sink/SinkGroup.java 0dffd69
flume-ng-core/src/main/java/org/apache/flume/sink/SinkProcessorFactory.java 6160a17
flume-ng-core/src/main/java/org/apache/flume/sink/SinkType.java 6b08c09
flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java 903889d
flume-ng-core/src/main/java/org/apache/flume/source/DefaultSourceFactory.java a610e6f
flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java b6b1181
flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java 94245ac
flume-ng-core/src/main/java/org/apache/flume/source/SequenceGeneratorSource.java 440c5a9
flume-ng-core/src/main/java/org/apache/flume/source/SourceType.java 9082470
flume-ng-core/src/main/java/org/apache/flume/source/SyslogTcpSource.java d78d27f
flume-ng-core/src/main/java/org/apache/flume/source/SyslogUDPSource.java 34818f0
flume-ng-core/src/test/java/org/apache/flume/TestContext.java 51c350f
flume-ng-core/src/test/java/org/apache/flume/channel/AbstractBasicChannelSemanticsTest.java 6e71e46
flume-ng-core/src/test/java/org/apache/flume/channel/MockChannel.java 24b01e2
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannel.java e070864
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelConcurrency.java 8dad0b2
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelTransaction.java bc81f26
flume-ng-core/src/test/java/org/apache/flume/sink/TestAvroSink.java 3765924
flume-ng-core/src/test/java/org/apache/flume/sink/TestFailoverSinkProcessor.java 195c121
flume-ng-core/src/test/java/org/apache/flume/sink/TestLoggerSink.java 92ff6fe
flume-ng-core/src/test/java/org/apache/flume/sink/TestRollingFileSink.java 7e26e2a
flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java c5c3f2f
flume-ng-core/src/test/java/org/apache/flume/source/TestExecSource.java 6035270
flume-ng-core/src/test/java/org/apache/flume/source/TestPollableSourceRunner.java c27f82c
flume-ng-core/src/test/java/org/apache/flume/source/TestSequenceGeneratorSource.java 579b257
flume-ng-core/src/test/java/org/apache/flume/source/TestSyslogUdpSource.java d1db49d
flume-ng-legacy-sources/flume-avro-source/src/main/java/org/apache/flume/source/avroLegacy/AvroLegacySource.java dde8f28
flume-ng-legacy-sources/flume-avro-source/src/test/java/org/apache/flume/source/avroLegacy/TestLegacyAvroSource.java 6e3eb53
flume-ng-legacy-sources/flume-thrift-source/src/main/java/org/apache/flume/source/thriftLegacy/ThriftLegacySource.java fbf7362
flume-ng-legacy-sources/flume-thrift-source/src/test/java/org/apache/flume/source/thriftLegacy/TestThriftLegacySource.java ddd9478
flume-ng-node/src/main/java/org/apache/flume/conf/properties/FlumeConfiguration.java d66f6d1
flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java 97f72e1
flume-ng-node/src/test/java/org/apache/flume/node/TestAbstractLogicalNodeManager.java 521b586
flume-ng-node/src/test/java/org/apache/flume/source/TestNetcatSource.java fb2a960
flume-ng-node/src/test/resources/flume-conf.properties 848caca
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java da82f7e
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 8fa72a1
flume-ng-sinks/flume-irc-sink/src/main/java/org/apache/flume/sink/irc/IRCSink.java 8e77218
pom.xml 794ebe3
Diff: https://reviews.apache.org/r/4115/diff
Testing
-------
All existing unit tests for the components whose configuration has now been moved to these stubs pass.
Thanks,
Hari
Re: Review Request: Component aware configuration managers for flume
components
Posted by Hari Shreedharan <hs...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4115/
-----------------------------------------------------------
(Updated 2012-03-15 05:07:23.274303)
Review request for Flume.
Changes
-------
Bug fix.
Summary
-------
Currently the configuration of each component sits deep inside the component themselves. There is no way to verify if a configuration is valid before run time. In most systems like Flume, it is likely that these confs will be deployed from one single host, on to the machines where flume agents are running. Only when each agent starts, invalid confs are identified because the Agents would terminate by throwing exceptions. This is a first attempt to make a component-aware configuration system which is independent of the Flume, and does not require the Flume jar to be installed. Each component needs a configuration manager, which configures the components.
I have not completed the validation part yet, but have completed the configuration managers for all the components that come in the flume jar. I will add support for the other components, such as HDFS sink, JDBC channel etc soon. I will also add the validation support to the classes, but would like a review of the model being proposed here.
Note that I do know of a couple of bugs in the existing diff(like that if the name of a config key starts with an Upper case letter, an exception is thrown) - I will fix and upload the review here soon.
This addresses bug FLUME-992.
https://issues.apache.org/jira/browse/FLUME-992
Diffs (updated)
-----
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannel.java bca0c50
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProvider.java e445d61
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProviderFactory.java 6fbd6ef
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java 307ae89
flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestJdbcChannelProvider.java 7710d46
flume-ng-clients/flume-ng-log4jappender/src/test/java/org/apache/flume/clients/log4jappender/TestLog4jAppender.java 68d95fb
flume-ng-configuration/pom.xml PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfigurationFactory.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ConfigurationException.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/Context.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationError.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationErrorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MemoryChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MultiplexingChannelSelectorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/PseudoTxnMemoryChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/AvroSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/FailoverSinkProcessorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/RollingFileSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkGroupConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkProcessorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkProcessorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/AvroSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/ExecSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/NetcatSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceType.java PRE-CREATION
flume-ng-configuration/src/test/resources/flume-ng-error-test PRE-CREATION
flume-ng-configuration/src/test/resources/flume-ng-test.conf PRE-CREATION
flume-ng-core/pom.xml 247df37
flume-ng-core/src/main/java/org/apache/flume/ChannelSelector.java fba2dcb
flume-ng-core/src/main/java/org/apache/flume/Context.java 5294e31
flume-ng-core/src/main/java/org/apache/flume/SinkProcessor.java ea6000b
flume-ng-core/src/main/java/org/apache/flume/SinkProcessorType.java be1891b
flume-ng-core/src/main/java/org/apache/flume/channel/AbstractChannel.java 352bf08
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorFactory.java 800f471
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorType.java 511fc65
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8
flume-ng-core/src/main/java/org/apache/flume/channel/DefaultChannelFactory.java 963a6a3
flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java bfa1fde
flume-ng-core/src/main/java/org/apache/flume/channel/MultiplexingChannelSelector.java 83928b7
flume-ng-core/src/main/java/org/apache/flume/channel/PseudoTxnMemoryChannel.java 489d3e5
flume-ng-core/src/main/java/org/apache/flume/channel/ReplicatingChannelSelector.java 8f22746
flume-ng-core/src/main/java/org/apache/flume/conf/Configurable.java 0fa4839
flume-ng-core/src/main/java/org/apache/flume/conf/ConfigurableComponent.java PRE-CREATION
flume-ng-core/src/main/java/org/apache/flume/conf/Configurables.java 84492e5
flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 3edc563
flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkFactory.java b89dfa0
flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkProcessor.java 257bab3
flume-ng-core/src/main/java/org/apache/flume/sink/FailoverSinkProcessor.java 9f5b856
flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java 7f1d3b3
flume-ng-core/src/main/java/org/apache/flume/sink/SinkGroup.java 0dffd69
flume-ng-core/src/main/java/org/apache/flume/sink/SinkProcessorFactory.java 6160a17
flume-ng-core/src/main/java/org/apache/flume/sink/SinkType.java 6b08c09
flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java 903889d
flume-ng-core/src/main/java/org/apache/flume/source/DefaultSourceFactory.java a610e6f
flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java b6b1181
flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java 94245ac
flume-ng-core/src/main/java/org/apache/flume/source/SequenceGeneratorSource.java 440c5a9
flume-ng-core/src/main/java/org/apache/flume/source/SourceType.java 9082470
flume-ng-core/src/main/java/org/apache/flume/source/SyslogTcpSource.java d78d27f
flume-ng-core/src/main/java/org/apache/flume/source/SyslogUDPSource.java 34818f0
flume-ng-core/src/test/java/org/apache/flume/TestContext.java 51c350f
flume-ng-core/src/test/java/org/apache/flume/channel/AbstractBasicChannelSemanticsTest.java 6e71e46
flume-ng-core/src/test/java/org/apache/flume/channel/MockChannel.java 24b01e2
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannel.java e070864
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelConcurrency.java 8dad0b2
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelTransaction.java bc81f26
flume-ng-core/src/test/java/org/apache/flume/sink/TestAvroSink.java 3765924
flume-ng-core/src/test/java/org/apache/flume/sink/TestFailoverSinkProcessor.java 195c121
flume-ng-core/src/test/java/org/apache/flume/sink/TestLoggerSink.java 92ff6fe
flume-ng-core/src/test/java/org/apache/flume/sink/TestRollingFileSink.java 7e26e2a
flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java c5c3f2f
flume-ng-core/src/test/java/org/apache/flume/source/TestExecSource.java 6035270
flume-ng-core/src/test/java/org/apache/flume/source/TestPollableSourceRunner.java c27f82c
flume-ng-core/src/test/java/org/apache/flume/source/TestSequenceGeneratorSource.java 579b257
flume-ng-core/src/test/java/org/apache/flume/source/TestSyslogUdpSource.java d1db49d
flume-ng-legacy-sources/flume-avro-source/src/main/java/org/apache/flume/source/avroLegacy/AvroLegacySource.java dde8f28
flume-ng-legacy-sources/flume-avro-source/src/test/java/org/apache/flume/source/avroLegacy/TestLegacyAvroSource.java 6e3eb53
flume-ng-legacy-sources/flume-thrift-source/src/main/java/org/apache/flume/source/thriftLegacy/ThriftLegacySource.java fbf7362
flume-ng-legacy-sources/flume-thrift-source/src/test/java/org/apache/flume/source/thriftLegacy/TestThriftLegacySource.java ddd9478
flume-ng-node/src/main/java/org/apache/flume/conf/properties/FlumeConfiguration.java d66f6d1
flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java 97f72e1
flume-ng-node/src/test/java/org/apache/flume/node/TestAbstractLogicalNodeManager.java 521b586
flume-ng-node/src/test/java/org/apache/flume/source/TestNetcatSource.java fb2a960
flume-ng-node/src/test/resources/flume-conf.properties 848caca
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java da82f7e
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 8fa72a1
flume-ng-sinks/flume-irc-sink/src/main/java/org/apache/flume/sink/irc/IRCSink.java 8e77218
pom.xml 794ebe3
Diff: https://reviews.apache.org/r/4115/diff
Testing
-------
All existing unit tests for the components whose configuration has now been moved to these stubs pass.
Thanks,
Hari
Re: Review Request: Component aware configuration managers for flume
components
Posted by Hari Shreedharan <hs...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4115/
-----------------------------------------------------------
(Updated 2012-03-15 01:55:45.546999)
Review request for Flume.
Changes
-------
Added headers to Sequence generator source to test multiplexing channel selector.
Summary
-------
Currently the configuration of each component sits deep inside the component themselves. There is no way to verify if a configuration is valid before run time. In most systems like Flume, it is likely that these confs will be deployed from one single host, on to the machines where flume agents are running. Only when each agent starts, invalid confs are identified because the Agents would terminate by throwing exceptions. This is a first attempt to make a component-aware configuration system which is independent of the Flume, and does not require the Flume jar to be installed. Each component needs a configuration manager, which configures the components.
I have not completed the validation part yet, but have completed the configuration managers for all the components that come in the flume jar. I will add support for the other components, such as HDFS sink, JDBC channel etc soon. I will also add the validation support to the classes, but would like a review of the model being proposed here.
Note that I do know of a couple of bugs in the existing diff(like that if the name of a config key starts with an Upper case letter, an exception is thrown) - I will fix and upload the review here soon.
This addresses bug FLUME-992.
https://issues.apache.org/jira/browse/FLUME-992
Diffs (updated)
-----
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannel.java bca0c50
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProvider.java e445d61
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProviderFactory.java 6fbd6ef
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java 307ae89
flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestJdbcChannelProvider.java 7710d46
flume-ng-clients/flume-ng-log4jappender/src/test/java/org/apache/flume/clients/log4jappender/TestLog4jAppender.java 68d95fb
flume-ng-configuration/pom.xml PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfigurationFactory.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ConfigurationException.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/Context.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationError.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationErrorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MemoryChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MultiplexingChannelSelectorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/PseudoTxnMemoryChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/AvroSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/FailoverSinkProcessorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/RollingFileSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkGroupConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkProcessorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkProcessorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/AvroSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/ExecSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/NetcatSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceType.java PRE-CREATION
flume-ng-configuration/src/test/resources/flume-ng-error-test PRE-CREATION
flume-ng-configuration/src/test/resources/flume-ng-test.conf PRE-CREATION
flume-ng-core/pom.xml 247df37
flume-ng-core/src/main/java/org/apache/flume/ChannelSelector.java fba2dcb
flume-ng-core/src/main/java/org/apache/flume/Context.java 5294e31
flume-ng-core/src/main/java/org/apache/flume/SinkProcessor.java ea6000b
flume-ng-core/src/main/java/org/apache/flume/SinkProcessorType.java be1891b
flume-ng-core/src/main/java/org/apache/flume/channel/AbstractChannel.java 352bf08
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorFactory.java 800f471
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorType.java 511fc65
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8
flume-ng-core/src/main/java/org/apache/flume/channel/DefaultChannelFactory.java 963a6a3
flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java bfa1fde
flume-ng-core/src/main/java/org/apache/flume/channel/MultiplexingChannelSelector.java 83928b7
flume-ng-core/src/main/java/org/apache/flume/channel/PseudoTxnMemoryChannel.java 489d3e5
flume-ng-core/src/main/java/org/apache/flume/channel/ReplicatingChannelSelector.java 8f22746
flume-ng-core/src/main/java/org/apache/flume/conf/Configurable.java 0fa4839
flume-ng-core/src/main/java/org/apache/flume/conf/ConfigurableComponent.java PRE-CREATION
flume-ng-core/src/main/java/org/apache/flume/conf/Configurables.java 84492e5
flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 3edc563
flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkFactory.java b89dfa0
flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkProcessor.java 257bab3
flume-ng-core/src/main/java/org/apache/flume/sink/FailoverSinkProcessor.java 9f5b856
flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java 7f1d3b3
flume-ng-core/src/main/java/org/apache/flume/sink/SinkGroup.java 0dffd69
flume-ng-core/src/main/java/org/apache/flume/sink/SinkProcessorFactory.java 6160a17
flume-ng-core/src/main/java/org/apache/flume/sink/SinkType.java 6b08c09
flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java 903889d
flume-ng-core/src/main/java/org/apache/flume/source/DefaultSourceFactory.java a610e6f
flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java b6b1181
flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java 94245ac
flume-ng-core/src/main/java/org/apache/flume/source/SequenceGeneratorSource.java 440c5a9
flume-ng-core/src/main/java/org/apache/flume/source/SourceType.java 9082470
flume-ng-core/src/main/java/org/apache/flume/source/SyslogTcpSource.java d78d27f
flume-ng-core/src/main/java/org/apache/flume/source/SyslogUDPSource.java 34818f0
flume-ng-core/src/test/java/org/apache/flume/TestContext.java 51c350f
flume-ng-core/src/test/java/org/apache/flume/channel/AbstractBasicChannelSemanticsTest.java 6e71e46
flume-ng-core/src/test/java/org/apache/flume/channel/MockChannel.java 24b01e2
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannel.java e070864
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelConcurrency.java 8dad0b2
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelTransaction.java bc81f26
flume-ng-core/src/test/java/org/apache/flume/sink/TestAvroSink.java 3765924
flume-ng-core/src/test/java/org/apache/flume/sink/TestFailoverSinkProcessor.java 195c121
flume-ng-core/src/test/java/org/apache/flume/sink/TestLoggerSink.java 92ff6fe
flume-ng-core/src/test/java/org/apache/flume/sink/TestRollingFileSink.java 7e26e2a
flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java c5c3f2f
flume-ng-core/src/test/java/org/apache/flume/source/TestExecSource.java 6035270
flume-ng-core/src/test/java/org/apache/flume/source/TestPollableSourceRunner.java c27f82c
flume-ng-core/src/test/java/org/apache/flume/source/TestSequenceGeneratorSource.java 579b257
flume-ng-core/src/test/java/org/apache/flume/source/TestSyslogUdpSource.java d1db49d
flume-ng-legacy-sources/flume-avro-source/src/main/java/org/apache/flume/source/avroLegacy/AvroLegacySource.java dde8f28
flume-ng-legacy-sources/flume-avro-source/src/test/java/org/apache/flume/source/avroLegacy/TestLegacyAvroSource.java 6e3eb53
flume-ng-legacy-sources/flume-thrift-source/src/main/java/org/apache/flume/source/thriftLegacy/ThriftLegacySource.java fbf7362
flume-ng-legacy-sources/flume-thrift-source/src/test/java/org/apache/flume/source/thriftLegacy/TestThriftLegacySource.java ddd9478
flume-ng-node/src/main/java/org/apache/flume/conf/properties/FlumeConfiguration.java d66f6d1
flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java 97f72e1
flume-ng-node/src/test/java/org/apache/flume/node/TestAbstractLogicalNodeManager.java 521b586
flume-ng-node/src/test/java/org/apache/flume/source/TestNetcatSource.java fb2a960
flume-ng-node/src/test/resources/flume-conf.properties 848caca
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java da82f7e
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 8fa72a1
flume-ng-sinks/flume-irc-sink/src/main/java/org/apache/flume/sink/irc/IRCSink.java 8e77218
pom.xml 794ebe3
Diff: https://reviews.apache.org/r/4115/diff
Testing
-------
All existing unit tests for the components whose configuration has now been moved to these stubs pass.
Thanks,
Hari
Re: Review Request: Component aware configuration managers for
flume components
Posted by Hari Shreedharan <hs...@cloudera.com>.
+ Multiplexing channel selector.
--
Hari Shreedharan
On Wednesday, March 14, 2012 at 3:34 PM, Hari Shreedharan wrote:
> Verified by functional testing:
> AvroSource
> AvroSink
> SeqSource
> NetcatSource
> ExecSource
> MemoryChannel
> JdbcChannel
> AvroSink
> LoggerSink
> RollingFileSink
>
> I will continue verification, as and when I get time.
>
> Thanks
> Hari
>
>
>
> --
> Hari Shreedharan
>
>
> On Wednesday, March 14, 2012 at 2:18 PM, Hari Shreedharan wrote:
>
> >
> > -----------------------------------------------------------
> > This is an automatically generated e-mail. To reply, visit:
> > https://reviews.apache.org/r/4115/
> > -----------------------------------------------------------
> >
> > (Updated 2012-03-14 21:18:41.095856)
> >
> >
> > Review request for Flume.
> >
> >
> > Changes
> > -------
> >
> > Added stubs for Jdbc channel and Hdfs sink which do not have any implementation. I am leaving it for the original implementer to update it.
> >
> >
> > Summary
> > -------
> >
> > Currently the configuration of each component sits deep inside the component themselves. There is no way to verify if a configuration is valid before run time. In most systems like Flume, it is likely that these confs will be deployed from one single host, on to the machines where flume agents are running. Only when each agent starts, invalid confs are identified because the Agents would terminate by throwing exceptions. This is a first attempt to make a component-aware configuration system which is independent of the Flume, and does not require the Flume jar to be installed. Each component needs a configuration manager, which configures the components.
> >
> > I have not completed the validation part yet, but have completed the configuration managers for all the components that come in the flume jar. I will add support for the other components, such as HDFS sink, JDBC channel etc soon. I will also add the validation support to the classes, but would like a review of the model being proposed here.
> >
> > Note that I do know of a couple of bugs in the existing diff(like that if the name of a config key starts with an Upper case letter, an exception is thrown) - I will fix and upload the review here soon.
> >
> >
> > This addresses bug FLUME-992.
> > https://issues.apache.org/jira/browse/FLUME-992
> >
> >
> > Diffs (updated)
> > -----
> >
> > flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453
> > flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannel.java bca0c50
> > flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProvider.java e445d61
> > flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProviderFactory.java 6fbd6ef
> > flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java 307ae89
> > flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestJdbcChannelProvider.java 7710d46
> > flume-ng-clients/flume-ng-log4jappender/src/test/java/org/apache/flume/clients/log4jappender/TestLog4jAppender.java 68d95fb
> > flume-ng-configuration/pom.xml PRE-CREATION
> > flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfiguration.java PRE-CREATION
> > flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfigurationFactory.java PRE-CREATION
> > flume-ng-configuration/src/main/java/org/apache/flume/conf/ConfigurationException.java PRE-CREATION
> > flume-ng-configuration/src/main/java/org/apache/flume/conf/Context.java PRE-CREATION
> > flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfiguration.java PRE-CREATION
> > flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationError.java PRE-CREATION
> > flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationErrorType.java PRE-CREATION
> > flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelConfiguration.java PRE-CREATION
> > flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorConfiguration.java PRE-CREATION
> > flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorType.java PRE-CREATION
> > flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelType.java PRE-CREATION
> > flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MemoryChannelConfiguration.java PRE-CREATION
> > flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MultiplexingChannelSelectorConfiguration.java PRE-CREATION
> > flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/PseudoTxnMemoryChannelConfiguration.java PRE-CREATION
> > flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/AvroSinkConfiguration.java PRE-CREATION
> > flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/FailoverSinkProcessorConfiguration.java PRE-CREATION
> > flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/RollingFileSinkConfiguration.java PRE-CREATION
> > flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkConfiguration.java PRE-CREATION
> > flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkGroupConfiguration.java PRE-CREATION
> > flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkProcessorConfiguration.java PRE-CREATION
> > flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkProcessorType.java PRE-CREATION
> > flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkType.java PRE-CREATION
> > flume-ng-configuration/src/main/java/org/apache/flume/conf/source/AvroSourceConfiguration.java PRE-CREATION
> > flume-ng-configuration/src/main/java/org/apache/flume/conf/source/ExecSourceConfiguration.java PRE-CREATION
> > flume-ng-configuration/src/main/java/org/apache/flume/conf/source/NetcatSourceConfiguration.java PRE-CREATION
> > flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceConfiguration.java PRE-CREATION
> > flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceType.java PRE-CREATION
> > flume-ng-configuration/src/test/resources/flume-ng-error-test PRE-CREATION
> > flume-ng-configuration/src/test/resources/flume-ng-test.conf PRE-CREATION
> > flume-ng-core/pom.xml 247df37
> > flume-ng-core/src/main/java/org/apache/flume/ChannelSelector.java fba2dcb
> > flume-ng-core/src/main/java/org/apache/flume/Context.java 5294e31
> > flume-ng-core/src/main/java/org/apache/flume/SinkProcessor.java ea6000b
> > flume-ng-core/src/main/java/org/apache/flume/SinkProcessorType.java be1891b
> > flume-ng-core/src/main/java/org/apache/flume/channel/AbstractChannel.java 352bf08
> > flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorFactory.java 800f471
> > flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorType.java 511fc65
> > flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8
> > flume-ng-core/src/main/java/org/apache/flume/channel/DefaultChannelFactory.java 963a6a3
> > flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java bfa1fde
> > flume-ng-core/src/main/java/org/apache/flume/channel/MultiplexingChannelSelector.java 83928b7
> > flume-ng-core/src/main/java/org/apache/flume/channel/PseudoTxnMemoryChannel.java 489d3e5
> > flume-ng-core/src/main/java/org/apache/flume/channel/ReplicatingChannelSelector.java 8f22746
> > flume-ng-core/src/main/java/org/apache/flume/conf/Configurable.java 0fa4839
> > flume-ng-core/src/main/java/org/apache/flume/conf/ConfigurableComponent.java PRE-CREATION
> > flume-ng-core/src/main/java/org/apache/flume/conf/Configurables.java 84492e5
> > flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 3edc563
> > flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkFactory.java b89dfa0
> > flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkProcessor.java 257bab3
> > flume-ng-core/src/main/java/org/apache/flume/sink/FailoverSinkProcessor.java 9f5b856
> > flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java 7f1d3b3
> > flume-ng-core/src/main/java/org/apache/flume/sink/SinkGroup.java 0dffd69
> > flume-ng-core/src/main/java/org/apache/flume/sink/SinkProcessorFactory.java 6160a17
> > flume-ng-core/src/main/java/org/apache/flume/sink/SinkType.java 6b08c09
> > flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java 903889d
> > flume-ng-core/src/main/java/org/apache/flume/source/DefaultSourceFactory.java a610e6f
> > flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java b6b1181
> > flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java 94245ac
> > flume-ng-core/src/main/java/org/apache/flume/source/SourceType.java 9082470
> > flume-ng-core/src/main/java/org/apache/flume/source/SyslogTcpSource.java d78d27f
> > flume-ng-core/src/main/java/org/apache/flume/source/SyslogUDPSource.java 34818f0
> > flume-ng-core/src/test/java/org/apache/flume/TestContext.java 51c350f
> > flume-ng-core/src/test/java/org/apache/flume/channel/AbstractBasicChannelSemanticsTest.java 6e71e46
> > flume-ng-core/src/test/java/org/apache/flume/channel/MockChannel.java 24b01e2
> > flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannel.java e070864
> > flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelConcurrency.java 8dad0b2
> > flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelTransaction.java bc81f26
> > flume-ng-core/src/test/java/org/apache/flume/sink/TestAvroSink.java 3765924
> > flume-ng-core/src/test/java/org/apache/flume/sink/TestFailoverSinkProcessor.java 195c121
> > flume-ng-core/src/test/java/org/apache/flume/sink/TestLoggerSink.java 92ff6fe
> > flume-ng-core/src/test/java/org/apache/flume/sink/TestRollingFileSink.java 7e26e2a
> > flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java c5c3f2f
> > flume-ng-core/src/test/java/org/apache/flume/source/TestExecSource.java 6035270
> > flume-ng-core/src/test/java/org/apache/flume/source/TestPollableSourceRunner.java c27f82c
> > flume-ng-core/src/test/java/org/apache/flume/source/TestSequenceGeneratorSource.java 579b257
> > flume-ng-core/src/test/java/org/apache/flume/source/TestSyslogUdpSource.java d1db49d
> > flume-ng-legacy-sources/flume-avro-source/src/main/java/org/apache/flume/source/avroLegacy/AvroLegacySource.java dde8f28
> > flume-ng-legacy-sources/flume-avro-source/src/test/java/org/apache/flume/source/avroLegacy/TestLegacyAvroSource.java 6e3eb53
> > flume-ng-legacy-sources/flume-thrift-source/src/main/java/org/apache/flume/source/thriftLegacy/ThriftLegacySource.java fbf7362
> > flume-ng-legacy-sources/flume-thrift-source/src/test/java/org/apache/flume/source/thriftLegacy/TestThriftLegacySource.java ddd9478
> > flume-ng-node/src/main/java/org/apache/flume/conf/properties/FlumeConfiguration.java d66f6d1
> > flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java 97f72e1
> > flume-ng-node/src/test/java/org/apache/flume/node/TestAbstractLogicalNodeManager.java 521b586
> > flume-ng-node/src/test/java/org/apache/flume/source/TestNetcatSource.java fb2a960
> > flume-ng-node/src/test/resources/flume-conf.properties 848caca
> > flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java da82f7e
> > flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 8fa72a1
> > flume-ng-sinks/flume-irc-sink/src/main/java/org/apache/flume/sink/irc/IRCSink.java 8e77218
> > pom.xml 794ebe3
> >
> > Diff: https://reviews.apache.org/r/4115/diff
> >
> >
> > Testing
> > -------
> >
> > All existing unit tests for the components whose configuration has now been moved to these stubs pass.
> >
> >
> > Thanks,
> >
> > Hari
>
Re: Review Request: Component aware configuration managers for
flume components
Posted by Hari Shreedharan <hs...@cloudera.com>.
Verified by functional testing:
AvroSource
AvroSink
SeqSource
NetcatSource
ExecSource
MemoryChannel
JdbcChannel
AvroSink
LoggerSink
RollingFileSink
I will continue verification, as and when I get time.
Thanks
Hari
--
Hari Shreedharan
On Wednesday, March 14, 2012 at 2:18 PM, Hari Shreedharan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/4115/
> -----------------------------------------------------------
>
> (Updated 2012-03-14 21:18:41.095856)
>
>
> Review request for Flume.
>
>
> Changes
> -------
>
> Added stubs for Jdbc channel and Hdfs sink which do not have any implementation. I am leaving it for the original implementer to update it.
>
>
> Summary
> -------
>
> Currently the configuration of each component sits deep inside the component themselves. There is no way to verify if a configuration is valid before run time. In most systems like Flume, it is likely that these confs will be deployed from one single host, on to the machines where flume agents are running. Only when each agent starts, invalid confs are identified because the Agents would terminate by throwing exceptions. This is a first attempt to make a component-aware configuration system which is independent of the Flume, and does not require the Flume jar to be installed. Each component needs a configuration manager, which configures the components.
>
> I have not completed the validation part yet, but have completed the configuration managers for all the components that come in the flume jar. I will add support for the other components, such as HDFS sink, JDBC channel etc soon. I will also add the validation support to the classes, but would like a review of the model being proposed here.
>
> Note that I do know of a couple of bugs in the existing diff(like that if the name of a config key starts with an Upper case letter, an exception is thrown) - I will fix and upload the review here soon.
>
>
> This addresses bug FLUME-992.
> https://issues.apache.org/jira/browse/FLUME-992
>
>
> Diffs (updated)
> -----
>
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453
> flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannel.java bca0c50
> flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProvider.java e445d61
> flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProviderFactory.java 6fbd6ef
> flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java 307ae89
> flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestJdbcChannelProvider.java 7710d46
> flume-ng-clients/flume-ng-log4jappender/src/test/java/org/apache/flume/clients/log4jappender/TestLog4jAppender.java 68d95fb
> flume-ng-configuration/pom.xml PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfigurationFactory.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/ConfigurationException.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/Context.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationError.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationErrorType.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorType.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelType.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MemoryChannelConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MultiplexingChannelSelectorConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/PseudoTxnMemoryChannelConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/AvroSinkConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/FailoverSinkProcessorConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/RollingFileSinkConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkGroupConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkProcessorConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkProcessorType.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkType.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/source/AvroSourceConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/source/ExecSourceConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/source/NetcatSourceConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceType.java PRE-CREATION
> flume-ng-configuration/src/test/resources/flume-ng-error-test PRE-CREATION
> flume-ng-configuration/src/test/resources/flume-ng-test.conf PRE-CREATION
> flume-ng-core/pom.xml 247df37
> flume-ng-core/src/main/java/org/apache/flume/ChannelSelector.java fba2dcb
> flume-ng-core/src/main/java/org/apache/flume/Context.java 5294e31
> flume-ng-core/src/main/java/org/apache/flume/SinkProcessor.java ea6000b
> flume-ng-core/src/main/java/org/apache/flume/SinkProcessorType.java be1891b
> flume-ng-core/src/main/java/org/apache/flume/channel/AbstractChannel.java 352bf08
> flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorFactory.java 800f471
> flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorType.java 511fc65
> flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8
> flume-ng-core/src/main/java/org/apache/flume/channel/DefaultChannelFactory.java 963a6a3
> flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java bfa1fde
> flume-ng-core/src/main/java/org/apache/flume/channel/MultiplexingChannelSelector.java 83928b7
> flume-ng-core/src/main/java/org/apache/flume/channel/PseudoTxnMemoryChannel.java 489d3e5
> flume-ng-core/src/main/java/org/apache/flume/channel/ReplicatingChannelSelector.java 8f22746
> flume-ng-core/src/main/java/org/apache/flume/conf/Configurable.java 0fa4839
> flume-ng-core/src/main/java/org/apache/flume/conf/ConfigurableComponent.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/conf/Configurables.java 84492e5
> flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 3edc563
> flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkFactory.java b89dfa0
> flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkProcessor.java 257bab3
> flume-ng-core/src/main/java/org/apache/flume/sink/FailoverSinkProcessor.java 9f5b856
> flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java 7f1d3b3
> flume-ng-core/src/main/java/org/apache/flume/sink/SinkGroup.java 0dffd69
> flume-ng-core/src/main/java/org/apache/flume/sink/SinkProcessorFactory.java 6160a17
> flume-ng-core/src/main/java/org/apache/flume/sink/SinkType.java 6b08c09
> flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java 903889d
> flume-ng-core/src/main/java/org/apache/flume/source/DefaultSourceFactory.java a610e6f
> flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java b6b1181
> flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java 94245ac
> flume-ng-core/src/main/java/org/apache/flume/source/SourceType.java 9082470
> flume-ng-core/src/main/java/org/apache/flume/source/SyslogTcpSource.java d78d27f
> flume-ng-core/src/main/java/org/apache/flume/source/SyslogUDPSource.java 34818f0
> flume-ng-core/src/test/java/org/apache/flume/TestContext.java 51c350f
> flume-ng-core/src/test/java/org/apache/flume/channel/AbstractBasicChannelSemanticsTest.java 6e71e46
> flume-ng-core/src/test/java/org/apache/flume/channel/MockChannel.java 24b01e2
> flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannel.java e070864
> flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelConcurrency.java 8dad0b2
> flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelTransaction.java bc81f26
> flume-ng-core/src/test/java/org/apache/flume/sink/TestAvroSink.java 3765924
> flume-ng-core/src/test/java/org/apache/flume/sink/TestFailoverSinkProcessor.java 195c121
> flume-ng-core/src/test/java/org/apache/flume/sink/TestLoggerSink.java 92ff6fe
> flume-ng-core/src/test/java/org/apache/flume/sink/TestRollingFileSink.java 7e26e2a
> flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java c5c3f2f
> flume-ng-core/src/test/java/org/apache/flume/source/TestExecSource.java 6035270
> flume-ng-core/src/test/java/org/apache/flume/source/TestPollableSourceRunner.java c27f82c
> flume-ng-core/src/test/java/org/apache/flume/source/TestSequenceGeneratorSource.java 579b257
> flume-ng-core/src/test/java/org/apache/flume/source/TestSyslogUdpSource.java d1db49d
> flume-ng-legacy-sources/flume-avro-source/src/main/java/org/apache/flume/source/avroLegacy/AvroLegacySource.java dde8f28
> flume-ng-legacy-sources/flume-avro-source/src/test/java/org/apache/flume/source/avroLegacy/TestLegacyAvroSource.java 6e3eb53
> flume-ng-legacy-sources/flume-thrift-source/src/main/java/org/apache/flume/source/thriftLegacy/ThriftLegacySource.java fbf7362
> flume-ng-legacy-sources/flume-thrift-source/src/test/java/org/apache/flume/source/thriftLegacy/TestThriftLegacySource.java ddd9478
> flume-ng-node/src/main/java/org/apache/flume/conf/properties/FlumeConfiguration.java d66f6d1
> flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java 97f72e1
> flume-ng-node/src/test/java/org/apache/flume/node/TestAbstractLogicalNodeManager.java 521b586
> flume-ng-node/src/test/java/org/apache/flume/source/TestNetcatSource.java fb2a960
> flume-ng-node/src/test/resources/flume-conf.properties 848caca
> flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java da82f7e
> flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 8fa72a1
> flume-ng-sinks/flume-irc-sink/src/main/java/org/apache/flume/sink/irc/IRCSink.java 8e77218
> pom.xml 794ebe3
>
> Diff: https://reviews.apache.org/r/4115/diff
>
>
> Testing
> -------
>
> All existing unit tests for the components whose configuration has now been moved to these stubs pass.
>
>
> Thanks,
>
> Hari
Re: Review Request: Component aware configuration managers for flume
components
Posted by Hari Shreedharan <hs...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4115/
-----------------------------------------------------------
(Updated 2012-03-14 21:18:41.095856)
Review request for Flume.
Changes
-------
Added stubs for Jdbc channel and Hdfs sink which do not have any implementation. I am leaving it for the original implementer to update it.
Summary
-------
Currently the configuration of each component sits deep inside the component themselves. There is no way to verify if a configuration is valid before run time. In most systems like Flume, it is likely that these confs will be deployed from one single host, on to the machines where flume agents are running. Only when each agent starts, invalid confs are identified because the Agents would terminate by throwing exceptions. This is a first attempt to make a component-aware configuration system which is independent of the Flume, and does not require the Flume jar to be installed. Each component needs a configuration manager, which configures the components.
I have not completed the validation part yet, but have completed the configuration managers for all the components that come in the flume jar. I will add support for the other components, such as HDFS sink, JDBC channel etc soon. I will also add the validation support to the classes, but would like a review of the model being proposed here.
Note that I do know of a couple of bugs in the existing diff(like that if the name of a config key starts with an Upper case letter, an exception is thrown) - I will fix and upload the review here soon.
This addresses bug FLUME-992.
https://issues.apache.org/jira/browse/FLUME-992
Diffs (updated)
-----
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannel.java bca0c50
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProvider.java e445d61
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProviderFactory.java 6fbd6ef
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java 307ae89
flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestJdbcChannelProvider.java 7710d46
flume-ng-clients/flume-ng-log4jappender/src/test/java/org/apache/flume/clients/log4jappender/TestLog4jAppender.java 68d95fb
flume-ng-configuration/pom.xml PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfigurationFactory.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ConfigurationException.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/Context.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationError.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationErrorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MemoryChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MultiplexingChannelSelectorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/PseudoTxnMemoryChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/AvroSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/FailoverSinkProcessorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/RollingFileSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkGroupConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkProcessorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkProcessorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/AvroSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/ExecSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/NetcatSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceType.java PRE-CREATION
flume-ng-configuration/src/test/resources/flume-ng-error-test PRE-CREATION
flume-ng-configuration/src/test/resources/flume-ng-test.conf PRE-CREATION
flume-ng-core/pom.xml 247df37
flume-ng-core/src/main/java/org/apache/flume/ChannelSelector.java fba2dcb
flume-ng-core/src/main/java/org/apache/flume/Context.java 5294e31
flume-ng-core/src/main/java/org/apache/flume/SinkProcessor.java ea6000b
flume-ng-core/src/main/java/org/apache/flume/SinkProcessorType.java be1891b
flume-ng-core/src/main/java/org/apache/flume/channel/AbstractChannel.java 352bf08
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorFactory.java 800f471
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorType.java 511fc65
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8
flume-ng-core/src/main/java/org/apache/flume/channel/DefaultChannelFactory.java 963a6a3
flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java bfa1fde
flume-ng-core/src/main/java/org/apache/flume/channel/MultiplexingChannelSelector.java 83928b7
flume-ng-core/src/main/java/org/apache/flume/channel/PseudoTxnMemoryChannel.java 489d3e5
flume-ng-core/src/main/java/org/apache/flume/channel/ReplicatingChannelSelector.java 8f22746
flume-ng-core/src/main/java/org/apache/flume/conf/Configurable.java 0fa4839
flume-ng-core/src/main/java/org/apache/flume/conf/ConfigurableComponent.java PRE-CREATION
flume-ng-core/src/main/java/org/apache/flume/conf/Configurables.java 84492e5
flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 3edc563
flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkFactory.java b89dfa0
flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkProcessor.java 257bab3
flume-ng-core/src/main/java/org/apache/flume/sink/FailoverSinkProcessor.java 9f5b856
flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java 7f1d3b3
flume-ng-core/src/main/java/org/apache/flume/sink/SinkGroup.java 0dffd69
flume-ng-core/src/main/java/org/apache/flume/sink/SinkProcessorFactory.java 6160a17
flume-ng-core/src/main/java/org/apache/flume/sink/SinkType.java 6b08c09
flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java 903889d
flume-ng-core/src/main/java/org/apache/flume/source/DefaultSourceFactory.java a610e6f
flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java b6b1181
flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java 94245ac
flume-ng-core/src/main/java/org/apache/flume/source/SourceType.java 9082470
flume-ng-core/src/main/java/org/apache/flume/source/SyslogTcpSource.java d78d27f
flume-ng-core/src/main/java/org/apache/flume/source/SyslogUDPSource.java 34818f0
flume-ng-core/src/test/java/org/apache/flume/TestContext.java 51c350f
flume-ng-core/src/test/java/org/apache/flume/channel/AbstractBasicChannelSemanticsTest.java 6e71e46
flume-ng-core/src/test/java/org/apache/flume/channel/MockChannel.java 24b01e2
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannel.java e070864
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelConcurrency.java 8dad0b2
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelTransaction.java bc81f26
flume-ng-core/src/test/java/org/apache/flume/sink/TestAvroSink.java 3765924
flume-ng-core/src/test/java/org/apache/flume/sink/TestFailoverSinkProcessor.java 195c121
flume-ng-core/src/test/java/org/apache/flume/sink/TestLoggerSink.java 92ff6fe
flume-ng-core/src/test/java/org/apache/flume/sink/TestRollingFileSink.java 7e26e2a
flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java c5c3f2f
flume-ng-core/src/test/java/org/apache/flume/source/TestExecSource.java 6035270
flume-ng-core/src/test/java/org/apache/flume/source/TestPollableSourceRunner.java c27f82c
flume-ng-core/src/test/java/org/apache/flume/source/TestSequenceGeneratorSource.java 579b257
flume-ng-core/src/test/java/org/apache/flume/source/TestSyslogUdpSource.java d1db49d
flume-ng-legacy-sources/flume-avro-source/src/main/java/org/apache/flume/source/avroLegacy/AvroLegacySource.java dde8f28
flume-ng-legacy-sources/flume-avro-source/src/test/java/org/apache/flume/source/avroLegacy/TestLegacyAvroSource.java 6e3eb53
flume-ng-legacy-sources/flume-thrift-source/src/main/java/org/apache/flume/source/thriftLegacy/ThriftLegacySource.java fbf7362
flume-ng-legacy-sources/flume-thrift-source/src/test/java/org/apache/flume/source/thriftLegacy/TestThriftLegacySource.java ddd9478
flume-ng-node/src/main/java/org/apache/flume/conf/properties/FlumeConfiguration.java d66f6d1
flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java 97f72e1
flume-ng-node/src/test/java/org/apache/flume/node/TestAbstractLogicalNodeManager.java 521b586
flume-ng-node/src/test/java/org/apache/flume/source/TestNetcatSource.java fb2a960
flume-ng-node/src/test/resources/flume-conf.properties 848caca
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java da82f7e
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 8fa72a1
flume-ng-sinks/flume-irc-sink/src/main/java/org/apache/flume/sink/irc/IRCSink.java 8e77218
pom.xml 794ebe3
Diff: https://reviews.apache.org/r/4115/diff
Testing
-------
All existing unit tests for the components whose configuration has now been moved to these stubs pass.
Thanks,
Hari
Re: Review Request: Component aware configuration managers for flume
components
Posted by Brock Noland <br...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4115/#review5961
-----------------------------------------------------------
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorConfiguration.java
<https://reviews.apache.org/r/4115/#comment12930>
In general I think errors like this should be thrown up.
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorConfiguration.java
<https://reviews.apache.org/r/4115/#comment12929>
We should be doing something with this exception. Else the only way to debug is a debugger.
- Brock
On 2012-03-14 19:21:20, Hari Shreedharan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/4115/
> -----------------------------------------------------------
>
> (Updated 2012-03-14 19:21:20)
>
>
> Review request for Flume.
>
>
> Summary
> -------
>
> Currently the configuration of each component sits deep inside the component themselves. There is no way to verify if a configuration is valid before run time. In most systems like Flume, it is likely that these confs will be deployed from one single host, on to the machines where flume agents are running. Only when each agent starts, invalid confs are identified because the Agents would terminate by throwing exceptions. This is a first attempt to make a component-aware configuration system which is independent of the Flume, and does not require the Flume jar to be installed. Each component needs a configuration manager, which configures the components.
>
> I have not completed the validation part yet, but have completed the configuration managers for all the components that come in the flume jar. I will add support for the other components, such as HDFS sink, JDBC channel etc soon. I will also add the validation support to the classes, but would like a review of the model being proposed here.
>
> Note that I do know of a couple of bugs in the existing diff(like that if the name of a config key starts with an Upper case letter, an exception is thrown) - I will fix and upload the review here soon.
>
>
> This addresses bug FLUME-992.
> https://issues.apache.org/jira/browse/FLUME-992
>
>
> Diffs
> -----
>
> flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationErrorType.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationError.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/Context.java PRE-CREATION
> flume-ng-clients/flume-ng-log4jappender/src/test/java/org/apache/flume/clients/log4jappender/TestLog4jAppender.java 68d95fb
> flume-ng-configuration/pom.xml PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfigurationFactory.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/ConfigurationException.java PRE-CREATION
> flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestJdbcChannelProvider.java 7710d46
> flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453
> flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannel.java bca0c50
> flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProvider.java e445d61
> flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProviderFactory.java 6fbd6ef
> flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java 307ae89
> flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorType.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelType.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MemoryChannelConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MultiplexingChannelSelectorConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/PseudoTxnMemoryChannelConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/AvroSinkConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/FailoverSinkProcessorConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/RollingFileSinkConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkGroupConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkProcessorConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkProcessorType.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkType.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/source/AvroSourceConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/source/ExecSourceConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/source/NetcatSourceConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceType.java PRE-CREATION
> flume-ng-configuration/src/test/resources/flume-ng-error-test PRE-CREATION
> flume-ng-configuration/src/test/resources/flume-ng-test.conf PRE-CREATION
> flume-ng-core/pom.xml 247df37
> flume-ng-core/src/main/java/org/apache/flume/ChannelSelector.java fba2dcb
> flume-ng-core/src/main/java/org/apache/flume/Context.java 5294e31
> flume-ng-core/src/main/java/org/apache/flume/SinkProcessor.java ea6000b
> flume-ng-core/src/main/java/org/apache/flume/SinkProcessorType.java be1891b
> flume-ng-core/src/main/java/org/apache/flume/channel/AbstractChannel.java 352bf08
> flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorFactory.java 800f471
> flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorType.java 511fc65
> flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8
> flume-ng-core/src/main/java/org/apache/flume/channel/DefaultChannelFactory.java 963a6a3
> flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java bfa1fde
> flume-ng-core/src/main/java/org/apache/flume/channel/MultiplexingChannelSelector.java 83928b7
> flume-ng-core/src/main/java/org/apache/flume/channel/PseudoTxnMemoryChannel.java 489d3e5
> flume-ng-core/src/main/java/org/apache/flume/channel/ReplicatingChannelSelector.java 8f22746
> flume-ng-core/src/main/java/org/apache/flume/conf/Configurable.java 0fa4839
> flume-ng-core/src/main/java/org/apache/flume/conf/ConfigurableComponent.java PRE-CREATION
> flume-ng-core/src/main/java/org/apache/flume/conf/Configurables.java 84492e5
> flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 3edc563
> flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkFactory.java b89dfa0
> flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkProcessor.java 257bab3
> flume-ng-core/src/main/java/org/apache/flume/sink/FailoverSinkProcessor.java 9f5b856
> flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java 7f1d3b3
> flume-ng-core/src/main/java/org/apache/flume/sink/SinkGroup.java 0dffd69
> flume-ng-core/src/main/java/org/apache/flume/sink/SinkProcessorFactory.java 6160a17
> flume-ng-core/src/main/java/org/apache/flume/sink/SinkType.java 6b08c09
> flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java 903889d
> flume-ng-core/src/main/java/org/apache/flume/source/DefaultSourceFactory.java a610e6f
> flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java b6b1181
> flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java 94245ac
> flume-ng-core/src/main/java/org/apache/flume/source/SourceType.java 9082470
> flume-ng-core/src/main/java/org/apache/flume/source/SyslogTcpSource.java d78d27f
> flume-ng-core/src/main/java/org/apache/flume/source/SyslogUDPSource.java 34818f0
> flume-ng-core/src/test/java/org/apache/flume/TestContext.java 51c350f
> flume-ng-core/src/test/java/org/apache/flume/channel/AbstractBasicChannelSemanticsTest.java 6e71e46
> flume-ng-core/src/test/java/org/apache/flume/channel/MockChannel.java 24b01e2
> flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannel.java e070864
> flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelConcurrency.java 8dad0b2
> flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelTransaction.java bc81f26
> flume-ng-core/src/test/java/org/apache/flume/sink/TestAvroSink.java 3765924
> flume-ng-core/src/test/java/org/apache/flume/sink/TestFailoverSinkProcessor.java 195c121
> flume-ng-core/src/test/java/org/apache/flume/sink/TestLoggerSink.java 92ff6fe
> flume-ng-core/src/test/java/org/apache/flume/sink/TestRollingFileSink.java 7e26e2a
> flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java c5c3f2f
> flume-ng-core/src/test/java/org/apache/flume/source/TestExecSource.java 6035270
> flume-ng-core/src/test/java/org/apache/flume/source/TestPollableSourceRunner.java c27f82c
> flume-ng-core/src/test/java/org/apache/flume/source/TestSequenceGeneratorSource.java 579b257
> flume-ng-core/src/test/java/org/apache/flume/source/TestSyslogUdpSource.java d1db49d
> flume-ng-legacy-sources/flume-avro-source/src/main/java/org/apache/flume/source/avroLegacy/AvroLegacySource.java dde8f28
> flume-ng-legacy-sources/flume-avro-source/src/test/java/org/apache/flume/source/avroLegacy/TestLegacyAvroSource.java 6e3eb53
> flume-ng-legacy-sources/flume-thrift-source/src/main/java/org/apache/flume/source/thriftLegacy/ThriftLegacySource.java fbf7362
> flume-ng-legacy-sources/flume-thrift-source/src/test/java/org/apache/flume/source/thriftLegacy/TestThriftLegacySource.java ddd9478
> flume-ng-node/src/main/java/org/apache/flume/conf/properties/FlumeConfiguration.java d66f6d1
> flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java 97f72e1
> flume-ng-node/src/test/java/org/apache/flume/node/TestAbstractLogicalNodeManager.java 521b586
> flume-ng-node/src/test/java/org/apache/flume/source/TestNetcatSource.java fb2a960
> flume-ng-node/src/test/resources/flume-conf.properties 848caca
> flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java da82f7e
> flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 8fa72a1
> flume-ng-sinks/flume-irc-sink/src/main/java/org/apache/flume/sink/irc/IRCSink.java 8e77218
> pom.xml 794ebe3
>
> Diff: https://reviews.apache.org/r/4115/diff
>
>
> Testing
> -------
>
> All existing unit tests for the components whose configuration has now been moved to these stubs pass.
>
>
> Thanks,
>
> Hari
>
>
Re: Review Request: Component aware configuration managers for flume
components
Posted by Hari Shreedharan <hs...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4115/
-----------------------------------------------------------
(Updated 2012-03-14 19:21:20.299543)
Review request for Flume.
Changes
-------
rebased on trunk. The toString functions provide the same functionality as it used to provide before. I didnt want to change it, since I am not sur eif it is used by someone else.
Summary
-------
Currently the configuration of each component sits deep inside the component themselves. There is no way to verify if a configuration is valid before run time. In most systems like Flume, it is likely that these confs will be deployed from one single host, on to the machines where flume agents are running. Only when each agent starts, invalid confs are identified because the Agents would terminate by throwing exceptions. This is a first attempt to make a component-aware configuration system which is independent of the Flume, and does not require the Flume jar to be installed. Each component needs a configuration manager, which configures the components.
I have not completed the validation part yet, but have completed the configuration managers for all the components that come in the flume jar. I will add support for the other components, such as HDFS sink, JDBC channel etc soon. I will also add the validation support to the classes, but would like a review of the model being proposed here.
Note that I do know of a couple of bugs in the existing diff(like that if the name of a config key starts with an Upper case letter, an exception is thrown) - I will fix and upload the review here soon.
This addresses bug FLUME-992.
https://issues.apache.org/jira/browse/FLUME-992
Diffs (updated)
-----
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationErrorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationError.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/Context.java PRE-CREATION
flume-ng-clients/flume-ng-log4jappender/src/test/java/org/apache/flume/clients/log4jappender/TestLog4jAppender.java 68d95fb
flume-ng-configuration/pom.xml PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfigurationFactory.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ConfigurationException.java PRE-CREATION
flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestJdbcChannelProvider.java 7710d46
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannel.java bca0c50
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProvider.java e445d61
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProviderFactory.java 6fbd6ef
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java 307ae89
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MemoryChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MultiplexingChannelSelectorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/PseudoTxnMemoryChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/AvroSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/FailoverSinkProcessorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/RollingFileSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkGroupConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkProcessorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkProcessorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/AvroSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/ExecSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/NetcatSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceType.java PRE-CREATION
flume-ng-configuration/src/test/resources/flume-ng-error-test PRE-CREATION
flume-ng-configuration/src/test/resources/flume-ng-test.conf PRE-CREATION
flume-ng-core/pom.xml 247df37
flume-ng-core/src/main/java/org/apache/flume/ChannelSelector.java fba2dcb
flume-ng-core/src/main/java/org/apache/flume/Context.java 5294e31
flume-ng-core/src/main/java/org/apache/flume/SinkProcessor.java ea6000b
flume-ng-core/src/main/java/org/apache/flume/SinkProcessorType.java be1891b
flume-ng-core/src/main/java/org/apache/flume/channel/AbstractChannel.java 352bf08
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorFactory.java 800f471
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorType.java 511fc65
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8
flume-ng-core/src/main/java/org/apache/flume/channel/DefaultChannelFactory.java 963a6a3
flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java bfa1fde
flume-ng-core/src/main/java/org/apache/flume/channel/MultiplexingChannelSelector.java 83928b7
flume-ng-core/src/main/java/org/apache/flume/channel/PseudoTxnMemoryChannel.java 489d3e5
flume-ng-core/src/main/java/org/apache/flume/channel/ReplicatingChannelSelector.java 8f22746
flume-ng-core/src/main/java/org/apache/flume/conf/Configurable.java 0fa4839
flume-ng-core/src/main/java/org/apache/flume/conf/ConfigurableComponent.java PRE-CREATION
flume-ng-core/src/main/java/org/apache/flume/conf/Configurables.java 84492e5
flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 3edc563
flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkFactory.java b89dfa0
flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkProcessor.java 257bab3
flume-ng-core/src/main/java/org/apache/flume/sink/FailoverSinkProcessor.java 9f5b856
flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java 7f1d3b3
flume-ng-core/src/main/java/org/apache/flume/sink/SinkGroup.java 0dffd69
flume-ng-core/src/main/java/org/apache/flume/sink/SinkProcessorFactory.java 6160a17
flume-ng-core/src/main/java/org/apache/flume/sink/SinkType.java 6b08c09
flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java 903889d
flume-ng-core/src/main/java/org/apache/flume/source/DefaultSourceFactory.java a610e6f
flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java b6b1181
flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java 94245ac
flume-ng-core/src/main/java/org/apache/flume/source/SourceType.java 9082470
flume-ng-core/src/main/java/org/apache/flume/source/SyslogTcpSource.java d78d27f
flume-ng-core/src/main/java/org/apache/flume/source/SyslogUDPSource.java 34818f0
flume-ng-core/src/test/java/org/apache/flume/TestContext.java 51c350f
flume-ng-core/src/test/java/org/apache/flume/channel/AbstractBasicChannelSemanticsTest.java 6e71e46
flume-ng-core/src/test/java/org/apache/flume/channel/MockChannel.java 24b01e2
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannel.java e070864
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelConcurrency.java 8dad0b2
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelTransaction.java bc81f26
flume-ng-core/src/test/java/org/apache/flume/sink/TestAvroSink.java 3765924
flume-ng-core/src/test/java/org/apache/flume/sink/TestFailoverSinkProcessor.java 195c121
flume-ng-core/src/test/java/org/apache/flume/sink/TestLoggerSink.java 92ff6fe
flume-ng-core/src/test/java/org/apache/flume/sink/TestRollingFileSink.java 7e26e2a
flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java c5c3f2f
flume-ng-core/src/test/java/org/apache/flume/source/TestExecSource.java 6035270
flume-ng-core/src/test/java/org/apache/flume/source/TestPollableSourceRunner.java c27f82c
flume-ng-core/src/test/java/org/apache/flume/source/TestSequenceGeneratorSource.java 579b257
flume-ng-core/src/test/java/org/apache/flume/source/TestSyslogUdpSource.java d1db49d
flume-ng-legacy-sources/flume-avro-source/src/main/java/org/apache/flume/source/avroLegacy/AvroLegacySource.java dde8f28
flume-ng-legacy-sources/flume-avro-source/src/test/java/org/apache/flume/source/avroLegacy/TestLegacyAvroSource.java 6e3eb53
flume-ng-legacy-sources/flume-thrift-source/src/main/java/org/apache/flume/source/thriftLegacy/ThriftLegacySource.java fbf7362
flume-ng-legacy-sources/flume-thrift-source/src/test/java/org/apache/flume/source/thriftLegacy/TestThriftLegacySource.java ddd9478
flume-ng-node/src/main/java/org/apache/flume/conf/properties/FlumeConfiguration.java d66f6d1
flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java 97f72e1
flume-ng-node/src/test/java/org/apache/flume/node/TestAbstractLogicalNodeManager.java 521b586
flume-ng-node/src/test/java/org/apache/flume/source/TestNetcatSource.java fb2a960
flume-ng-node/src/test/resources/flume-conf.properties 848caca
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java da82f7e
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 8fa72a1
flume-ng-sinks/flume-irc-sink/src/main/java/org/apache/flume/sink/irc/IRCSink.java 8e77218
pom.xml 794ebe3
Diff: https://reviews.apache.org/r/4115/diff
Testing
-------
All existing unit tests for the components whose configuration has now been moved to these stubs pass.
Thanks,
Hari
Re: Review Request: Component aware configuration managers for flume
components
Posted by Hari Shreedharan <hs...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4115/
-----------------------------------------------------------
(Updated 2012-03-13 20:49:17.095984)
Review request for Flume.
Changes
-------
Minor fixes, more logging.
Summary
-------
Currently the configuration of each component sits deep inside the component themselves. There is no way to verify if a configuration is valid before run time. In most systems like Flume, it is likely that these confs will be deployed from one single host, on to the machines where flume agents are running. Only when each agent starts, invalid confs are identified because the Agents would terminate by throwing exceptions. This is a first attempt to make a component-aware configuration system which is independent of the Flume, and does not require the Flume jar to be installed. Each component needs a configuration manager, which configures the components.
I have not completed the validation part yet, but have completed the configuration managers for all the components that come in the flume jar. I will add support for the other components, such as HDFS sink, JDBC channel etc soon. I will also add the validation support to the classes, but would like a review of the model being proposed here.
Note that I do know of a couple of bugs in the existing diff(like that if the name of a config key starts with an Upper case letter, an exception is thrown) - I will fix and upload the review here soon.
This addresses bug FLUME-992.
https://issues.apache.org/jira/browse/FLUME-992
Diffs (updated)
-----
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannel.java bca0c50
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProvider.java e445d61
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProviderFactory.java 6fbd6ef
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java 307ae89
flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestJdbcChannelProvider.java 7710d46
flume-ng-clients/flume-ng-log4jappender/src/test/java/org/apache/flume/clients/log4jappender/TestLog4jAppender.java 68d95fb
flume-ng-configuration/pom.xml PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfigurationFactory.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ConfigurationException.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/Context.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationError.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationErrorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MemoryChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MultiplexingChannelSelectorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/PseudoTxnMemoryChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/AvroSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/FailoverSinkProcessorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/RollingFileSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkGroupConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkProcessorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkProcessorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/AvroSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/ExecSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/NetcatSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceType.java PRE-CREATION
flume-ng-configuration/src/test/resources/flume-ng-error-test PRE-CREATION
flume-ng-configuration/src/test/resources/flume-ng-test.conf PRE-CREATION
flume-ng-core/pom.xml d753fa1
flume-ng-core/src/main/java/org/apache/flume/ChannelSelector.java fba2dcb
flume-ng-core/src/main/java/org/apache/flume/Context.java a6341a5
flume-ng-core/src/main/java/org/apache/flume/SinkProcessor.java 11651ad
flume-ng-core/src/main/java/org/apache/flume/SinkProcessorType.java be1891b
flume-ng-core/src/main/java/org/apache/flume/channel/AbstractChannel.java 352bf08
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorFactory.java 800f471
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorType.java 511fc65
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8
flume-ng-core/src/main/java/org/apache/flume/channel/DefaultChannelFactory.java 963a6a3
flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java bfa1fde
flume-ng-core/src/main/java/org/apache/flume/channel/MultiplexingChannelSelector.java 83928b7
flume-ng-core/src/main/java/org/apache/flume/channel/PseudoTxnMemoryChannel.java 489d3e5
flume-ng-core/src/main/java/org/apache/flume/channel/ReplicatingChannelSelector.java 8f22746
flume-ng-core/src/main/java/org/apache/flume/conf/Configurable.java 0fa4839
flume-ng-core/src/main/java/org/apache/flume/conf/ConfigurableComponent.java PRE-CREATION
flume-ng-core/src/main/java/org/apache/flume/conf/Configurables.java 84492e5
flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 7386d06
flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkFactory.java b89dfa0
flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkProcessor.java 257bab3
flume-ng-core/src/main/java/org/apache/flume/sink/FailoverSinkProcessor.java 9f5b856
flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java 7f1d3b3
flume-ng-core/src/main/java/org/apache/flume/sink/SinkGroup.java 2e80a56
flume-ng-core/src/main/java/org/apache/flume/sink/SinkProcessorFactory.java 10f9f4e
flume-ng-core/src/main/java/org/apache/flume/sink/SinkType.java 6b08c09
flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java a3f6640
flume-ng-core/src/main/java/org/apache/flume/source/DefaultSourceFactory.java a610e6f
flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java b6b1181
flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java 94245ac
flume-ng-core/src/main/java/org/apache/flume/source/SourceType.java 9082470
flume-ng-core/src/test/java/org/apache/flume/TestContext.java 51c350f
flume-ng-core/src/test/java/org/apache/flume/channel/AbstractBasicChannelSemanticsTest.java 6e71e46
flume-ng-core/src/test/java/org/apache/flume/channel/MockChannel.java 24b01e2
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannel.java e070864
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelConcurrency.java 8dad0b2
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelTransaction.java bc81f26
flume-ng-core/src/test/java/org/apache/flume/sink/TestAvroSink.java 467785f
flume-ng-core/src/test/java/org/apache/flume/sink/TestFailoverSinkProcessor.java 195c121
flume-ng-core/src/test/java/org/apache/flume/sink/TestLoggerSink.java 92ff6fe
flume-ng-core/src/test/java/org/apache/flume/sink/TestRollingFileSink.java 7e26e2a
flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java c5c3f2f
flume-ng-core/src/test/java/org/apache/flume/source/TestExecSource.java 6035270
flume-ng-core/src/test/java/org/apache/flume/source/TestPollableSourceRunner.java c27f82c
flume-ng-core/src/test/java/org/apache/flume/source/TestSequenceGeneratorSource.java 579b257
flume-ng-legacy-sources/flume-avro-source/src/main/java/org/apache/flume/source/avroLegacy/AvroLegacySource.java 65d9142
flume-ng-legacy-sources/flume-avro-source/src/test/java/org/apache/flume/source/avroLegacy/TestLegacyAvroSource.java 6e3eb53
flume-ng-legacy-sources/flume-thrift-source/src/main/java/org/apache/flume/source/thriftLegacy/ThriftLegacySource.java fbf7362
flume-ng-legacy-sources/flume-thrift-source/src/test/java/org/apache/flume/source/thriftLegacy/TestThriftLegacySource.java ddd9478
flume-ng-node/src/main/java/org/apache/flume/conf/properties/FlumeConfiguration.java d66f6d1
flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java 97f72e1
flume-ng-node/src/test/java/org/apache/flume/node/TestAbstractLogicalNodeManager.java 521b586
flume-ng-node/src/test/java/org/apache/flume/source/TestNetcatSource.java fb2a960
flume-ng-node/src/test/resources/flume-conf.properties 848caca
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 524b69c
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 7d8ee8a
flume-ng-sinks/flume-irc-sink/src/main/java/org/apache/flume/sink/irc/IRCSink.java 8e77218
pom.xml d785762
Diff: https://reviews.apache.org/r/4115/diff
Testing
-------
All existing unit tests for the components whose configuration has now been moved to these stubs pass.
Thanks,
Hari
Re: Review Request: Component aware configuration managers for flume
components
Posted by Hari Shreedharan <hs...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4115/
-----------------------------------------------------------
(Updated 2012-03-13 18:34:28.437390)
Review request for Flume.
Changes
-------
Adding support for context based configuration components
Summary
-------
Currently the configuration of each component sits deep inside the component themselves. There is no way to verify if a configuration is valid before run time. In most systems like Flume, it is likely that these confs will be deployed from one single host, on to the machines where flume agents are running. Only when each agent starts, invalid confs are identified because the Agents would terminate by throwing exceptions. This is a first attempt to make a component-aware configuration system which is independent of the Flume, and does not require the Flume jar to be installed. Each component needs a configuration manager, which configures the components.
I have not completed the validation part yet, but have completed the configuration managers for all the components that come in the flume jar. I will add support for the other components, such as HDFS sink, JDBC channel etc soon. I will also add the validation support to the classes, but would like a review of the model being proposed here.
Note that I do know of a couple of bugs in the existing diff(like that if the name of a config key starts with an Upper case letter, an exception is thrown) - I will fix and upload the review here soon.
This addresses bug FLUME-992.
https://issues.apache.org/jira/browse/FLUME-992
Diffs (updated)
-----
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannel.java bca0c50
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProvider.java e445d61
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProviderFactory.java 6fbd6ef
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java 307ae89
flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestJdbcChannelProvider.java 7710d46
flume-ng-clients/flume-ng-log4jappender/src/test/java/org/apache/flume/clients/log4jappender/TestLog4jAppender.java 68d95fb
flume-ng-configuration/pom.xml PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfigurationFactory.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ConfigurationException.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/Context.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationError.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationErrorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MemoryChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MultiplexingChannelSelectorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/PseudoTxnMemoryChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/AvroSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/FailoverSinkProcessorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/RollingFileSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkGroupConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkProcessorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkProcessorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/AvroSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/ExecSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/NetcatSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceType.java PRE-CREATION
flume-ng-configuration/src/test/resources/flume-ng-error-test PRE-CREATION
flume-ng-configuration/src/test/resources/flume-ng-test.conf PRE-CREATION
flume-ng-core/pom.xml d753fa1
flume-ng-core/src/main/java/org/apache/flume/ChannelSelector.java fba2dcb
flume-ng-core/src/main/java/org/apache/flume/Context.java a6341a5
flume-ng-core/src/main/java/org/apache/flume/SinkProcessor.java 11651ad
flume-ng-core/src/main/java/org/apache/flume/SinkProcessorType.java be1891b
flume-ng-core/src/main/java/org/apache/flume/channel/AbstractChannel.java 352bf08
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorFactory.java 800f471
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorType.java 511fc65
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8
flume-ng-core/src/main/java/org/apache/flume/channel/DefaultChannelFactory.java 963a6a3
flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java bfa1fde
flume-ng-core/src/main/java/org/apache/flume/channel/MultiplexingChannelSelector.java 83928b7
flume-ng-core/src/main/java/org/apache/flume/channel/PseudoTxnMemoryChannel.java 489d3e5
flume-ng-core/src/main/java/org/apache/flume/channel/ReplicatingChannelSelector.java 8f22746
flume-ng-core/src/main/java/org/apache/flume/conf/Configurable.java 0fa4839
flume-ng-core/src/main/java/org/apache/flume/conf/ConfigurableComponent.java PRE-CREATION
flume-ng-core/src/main/java/org/apache/flume/conf/Configurables.java 84492e5
flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 7386d06
flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkFactory.java b89dfa0
flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkProcessor.java 257bab3
flume-ng-core/src/main/java/org/apache/flume/sink/FailoverSinkProcessor.java 9f5b856
flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java 7f1d3b3
flume-ng-core/src/main/java/org/apache/flume/sink/SinkGroup.java 2e80a56
flume-ng-core/src/main/java/org/apache/flume/sink/SinkProcessorFactory.java 10f9f4e
flume-ng-core/src/main/java/org/apache/flume/sink/SinkType.java 6b08c09
flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java a3f6640
flume-ng-core/src/main/java/org/apache/flume/source/DefaultSourceFactory.java a610e6f
flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java b6b1181
flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java 94245ac
flume-ng-core/src/main/java/org/apache/flume/source/SourceType.java 9082470
flume-ng-core/src/test/java/org/apache/flume/TestContext.java 51c350f
flume-ng-core/src/test/java/org/apache/flume/channel/AbstractBasicChannelSemanticsTest.java 6e71e46
flume-ng-core/src/test/java/org/apache/flume/channel/MockChannel.java 24b01e2
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannel.java e070864
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelConcurrency.java 8dad0b2
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelTransaction.java bc81f26
flume-ng-core/src/test/java/org/apache/flume/sink/TestAvroSink.java 467785f
flume-ng-core/src/test/java/org/apache/flume/sink/TestFailoverSinkProcessor.java 195c121
flume-ng-core/src/test/java/org/apache/flume/sink/TestLoggerSink.java 92ff6fe
flume-ng-core/src/test/java/org/apache/flume/sink/TestRollingFileSink.java 7e26e2a
flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java c5c3f2f
flume-ng-core/src/test/java/org/apache/flume/source/TestExecSource.java 6035270
flume-ng-core/src/test/java/org/apache/flume/source/TestPollableSourceRunner.java c27f82c
flume-ng-core/src/test/java/org/apache/flume/source/TestSequenceGeneratorSource.java 579b257
flume-ng-legacy-sources/flume-avro-source/src/main/java/org/apache/flume/source/avroLegacy/AvroLegacySource.java 65d9142
flume-ng-legacy-sources/flume-avro-source/src/test/java/org/apache/flume/source/avroLegacy/TestLegacyAvroSource.java 6e3eb53
flume-ng-legacy-sources/flume-thrift-source/src/main/java/org/apache/flume/source/thriftLegacy/ThriftLegacySource.java fbf7362
flume-ng-legacy-sources/flume-thrift-source/src/test/java/org/apache/flume/source/thriftLegacy/TestThriftLegacySource.java ddd9478
flume-ng-node/src/main/java/org/apache/flume/conf/properties/FlumeConfiguration.java d66f6d1
flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java 97f72e1
flume-ng-node/src/test/java/org/apache/flume/node/TestAbstractLogicalNodeManager.java 521b586
flume-ng-node/src/test/java/org/apache/flume/source/TestNetcatSource.java fb2a960
flume-ng-node/src/test/resources/flume-conf.properties 848caca
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 524b69c
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 7d8ee8a
flume-ng-sinks/flume-irc-sink/src/main/java/org/apache/flume/sink/irc/IRCSink.java 8e77218
pom.xml d785762
Diff: https://reviews.apache.org/r/4115/diff
Testing
-------
All existing unit tests for the components whose configuration has now been moved to these stubs pass.
Thanks,
Hari
Re: Review Request: Component aware configuration managers for flume
components
Posted by Hari Shreedharan <hs...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4115/
-----------------------------------------------------------
(Updated 2012-03-13 07:45:16.395171)
Review request for Flume.
Changes
-------
Incorporate much of the feedback. Errors now pushed up to higher level. Need to discuss plan to test this, and then will add tests - will need more system tests, rather than unit tests.
Summary
-------
Currently the configuration of each component sits deep inside the component themselves. There is no way to verify if a configuration is valid before run time. In most systems like Flume, it is likely that these confs will be deployed from one single host, on to the machines where flume agents are running. Only when each agent starts, invalid confs are identified because the Agents would terminate by throwing exceptions. This is a first attempt to make a component-aware configuration system which is independent of the Flume, and does not require the Flume jar to be installed. Each component needs a configuration manager, which configures the components.
I have not completed the validation part yet, but have completed the configuration managers for all the components that come in the flume jar. I will add support for the other components, such as HDFS sink, JDBC channel etc soon. I will also add the validation support to the classes, but would like a review of the model being proposed here.
Note that I do know of a couple of bugs in the existing diff(like that if the name of a config key starts with an Upper case letter, an exception is thrown) - I will fix and upload the review here soon.
This addresses bug FLUME-992.
https://issues.apache.org/jira/browse/FLUME-992
Diffs (updated)
-----
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannel.java bca0c50
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProvider.java e445d61
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProviderFactory.java 6fbd6ef
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java 307ae89
flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestJdbcChannelProvider.java 7710d46
flume-ng-clients/flume-ng-log4jappender/src/test/java/org/apache/flume/clients/log4jappender/TestLog4jAppender.java 68d95fb
flume-ng-configuration/pom.xml PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfigurationFactory.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ConfigurationException.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/Context.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationError.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationErrorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MemoryChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MultiplexingChannelSelectorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/PseudoTxnMemoryChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/AvroSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/FailoverSinkProcessorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/RollingFileSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkGroupConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkProcessorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkProcessorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/AvroSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/ExecSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/NetcatSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceType.java PRE-CREATION
flume-ng-configuration/src/test/resources/flume-ng-error-test PRE-CREATION
flume-ng-configuration/src/test/resources/flume-ng-test.conf PRE-CREATION
flume-ng-core/pom.xml d753fa1
flume-ng-core/src/main/java/org/apache/flume/ChannelSelector.java fba2dcb
flume-ng-core/src/main/java/org/apache/flume/Context.java a6341a5
flume-ng-core/src/main/java/org/apache/flume/SinkProcessor.java 11651ad
flume-ng-core/src/main/java/org/apache/flume/SinkProcessorType.java be1891b
flume-ng-core/src/main/java/org/apache/flume/channel/AbstractChannel.java 352bf08
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorFactory.java 800f471
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorType.java 511fc65
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8
flume-ng-core/src/main/java/org/apache/flume/channel/DefaultChannelFactory.java 963a6a3
flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java bfa1fde
flume-ng-core/src/main/java/org/apache/flume/channel/MultiplexingChannelSelector.java 83928b7
flume-ng-core/src/main/java/org/apache/flume/channel/PseudoTxnMemoryChannel.java 489d3e5
flume-ng-core/src/main/java/org/apache/flume/channel/ReplicatingChannelSelector.java 8f22746
flume-ng-core/src/main/java/org/apache/flume/conf/Configurable.java 0fa4839
flume-ng-core/src/main/java/org/apache/flume/conf/ConfigurableComponent.java PRE-CREATION
flume-ng-core/src/main/java/org/apache/flume/conf/Configurables.java 84492e5
flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 7386d06
flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkFactory.java b89dfa0
flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkProcessor.java 257bab3
flume-ng-core/src/main/java/org/apache/flume/sink/FailoverSinkProcessor.java 9f5b856
flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java 7f1d3b3
flume-ng-core/src/main/java/org/apache/flume/sink/SinkGroup.java 2e80a56
flume-ng-core/src/main/java/org/apache/flume/sink/SinkProcessorFactory.java 10f9f4e
flume-ng-core/src/main/java/org/apache/flume/sink/SinkType.java 6b08c09
flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java a3f6640
flume-ng-core/src/main/java/org/apache/flume/source/DefaultSourceFactory.java a610e6f
flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java b6b1181
flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java 94245ac
flume-ng-core/src/main/java/org/apache/flume/source/SourceType.java 9082470
flume-ng-core/src/test/java/org/apache/flume/TestContext.java 51c350f
flume-ng-core/src/test/java/org/apache/flume/channel/AbstractBasicChannelSemanticsTest.java 6e71e46
flume-ng-core/src/test/java/org/apache/flume/channel/MockChannel.java 24b01e2
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannel.java e070864
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelConcurrency.java 8dad0b2
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelTransaction.java bc81f26
flume-ng-core/src/test/java/org/apache/flume/sink/TestAvroSink.java 467785f
flume-ng-core/src/test/java/org/apache/flume/sink/TestFailoverSinkProcessor.java 195c121
flume-ng-core/src/test/java/org/apache/flume/sink/TestLoggerSink.java 92ff6fe
flume-ng-core/src/test/java/org/apache/flume/sink/TestRollingFileSink.java 7e26e2a
flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java c5c3f2f
flume-ng-core/src/test/java/org/apache/flume/source/TestExecSource.java 6035270
flume-ng-core/src/test/java/org/apache/flume/source/TestPollableSourceRunner.java c27f82c
flume-ng-core/src/test/java/org/apache/flume/source/TestSequenceGeneratorSource.java 579b257
flume-ng-legacy-sources/flume-avro-source/src/main/java/org/apache/flume/source/avroLegacy/AvroLegacySource.java 65d9142
flume-ng-legacy-sources/flume-avro-source/src/test/java/org/apache/flume/source/avroLegacy/TestLegacyAvroSource.java 6e3eb53
flume-ng-legacy-sources/flume-thrift-source/src/main/java/org/apache/flume/source/thriftLegacy/ThriftLegacySource.java fbf7362
flume-ng-legacy-sources/flume-thrift-source/src/test/java/org/apache/flume/source/thriftLegacy/TestThriftLegacySource.java ddd9478
flume-ng-node/src/main/java/org/apache/flume/conf/properties/FlumeConfiguration.java d66f6d1
flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java 97f72e1
flume-ng-node/src/test/java/org/apache/flume/node/TestAbstractLogicalNodeManager.java 521b586
flume-ng-node/src/test/java/org/apache/flume/source/TestNetcatSource.java fb2a960
flume-ng-node/src/test/resources/flume-conf.properties 848caca
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 524b69c
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 7d8ee8a
flume-ng-sinks/flume-irc-sink/src/main/java/org/apache/flume/sink/irc/IRCSink.java 8e77218
pom.xml d785762
Diff: https://reviews.apache.org/r/4115/diff
Testing
-------
All existing unit tests for the components whose configuration has now been moved to these stubs pass.
Thanks,
Hari
Re: Review Request: Component aware configuration managers for flume
components
Posted by Hari Shreedharan <hs...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4115/
-----------------------------------------------------------
(Updated 2012-03-09 08:47:00.765150)
Review request for Flume.
Changes
-------
Unit tests pass. End-to-end testing to be done. Will update patch if needed.
Summary
-------
Currently the configuration of each component sits deep inside the component themselves. There is no way to verify if a configuration is valid before run time. In most systems like Flume, it is likely that these confs will be deployed from one single host, on to the machines where flume agents are running. Only when each agent starts, invalid confs are identified because the Agents would terminate by throwing exceptions. This is a first attempt to make a component-aware configuration system which is independent of the Flume, and does not require the Flume jar to be installed. Each component needs a configuration manager, which configures the components.
I have not completed the validation part yet, but have completed the configuration managers for all the components that come in the flume jar. I will add support for the other components, such as HDFS sink, JDBC channel etc soon. I will also add the validation support to the classes, but would like a review of the model being proposed here.
Note that I do know of a couple of bugs in the existing diff(like that if the name of a config key starts with an Upper case letter, an exception is thrown) - I will fix and upload the review here soon.
This addresses bug FLUME-992.
https://issues.apache.org/jira/browse/FLUME-992
Diffs (updated)
-----
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/NetcatSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/ExecSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/AvroSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkProcessorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkProcessorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkGroupConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/RollingFileSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationError.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationErrorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MemoryChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MultiplexingChannelSelectorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/PseudoTxnMemoryChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/AvroSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/FailoverSinkProcessorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/Context.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ConfigurationException.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfigurationFactory.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfiguration.java PRE-CREATION
flume-ng-clients/flume-ng-log4jappender/src/test/java/org/apache/flume/clients/log4jappender/TestLog4jAppender.java 68d95fb
flume-ng-configuration/pom.xml PRE-CREATION
flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestJdbcChannelProvider.java 7710d46
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannel.java bca0c50
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProvider.java e445d61
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProviderFactory.java 6fbd6ef
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java 307ae89
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceType.java PRE-CREATION
flume-ng-configuration/src/test/resources/flume-ng-error-test PRE-CREATION
flume-ng-configuration/src/test/resources/flume-ng-test.conf PRE-CREATION
flume-ng-core/pom.xml d753fa1
flume-ng-core/src/main/java/org/apache/flume/ChannelSelector.java fba2dcb
flume-ng-core/src/main/java/org/apache/flume/Context.java a6341a5
flume-ng-core/src/main/java/org/apache/flume/SinkProcessor.java 11651ad
flume-ng-core/src/main/java/org/apache/flume/SinkProcessorType.java be1891b
flume-ng-core/src/main/java/org/apache/flume/channel/AbstractChannel.java 352bf08
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorFactory.java 800f471
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorType.java 511fc65
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8
flume-ng-core/src/main/java/org/apache/flume/channel/DefaultChannelFactory.java 963a6a3
flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java bfa1fde
flume-ng-core/src/main/java/org/apache/flume/channel/MultiplexingChannelSelector.java 83928b7
flume-ng-core/src/main/java/org/apache/flume/channel/PseudoTxnMemoryChannel.java 489d3e5
flume-ng-core/src/main/java/org/apache/flume/channel/ReplicatingChannelSelector.java 8f22746
flume-ng-core/src/main/java/org/apache/flume/conf/Configurable.java 0fa4839
flume-ng-core/src/main/java/org/apache/flume/conf/ConfigurableComponent.java PRE-CREATION
flume-ng-core/src/main/java/org/apache/flume/conf/Configurables.java 84492e5
flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 7386d06
flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkFactory.java b89dfa0
flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkProcessor.java 257bab3
flume-ng-core/src/main/java/org/apache/flume/sink/FailoverSinkProcessor.java 9f5b856
flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java 7f1d3b3
flume-ng-core/src/main/java/org/apache/flume/sink/SinkGroup.java 2e80a56
flume-ng-core/src/main/java/org/apache/flume/sink/SinkProcessorFactory.java 10f9f4e
flume-ng-core/src/main/java/org/apache/flume/sink/SinkType.java 6b08c09
flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java a3f6640
flume-ng-core/src/main/java/org/apache/flume/source/DefaultSourceFactory.java a610e6f
flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java b6b1181
flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java 94245ac
flume-ng-core/src/main/java/org/apache/flume/source/SourceType.java 9082470
flume-ng-core/src/test/java/org/apache/flume/TestContext.java 51c350f
flume-ng-core/src/test/java/org/apache/flume/channel/AbstractBasicChannelSemanticsTest.java 6e71e46
flume-ng-core/src/test/java/org/apache/flume/channel/MockChannel.java 24b01e2
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannel.java e070864
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelConcurrency.java 8dad0b2
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelTransaction.java bc81f26
flume-ng-core/src/test/java/org/apache/flume/sink/TestAvroSink.java 467785f
flume-ng-core/src/test/java/org/apache/flume/sink/TestFailoverSinkProcessor.java 195c121
flume-ng-core/src/test/java/org/apache/flume/sink/TestLoggerSink.java 92ff6fe
flume-ng-core/src/test/java/org/apache/flume/sink/TestRollingFileSink.java 7e26e2a
flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java c5c3f2f
flume-ng-core/src/test/java/org/apache/flume/source/TestExecSource.java 6035270
flume-ng-core/src/test/java/org/apache/flume/source/TestPollableSourceRunner.java c27f82c
flume-ng-core/src/test/java/org/apache/flume/source/TestSequenceGeneratorSource.java 579b257
flume-ng-legacy-sources/flume-avro-source/src/main/java/org/apache/flume/source/avroLegacy/AvroLegacySource.java 65d9142
flume-ng-legacy-sources/flume-avro-source/src/test/java/org/apache/flume/source/avroLegacy/TestLegacyAvroSource.java 6e3eb53
flume-ng-legacy-sources/flume-thrift-source/src/main/java/org/apache/flume/source/thriftLegacy/ThriftLegacySource.java fbf7362
flume-ng-legacy-sources/flume-thrift-source/src/test/java/org/apache/flume/source/thriftLegacy/TestThriftLegacySource.java ddd9478
flume-ng-node/src/main/java/org/apache/flume/conf/properties/FlumeConfiguration.java d66f6d1
flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java 97f72e1
flume-ng-node/src/test/java/org/apache/flume/node/TestAbstractLogicalNodeManager.java 521b586
flume-ng-node/src/test/java/org/apache/flume/source/TestNetcatSource.java fb2a960
flume-ng-node/src/test/resources/flume-conf.properties 848caca
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 524b69c
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 7d8ee8a
flume-ng-sinks/flume-irc-sink/src/main/java/org/apache/flume/sink/irc/IRCSink.java 8e77218
pom.xml d785762
Diff: https://reviews.apache.org/r/4115/diff
Testing
-------
All existing unit tests for the components whose configuration has now been moved to these stubs pass.
Thanks,
Hari
Re: Review Request: Component aware configuration managers for flume
components
Posted by Hari Shreedharan <hs...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4115/
-----------------------------------------------------------
(Updated 2012-03-09 01:04:54.371110)
Review request for Flume.
Changes
-------
Updated licenses.
Summary
-------
Currently the configuration of each component sits deep inside the component themselves. There is no way to verify if a configuration is valid before run time. In most systems like Flume, it is likely that these confs will be deployed from one single host, on to the machines where flume agents are running. Only when each agent starts, invalid confs are identified because the Agents would terminate by throwing exceptions. This is a first attempt to make a component-aware configuration system which is independent of the Flume, and does not require the Flume jar to be installed. Each component needs a configuration manager, which configures the components.
I have not completed the validation part yet, but have completed the configuration managers for all the components that come in the flume jar. I will add support for the other components, such as HDFS sink, JDBC channel etc soon. I will also add the validation support to the classes, but would like a review of the model being proposed here.
Note that I do know of a couple of bugs in the existing diff(like that if the name of a config key starts with an Upper case letter, an exception is thrown) - I will fix and upload the review here soon.
This addresses bug FLUME-992.
https://issues.apache.org/jira/browse/FLUME-992
Diffs (updated)
-----
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannel.java bca0c50
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProvider.java e445d61
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProviderFactory.java 6fbd6ef
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java 307ae89
flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestJdbcChannelProvider.java 7710d46
flume-ng-clients/flume-ng-log4jappender/src/test/java/org/apache/flume/clients/log4jappender/TestLog4jAppender.java 68d95fb
flume-ng-configuration/pom.xml PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfigurationFactory.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ConfigurationException.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/Context.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationError.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationErrorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MemoryChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MultiplexingChannelSelectorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/PseudoTxnMemoryChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/AvroSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/FailoverSinkProcessorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/RollingFileSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkGroupConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkProcessorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkProcessorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/AvroSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/ExecSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/NetcatSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceType.java PRE-CREATION
flume-ng-configuration/src/test/resources/flume-ng-error-test PRE-CREATION
flume-ng-configuration/src/test/resources/flume-ng-test.conf PRE-CREATION
flume-ng-core/pom.xml d753fa1
flume-ng-core/src/main/java/org/apache/flume/ChannelSelector.java fba2dcb
flume-ng-core/src/main/java/org/apache/flume/Context.java a6341a5
flume-ng-core/src/main/java/org/apache/flume/SinkProcessor.java 11651ad
flume-ng-core/src/main/java/org/apache/flume/SinkProcessorType.java be1891b
flume-ng-core/src/main/java/org/apache/flume/channel/AbstractChannel.java 352bf08
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorFactory.java 800f471
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorType.java 511fc65
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8
flume-ng-core/src/main/java/org/apache/flume/channel/DefaultChannelFactory.java 963a6a3
flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java bfa1fde
flume-ng-core/src/main/java/org/apache/flume/channel/MultiplexingChannelSelector.java 83928b7
flume-ng-core/src/main/java/org/apache/flume/channel/PseudoTxnMemoryChannel.java 489d3e5
flume-ng-core/src/main/java/org/apache/flume/channel/ReplicatingChannelSelector.java 8f22746
flume-ng-core/src/main/java/org/apache/flume/conf/Configurable.java 0fa4839
flume-ng-core/src/main/java/org/apache/flume/conf/ConfigurableComponent.java PRE-CREATION
flume-ng-core/src/main/java/org/apache/flume/conf/Configurables.java 84492e5
flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 7386d06
flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkFactory.java b89dfa0
flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkProcessor.java 257bab3
flume-ng-core/src/main/java/org/apache/flume/sink/FailoverSinkProcessor.java 9f5b856
flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java 7f1d3b3
flume-ng-core/src/main/java/org/apache/flume/sink/SinkGroup.java 2e80a56
flume-ng-core/src/main/java/org/apache/flume/sink/SinkProcessorFactory.java 10f9f4e
flume-ng-core/src/main/java/org/apache/flume/sink/SinkType.java 6b08c09
flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java a3f6640
flume-ng-core/src/main/java/org/apache/flume/source/DefaultSourceFactory.java a610e6f
flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java b6b1181
flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java 94245ac
flume-ng-core/src/main/java/org/apache/flume/source/SourceType.java 9082470
flume-ng-core/src/test/java/org/apache/flume/TestContext.java 51c350f
flume-ng-core/src/test/java/org/apache/flume/channel/AbstractBasicChannelSemanticsTest.java 6e71e46
flume-ng-core/src/test/java/org/apache/flume/channel/MockChannel.java 24b01e2
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannel.java e070864
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelConcurrency.java 8dad0b2
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelTransaction.java bc81f26
flume-ng-core/src/test/java/org/apache/flume/sink/TestAvroSink.java 467785f
flume-ng-core/src/test/java/org/apache/flume/sink/TestFailoverSinkProcessor.java 195c121
flume-ng-core/src/test/java/org/apache/flume/sink/TestLoggerSink.java 92ff6fe
flume-ng-core/src/test/java/org/apache/flume/sink/TestRollingFileSink.java 7e26e2a
flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java c5c3f2f
flume-ng-core/src/test/java/org/apache/flume/source/TestExecSource.java 6035270
flume-ng-core/src/test/java/org/apache/flume/source/TestPollableSourceRunner.java c27f82c
flume-ng-core/src/test/java/org/apache/flume/source/TestSequenceGeneratorSource.java 579b257
flume-ng-legacy-sources/flume-avro-source/src/main/java/org/apache/flume/source/avroLegacy/AvroLegacySource.java 65d9142
flume-ng-legacy-sources/flume-avro-source/src/test/java/org/apache/flume/source/avroLegacy/TestLegacyAvroSource.java 6e3eb53
flume-ng-legacy-sources/flume-thrift-source/src/main/java/org/apache/flume/source/thriftLegacy/ThriftLegacySource.java fbf7362
flume-ng-legacy-sources/flume-thrift-source/src/test/java/org/apache/flume/source/thriftLegacy/TestThriftLegacySource.java ddd9478
flume-ng-node/src/main/java/org/apache/flume/conf/properties/FlumeConfiguration.java d66f6d1
flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java 97f72e1
flume-ng-node/src/test/java/org/apache/flume/node/TestAbstractLogicalNodeManager.java 521b586
flume-ng-node/src/test/java/org/apache/flume/source/TestNetcatSource.java fb2a960
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 524b69c
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 7d8ee8a
flume-ng-sinks/flume-irc-sink/src/main/java/org/apache/flume/sink/irc/IRCSink.java 8e77218
pom.xml d785762
Diff: https://reviews.apache.org/r/4115/diff
Testing
-------
All existing unit tests for the components whose configuration has now been moved to these stubs pass.
Thanks,
Hari
Re: Review Request: Component aware configuration managers for flume
components
Posted by Hari Shreedharan <hs...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4115/
-----------------------------------------------------------
(Updated 2012-03-09 00:48:11.972265)
Review request for Flume.
Changes
-------
This is being submitted for early review. It builds, but is not based off of the latest commits. Currently it is not tested yet.
I am requesting a review of the new configuration model, rather than the code itself. I am sure there are a ton of fixes we can make to this.
Not all components have been updated(like JdbcChannel, HDFS and IRC sinks have not been updated yet). There probably are several bugs which I will iron out once I start testing the changes rigorously, but please do point out major logical bugs you may find.
Summary
-------
Currently the configuration of each component sits deep inside the component themselves. There is no way to verify if a configuration is valid before run time. In most systems like Flume, it is likely that these confs will be deployed from one single host, on to the machines where flume agents are running. Only when each agent starts, invalid confs are identified because the Agents would terminate by throwing exceptions. This is a first attempt to make a component-aware configuration system which is independent of the Flume, and does not require the Flume jar to be installed. Each component needs a configuration manager, which configures the components.
I have not completed the validation part yet, but have completed the configuration managers for all the components that come in the flume jar. I will add support for the other components, such as HDFS sink, JDBC channel etc soon. I will also add the validation support to the classes, but would like a review of the model being proposed here.
Note that I do know of a couple of bugs in the existing diff(like that if the name of a config key starts with an Upper case letter, an exception is thrown) - I will fix and upload the review here soon.
This addresses bug FLUME-992.
https://issues.apache.org/jira/browse/FLUME-992
Diffs (updated)
-----
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannel.java bca0c50
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProvider.java e445d61
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProviderFactory.java 6fbd6ef
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java 307ae89
flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestJdbcChannelProvider.java 7710d46
flume-ng-clients/flume-ng-log4jappender/src/test/java/org/apache/flume/clients/log4jappender/TestLog4jAppender.java 68d95fb
flume-ng-configuration/pom.xml PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfigurationFactory.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ConfigurationException.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/Context.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationError.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationErrorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MemoryChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MultiplexingChannelSelectorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/PseudoTxnMemoryChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/AvroSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/FailoverSinkProcessorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/RollingFileSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkGroupConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkProcessorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkProcessorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/AvroSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/ExecSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/NetcatSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceType.java PRE-CREATION
flume-ng-configuration/src/test/resources/flume-ng-error-test PRE-CREATION
flume-ng-configuration/src/test/resources/flume-ng-test.conf PRE-CREATION
flume-ng-core/pom.xml d753fa1
flume-ng-core/src/main/java/org/apache/flume/ChannelSelector.java fba2dcb
flume-ng-core/src/main/java/org/apache/flume/Context.java a6341a5
flume-ng-core/src/main/java/org/apache/flume/SinkProcessor.java 11651ad
flume-ng-core/src/main/java/org/apache/flume/SinkProcessorType.java be1891b
flume-ng-core/src/main/java/org/apache/flume/channel/AbstractChannel.java 352bf08
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorFactory.java 800f471
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8
flume-ng-core/src/main/java/org/apache/flume/channel/DefaultChannelFactory.java 963a6a3
flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java bfa1fde
flume-ng-core/src/main/java/org/apache/flume/channel/MultiplexingChannelSelector.java 83928b7
flume-ng-core/src/main/java/org/apache/flume/channel/PseudoTxnMemoryChannel.java 489d3e5
flume-ng-core/src/main/java/org/apache/flume/channel/ReplicatingChannelSelector.java 8f22746
flume-ng-core/src/main/java/org/apache/flume/conf/Configurable.java 0fa4839
flume-ng-core/src/main/java/org/apache/flume/conf/ConfigurableComponent.java PRE-CREATION
flume-ng-core/src/main/java/org/apache/flume/conf/Configurables.java 84492e5
flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 7386d06
flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkFactory.java b89dfa0
flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkProcessor.java 257bab3
flume-ng-core/src/main/java/org/apache/flume/sink/FailoverSinkProcessor.java 9f5b856
flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java 7f1d3b3
flume-ng-core/src/main/java/org/apache/flume/sink/SinkGroup.java 2e80a56
flume-ng-core/src/main/java/org/apache/flume/sink/SinkProcessorFactory.java 10f9f4e
flume-ng-core/src/main/java/org/apache/flume/sink/SinkType.java 6b08c09
flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java a3f6640
flume-ng-core/src/main/java/org/apache/flume/source/DefaultSourceFactory.java a610e6f
flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java b6b1181
flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java 94245ac
flume-ng-core/src/main/java/org/apache/flume/source/SourceType.java 9082470
flume-ng-core/src/test/java/org/apache/flume/TestContext.java 51c350f
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannel.java e070864
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelConcurrency.java 8dad0b2
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelTransaction.java bc81f26
flume-ng-core/src/test/java/org/apache/flume/sink/TestAvroSink.java 467785f
flume-ng-core/src/test/java/org/apache/flume/sink/TestFailoverSinkProcessor.java 195c121
flume-ng-core/src/test/java/org/apache/flume/sink/TestLoggerSink.java 92ff6fe
flume-ng-core/src/test/java/org/apache/flume/sink/TestRollingFileSink.java 7e26e2a
flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java c5c3f2f
flume-ng-core/src/test/java/org/apache/flume/source/TestExecSource.java 6035270
flume-ng-core/src/test/java/org/apache/flume/source/TestPollableSourceRunner.java c27f82c
flume-ng-core/src/test/java/org/apache/flume/source/TestSequenceGeneratorSource.java 579b257
flume-ng-legacy-sources/flume-avro-source/src/main/java/org/apache/flume/source/avroLegacy/AvroLegacySource.java 65d9142
flume-ng-legacy-sources/flume-avro-source/src/test/java/org/apache/flume/source/avroLegacy/TestLegacyAvroSource.java 6e3eb53
flume-ng-legacy-sources/flume-thrift-source/src/main/java/org/apache/flume/source/thriftLegacy/ThriftLegacySource.java fbf7362
flume-ng-legacy-sources/flume-thrift-source/src/test/java/org/apache/flume/source/thriftLegacy/TestThriftLegacySource.java ddd9478
flume-ng-node/src/main/java/org/apache/flume/conf/properties/FlumeConfiguration.java d66f6d1
flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java 97f72e1
flume-ng-node/src/test/java/org/apache/flume/node/TestAbstractLogicalNodeManager.java 521b586
flume-ng-node/src/test/java/org/apache/flume/source/TestNetcatSource.java fb2a960
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 524b69c
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 7d8ee8a
flume-ng-sinks/flume-irc-sink/src/main/java/org/apache/flume/sink/irc/IRCSink.java 8e77218
pom.xml d785762
Diff: https://reviews.apache.org/r/4115/diff
Testing
-------
All existing unit tests for the components whose configuration has now been moved to these stubs pass.
Thanks,
Hari
Re: Review Request: Component aware configuration managers for flume components
Posted by Hari Shreedharan <hs...@cloudera.com>.
I did add a way to return the errors from FlumeConfiguration, which were
just being logged, there was no change in functionality.
On Fri, Mar 2, 2012 at 11:55 PM, Hari Shreedharan <hshreedharan@cloudera.com
> wrote:
>
>
> On Fri, Mar 2, 2012 at 5:54 PM, Eric Sammer <es...@cloudera.com> wrote:
>
>> I just read the diff and I strongly object to it for the following
>> reasons:
>>
>> * It's *terribly* overcomplicated / overengineered. You're very close to
>> having reimplemented Spring bean configuration (without strict DI).
>>
> - Yes, it is a bit complex, but if we need to make sure configuration is
> validated without having access to the components themselves, the
> properties of the components have to be separated out into another class. I
> am making changes to make it simpler, removing reflection from the
> validation logic which was used to remove the code repetition.
>
>
>> * There's significant logic in constructors (a code smell).
>>
> This was a first cut, I am moving the logic into to other methods.
>
> * Configuration is now highly coupled to the storage format (java
>> properties)
>>
> I am not sure how this is different from the current implementation. I did
> not add any of the Context methods or FlumeConfiguration classes (all I did
> in this patch was move it to the new package) - these are already based on
> Properties..
>
> Please see:
> http://svn.apache.org/viewvc/incubator/flume/branches/flume-728/flume-ng-node/src/main/java/org/apache/flume/conf/properties/and
> http://svn.apache.org/viewvc/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/Context.java?revision=1294969&view=markup
>
> * FlumeConfiguration has become a God Object.
>>
> No changes were made to FlumeConfiguration at all. The only thing I did
> was to move it from the current package to the conf package. This is the
> exact same code as it is currently. It was already a god object.
>
>
>> * A bunch of hoops are jumped to remain type safe, and then the rest of
>> the code breaks all type safety by stringifying a ton of stuff. I'm not
>> specifically against dynamic techniques or reflection, but I'd like to see
>> it only when there's a true need and it adds real value.
>>
> Can you give me an example of the stringification that you have mentioned?
> I did not get this comment. If you are talking about reflection to convert
> the data into the correct type: Since the data would be read in through
> Context and it is type safe and since we use the same type it should not
> cause an issue. Yes, this can break if the classes are specified in
> correctly, but that is likely due to a bug. Either way, I have removed the
> reflection from the validation logic, as performance can be bad, but I
> don't specifically see a problem with it, as long as the code is correct.
>
> I am moving the properties for each component into the Conf classes(only
> the properties which are read from a configuration file or configured in
> some other way), from where the component reads this data using getters(To
> avoid 2 copies of the properties, they will not be duplicated in the
> components themselves).
> This will allow the validation to happen without having access to
> components.
>
> If you feel separating out the configured properties from the component
> is not a good idea, I am open to any suggestions on how to do the
> validation without actually being dependent on flume-core.
>
> Thanks
> Hari
>
>
>> On Thu, Mar 1, 2012 at 1:35 PM, Hari Shreedharan <
>> hshreedharan@cloudera.com> wrote:
>>
>>>
>>> -----------------------------------------------------------
>>> This is an automatically generated e-mail. To reply, visit:
>>> https://reviews.apache.org/r/4115/
>>> -----------------------------------------------------------
>>>
>>> (Updated 2012-03-01 21:35:41.679397)
>>>
>>>
>>> Review request for Flume.
>>>
>>>
>>> Changes
>>> -------
>>>
>>> Note that I do know of a couple of bugs in the existing diff(like that
>>> if the name of a config key starts with an Upper case letter, an exception
>>> is thrown) - I will fix and upload the review here soon.
>>>
>>>
>>> Summary (updated)
>>> -------
>>>
>>> Currently the configuration of each component sits deep inside the
>>> component themselves. There is no way to verify if a configuration is valid
>>> before run time. In most systems like Flume, it is likely that these confs
>>> will be deployed from one single host, on to the machines where flume
>>> agents are running. Only when each agent starts, invalid confs are
>>> identified because the Agents would terminate by throwing exceptions. This
>>> is a first attempt to make a component-aware configuration system which is
>>> independent of the Flume, and does not require the Flume jar to be
>>> installed. Each component needs a configuration manager, which configures
>>> the components.
>>>
>>> I have not completed the validation part yet, but have completed the
>>> configuration managers for all the components that come in the flume jar. I
>>> will add support for the other components, such as HDFS sink, JDBC channel
>>> etc soon. I will also add the validation support to the classes, but would
>>> like a review of the model being proposed here.
>>>
>>> Note that I do know of a couple of bugs in the existing diff(like that
>>> if the name of a config key starts with an Upper case letter, an exception
>>> is thrown) - I will fix and upload the review here soon.
>>>
>>>
>>> This addresses bug FLUME-992.
>>> https://issues.apache.org/jira/browse/FLUME-992
>>>
>>>
>>> Diffs
>>> -----
>>>
>>> flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannel.java
>>> bca0c50
>>> flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProvider.java
>>> e445d61
>>> flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProviderFactory.java
>>> 6fbd6ef
>>> flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java
>>> 1cf1c0c
>>> flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestJdbcChannelProvider.java
>>> 7710d46
>>> flume-ng-clients/flume-ng-log4jappender/src/test/java/org/apache/flume/clients/log4jappender/TestLog4jAppender.java
>>> 68d95fb
>>> flume-ng-configuration/pom.xml PRE-CREATION
>>> flume-ng-configuration/src/main/java/org/apache/flume/conf/AbstractComponentConfiguration.java
>>> PRE-CREATION
>>> flume-ng-configuration/src/main/java/org/apache/flume/conf/ConfigurationException.java
>>> PRE-CREATION
>>> flume-ng-configuration/src/main/java/org/apache/flume/conf/Context.java
>>> PRE-CREATION
>>> flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfiguration.java
>>> PRE-CREATION
>>> flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationError.java
>>> PRE-CREATION
>>> flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationErrorType.java
>>> PRE-CREATION
>>> flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MemoryChannelConfiguration.java
>>> PRE-CREATION
>>> flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MultiplexingChannelSelectorConfiguration.java
>>> PRE-CREATION
>>> flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/PseudoTxnMemoryChannelConfiguration.java
>>> PRE-CREATION
>>> flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/AvroSinkConfiguration.java
>>> PRE-CREATION
>>> flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/FailoverSinkProcessorConfiguration.java
>>> PRE-CREATION
>>> flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/RollingFileSinkConfiguration.java
>>> PRE-CREATION
>>> flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkGroupConfiguration.java
>>> PRE-CREATION
>>> flume-ng-configuration/src/main/java/org/apache/flume/conf/source/AvroSourceConfiguration.java
>>> PRE-CREATION
>>> flume-ng-configuration/src/main/java/org/apache/flume/conf/source/ExecSourceConfiguration.java
>>> PRE-CREATION
>>> flume-ng-configuration/src/main/java/org/apache/flume/conf/source/NetcatSourceConfiguration.java
>>> PRE-CREATION
>>> flume-ng-configuration/src/test/resources/flume-ng-error-test
>>> PRE-CREATION
>>> flume-ng-configuration/src/test/resources/flume-ng-test.conf
>>> PRE-CREATION
>>> flume-ng-core/pom.xml d753fa1
>>> flume-ng-core/src/main/java/org/apache/flume/Context.java f1c8f85
>>> flume-ng-core/src/main/java/org/apache/flume/channel/AbstractChannel.java
>>> 352bf08
>>> flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorFactory.java
>>> a7d5f94
>>> flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java
>>> e79490e
>>> flume-ng-core/src/main/java/org/apache/flume/channel/MultiplexingChannelSelector.java
>>> c63d0a1
>>> flume-ng-core/src/main/java/org/apache/flume/channel/PseudoTxnMemoryChannel.java
>>> 1df580e
>>> flume-ng-core/src/main/java/org/apache/flume/channel/ReplicatingChannelSelector.java
>>> 8f22746
>>> flume-ng-core/src/main/java/org/apache/flume/conf/Configurable.java
>>> 0fa4839
>>> flume-ng-core/src/main/java/org/apache/flume/conf/Configurables.java
>>> 84492e5
>>> flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 5440631
>>> flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkProcessor.java
>>> 257bab3
>>> flume-ng-core/src/main/java/org/apache/flume/sink/FailoverSinkProcessor.java
>>> 9f5b856
>>> flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java
>>> 45c031d
>>> flume-ng-core/src/main/java/org/apache/flume/sink/SinkGroup.java 1adc5ff
>>> flume-ng-core/src/main/java/org/apache/flume/sink/SinkProcessorFactory.java
>>> 859f4fd
>>> flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java
>>> 7b079f9
>>> flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java
>>> a96016c
>>> flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java
>>> d205bbc
>>> flume-ng-core/src/test/java/org/apache/flume/TestContext.java a5e6aa8
>>> flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannel.java
>>> 3392dff
>>> flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelConcurrency.java
>>> 8dad0b2
>>> flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelTransaction.java
>>> bc81f26
>>> flume-ng-core/src/test/java/org/apache/flume/sink/TestAvroSink.java
>>> 467785f
>>> flume-ng-core/src/test/java/org/apache/flume/sink/TestFailoverSinkProcessor.java
>>> 93ad3bf
>>> flume-ng-core/src/test/java/org/apache/flume/sink/TestLoggerSink.java
>>> 92ff6fe
>>> flume-ng-core/src/test/java/org/apache/flume/sink/TestRollingFileSink.java
>>> 7e26e2a
>>> flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java
>>> c5c3f2f
>>> flume-ng-core/src/test/java/org/apache/flume/source/TestExecSource.java
>>> 6035270
>>> flume-ng-core/src/test/java/org/apache/flume/source/TestPollableSourceRunner.java
>>> c27f82c
>>> flume-ng-core/src/test/java/org/apache/flume/source/TestSequenceGeneratorSource.java
>>> 579b257
>>> flume-ng-legacy-sources/flume-avro-source/src/main/java/org/apache/flume/source/avroLegacy/AvroLegacySource.java
>>> b1e67f7
>>> flume-ng-legacy-sources/flume-avro-source/src/test/java/org/apache/flume/source/avroLegacy/TestLegacyAvroSource.java
>>> 6e3eb53
>>> flume-ng-legacy-sources/flume-thrift-source/src/main/java/org/apache/flume/source/thriftLegacy/ThriftLegacySource.java
>>> 5fe270a
>>> flume-ng-legacy-sources/flume-thrift-source/src/test/java/org/apache/flume/source/thriftLegacy/TestThriftLegacySource.java
>>> ddd9478
>>> flume-ng-node/pom.xml b9b062e
>>> flume-ng-node/src/main/java/org/apache/flume/conf/properties/FlumeConfiguration.java
>>> d66f6d1
>>> flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java
>>> 97f72e1
>>> flume-ng-node/src/test/java/org/apache/flume/node/TestAbstractLogicalNodeManager.java
>>> 521b586
>>> flume-ng-node/src/test/java/org/apache/flume/source/TestNetcatSource.java
>>> fb2a960
>>> flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java
>>> 3da90a5
>>> flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java
>>> 7d8ee8a
>>> flume-ng-sinks/flume-irc-sink/src/main/java/org/apache/flume/sink/irc/IRCSink.java
>>> 0a5498f
>>> pom.xml d785762
>>>
>>> Diff: https://reviews.apache.org/r/4115/diff
>>>
>>>
>>> Testing
>>> -------
>>>
>>> All existing unit tests for the components whose configuration has now
>>> been moved to these stubs pass.
>>>
>>>
>>> Thanks,
>>>
>>> Hari
>>>
>>>
>>
>>
>> --
>> Eric Sammer
>> twitter: esammer
>> data: www.cloudera.com
>>
>
>
Re: Review Request: Component aware configuration managers for flume components
Posted by Hari Shreedharan <hs...@cloudera.com>.
On Fri, Mar 2, 2012 at 5:54 PM, Eric Sammer <es...@cloudera.com> wrote:
> I just read the diff and I strongly object to it for the following reasons:
>
> * It's *terribly* overcomplicated / overengineered. You're very close to
> having reimplemented Spring bean configuration (without strict DI).
>
- Yes, it is a bit complex, but if we need to make sure configuration is
validated without having access to the components themselves, the
properties of the components have to be separated out into another class. I
am making changes to make it simpler, removing reflection from the
validation logic which was used to remove the code repetition.
> * There's significant logic in constructors (a code smell).
>
This was a first cut, I am moving the logic into to other methods.
* Configuration is now highly coupled to the storage format (java
> properties)
>
I am not sure how this is different from the current implementation. I did
not add any of the Context methods or FlumeConfiguration classes (all I did
in this patch was move it to the new package) - these are already based on
Properties..
Please see:
http://svn.apache.org/viewvc/incubator/flume/branches/flume-728/flume-ng-node/src/main/java/org/apache/flume/conf/properties/and
http://svn.apache.org/viewvc/incubator/flume/branches/flume-728/flume-ng-core/src/main/java/org/apache/flume/Context.java?revision=1294969&view=markup
* FlumeConfiguration has become a God Object.
>
No changes were made to FlumeConfiguration at all. The only thing I did was
to move it from the current package to the conf package. This is the exact
same code as it is currently. It was already a god object.
> * A bunch of hoops are jumped to remain type safe, and then the rest of
> the code breaks all type safety by stringifying a ton of stuff. I'm not
> specifically against dynamic techniques or reflection, but I'd like to see
> it only when there's a true need and it adds real value.
>
Can you give me an example of the stringification that you have mentioned?
I did not get this comment. If you are talking about reflection to convert
the data into the correct type: Since the data would be read in through
Context and it is type safe and since we use the same type it should not
cause an issue. Yes, this can break if the classes are specified in
correctly, but that is likely due to a bug. Either way, I have removed the
reflection from the validation logic, as performance can be bad, but I
don't specifically see a problem with it, as long as the code is correct.
I am moving the properties for each component into the Conf classes(only
the properties which are read from a configuration file or configured in
some other way), from where the component reads this data using getters(To
avoid 2 copies of the properties, they will not be duplicated in the
components themselves).
This will allow the validation to happen without having access to
components.
If you feel separating out the configured properties from the component is
not a good idea, I am open to any suggestions on how to do the validation
without actually being dependent on flume-core.
Thanks
Hari
> On Thu, Mar 1, 2012 at 1:35 PM, Hari Shreedharan <
> hshreedharan@cloudera.com> wrote:
>
>>
>> -----------------------------------------------------------
>> This is an automatically generated e-mail. To reply, visit:
>> https://reviews.apache.org/r/4115/
>> -----------------------------------------------------------
>>
>> (Updated 2012-03-01 21:35:41.679397)
>>
>>
>> Review request for Flume.
>>
>>
>> Changes
>> -------
>>
>> Note that I do know of a couple of bugs in the existing diff(like that if
>> the name of a config key starts with an Upper case letter, an exception is
>> thrown) - I will fix and upload the review here soon.
>>
>>
>> Summary (updated)
>> -------
>>
>> Currently the configuration of each component sits deep inside the
>> component themselves. There is no way to verify if a configuration is valid
>> before run time. In most systems like Flume, it is likely that these confs
>> will be deployed from one single host, on to the machines where flume
>> agents are running. Only when each agent starts, invalid confs are
>> identified because the Agents would terminate by throwing exceptions. This
>> is a first attempt to make a component-aware configuration system which is
>> independent of the Flume, and does not require the Flume jar to be
>> installed. Each component needs a configuration manager, which configures
>> the components.
>>
>> I have not completed the validation part yet, but have completed the
>> configuration managers for all the components that come in the flume jar. I
>> will add support for the other components, such as HDFS sink, JDBC channel
>> etc soon. I will also add the validation support to the classes, but would
>> like a review of the model being proposed here.
>>
>> Note that I do know of a couple of bugs in the existing diff(like that if
>> the name of a config key starts with an Upper case letter, an exception is
>> thrown) - I will fix and upload the review here soon.
>>
>>
>> This addresses bug FLUME-992.
>> https://issues.apache.org/jira/browse/FLUME-992
>>
>>
>> Diffs
>> -----
>>
>> flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannel.java
>> bca0c50
>> flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProvider.java
>> e445d61
>> flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProviderFactory.java
>> 6fbd6ef
>> flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java
>> 1cf1c0c
>> flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestJdbcChannelProvider.java
>> 7710d46
>> flume-ng-clients/flume-ng-log4jappender/src/test/java/org/apache/flume/clients/log4jappender/TestLog4jAppender.java
>> 68d95fb
>> flume-ng-configuration/pom.xml PRE-CREATION
>> flume-ng-configuration/src/main/java/org/apache/flume/conf/AbstractComponentConfiguration.java
>> PRE-CREATION
>> flume-ng-configuration/src/main/java/org/apache/flume/conf/ConfigurationException.java
>> PRE-CREATION
>> flume-ng-configuration/src/main/java/org/apache/flume/conf/Context.java
>> PRE-CREATION
>> flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfiguration.java
>> PRE-CREATION
>> flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationError.java
>> PRE-CREATION
>> flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationErrorType.java
>> PRE-CREATION
>> flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MemoryChannelConfiguration.java
>> PRE-CREATION
>> flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MultiplexingChannelSelectorConfiguration.java
>> PRE-CREATION
>> flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/PseudoTxnMemoryChannelConfiguration.java
>> PRE-CREATION
>> flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/AvroSinkConfiguration.java
>> PRE-CREATION
>> flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/FailoverSinkProcessorConfiguration.java
>> PRE-CREATION
>> flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/RollingFileSinkConfiguration.java
>> PRE-CREATION
>> flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkGroupConfiguration.java
>> PRE-CREATION
>> flume-ng-configuration/src/main/java/org/apache/flume/conf/source/AvroSourceConfiguration.java
>> PRE-CREATION
>> flume-ng-configuration/src/main/java/org/apache/flume/conf/source/ExecSourceConfiguration.java
>> PRE-CREATION
>> flume-ng-configuration/src/main/java/org/apache/flume/conf/source/NetcatSourceConfiguration.java
>> PRE-CREATION
>> flume-ng-configuration/src/test/resources/flume-ng-error-test
>> PRE-CREATION
>> flume-ng-configuration/src/test/resources/flume-ng-test.conf PRE-CREATION
>> flume-ng-core/pom.xml d753fa1
>> flume-ng-core/src/main/java/org/apache/flume/Context.java f1c8f85
>> flume-ng-core/src/main/java/org/apache/flume/channel/AbstractChannel.java
>> 352bf08
>> flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorFactory.java
>> a7d5f94
>> flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java
>> e79490e
>> flume-ng-core/src/main/java/org/apache/flume/channel/MultiplexingChannelSelector.java
>> c63d0a1
>> flume-ng-core/src/main/java/org/apache/flume/channel/PseudoTxnMemoryChannel.java
>> 1df580e
>> flume-ng-core/src/main/java/org/apache/flume/channel/ReplicatingChannelSelector.java
>> 8f22746
>> flume-ng-core/src/main/java/org/apache/flume/conf/Configurable.java
>> 0fa4839
>> flume-ng-core/src/main/java/org/apache/flume/conf/Configurables.java
>> 84492e5
>> flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 5440631
>> flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkProcessor.java
>> 257bab3
>> flume-ng-core/src/main/java/org/apache/flume/sink/FailoverSinkProcessor.java
>> 9f5b856
>> flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java
>> 45c031d
>> flume-ng-core/src/main/java/org/apache/flume/sink/SinkGroup.java 1adc5ff
>> flume-ng-core/src/main/java/org/apache/flume/sink/SinkProcessorFactory.java
>> 859f4fd
>> flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java
>> 7b079f9
>> flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java
>> a96016c
>> flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java
>> d205bbc
>> flume-ng-core/src/test/java/org/apache/flume/TestContext.java a5e6aa8
>> flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannel.java
>> 3392dff
>> flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelConcurrency.java
>> 8dad0b2
>> flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelTransaction.java
>> bc81f26
>> flume-ng-core/src/test/java/org/apache/flume/sink/TestAvroSink.java
>> 467785f
>> flume-ng-core/src/test/java/org/apache/flume/sink/TestFailoverSinkProcessor.java
>> 93ad3bf
>> flume-ng-core/src/test/java/org/apache/flume/sink/TestLoggerSink.java
>> 92ff6fe
>> flume-ng-core/src/test/java/org/apache/flume/sink/TestRollingFileSink.java
>> 7e26e2a
>> flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java
>> c5c3f2f
>> flume-ng-core/src/test/java/org/apache/flume/source/TestExecSource.java
>> 6035270
>> flume-ng-core/src/test/java/org/apache/flume/source/TestPollableSourceRunner.java
>> c27f82c
>> flume-ng-core/src/test/java/org/apache/flume/source/TestSequenceGeneratorSource.java
>> 579b257
>> flume-ng-legacy-sources/flume-avro-source/src/main/java/org/apache/flume/source/avroLegacy/AvroLegacySource.java
>> b1e67f7
>> flume-ng-legacy-sources/flume-avro-source/src/test/java/org/apache/flume/source/avroLegacy/TestLegacyAvroSource.java
>> 6e3eb53
>> flume-ng-legacy-sources/flume-thrift-source/src/main/java/org/apache/flume/source/thriftLegacy/ThriftLegacySource.java
>> 5fe270a
>> flume-ng-legacy-sources/flume-thrift-source/src/test/java/org/apache/flume/source/thriftLegacy/TestThriftLegacySource.java
>> ddd9478
>> flume-ng-node/pom.xml b9b062e
>> flume-ng-node/src/main/java/org/apache/flume/conf/properties/FlumeConfiguration.java
>> d66f6d1
>> flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java
>> 97f72e1
>> flume-ng-node/src/test/java/org/apache/flume/node/TestAbstractLogicalNodeManager.java
>> 521b586
>> flume-ng-node/src/test/java/org/apache/flume/source/TestNetcatSource.java
>> fb2a960
>> flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java
>> 3da90a5
>> flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java
>> 7d8ee8a
>> flume-ng-sinks/flume-irc-sink/src/main/java/org/apache/flume/sink/irc/IRCSink.java
>> 0a5498f
>> pom.xml d785762
>>
>> Diff: https://reviews.apache.org/r/4115/diff
>>
>>
>> Testing
>> -------
>>
>> All existing unit tests for the components whose configuration has now
>> been moved to these stubs pass.
>>
>>
>> Thanks,
>>
>> Hari
>>
>>
>
>
> --
> Eric Sammer
> twitter: esammer
> data: www.cloudera.com
>
Re: Review Request: Component aware configuration managers for flume components
Posted by Eric Sammer <es...@cloudera.com>.
I just read the diff and I strongly object to it for the following reasons:
* It's *terribly* overcomplicated / overengineered. You're very close to
having reimplemented Spring bean configuration (without strict DI).
* There's significant logic in constructors (a code smell).
* Configuration is now highly coupled to the storage format (java
properties)
* FlumeConfiguration has become a God Object.
* A bunch of hoops are jumped to remain type safe, and then the rest of the
code breaks all type safety by stringifying a ton of stuff. I'm not
specifically against dynamic techniques or reflection, but I'd like to see
it only when there's a true need and it adds real value.
On Thu, Mar 1, 2012 at 1:35 PM, Hari Shreedharan
<hs...@cloudera.com>wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/4115/
> -----------------------------------------------------------
>
> (Updated 2012-03-01 21:35:41.679397)
>
>
> Review request for Flume.
>
>
> Changes
> -------
>
> Note that I do know of a couple of bugs in the existing diff(like that if
> the name of a config key starts with an Upper case letter, an exception is
> thrown) - I will fix and upload the review here soon.
>
>
> Summary (updated)
> -------
>
> Currently the configuration of each component sits deep inside the
> component themselves. There is no way to verify if a configuration is valid
> before run time. In most systems like Flume, it is likely that these confs
> will be deployed from one single host, on to the machines where flume
> agents are running. Only when each agent starts, invalid confs are
> identified because the Agents would terminate by throwing exceptions. This
> is a first attempt to make a component-aware configuration system which is
> independent of the Flume, and does not require the Flume jar to be
> installed. Each component needs a configuration manager, which configures
> the components.
>
> I have not completed the validation part yet, but have completed the
> configuration managers for all the components that come in the flume jar. I
> will add support for the other components, such as HDFS sink, JDBC channel
> etc soon. I will also add the validation support to the classes, but would
> like a review of the model being proposed here.
>
> Note that I do know of a couple of bugs in the existing diff(like that if
> the name of a config key starts with an Upper case letter, an exception is
> thrown) - I will fix and upload the review here soon.
>
>
> This addresses bug FLUME-992.
> https://issues.apache.org/jira/browse/FLUME-992
>
>
> Diffs
> -----
>
> flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannel.java
> bca0c50
> flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProvider.java
> e445d61
> flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProviderFactory.java
> 6fbd6ef
> flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java
> 1cf1c0c
> flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestJdbcChannelProvider.java
> 7710d46
> flume-ng-clients/flume-ng-log4jappender/src/test/java/org/apache/flume/clients/log4jappender/TestLog4jAppender.java
> 68d95fb
> flume-ng-configuration/pom.xml PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/AbstractComponentConfiguration.java
> PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/ConfigurationException.java
> PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/Context.java
> PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfiguration.java
> PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationError.java
> PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationErrorType.java
> PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MemoryChannelConfiguration.java
> PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MultiplexingChannelSelectorConfiguration.java
> PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/PseudoTxnMemoryChannelConfiguration.java
> PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/AvroSinkConfiguration.java
> PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/FailoverSinkProcessorConfiguration.java
> PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/RollingFileSinkConfiguration.java
> PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkGroupConfiguration.java
> PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/source/AvroSourceConfiguration.java
> PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/source/ExecSourceConfiguration.java
> PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/source/NetcatSourceConfiguration.java
> PRE-CREATION
> flume-ng-configuration/src/test/resources/flume-ng-error-test PRE-CREATION
> flume-ng-configuration/src/test/resources/flume-ng-test.conf PRE-CREATION
> flume-ng-core/pom.xml d753fa1
> flume-ng-core/src/main/java/org/apache/flume/Context.java f1c8f85
> flume-ng-core/src/main/java/org/apache/flume/channel/AbstractChannel.java
> 352bf08
> flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorFactory.java
> a7d5f94
> flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java
> e79490e
> flume-ng-core/src/main/java/org/apache/flume/channel/MultiplexingChannelSelector.java
> c63d0a1
> flume-ng-core/src/main/java/org/apache/flume/channel/PseudoTxnMemoryChannel.java
> 1df580e
> flume-ng-core/src/main/java/org/apache/flume/channel/ReplicatingChannelSelector.java
> 8f22746
> flume-ng-core/src/main/java/org/apache/flume/conf/Configurable.java
> 0fa4839
> flume-ng-core/src/main/java/org/apache/flume/conf/Configurables.java
> 84492e5
> flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 5440631
> flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkProcessor.java
> 257bab3
> flume-ng-core/src/main/java/org/apache/flume/sink/FailoverSinkProcessor.java
> 9f5b856
> flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java
> 45c031d
> flume-ng-core/src/main/java/org/apache/flume/sink/SinkGroup.java 1adc5ff
> flume-ng-core/src/main/java/org/apache/flume/sink/SinkProcessorFactory.java
> 859f4fd
> flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java
> 7b079f9
> flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java
> a96016c
> flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java
> d205bbc
> flume-ng-core/src/test/java/org/apache/flume/TestContext.java a5e6aa8
> flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannel.java
> 3392dff
> flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelConcurrency.java
> 8dad0b2
> flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelTransaction.java
> bc81f26
> flume-ng-core/src/test/java/org/apache/flume/sink/TestAvroSink.java
> 467785f
> flume-ng-core/src/test/java/org/apache/flume/sink/TestFailoverSinkProcessor.java
> 93ad3bf
> flume-ng-core/src/test/java/org/apache/flume/sink/TestLoggerSink.java
> 92ff6fe
> flume-ng-core/src/test/java/org/apache/flume/sink/TestRollingFileSink.java
> 7e26e2a
> flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java
> c5c3f2f
> flume-ng-core/src/test/java/org/apache/flume/source/TestExecSource.java
> 6035270
> flume-ng-core/src/test/java/org/apache/flume/source/TestPollableSourceRunner.java
> c27f82c
> flume-ng-core/src/test/java/org/apache/flume/source/TestSequenceGeneratorSource.java
> 579b257
> flume-ng-legacy-sources/flume-avro-source/src/main/java/org/apache/flume/source/avroLegacy/AvroLegacySource.java
> b1e67f7
> flume-ng-legacy-sources/flume-avro-source/src/test/java/org/apache/flume/source/avroLegacy/TestLegacyAvroSource.java
> 6e3eb53
> flume-ng-legacy-sources/flume-thrift-source/src/main/java/org/apache/flume/source/thriftLegacy/ThriftLegacySource.java
> 5fe270a
> flume-ng-legacy-sources/flume-thrift-source/src/test/java/org/apache/flume/source/thriftLegacy/TestThriftLegacySource.java
> ddd9478
> flume-ng-node/pom.xml b9b062e
> flume-ng-node/src/main/java/org/apache/flume/conf/properties/FlumeConfiguration.java
> d66f6d1
> flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java
> 97f72e1
> flume-ng-node/src/test/java/org/apache/flume/node/TestAbstractLogicalNodeManager.java
> 521b586
> flume-ng-node/src/test/java/org/apache/flume/source/TestNetcatSource.java
> fb2a960
> flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java
> 3da90a5
> flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java
> 7d8ee8a
> flume-ng-sinks/flume-irc-sink/src/main/java/org/apache/flume/sink/irc/IRCSink.java
> 0a5498f
> pom.xml d785762
>
> Diff: https://reviews.apache.org/r/4115/diff
>
>
> Testing
> -------
>
> All existing unit tests for the components whose configuration has now
> been moved to these stubs pass.
>
>
> Thanks,
>
> Hari
>
>
--
Eric Sammer
twitter: esammer
data: www.cloudera.com
Re: Review Request: Component aware configuration managers for flume
components
Posted by Hari Shreedharan <hs...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4115/
-----------------------------------------------------------
(Updated 2012-03-01 21:35:41.679397)
Review request for Flume.
Changes
-------
Note that I do know of a couple of bugs in the existing diff(like that if the name of a config key starts with an Upper case letter, an exception is thrown) - I will fix and upload the review here soon.
Summary (updated)
-------
Currently the configuration of each component sits deep inside the component themselves. There is no way to verify if a configuration is valid before run time. In most systems like Flume, it is likely that these confs will be deployed from one single host, on to the machines where flume agents are running. Only when each agent starts, invalid confs are identified because the Agents would terminate by throwing exceptions. This is a first attempt to make a component-aware configuration system which is independent of the Flume, and does not require the Flume jar to be installed. Each component needs a configuration manager, which configures the components.
I have not completed the validation part yet, but have completed the configuration managers for all the components that come in the flume jar. I will add support for the other components, such as HDFS sink, JDBC channel etc soon. I will also add the validation support to the classes, but would like a review of the model being proposed here.
Note that I do know of a couple of bugs in the existing diff(like that if the name of a config key starts with an Upper case letter, an exception is thrown) - I will fix and upload the review here soon.
This addresses bug FLUME-992.
https://issues.apache.org/jira/browse/FLUME-992
Diffs
-----
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannel.java bca0c50
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProvider.java e445d61
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProviderFactory.java 6fbd6ef
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java 1cf1c0c
flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestJdbcChannelProvider.java 7710d46
flume-ng-clients/flume-ng-log4jappender/src/test/java/org/apache/flume/clients/log4jappender/TestLog4jAppender.java 68d95fb
flume-ng-configuration/pom.xml PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/AbstractComponentConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ConfigurationException.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/Context.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationError.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationErrorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MemoryChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MultiplexingChannelSelectorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/PseudoTxnMemoryChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/AvroSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/FailoverSinkProcessorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/RollingFileSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkGroupConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/AvroSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/ExecSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/NetcatSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/test/resources/flume-ng-error-test PRE-CREATION
flume-ng-configuration/src/test/resources/flume-ng-test.conf PRE-CREATION
flume-ng-core/pom.xml d753fa1
flume-ng-core/src/main/java/org/apache/flume/Context.java f1c8f85
flume-ng-core/src/main/java/org/apache/flume/channel/AbstractChannel.java 352bf08
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorFactory.java a7d5f94
flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java e79490e
flume-ng-core/src/main/java/org/apache/flume/channel/MultiplexingChannelSelector.java c63d0a1
flume-ng-core/src/main/java/org/apache/flume/channel/PseudoTxnMemoryChannel.java 1df580e
flume-ng-core/src/main/java/org/apache/flume/channel/ReplicatingChannelSelector.java 8f22746
flume-ng-core/src/main/java/org/apache/flume/conf/Configurable.java 0fa4839
flume-ng-core/src/main/java/org/apache/flume/conf/Configurables.java 84492e5
flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 5440631
flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkProcessor.java 257bab3
flume-ng-core/src/main/java/org/apache/flume/sink/FailoverSinkProcessor.java 9f5b856
flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java 45c031d
flume-ng-core/src/main/java/org/apache/flume/sink/SinkGroup.java 1adc5ff
flume-ng-core/src/main/java/org/apache/flume/sink/SinkProcessorFactory.java 859f4fd
flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java 7b079f9
flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java a96016c
flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java d205bbc
flume-ng-core/src/test/java/org/apache/flume/TestContext.java a5e6aa8
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannel.java 3392dff
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelConcurrency.java 8dad0b2
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelTransaction.java bc81f26
flume-ng-core/src/test/java/org/apache/flume/sink/TestAvroSink.java 467785f
flume-ng-core/src/test/java/org/apache/flume/sink/TestFailoverSinkProcessor.java 93ad3bf
flume-ng-core/src/test/java/org/apache/flume/sink/TestLoggerSink.java 92ff6fe
flume-ng-core/src/test/java/org/apache/flume/sink/TestRollingFileSink.java 7e26e2a
flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java c5c3f2f
flume-ng-core/src/test/java/org/apache/flume/source/TestExecSource.java 6035270
flume-ng-core/src/test/java/org/apache/flume/source/TestPollableSourceRunner.java c27f82c
flume-ng-core/src/test/java/org/apache/flume/source/TestSequenceGeneratorSource.java 579b257
flume-ng-legacy-sources/flume-avro-source/src/main/java/org/apache/flume/source/avroLegacy/AvroLegacySource.java b1e67f7
flume-ng-legacy-sources/flume-avro-source/src/test/java/org/apache/flume/source/avroLegacy/TestLegacyAvroSource.java 6e3eb53
flume-ng-legacy-sources/flume-thrift-source/src/main/java/org/apache/flume/source/thriftLegacy/ThriftLegacySource.java 5fe270a
flume-ng-legacy-sources/flume-thrift-source/src/test/java/org/apache/flume/source/thriftLegacy/TestThriftLegacySource.java ddd9478
flume-ng-node/pom.xml b9b062e
flume-ng-node/src/main/java/org/apache/flume/conf/properties/FlumeConfiguration.java d66f6d1
flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java 97f72e1
flume-ng-node/src/test/java/org/apache/flume/node/TestAbstractLogicalNodeManager.java 521b586
flume-ng-node/src/test/java/org/apache/flume/source/TestNetcatSource.java fb2a960
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 3da90a5
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 7d8ee8a
flume-ng-sinks/flume-irc-sink/src/main/java/org/apache/flume/sink/irc/IRCSink.java 0a5498f
pom.xml d785762
Diff: https://reviews.apache.org/r/4115/diff
Testing
-------
All existing unit tests for the components whose configuration has now been moved to these stubs pass.
Thanks,
Hari
Re: Review Request: Component aware configuration managers for flume components
Posted by Eric Sammer <es...@cloudera.com>.
You're always going to have some amount of Flume code in process to
validate Flume configs. I can understand breaking them out into a module if
that's the goal, but I'd still use validators (i.e. classes called
FooValidator with a single validate() method that takes a Context for each
component that is configurable). That was the only point I wanted to make.
On Thu, Mar 1, 2012 at 1:25 PM, Hari Shreedharan
<hs...@cloudera.com>wrote:
> Hi Eric,
>
> I had considered the validator pattern. What we are trying to do is to be
> able to validate the configuration for specific components specified in the
> conf, without actually depending on any of the Flume API themselves.
>
> The use case I am trying to address: Say you have a 100 node cluster on
> which we want to install and run flume and we use some automated way of
> starting up flume on these clusters, usually from one box. The idea is that
> the conf file be verified before actually deploying it on the cluster
> itself. Requiring Flume to be installed on these boxes(which could as well
> be on the corp network) should not be a requirement. Using the validator
> pattern directly would require us to actually have the flume components on
> the box validating the conf.
>
> We could use a stub that stores all the required parameters and run the
> validator on that, but I think that is pretty much the same as what is
> already done here.
>
> Thanks!
> Hari
>
> --
> Hari Shreedharan
>
>
> On Thursday, March 1, 2012 at 11:20 AM, Eric Sammer wrote:
>
> > Why not just follow the Validator pattern? See Spring's validators for
> > reference.
> >
> >
> >
> > On Feb 29, 2012, at 4:31 PM, Hari Shreedharan <hshreedharan@cloudera.com(mailto:
> hshreedharan@cloudera.com)> wrote:
> >
> > >
> > > -----------------------------------------------------------
> > > This is an automatically generated e-mail. To reply, visit:
> > > https://reviews.apache.org/r/4115/
> > > -----------------------------------------------------------
> > >
> > > (Updated 2012-03-01 00:31:40.789559)
> > >
> > >
> > > Review request for Flume.
> > >
> > >
> > > Changes
> > > -------
> > >
> > > Last diff included changes not intended for this jira.
> > >
> > >
> > > Summary
> > > -------
> > >
> > > Currently the configuration of each component sits deep inside the
> component themselves. There is no way to verify if a configuration is valid
> before run time. In most systems like Flume, it is likely that these confs
> will be deployed from one single host, on to the machines where flume
> agents are running. Only when each agent starts, invalid confs are
> identified because the Agents would terminate by throwing exceptions. This
> is a first attempt to make a component-aware configuration system which is
> independent of the Flume, and does not require the Flume jar to be
> installed. Each component needs a configuration manager, which configures
> the components.
> > >
> > > I have not completed the validation part yet, but have completed the
> configuration managers for all the components that come in the flume jar. I
> will add support for the other components, such as HDFS sink, JDBC channel
> etc soon. I will also add the validation support to the classes, but would
> like a review of the model being proposed here.
> > >
> > >
> > > This addresses bug FLUME-992.
> > > https://issues.apache.org/jira/browse/FLUME-992
> > >
> > >
> > > Diffs (updated)
> > > -----
> > >
> > >
> flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannel.java
> bca0c50
> > >
> flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProvider.java
> e445d61
> > >
> flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProviderFactory.java
> 6fbd6ef
> > >
> flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java
> 1cf1c0c
> > >
> flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestJdbcChannelProvider.java
> 7710d46
> > >
> flume-ng-clients/flume-ng-log4jappender/src/test/java/org/apache/flume/clients/log4jappender/TestLog4jAppender.java
> 68d95fb
> > > flume-ng-configuration/pom.xml PRE-CREATION
> > >
> flume-ng-configuration/src/main/java/org/apache/flume/conf/AbstractComponentConfiguration.java
> PRE-CREATION
> > >
> flume-ng-configuration/src/main/java/org/apache/flume/conf/ConfigurationException.java
> PRE-CREATION
> > >
> flume-ng-configuration/src/main/java/org/apache/flume/conf/Context.java
> PRE-CREATION
> > >
> flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfiguration.java
> PRE-CREATION
> > >
> flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationError.java
> PRE-CREATION
> > >
> flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationErrorType.java
> PRE-CREATION
> > >
> flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MemoryChannelConfiguration.java
> PRE-CREATION
> > >
> flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MultiplexingChannelSelectorConfiguration.java
> PRE-CREATION
> > >
> flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/PseudoTxnMemoryChannelConfiguration.java
> PRE-CREATION
> > >
> flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/AvroSinkConfiguration.java
> PRE-CREATION
> > >
> flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/FailoverSinkProcessorConfiguration.java
> PRE-CREATION
> > >
> flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/RollingFileSinkConfiguration.java
> PRE-CREATION
> > >
> flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkGroupConfiguration.java
> PRE-CREATION
> > >
> flume-ng-configuration/src/main/java/org/apache/flume/conf/source/AvroSourceConfiguration.java
> PRE-CREATION
> > >
> flume-ng-configuration/src/main/java/org/apache/flume/conf/source/ExecSourceConfiguration.java
> PRE-CREATION
> > >
> flume-ng-configuration/src/main/java/org/apache/flume/conf/source/NetcatSourceConfiguration.java
> PRE-CREATION
> > > flume-ng-configuration/src/test/resources/flume-ng-error-test
> PRE-CREATION
> > > flume-ng-configuration/src/test/resources/flume-ng-test.conf
> PRE-CREATION
> > > flume-ng-core/pom.xml d753fa1
> > > flume-ng-core/src/main/java/org/apache/flume/Context.java f1c8f85
> > >
> flume-ng-core/src/main/java/org/apache/flume/channel/AbstractChannel.java
> 352bf08
> > >
> flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorFactory.java
> a7d5f94
> > >
> flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java
> e79490e
> > >
> flume-ng-core/src/main/java/org/apache/flume/channel/MultiplexingChannelSelector.java
> c63d0a1
> > >
> flume-ng-core/src/main/java/org/apache/flume/channel/PseudoTxnMemoryChannel.java
> 1df580e
> > >
> flume-ng-core/src/main/java/org/apache/flume/channel/ReplicatingChannelSelector.java
> 8f22746
> > > flume-ng-core/src/main/java/org/apache/flume/conf/Configurable.java
> 0fa4839
> > > flume-ng-core/src/main/java/org/apache/flume/conf/Configurables.java
> 84492e5
> > > flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 5440631
> > >
> flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkProcessor.java
> 257bab3
> > >
> flume-ng-core/src/main/java/org/apache/flume/sink/FailoverSinkProcessor.java
> 9f5b856
> > > flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java
> 45c031d
> > > flume-ng-core/src/main/java/org/apache/flume/sink/SinkGroup.java
> 1adc5ff
> > >
> flume-ng-core/src/main/java/org/apache/flume/sink/SinkProcessorFactory.java
> 859f4fd
> > > flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java
> 7b079f9
> > > flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java
> a96016c
> > > flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java
> d205bbc
> > > flume-ng-core/src/test/java/org/apache/flume/TestContext.java a5e6aa8
> > >
> flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannel.java
> 3392dff
> > >
> flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelConcurrency.java
> 8dad0b2
> > >
> flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelTransaction.java
> bc81f26
> > > flume-ng-core/src/test/java/org/apache/flume/sink/TestAvroSink.java
> 467785f
> > >
> flume-ng-core/src/test/java/org/apache/flume/sink/TestFailoverSinkProcessor.java
> 93ad3bf
> > > flume-ng-core/src/test/java/org/apache/flume/sink/TestLoggerSink.java
> 92ff6fe
> > >
> flume-ng-core/src/test/java/org/apache/flume/sink/TestRollingFileSink.java
> 7e26e2a
> > >
> flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java
> c5c3f2f
> > >
> flume-ng-core/src/test/java/org/apache/flume/source/TestExecSource.java
> 6035270
> > >
> flume-ng-core/src/test/java/org/apache/flume/source/TestPollableSourceRunner.java
> c27f82c
> > >
> flume-ng-core/src/test/java/org/apache/flume/source/TestSequenceGeneratorSource.java
> 579b257
> > >
> flume-ng-legacy-sources/flume-avro-source/src/main/java/org/apache/flume/source/avroLegacy/AvroLegacySource.java
> b1e67f7
> > >
> flume-ng-legacy-sources/flume-avro-source/src/test/java/org/apache/flume/source/avroLegacy/TestLegacyAvroSource.java
> 6e3eb53
> > >
> flume-ng-legacy-sources/flume-thrift-source/src/main/java/org/apache/flume/source/thriftLegacy/ThriftLegacySource.java
> 5fe270a
> > >
> flume-ng-legacy-sources/flume-thrift-source/src/test/java/org/apache/flume/source/thriftLegacy/TestThriftLegacySource.java
> ddd9478
> > > flume-ng-node/pom.xml b9b062e
> > >
> flume-ng-node/src/main/java/org/apache/flume/conf/properties/FlumeConfiguration.java
> d66f6d1
> > >
> flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java
> 97f72e1
> > >
> flume-ng-node/src/test/java/org/apache/flume/node/TestAbstractLogicalNodeManager.java
> 521b586
> > >
> flume-ng-node/src/test/java/org/apache/flume/source/TestNetcatSource.java
> fb2a960
> > >
> flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java
> 3da90a5
> > >
> flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java
> 7d8ee8a
> > >
> flume-ng-sinks/flume-irc-sink/src/main/java/org/apache/flume/sink/irc/IRCSink.java
> 0a5498f
> > > pom.xml d785762
> > >
> > > Diff: https://reviews.apache.org/r/4115/diff
> > >
> > >
> > > Testing
> > > -------
> > >
> > > All existing unit tests for the components whose configuration has now
> been moved to these stubs pass.
> > >
> > >
> > > Thanks,
> > >
> > > Hari
>
>
--
Eric Sammer
twitter: esammer
data: www.cloudera.com
Re: Review Request: Component aware configuration managers for
flume components
Posted by Hari Shreedharan <hs...@cloudera.com>.
Hi Eric,
I had considered the validator pattern. What we are trying to do is to be able to validate the configuration for specific components specified in the conf, without actually depending on any of the Flume API themselves.
The use case I am trying to address: Say you have a 100 node cluster on which we want to install and run flume and we use some automated way of starting up flume on these clusters, usually from one box. The idea is that the conf file be verified before actually deploying it on the cluster itself. Requiring Flume to be installed on these boxes(which could as well be on the corp network) should not be a requirement. Using the validator pattern directly would require us to actually have the flume components on the box validating the conf.
We could use a stub that stores all the required parameters and run the validator on that, but I think that is pretty much the same as what is already done here.
Thanks!
Hari
--
Hari Shreedharan
On Thursday, March 1, 2012 at 11:20 AM, Eric Sammer wrote:
> Why not just follow the Validator pattern? See Spring's validators for
> reference.
>
>
>
> On Feb 29, 2012, at 4:31 PM, Hari Shreedharan <hshreedharan@cloudera.com (mailto:hshreedharan@cloudera.com)> wrote:
>
> >
> > -----------------------------------------------------------
> > This is an automatically generated e-mail. To reply, visit:
> > https://reviews.apache.org/r/4115/
> > -----------------------------------------------------------
> >
> > (Updated 2012-03-01 00:31:40.789559)
> >
> >
> > Review request for Flume.
> >
> >
> > Changes
> > -------
> >
> > Last diff included changes not intended for this jira.
> >
> >
> > Summary
> > -------
> >
> > Currently the configuration of each component sits deep inside the component themselves. There is no way to verify if a configuration is valid before run time. In most systems like Flume, it is likely that these confs will be deployed from one single host, on to the machines where flume agents are running. Only when each agent starts, invalid confs are identified because the Agents would terminate by throwing exceptions. This is a first attempt to make a component-aware configuration system which is independent of the Flume, and does not require the Flume jar to be installed. Each component needs a configuration manager, which configures the components.
> >
> > I have not completed the validation part yet, but have completed the configuration managers for all the components that come in the flume jar. I will add support for the other components, such as HDFS sink, JDBC channel etc soon. I will also add the validation support to the classes, but would like a review of the model being proposed here.
> >
> >
> > This addresses bug FLUME-992.
> > https://issues.apache.org/jira/browse/FLUME-992
> >
> >
> > Diffs (updated)
> > -----
> >
> > flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannel.java bca0c50
> > flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProvider.java e445d61
> > flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProviderFactory.java 6fbd6ef
> > flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java 1cf1c0c
> > flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestJdbcChannelProvider.java 7710d46
> > flume-ng-clients/flume-ng-log4jappender/src/test/java/org/apache/flume/clients/log4jappender/TestLog4jAppender.java 68d95fb
> > flume-ng-configuration/pom.xml PRE-CREATION
> > flume-ng-configuration/src/main/java/org/apache/flume/conf/AbstractComponentConfiguration.java PRE-CREATION
> > flume-ng-configuration/src/main/java/org/apache/flume/conf/ConfigurationException.java PRE-CREATION
> > flume-ng-configuration/src/main/java/org/apache/flume/conf/Context.java PRE-CREATION
> > flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfiguration.java PRE-CREATION
> > flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationError.java PRE-CREATION
> > flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationErrorType.java PRE-CREATION
> > flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MemoryChannelConfiguration.java PRE-CREATION
> > flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MultiplexingChannelSelectorConfiguration.java PRE-CREATION
> > flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/PseudoTxnMemoryChannelConfiguration.java PRE-CREATION
> > flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/AvroSinkConfiguration.java PRE-CREATION
> > flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/FailoverSinkProcessorConfiguration.java PRE-CREATION
> > flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/RollingFileSinkConfiguration.java PRE-CREATION
> > flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkGroupConfiguration.java PRE-CREATION
> > flume-ng-configuration/src/main/java/org/apache/flume/conf/source/AvroSourceConfiguration.java PRE-CREATION
> > flume-ng-configuration/src/main/java/org/apache/flume/conf/source/ExecSourceConfiguration.java PRE-CREATION
> > flume-ng-configuration/src/main/java/org/apache/flume/conf/source/NetcatSourceConfiguration.java PRE-CREATION
> > flume-ng-configuration/src/test/resources/flume-ng-error-test PRE-CREATION
> > flume-ng-configuration/src/test/resources/flume-ng-test.conf PRE-CREATION
> > flume-ng-core/pom.xml d753fa1
> > flume-ng-core/src/main/java/org/apache/flume/Context.java f1c8f85
> > flume-ng-core/src/main/java/org/apache/flume/channel/AbstractChannel.java 352bf08
> > flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorFactory.java a7d5f94
> > flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java e79490e
> > flume-ng-core/src/main/java/org/apache/flume/channel/MultiplexingChannelSelector.java c63d0a1
> > flume-ng-core/src/main/java/org/apache/flume/channel/PseudoTxnMemoryChannel.java 1df580e
> > flume-ng-core/src/main/java/org/apache/flume/channel/ReplicatingChannelSelector.java 8f22746
> > flume-ng-core/src/main/java/org/apache/flume/conf/Configurable.java 0fa4839
> > flume-ng-core/src/main/java/org/apache/flume/conf/Configurables.java 84492e5
> > flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 5440631
> > flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkProcessor.java 257bab3
> > flume-ng-core/src/main/java/org/apache/flume/sink/FailoverSinkProcessor.java 9f5b856
> > flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java 45c031d
> > flume-ng-core/src/main/java/org/apache/flume/sink/SinkGroup.java 1adc5ff
> > flume-ng-core/src/main/java/org/apache/flume/sink/SinkProcessorFactory.java 859f4fd
> > flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java 7b079f9
> > flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java a96016c
> > flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java d205bbc
> > flume-ng-core/src/test/java/org/apache/flume/TestContext.java a5e6aa8
> > flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannel.java 3392dff
> > flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelConcurrency.java 8dad0b2
> > flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelTransaction.java bc81f26
> > flume-ng-core/src/test/java/org/apache/flume/sink/TestAvroSink.java 467785f
> > flume-ng-core/src/test/java/org/apache/flume/sink/TestFailoverSinkProcessor.java 93ad3bf
> > flume-ng-core/src/test/java/org/apache/flume/sink/TestLoggerSink.java 92ff6fe
> > flume-ng-core/src/test/java/org/apache/flume/sink/TestRollingFileSink.java 7e26e2a
> > flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java c5c3f2f
> > flume-ng-core/src/test/java/org/apache/flume/source/TestExecSource.java 6035270
> > flume-ng-core/src/test/java/org/apache/flume/source/TestPollableSourceRunner.java c27f82c
> > flume-ng-core/src/test/java/org/apache/flume/source/TestSequenceGeneratorSource.java 579b257
> > flume-ng-legacy-sources/flume-avro-source/src/main/java/org/apache/flume/source/avroLegacy/AvroLegacySource.java b1e67f7
> > flume-ng-legacy-sources/flume-avro-source/src/test/java/org/apache/flume/source/avroLegacy/TestLegacyAvroSource.java 6e3eb53
> > flume-ng-legacy-sources/flume-thrift-source/src/main/java/org/apache/flume/source/thriftLegacy/ThriftLegacySource.java 5fe270a
> > flume-ng-legacy-sources/flume-thrift-source/src/test/java/org/apache/flume/source/thriftLegacy/TestThriftLegacySource.java ddd9478
> > flume-ng-node/pom.xml b9b062e
> > flume-ng-node/src/main/java/org/apache/flume/conf/properties/FlumeConfiguration.java d66f6d1
> > flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java 97f72e1
> > flume-ng-node/src/test/java/org/apache/flume/node/TestAbstractLogicalNodeManager.java 521b586
> > flume-ng-node/src/test/java/org/apache/flume/source/TestNetcatSource.java fb2a960
> > flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 3da90a5
> > flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 7d8ee8a
> > flume-ng-sinks/flume-irc-sink/src/main/java/org/apache/flume/sink/irc/IRCSink.java 0a5498f
> > pom.xml d785762
> >
> > Diff: https://reviews.apache.org/r/4115/diff
> >
> >
> > Testing
> > -------
> >
> > All existing unit tests for the components whose configuration has now been moved to these stubs pass.
> >
> >
> > Thanks,
> >
> > Hari
Re: Review Request: Component aware configuration managers for flume components
Posted by Eric Sammer <es...@cloudera.com>.
Why not just follow the Validator pattern? See Spring's validators for
reference.
On Feb 29, 2012, at 4:31 PM, Hari Shreedharan <hs...@cloudera.com> wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/4115/
> -----------------------------------------------------------
>
> (Updated 2012-03-01 00:31:40.789559)
>
>
> Review request for Flume.
>
>
> Changes
> -------
>
> Last diff included changes not intended for this jira.
>
>
> Summary
> -------
>
> Currently the configuration of each component sits deep inside the component themselves. There is no way to verify if a configuration is valid before run time. In most systems like Flume, it is likely that these confs will be deployed from one single host, on to the machines where flume agents are running. Only when each agent starts, invalid confs are identified because the Agents would terminate by throwing exceptions. This is a first attempt to make a component-aware configuration system which is independent of the Flume, and does not require the Flume jar to be installed. Each component needs a configuration manager, which configures the components.
>
> I have not completed the validation part yet, but have completed the configuration managers for all the components that come in the flume jar. I will add support for the other components, such as HDFS sink, JDBC channel etc soon. I will also add the validation support to the classes, but would like a review of the model being proposed here.
>
>
> This addresses bug FLUME-992.
> https://issues.apache.org/jira/browse/FLUME-992
>
>
> Diffs (updated)
> -----
>
> flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannel.java bca0c50
> flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProvider.java e445d61
> flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProviderFactory.java 6fbd6ef
> flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java 1cf1c0c
> flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestJdbcChannelProvider.java 7710d46
> flume-ng-clients/flume-ng-log4jappender/src/test/java/org/apache/flume/clients/log4jappender/TestLog4jAppender.java 68d95fb
> flume-ng-configuration/pom.xml PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/AbstractComponentConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/ConfigurationException.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/Context.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationError.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationErrorType.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MemoryChannelConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MultiplexingChannelSelectorConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/PseudoTxnMemoryChannelConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/AvroSinkConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/FailoverSinkProcessorConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/RollingFileSinkConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkGroupConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/source/AvroSourceConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/source/ExecSourceConfiguration.java PRE-CREATION
> flume-ng-configuration/src/main/java/org/apache/flume/conf/source/NetcatSourceConfiguration.java PRE-CREATION
> flume-ng-configuration/src/test/resources/flume-ng-error-test PRE-CREATION
> flume-ng-configuration/src/test/resources/flume-ng-test.conf PRE-CREATION
> flume-ng-core/pom.xml d753fa1
> flume-ng-core/src/main/java/org/apache/flume/Context.java f1c8f85
> flume-ng-core/src/main/java/org/apache/flume/channel/AbstractChannel.java 352bf08
> flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorFactory.java a7d5f94
> flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java e79490e
> flume-ng-core/src/main/java/org/apache/flume/channel/MultiplexingChannelSelector.java c63d0a1
> flume-ng-core/src/main/java/org/apache/flume/channel/PseudoTxnMemoryChannel.java 1df580e
> flume-ng-core/src/main/java/org/apache/flume/channel/ReplicatingChannelSelector.java 8f22746
> flume-ng-core/src/main/java/org/apache/flume/conf/Configurable.java 0fa4839
> flume-ng-core/src/main/java/org/apache/flume/conf/Configurables.java 84492e5
> flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 5440631
> flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkProcessor.java 257bab3
> flume-ng-core/src/main/java/org/apache/flume/sink/FailoverSinkProcessor.java 9f5b856
> flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java 45c031d
> flume-ng-core/src/main/java/org/apache/flume/sink/SinkGroup.java 1adc5ff
> flume-ng-core/src/main/java/org/apache/flume/sink/SinkProcessorFactory.java 859f4fd
> flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java 7b079f9
> flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java a96016c
> flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java d205bbc
> flume-ng-core/src/test/java/org/apache/flume/TestContext.java a5e6aa8
> flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannel.java 3392dff
> flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelConcurrency.java 8dad0b2
> flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelTransaction.java bc81f26
> flume-ng-core/src/test/java/org/apache/flume/sink/TestAvroSink.java 467785f
> flume-ng-core/src/test/java/org/apache/flume/sink/TestFailoverSinkProcessor.java 93ad3bf
> flume-ng-core/src/test/java/org/apache/flume/sink/TestLoggerSink.java 92ff6fe
> flume-ng-core/src/test/java/org/apache/flume/sink/TestRollingFileSink.java 7e26e2a
> flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java c5c3f2f
> flume-ng-core/src/test/java/org/apache/flume/source/TestExecSource.java 6035270
> flume-ng-core/src/test/java/org/apache/flume/source/TestPollableSourceRunner.java c27f82c
> flume-ng-core/src/test/java/org/apache/flume/source/TestSequenceGeneratorSource.java 579b257
> flume-ng-legacy-sources/flume-avro-source/src/main/java/org/apache/flume/source/avroLegacy/AvroLegacySource.java b1e67f7
> flume-ng-legacy-sources/flume-avro-source/src/test/java/org/apache/flume/source/avroLegacy/TestLegacyAvroSource.java 6e3eb53
> flume-ng-legacy-sources/flume-thrift-source/src/main/java/org/apache/flume/source/thriftLegacy/ThriftLegacySource.java 5fe270a
> flume-ng-legacy-sources/flume-thrift-source/src/test/java/org/apache/flume/source/thriftLegacy/TestThriftLegacySource.java ddd9478
> flume-ng-node/pom.xml b9b062e
> flume-ng-node/src/main/java/org/apache/flume/conf/properties/FlumeConfiguration.java d66f6d1
> flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java 97f72e1
> flume-ng-node/src/test/java/org/apache/flume/node/TestAbstractLogicalNodeManager.java 521b586
> flume-ng-node/src/test/java/org/apache/flume/source/TestNetcatSource.java fb2a960
> flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 3da90a5
> flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 7d8ee8a
> flume-ng-sinks/flume-irc-sink/src/main/java/org/apache/flume/sink/irc/IRCSink.java 0a5498f
> pom.xml d785762
>
> Diff: https://reviews.apache.org/r/4115/diff
>
>
> Testing
> -------
>
> All existing unit tests for the components whose configuration has now been moved to these stubs pass.
>
>
> Thanks,
>
> Hari
>
Re: Review Request: Component aware configuration managers for flume
components
Posted by Hari Shreedharan <hs...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4115/
-----------------------------------------------------------
(Updated 2012-03-01 00:31:40.789559)
Review request for Flume.
Changes
-------
Last diff included changes not intended for this jira.
Summary
-------
Currently the configuration of each component sits deep inside the component themselves. There is no way to verify if a configuration is valid before run time. In most systems like Flume, it is likely that these confs will be deployed from one single host, on to the machines where flume agents are running. Only when each agent starts, invalid confs are identified because the Agents would terminate by throwing exceptions. This is a first attempt to make a component-aware configuration system which is independent of the Flume, and does not require the Flume jar to be installed. Each component needs a configuration manager, which configures the components.
I have not completed the validation part yet, but have completed the configuration managers for all the components that come in the flume jar. I will add support for the other components, such as HDFS sink, JDBC channel etc soon. I will also add the validation support to the classes, but would like a review of the model being proposed here.
This addresses bug FLUME-992.
https://issues.apache.org/jira/browse/FLUME-992
Diffs (updated)
-----
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannel.java bca0c50
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProvider.java e445d61
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProviderFactory.java 6fbd6ef
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java 1cf1c0c
flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestJdbcChannelProvider.java 7710d46
flume-ng-clients/flume-ng-log4jappender/src/test/java/org/apache/flume/clients/log4jappender/TestLog4jAppender.java 68d95fb
flume-ng-configuration/pom.xml PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/AbstractComponentConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ConfigurationException.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/Context.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationError.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationErrorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MemoryChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MultiplexingChannelSelectorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/PseudoTxnMemoryChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/AvroSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/FailoverSinkProcessorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/RollingFileSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkGroupConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/AvroSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/ExecSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/NetcatSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/test/resources/flume-ng-error-test PRE-CREATION
flume-ng-configuration/src/test/resources/flume-ng-test.conf PRE-CREATION
flume-ng-core/pom.xml d753fa1
flume-ng-core/src/main/java/org/apache/flume/Context.java f1c8f85
flume-ng-core/src/main/java/org/apache/flume/channel/AbstractChannel.java 352bf08
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorFactory.java a7d5f94
flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java e79490e
flume-ng-core/src/main/java/org/apache/flume/channel/MultiplexingChannelSelector.java c63d0a1
flume-ng-core/src/main/java/org/apache/flume/channel/PseudoTxnMemoryChannel.java 1df580e
flume-ng-core/src/main/java/org/apache/flume/channel/ReplicatingChannelSelector.java 8f22746
flume-ng-core/src/main/java/org/apache/flume/conf/Configurable.java 0fa4839
flume-ng-core/src/main/java/org/apache/flume/conf/Configurables.java 84492e5
flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 5440631
flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkProcessor.java 257bab3
flume-ng-core/src/main/java/org/apache/flume/sink/FailoverSinkProcessor.java 9f5b856
flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java 45c031d
flume-ng-core/src/main/java/org/apache/flume/sink/SinkGroup.java 1adc5ff
flume-ng-core/src/main/java/org/apache/flume/sink/SinkProcessorFactory.java 859f4fd
flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java 7b079f9
flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java a96016c
flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java d205bbc
flume-ng-core/src/test/java/org/apache/flume/TestContext.java a5e6aa8
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannel.java 3392dff
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelConcurrency.java 8dad0b2
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelTransaction.java bc81f26
flume-ng-core/src/test/java/org/apache/flume/sink/TestAvroSink.java 467785f
flume-ng-core/src/test/java/org/apache/flume/sink/TestFailoverSinkProcessor.java 93ad3bf
flume-ng-core/src/test/java/org/apache/flume/sink/TestLoggerSink.java 92ff6fe
flume-ng-core/src/test/java/org/apache/flume/sink/TestRollingFileSink.java 7e26e2a
flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java c5c3f2f
flume-ng-core/src/test/java/org/apache/flume/source/TestExecSource.java 6035270
flume-ng-core/src/test/java/org/apache/flume/source/TestPollableSourceRunner.java c27f82c
flume-ng-core/src/test/java/org/apache/flume/source/TestSequenceGeneratorSource.java 579b257
flume-ng-legacy-sources/flume-avro-source/src/main/java/org/apache/flume/source/avroLegacy/AvroLegacySource.java b1e67f7
flume-ng-legacy-sources/flume-avro-source/src/test/java/org/apache/flume/source/avroLegacy/TestLegacyAvroSource.java 6e3eb53
flume-ng-legacy-sources/flume-thrift-source/src/main/java/org/apache/flume/source/thriftLegacy/ThriftLegacySource.java 5fe270a
flume-ng-legacy-sources/flume-thrift-source/src/test/java/org/apache/flume/source/thriftLegacy/TestThriftLegacySource.java ddd9478
flume-ng-node/pom.xml b9b062e
flume-ng-node/src/main/java/org/apache/flume/conf/properties/FlumeConfiguration.java d66f6d1
flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java 97f72e1
flume-ng-node/src/test/java/org/apache/flume/node/TestAbstractLogicalNodeManager.java 521b586
flume-ng-node/src/test/java/org/apache/flume/source/TestNetcatSource.java fb2a960
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 3da90a5
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 7d8ee8a
flume-ng-sinks/flume-irc-sink/src/main/java/org/apache/flume/sink/irc/IRCSink.java 0a5498f
pom.xml d785762
Diff: https://reviews.apache.org/r/4115/diff
Testing
-------
All existing unit tests for the components whose configuration has now been moved to these stubs pass.
Thanks,
Hari