You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by "Hari Shreedharan (Created) (JIRA)" <ji...@apache.org> on 2012/02/23 21:16:48 UTC
[jira] [Created] (FLUME-992) Creat configuration stubs for sources,
channels, sinks etc
Creat configuration stubs for sources, channels, sinks etc
----------------------------------------------------------
Key: FLUME-992
URL: https://issues.apache.org/jira/browse/FLUME-992
Project: Flume
Issue Type: Sub-task
Affects Versions: v1.0.0
Reporter: Hari Shreedharan
Fix For: v1.1.0
Currently the configuration provider implementation encompasses all the syntactic and structural validation rules for loading the configuration. Externalizing this functionality to a library will allow external tools to easily operate on flume configuration files and be able to help parse and validate these files.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (FLUME-992) Create configuration stubs for
sources, channels, sinks etc
Posted by "Hari Shreedharan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hari Shreedharan resolved FLUME-992.
------------------------------------
Resolution: Fixed
> Create configuration stubs for sources, channels, sinks etc
> -----------------------------------------------------------
>
> Key: FLUME-992
> URL: https://issues.apache.org/jira/browse/FLUME-992
> Project: Flume
> Issue Type: Sub-task
> Affects Versions: v1.0.0
> Reporter: Hari Shreedharan
> Assignee: Hari Shreedharan
> Fix For: v1.2.0
>
> Original Estimate: 0h
> Remaining Estimate: 0h
>
> Currently the configuration of each component sits deep inside the component themselves. There is no way to verify if a configuration is valid before run time. In most systems like Flume, it is likely that these confs will be deployed from one single host, on to the machines where flume agents are running. Only when each agent starts, invalid confs are identified because the Agents would terminate by throwing exceptions. This is a first attempt to make a component-aware configuration system which is independent of the Flume, and does not require the Flume jar to be installed. Each component needs a configuration manager, which configures the components.
> Provide abstract Configuration stubs for each component type, sources, channels, sinks, selectors etc, which are in the new package, independent on ng-core. Now for each of the channels extend the abstract class and check the config properties for each of the required parameters for that component, for example: MultiplexingChannelSelectorConfigurator would look for default channel etc. If a particular component does not have a configuration class then let the current mechanism continue.
> This will require implementation for each component, but it should not be too complex. One additional advantage we get from this is that we can separate out the config validation from the components into these stubs, but we will still need to read the values out of the Context as we do currently(else we end up making the configuration dependent on flume-ng-core itself which we wanted to avoid).
> A problem with this is making a change to the configuration would require changes in the configuration classes and in the components also(where the configuration is read and the component is actually configured). I could not figure out a way of avoiding this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-992) Create configuration stubs for
sources, channels, sinks etc
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13229526#comment-13229526 ]
jiraposter@reviews.apache.org commented on FLUME-992:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4115/
-----------------------------------------------------------
(Updated 2012-03-14 19:21:20.299543)
Review request for Flume.
Changes
-------
rebased on trunk. The toString functions provide the same functionality as it used to provide before. I didnt want to change it, since I am not sur eif it is used by someone else.
Summary
-------
Currently the configuration of each component sits deep inside the component themselves. There is no way to verify if a configuration is valid before run time. In most systems like Flume, it is likely that these confs will be deployed from one single host, on to the machines where flume agents are running. Only when each agent starts, invalid confs are identified because the Agents would terminate by throwing exceptions. This is a first attempt to make a component-aware configuration system which is independent of the Flume, and does not require the Flume jar to be installed. Each component needs a configuration manager, which configures the components.
I have not completed the validation part yet, but have completed the configuration managers for all the components that come in the flume jar. I will add support for the other components, such as HDFS sink, JDBC channel etc soon. I will also add the validation support to the classes, but would like a review of the model being proposed here.
Note that I do know of a couple of bugs in the existing diff(like that if the name of a config key starts with an Upper case letter, an exception is thrown) - I will fix and upload the review here soon.
This addresses bug FLUME-992.
https://issues.apache.org/jira/browse/FLUME-992
Diffs (updated)
-----
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationErrorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationError.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/Context.java PRE-CREATION
flume-ng-clients/flume-ng-log4jappender/src/test/java/org/apache/flume/clients/log4jappender/TestLog4jAppender.java 68d95fb
flume-ng-configuration/pom.xml PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfigurationFactory.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ConfigurationException.java PRE-CREATION
flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestJdbcChannelProvider.java 7710d46
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannel.java bca0c50
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProvider.java e445d61
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProviderFactory.java 6fbd6ef
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java 307ae89
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MemoryChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MultiplexingChannelSelectorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/PseudoTxnMemoryChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/AvroSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/FailoverSinkProcessorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/RollingFileSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkGroupConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkProcessorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkProcessorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/AvroSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/ExecSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/NetcatSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceType.java PRE-CREATION
flume-ng-configuration/src/test/resources/flume-ng-error-test PRE-CREATION
flume-ng-configuration/src/test/resources/flume-ng-test.conf PRE-CREATION
flume-ng-core/pom.xml 247df37
flume-ng-core/src/main/java/org/apache/flume/ChannelSelector.java fba2dcb
flume-ng-core/src/main/java/org/apache/flume/Context.java 5294e31
flume-ng-core/src/main/java/org/apache/flume/SinkProcessor.java ea6000b
flume-ng-core/src/main/java/org/apache/flume/SinkProcessorType.java be1891b
flume-ng-core/src/main/java/org/apache/flume/channel/AbstractChannel.java 352bf08
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorFactory.java 800f471
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorType.java 511fc65
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8
flume-ng-core/src/main/java/org/apache/flume/channel/DefaultChannelFactory.java 963a6a3
flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java bfa1fde
flume-ng-core/src/main/java/org/apache/flume/channel/MultiplexingChannelSelector.java 83928b7
flume-ng-core/src/main/java/org/apache/flume/channel/PseudoTxnMemoryChannel.java 489d3e5
flume-ng-core/src/main/java/org/apache/flume/channel/ReplicatingChannelSelector.java 8f22746
flume-ng-core/src/main/java/org/apache/flume/conf/Configurable.java 0fa4839
flume-ng-core/src/main/java/org/apache/flume/conf/ConfigurableComponent.java PRE-CREATION
flume-ng-core/src/main/java/org/apache/flume/conf/Configurables.java 84492e5
flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 3edc563
flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkFactory.java b89dfa0
flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkProcessor.java 257bab3
flume-ng-core/src/main/java/org/apache/flume/sink/FailoverSinkProcessor.java 9f5b856
flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java 7f1d3b3
flume-ng-core/src/main/java/org/apache/flume/sink/SinkGroup.java 0dffd69
flume-ng-core/src/main/java/org/apache/flume/sink/SinkProcessorFactory.java 6160a17
flume-ng-core/src/main/java/org/apache/flume/sink/SinkType.java 6b08c09
flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java 903889d
flume-ng-core/src/main/java/org/apache/flume/source/DefaultSourceFactory.java a610e6f
flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java b6b1181
flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java 94245ac
flume-ng-core/src/main/java/org/apache/flume/source/SourceType.java 9082470
flume-ng-core/src/main/java/org/apache/flume/source/SyslogTcpSource.java d78d27f
flume-ng-core/src/main/java/org/apache/flume/source/SyslogUDPSource.java 34818f0
flume-ng-core/src/test/java/org/apache/flume/TestContext.java 51c350f
flume-ng-core/src/test/java/org/apache/flume/channel/AbstractBasicChannelSemanticsTest.java 6e71e46
flume-ng-core/src/test/java/org/apache/flume/channel/MockChannel.java 24b01e2
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannel.java e070864
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelConcurrency.java 8dad0b2
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelTransaction.java bc81f26
flume-ng-core/src/test/java/org/apache/flume/sink/TestAvroSink.java 3765924
flume-ng-core/src/test/java/org/apache/flume/sink/TestFailoverSinkProcessor.java 195c121
flume-ng-core/src/test/java/org/apache/flume/sink/TestLoggerSink.java 92ff6fe
flume-ng-core/src/test/java/org/apache/flume/sink/TestRollingFileSink.java 7e26e2a
flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java c5c3f2f
flume-ng-core/src/test/java/org/apache/flume/source/TestExecSource.java 6035270
flume-ng-core/src/test/java/org/apache/flume/source/TestPollableSourceRunner.java c27f82c
flume-ng-core/src/test/java/org/apache/flume/source/TestSequenceGeneratorSource.java 579b257
flume-ng-core/src/test/java/org/apache/flume/source/TestSyslogUdpSource.java d1db49d
flume-ng-legacy-sources/flume-avro-source/src/main/java/org/apache/flume/source/avroLegacy/AvroLegacySource.java dde8f28
flume-ng-legacy-sources/flume-avro-source/src/test/java/org/apache/flume/source/avroLegacy/TestLegacyAvroSource.java 6e3eb53
flume-ng-legacy-sources/flume-thrift-source/src/main/java/org/apache/flume/source/thriftLegacy/ThriftLegacySource.java fbf7362
flume-ng-legacy-sources/flume-thrift-source/src/test/java/org/apache/flume/source/thriftLegacy/TestThriftLegacySource.java ddd9478
flume-ng-node/src/main/java/org/apache/flume/conf/properties/FlumeConfiguration.java d66f6d1
flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java 97f72e1
flume-ng-node/src/test/java/org/apache/flume/node/TestAbstractLogicalNodeManager.java 521b586
flume-ng-node/src/test/java/org/apache/flume/source/TestNetcatSource.java fb2a960
flume-ng-node/src/test/resources/flume-conf.properties 848caca
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java da82f7e
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 8fa72a1
flume-ng-sinks/flume-irc-sink/src/main/java/org/apache/flume/sink/irc/IRCSink.java 8e77218
pom.xml 794ebe3
Diff: https://reviews.apache.org/r/4115/diff
Testing
-------
All existing unit tests for the components whose configuration has now been moved to these stubs pass.
Thanks,
Hari
> Create configuration stubs for sources, channels, sinks etc
> -----------------------------------------------------------
>
> Key: FLUME-992
> URL: https://issues.apache.org/jira/browse/FLUME-992
> Project: Flume
> Issue Type: Sub-task
> Affects Versions: v1.0.0
> Reporter: Hari Shreedharan
> Assignee: Hari Shreedharan
> Fix For: v1.2.0
>
>
> Currently the configuration of each component sits deep inside the component themselves. There is no way to verify if a configuration is valid before run time. In most systems like Flume, it is likely that these confs will be deployed from one single host, on to the machines where flume agents are running. Only when each agent starts, invalid confs are identified because the Agents would terminate by throwing exceptions. This is a first attempt to make a component-aware configuration system which is independent of the Flume, and does not require the Flume jar to be installed. Each component needs a configuration manager, which configures the components.
> Provide abstract Configuration stubs for each component type, sources, channels, sinks, selectors etc, which are in the new package, independent on ng-core. Now for each of the channels extend the abstract class and check the config properties for each of the required parameters for that component, for example: MultiplexingChannelSelectorConfigurator would look for default channel etc. If a particular component does not have a configuration class then let the current mechanism continue.
> This will require implementation for each component, but it should not be too complex. One additional advantage we get from this is that we can separate out the config validation from the components into these stubs, but we will still need to read the values out of the Context as we do currently(else we end up making the configuration dependent on flume-ng-core itself which we wanted to avoid).
> A problem with this is making a change to the configuration would require changes in the configuration classes and in the components also(where the configuration is read and the component is actually configured). I could not figure out a way of avoiding this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-992) Create configuration stubs for
sources, channels, sinks etc
Posted by "Hari Shreedharan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hari Shreedharan updated FLUME-992:
-----------------------------------
Remaining Estimate: 0h
Original Estimate: 0h
> Create configuration stubs for sources, channels, sinks etc
> -----------------------------------------------------------
>
> Key: FLUME-992
> URL: https://issues.apache.org/jira/browse/FLUME-992
> Project: Flume
> Issue Type: Sub-task
> Affects Versions: v1.0.0
> Reporter: Hari Shreedharan
> Assignee: Hari Shreedharan
> Fix For: v1.2.0
>
> Original Estimate: 0h
> Remaining Estimate: 0h
>
> Currently the configuration of each component sits deep inside the component themselves. There is no way to verify if a configuration is valid before run time. In most systems like Flume, it is likely that these confs will be deployed from one single host, on to the machines where flume agents are running. Only when each agent starts, invalid confs are identified because the Agents would terminate by throwing exceptions. This is a first attempt to make a component-aware configuration system which is independent of the Flume, and does not require the Flume jar to be installed. Each component needs a configuration manager, which configures the components.
> Provide abstract Configuration stubs for each component type, sources, channels, sinks, selectors etc, which are in the new package, independent on ng-core. Now for each of the channels extend the abstract class and check the config properties for each of the required parameters for that component, for example: MultiplexingChannelSelectorConfigurator would look for default channel etc. If a particular component does not have a configuration class then let the current mechanism continue.
> This will require implementation for each component, but it should not be too complex. One additional advantage we get from this is that we can separate out the config validation from the components into these stubs, but we will still need to read the values out of the Context as we do currently(else we end up making the configuration dependent on flume-ng-core itself which we wanted to avoid).
> A problem with this is making a change to the configuration would require changes in the configuration classes and in the components also(where the configuration is read and the component is actually configured). I could not figure out a way of avoiding this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-992) Create configuration stubs for
sources, channels, sinks etc
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13229948#comment-13229948 ]
jiraposter@reviews.apache.org commented on FLUME-992:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4115/
-----------------------------------------------------------
(Updated 2012-03-15 07:12:25.261211)
Review request for Flume.
Changes
-------
Fixing a bug which I found.
Summary
-------
Currently the configuration of each component sits deep inside the component themselves. There is no way to verify if a configuration is valid before run time. In most systems like Flume, it is likely that these confs will be deployed from one single host, on to the machines where flume agents are running. Only when each agent starts, invalid confs are identified because the Agents would terminate by throwing exceptions. This is a first attempt to make a component-aware configuration system which is independent of the Flume, and does not require the Flume jar to be installed. Each component needs a configuration manager, which configures the components.
I have not completed the validation part yet, but have completed the configuration managers for all the components that come in the flume jar. I will add support for the other components, such as HDFS sink, JDBC channel etc soon. I will also add the validation support to the classes, but would like a review of the model being proposed here.
Note that I do know of a couple of bugs in the existing diff(like that if the name of a config key starts with an Upper case letter, an exception is thrown) - I will fix and upload the review here soon.
This addresses bug FLUME-992.
https://issues.apache.org/jira/browse/FLUME-992
Diffs (updated)
-----
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannel.java bca0c50
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProvider.java e445d61
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProviderFactory.java 6fbd6ef
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java 307ae89
flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestJdbcChannelProvider.java 7710d46
flume-ng-clients/flume-ng-log4jappender/src/test/java/org/apache/flume/clients/log4jappender/TestLog4jAppender.java 68d95fb
flume-ng-configuration/pom.xml PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfigurationFactory.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ConfigurationException.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/Context.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationError.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationErrorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MemoryChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MultiplexingChannelSelectorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/PseudoTxnMemoryChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/AvroSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/FailoverSinkProcessorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/RollingFileSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkGroupConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkProcessorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkProcessorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/AvroSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/ExecSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/NetcatSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceType.java PRE-CREATION
flume-ng-configuration/src/test/resources/flume-ng-error-test PRE-CREATION
flume-ng-configuration/src/test/resources/flume-ng-test.conf PRE-CREATION
flume-ng-core/pom.xml 247df37
flume-ng-core/src/main/java/org/apache/flume/ChannelSelector.java fba2dcb
flume-ng-core/src/main/java/org/apache/flume/Context.java 5294e31
flume-ng-core/src/main/java/org/apache/flume/SinkProcessor.java ea6000b
flume-ng-core/src/main/java/org/apache/flume/SinkProcessorType.java be1891b
flume-ng-core/src/main/java/org/apache/flume/channel/AbstractChannel.java 352bf08
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorFactory.java 800f471
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorType.java 511fc65
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8
flume-ng-core/src/main/java/org/apache/flume/channel/DefaultChannelFactory.java 963a6a3
flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java bfa1fde
flume-ng-core/src/main/java/org/apache/flume/channel/MultiplexingChannelSelector.java 83928b7
flume-ng-core/src/main/java/org/apache/flume/channel/PseudoTxnMemoryChannel.java 489d3e5
flume-ng-core/src/main/java/org/apache/flume/channel/ReplicatingChannelSelector.java 8f22746
flume-ng-core/src/main/java/org/apache/flume/conf/Configurable.java 0fa4839
flume-ng-core/src/main/java/org/apache/flume/conf/ConfigurableComponent.java PRE-CREATION
flume-ng-core/src/main/java/org/apache/flume/conf/Configurables.java 84492e5
flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 3edc563
flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkFactory.java b89dfa0
flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkProcessor.java 257bab3
flume-ng-core/src/main/java/org/apache/flume/sink/FailoverSinkProcessor.java 9f5b856
flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java 7f1d3b3
flume-ng-core/src/main/java/org/apache/flume/sink/SinkGroup.java 0dffd69
flume-ng-core/src/main/java/org/apache/flume/sink/SinkProcessorFactory.java 6160a17
flume-ng-core/src/main/java/org/apache/flume/sink/SinkType.java 6b08c09
flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java 903889d
flume-ng-core/src/main/java/org/apache/flume/source/DefaultSourceFactory.java a610e6f
flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java b6b1181
flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java 94245ac
flume-ng-core/src/main/java/org/apache/flume/source/SequenceGeneratorSource.java 440c5a9
flume-ng-core/src/main/java/org/apache/flume/source/SourceType.java 9082470
flume-ng-core/src/main/java/org/apache/flume/source/SyslogTcpSource.java d78d27f
flume-ng-core/src/main/java/org/apache/flume/source/SyslogUDPSource.java 34818f0
flume-ng-core/src/test/java/org/apache/flume/TestContext.java 51c350f
flume-ng-core/src/test/java/org/apache/flume/channel/AbstractBasicChannelSemanticsTest.java 6e71e46
flume-ng-core/src/test/java/org/apache/flume/channel/MockChannel.java 24b01e2
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannel.java e070864
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelConcurrency.java 8dad0b2
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelTransaction.java bc81f26
flume-ng-core/src/test/java/org/apache/flume/sink/TestAvroSink.java 3765924
flume-ng-core/src/test/java/org/apache/flume/sink/TestFailoverSinkProcessor.java 195c121
flume-ng-core/src/test/java/org/apache/flume/sink/TestLoggerSink.java 92ff6fe
flume-ng-core/src/test/java/org/apache/flume/sink/TestRollingFileSink.java 7e26e2a
flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java c5c3f2f
flume-ng-core/src/test/java/org/apache/flume/source/TestExecSource.java 6035270
flume-ng-core/src/test/java/org/apache/flume/source/TestPollableSourceRunner.java c27f82c
flume-ng-core/src/test/java/org/apache/flume/source/TestSequenceGeneratorSource.java 579b257
flume-ng-core/src/test/java/org/apache/flume/source/TestSyslogUdpSource.java d1db49d
flume-ng-legacy-sources/flume-avro-source/src/main/java/org/apache/flume/source/avroLegacy/AvroLegacySource.java dde8f28
flume-ng-legacy-sources/flume-avro-source/src/test/java/org/apache/flume/source/avroLegacy/TestLegacyAvroSource.java 6e3eb53
flume-ng-legacy-sources/flume-thrift-source/src/main/java/org/apache/flume/source/thriftLegacy/ThriftLegacySource.java fbf7362
flume-ng-legacy-sources/flume-thrift-source/src/test/java/org/apache/flume/source/thriftLegacy/TestThriftLegacySource.java ddd9478
flume-ng-node/src/main/java/org/apache/flume/conf/properties/FlumeConfiguration.java d66f6d1
flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java 97f72e1
flume-ng-node/src/test/java/org/apache/flume/node/TestAbstractLogicalNodeManager.java 521b586
flume-ng-node/src/test/java/org/apache/flume/source/TestNetcatSource.java fb2a960
flume-ng-node/src/test/resources/flume-conf.properties 848caca
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java da82f7e
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 8fa72a1
flume-ng-sinks/flume-irc-sink/src/main/java/org/apache/flume/sink/irc/IRCSink.java 8e77218
pom.xml 794ebe3
Diff: https://reviews.apache.org/r/4115/diff
Testing
-------
All existing unit tests for the components whose configuration has now been moved to these stubs pass.
Thanks,
Hari
> Create configuration stubs for sources, channels, sinks etc
> -----------------------------------------------------------
>
> Key: FLUME-992
> URL: https://issues.apache.org/jira/browse/FLUME-992
> Project: Flume
> Issue Type: Sub-task
> Affects Versions: v1.0.0
> Reporter: Hari Shreedharan
> Assignee: Hari Shreedharan
> Fix For: v1.2.0
>
>
> Currently the configuration of each component sits deep inside the component themselves. There is no way to verify if a configuration is valid before run time. In most systems like Flume, it is likely that these confs will be deployed from one single host, on to the machines where flume agents are running. Only when each agent starts, invalid confs are identified because the Agents would terminate by throwing exceptions. This is a first attempt to make a component-aware configuration system which is independent of the Flume, and does not require the Flume jar to be installed. Each component needs a configuration manager, which configures the components.
> Provide abstract Configuration stubs for each component type, sources, channels, sinks, selectors etc, which are in the new package, independent on ng-core. Now for each of the channels extend the abstract class and check the config properties for each of the required parameters for that component, for example: MultiplexingChannelSelectorConfigurator would look for default channel etc. If a particular component does not have a configuration class then let the current mechanism continue.
> This will require implementation for each component, but it should not be too complex. One additional advantage we get from this is that we can separate out the config validation from the components into these stubs, but we will still need to read the values out of the Context as we do currently(else we end up making the configuration dependent on flume-ng-core itself which we wanted to avoid).
> A problem with this is making a change to the configuration would require changes in the configuration classes and in the components also(where the configuration is read and the component is actually configured). I could not figure out a way of avoiding this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-992) Create configuration stubs for
sources, channels, sinks etc
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13228262#comment-13228262 ]
jiraposter@reviews.apache.org commented on FLUME-992:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4115/
-----------------------------------------------------------
(Updated 2012-03-13 07:45:16.395171)
Review request for Flume.
Changes
-------
Incorporate much of the feedback. Errors now pushed up to higher level. Need to discuss plan to test this, and then will add tests - will need more system tests, rather than unit tests.
Summary
-------
Currently the configuration of each component sits deep inside the component themselves. There is no way to verify if a configuration is valid before run time. In most systems like Flume, it is likely that these confs will be deployed from one single host, on to the machines where flume agents are running. Only when each agent starts, invalid confs are identified because the Agents would terminate by throwing exceptions. This is a first attempt to make a component-aware configuration system which is independent of the Flume, and does not require the Flume jar to be installed. Each component needs a configuration manager, which configures the components.
I have not completed the validation part yet, but have completed the configuration managers for all the components that come in the flume jar. I will add support for the other components, such as HDFS sink, JDBC channel etc soon. I will also add the validation support to the classes, but would like a review of the model being proposed here.
Note that I do know of a couple of bugs in the existing diff(like that if the name of a config key starts with an Upper case letter, an exception is thrown) - I will fix and upload the review here soon.
This addresses bug FLUME-992.
https://issues.apache.org/jira/browse/FLUME-992
Diffs (updated)
-----
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannel.java bca0c50
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProvider.java e445d61
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProviderFactory.java 6fbd6ef
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java 307ae89
flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestJdbcChannelProvider.java 7710d46
flume-ng-clients/flume-ng-log4jappender/src/test/java/org/apache/flume/clients/log4jappender/TestLog4jAppender.java 68d95fb
flume-ng-configuration/pom.xml PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfigurationFactory.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ConfigurationException.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/Context.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationError.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationErrorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MemoryChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MultiplexingChannelSelectorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/PseudoTxnMemoryChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/AvroSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/FailoverSinkProcessorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/RollingFileSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkGroupConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkProcessorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkProcessorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/AvroSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/ExecSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/NetcatSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceType.java PRE-CREATION
flume-ng-configuration/src/test/resources/flume-ng-error-test PRE-CREATION
flume-ng-configuration/src/test/resources/flume-ng-test.conf PRE-CREATION
flume-ng-core/pom.xml d753fa1
flume-ng-core/src/main/java/org/apache/flume/ChannelSelector.java fba2dcb
flume-ng-core/src/main/java/org/apache/flume/Context.java a6341a5
flume-ng-core/src/main/java/org/apache/flume/SinkProcessor.java 11651ad
flume-ng-core/src/main/java/org/apache/flume/SinkProcessorType.java be1891b
flume-ng-core/src/main/java/org/apache/flume/channel/AbstractChannel.java 352bf08
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorFactory.java 800f471
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorType.java 511fc65
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8
flume-ng-core/src/main/java/org/apache/flume/channel/DefaultChannelFactory.java 963a6a3
flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java bfa1fde
flume-ng-core/src/main/java/org/apache/flume/channel/MultiplexingChannelSelector.java 83928b7
flume-ng-core/src/main/java/org/apache/flume/channel/PseudoTxnMemoryChannel.java 489d3e5
flume-ng-core/src/main/java/org/apache/flume/channel/ReplicatingChannelSelector.java 8f22746
flume-ng-core/src/main/java/org/apache/flume/conf/Configurable.java 0fa4839
flume-ng-core/src/main/java/org/apache/flume/conf/ConfigurableComponent.java PRE-CREATION
flume-ng-core/src/main/java/org/apache/flume/conf/Configurables.java 84492e5
flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 7386d06
flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkFactory.java b89dfa0
flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkProcessor.java 257bab3
flume-ng-core/src/main/java/org/apache/flume/sink/FailoverSinkProcessor.java 9f5b856
flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java 7f1d3b3
flume-ng-core/src/main/java/org/apache/flume/sink/SinkGroup.java 2e80a56
flume-ng-core/src/main/java/org/apache/flume/sink/SinkProcessorFactory.java 10f9f4e
flume-ng-core/src/main/java/org/apache/flume/sink/SinkType.java 6b08c09
flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java a3f6640
flume-ng-core/src/main/java/org/apache/flume/source/DefaultSourceFactory.java a610e6f
flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java b6b1181
flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java 94245ac
flume-ng-core/src/main/java/org/apache/flume/source/SourceType.java 9082470
flume-ng-core/src/test/java/org/apache/flume/TestContext.java 51c350f
flume-ng-core/src/test/java/org/apache/flume/channel/AbstractBasicChannelSemanticsTest.java 6e71e46
flume-ng-core/src/test/java/org/apache/flume/channel/MockChannel.java 24b01e2
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannel.java e070864
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelConcurrency.java 8dad0b2
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelTransaction.java bc81f26
flume-ng-core/src/test/java/org/apache/flume/sink/TestAvroSink.java 467785f
flume-ng-core/src/test/java/org/apache/flume/sink/TestFailoverSinkProcessor.java 195c121
flume-ng-core/src/test/java/org/apache/flume/sink/TestLoggerSink.java 92ff6fe
flume-ng-core/src/test/java/org/apache/flume/sink/TestRollingFileSink.java 7e26e2a
flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java c5c3f2f
flume-ng-core/src/test/java/org/apache/flume/source/TestExecSource.java 6035270
flume-ng-core/src/test/java/org/apache/flume/source/TestPollableSourceRunner.java c27f82c
flume-ng-core/src/test/java/org/apache/flume/source/TestSequenceGeneratorSource.java 579b257
flume-ng-legacy-sources/flume-avro-source/src/main/java/org/apache/flume/source/avroLegacy/AvroLegacySource.java 65d9142
flume-ng-legacy-sources/flume-avro-source/src/test/java/org/apache/flume/source/avroLegacy/TestLegacyAvroSource.java 6e3eb53
flume-ng-legacy-sources/flume-thrift-source/src/main/java/org/apache/flume/source/thriftLegacy/ThriftLegacySource.java fbf7362
flume-ng-legacy-sources/flume-thrift-source/src/test/java/org/apache/flume/source/thriftLegacy/TestThriftLegacySource.java ddd9478
flume-ng-node/src/main/java/org/apache/flume/conf/properties/FlumeConfiguration.java d66f6d1
flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java 97f72e1
flume-ng-node/src/test/java/org/apache/flume/node/TestAbstractLogicalNodeManager.java 521b586
flume-ng-node/src/test/java/org/apache/flume/source/TestNetcatSource.java fb2a960
flume-ng-node/src/test/resources/flume-conf.properties 848caca
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 524b69c
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 7d8ee8a
flume-ng-sinks/flume-irc-sink/src/main/java/org/apache/flume/sink/irc/IRCSink.java 8e77218
pom.xml d785762
Diff: https://reviews.apache.org/r/4115/diff
Testing
-------
All existing unit tests for the components whose configuration has now been moved to these stubs pass.
Thanks,
Hari
> Create configuration stubs for sources, channels, sinks etc
> -----------------------------------------------------------
>
> Key: FLUME-992
> URL: https://issues.apache.org/jira/browse/FLUME-992
> Project: Flume
> Issue Type: Sub-task
> Affects Versions: v1.0.0
> Reporter: Hari Shreedharan
> Assignee: Hari Shreedharan
> Fix For: v1.2.0
>
>
> Currently the configuration of each component sits deep inside the component themselves. There is no way to verify if a configuration is valid before run time. In most systems like Flume, it is likely that these confs will be deployed from one single host, on to the machines where flume agents are running. Only when each agent starts, invalid confs are identified because the Agents would terminate by throwing exceptions. This is a first attempt to make a component-aware configuration system which is independent of the Flume, and does not require the Flume jar to be installed. Each component needs a configuration manager, which configures the components.
> Provide abstract Configuration stubs for each component type, sources, channels, sinks, selectors etc, which are in the new package, independent on ng-core. Now for each of the channels extend the abstract class and check the config properties for each of the required parameters for that component, for example: MultiplexingChannelSelectorConfigurator would look for default channel etc. If a particular component does not have a configuration class then let the current mechanism continue.
> This will require implementation for each component, but it should not be too complex. One additional advantage we get from this is that we can separate out the config validation from the components into these stubs, but we will still need to read the values out of the Context as we do currently(else we end up making the configuration dependent on flume-ng-core itself which we wanted to avoid).
> A problem with this is making a change to the configuration would require changes in the configuration classes and in the components also(where the configuration is read and the component is actually configured). I could not figure out a way of avoiding this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-992) Create configuration stubs for
sources, channels, sinks etc
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13238175#comment-13238175 ]
jiraposter@reviews.apache.org commented on FLUME-992:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4115/
-----------------------------------------------------------
(Updated 2012-03-26 07:34:11.739683)
Review request for Flume.
Changes
-------
Removed most whitespace changes. I will split this up soon, for now, just trying to make sure all whitespaces changes were reverted.
Summary
-------
Currently the configuration of each component sits deep inside the component themselves. There is no way to verify if a configuration is valid before run time. In most systems like Flume, it is likely that these confs will be deployed from one single host, on to the machines where flume agents are running. Only when each agent starts, invalid confs are identified because the Agents would terminate by throwing exceptions. This is a first attempt to make a component-aware configuration system which is independent of the Flume, and does not require the Flume jar to be installed. Each component needs a configuration manager, which configures the components.
I have not completed the validation part yet, but have completed the configuration managers for all the components that come in the flume jar. I will add support for the other components, such as HDFS sink, JDBC channel etc soon. I will also add the validation support to the classes, but would like a review of the model being proposed here.
Note that I do know of a couple of bugs in the existing diff(like that if the name of a config key starts with an Upper case letter, an exception is thrown) - I will fix and upload the review here soon.
This addresses bug FLUME-992.
https://issues.apache.org/jira/browse/FLUME-992
Diffs (updated)
-----
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannel.java bca0c50
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProvider.java e445d61
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProviderFactory.java 6fbd6ef
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java 307ae89
flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestJdbcChannelProvider.java 7710d46
flume-ng-clients/flume-ng-log4jappender/src/test/java/org/apache/flume/clients/log4jappender/TestLog4jAppender.java 68d95fb
flume-ng-configuration/pom.xml PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfigurationFactory.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ConfigurationException.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/Context.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationError.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationErrorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/JdbcChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MemoryChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MultiplexingChannelSelectorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/PseudoTxnMemoryChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/AvroSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/FailoverSinkProcessorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/HDFSSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/RollingFileSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkGroupConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkProcessorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkProcessorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/AvroSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/ExecSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/ExecSourceConfigurationConstants.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/NetcatSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceType.java PRE-CREATION
flume-ng-configuration/src/test/resources/flume-ng-error-test PRE-CREATION
flume-ng-configuration/src/test/resources/flume-ng-test.conf PRE-CREATION
flume-ng-core/pom.xml 37fb112
flume-ng-core/src/main/java/org/apache/flume/ChannelSelector.java fba2dcb
flume-ng-core/src/main/java/org/apache/flume/Context.java 5294e31
flume-ng-core/src/main/java/org/apache/flume/SinkProcessor.java d863ed0
flume-ng-core/src/main/java/org/apache/flume/SinkProcessorType.java be1891b
flume-ng-core/src/main/java/org/apache/flume/channel/AbstractChannel.java 352bf08
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorFactory.java 800f471
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorType.java 511fc65
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8
flume-ng-core/src/main/java/org/apache/flume/channel/DefaultChannelFactory.java 963a6a3
flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java bfa1fde
flume-ng-core/src/main/java/org/apache/flume/channel/MultiplexingChannelSelector.java 83928b7
flume-ng-core/src/main/java/org/apache/flume/channel/PseudoTxnMemoryChannel.java 489d3e5
flume-ng-core/src/main/java/org/apache/flume/channel/ReplicatingChannelSelector.java 8f22746
flume-ng-core/src/main/java/org/apache/flume/conf/Configurable.java 1d69112
flume-ng-core/src/main/java/org/apache/flume/conf/ConfigurableComponent.java PRE-CREATION
flume-ng-core/src/main/java/org/apache/flume/conf/Configurables.java 84492e5
flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 3edc563
flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkFactory.java b89dfa0
flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkProcessor.java 257bab3
flume-ng-core/src/main/java/org/apache/flume/sink/FailoverSinkProcessor.java 066ee37
flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java 7f1d3b3
flume-ng-core/src/main/java/org/apache/flume/sink/SinkGroup.java 0dffd69
flume-ng-core/src/main/java/org/apache/flume/sink/SinkProcessorFactory.java 6160a17
flume-ng-core/src/main/java/org/apache/flume/sink/SinkType.java 6b08c09
flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java 903889d
flume-ng-core/src/main/java/org/apache/flume/source/DefaultSourceFactory.java a610e6f
flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java dbf79e0
flume-ng-core/src/main/java/org/apache/flume/source/ExecSourceConfigurationConstants.java 73c985e
flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java a841b0e
flume-ng-core/src/main/java/org/apache/flume/source/SequenceGeneratorSource.java 440c5a9
flume-ng-core/src/main/java/org/apache/flume/source/SourceType.java 9082470
flume-ng-core/src/main/java/org/apache/flume/source/SyslogTcpSource.java d78d27f
flume-ng-core/src/main/java/org/apache/flume/source/SyslogUDPSource.java 34818f0
flume-ng-core/src/test/java/org/apache/flume/TestContext.java 51c350f
flume-ng-core/src/test/java/org/apache/flume/channel/AbstractBasicChannelSemanticsTest.java 6e71e46
flume-ng-core/src/test/java/org/apache/flume/channel/MockChannel.java 24b01e2
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannel.java e070864
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelConcurrency.java 8dad0b2
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelTransaction.java bc81f26
flume-ng-core/src/test/java/org/apache/flume/sink/TestAvroSink.java 3765924
flume-ng-core/src/test/java/org/apache/flume/sink/TestFailoverSinkProcessor.java 3358cf4
flume-ng-core/src/test/java/org/apache/flume/sink/TestLoggerSink.java 92ff6fe
flume-ng-core/src/test/java/org/apache/flume/sink/TestRollingFileSink.java 7e26e2a
flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java c5c3f2f
flume-ng-core/src/test/java/org/apache/flume/source/TestExecSource.java 615f2a3
flume-ng-core/src/test/java/org/apache/flume/source/TestPollableSourceRunner.java c27f82c
flume-ng-core/src/test/java/org/apache/flume/source/TestSequenceGeneratorSource.java 579b257
flume-ng-core/src/test/java/org/apache/flume/source/TestSyslogUdpSource.java d1db49d
flume-ng-legacy-sources/flume-avro-source/src/main/java/org/apache/flume/source/avroLegacy/AvroLegacySource.java dde8f28
flume-ng-legacy-sources/flume-avro-source/src/test/java/org/apache/flume/source/avroLegacy/TestLegacyAvroSource.java 6e3eb53
flume-ng-legacy-sources/flume-thrift-source/src/main/java/org/apache/flume/source/thriftLegacy/ThriftLegacySource.java 00df7e4
flume-ng-legacy-sources/flume-thrift-source/src/test/java/org/apache/flume/source/thriftLegacy/TestThriftLegacySource.java ddd9478
flume-ng-node/src/main/java/org/apache/flume/conf/properties/FlumeConfiguration.java d66f6d1
flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java 50b9f0c
flume-ng-node/src/test/java/org/apache/flume/node/TestAbstractLogicalNodeManager.java 521b586
flume-ng-node/src/test/java/org/apache/flume/source/TestNetcatSource.java fb2a960
flume-ng-node/src/test/resources/flume-conf.properties 848caca
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 203886e
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 6ff3737
flume-ng-sinks/flume-irc-sink/src/main/java/org/apache/flume/sink/irc/IRCSink.java 8e77218
pom.xml 19798b2
Diff: https://reviews.apache.org/r/4115/diff
Testing
-------
All existing unit tests for the components whose configuration has now been moved to these stubs pass.
Thanks,
Hari
> Create configuration stubs for sources, channels, sinks etc
> -----------------------------------------------------------
>
> Key: FLUME-992
> URL: https://issues.apache.org/jira/browse/FLUME-992
> Project: Flume
> Issue Type: Sub-task
> Affects Versions: v1.0.0
> Reporter: Hari Shreedharan
> Assignee: Hari Shreedharan
> Fix For: v1.2.0
>
>
> Currently the configuration of each component sits deep inside the component themselves. There is no way to verify if a configuration is valid before run time. In most systems like Flume, it is likely that these confs will be deployed from one single host, on to the machines where flume agents are running. Only when each agent starts, invalid confs are identified because the Agents would terminate by throwing exceptions. This is a first attempt to make a component-aware configuration system which is independent of the Flume, and does not require the Flume jar to be installed. Each component needs a configuration manager, which configures the components.
> Provide abstract Configuration stubs for each component type, sources, channels, sinks, selectors etc, which are in the new package, independent on ng-core. Now for each of the channels extend the abstract class and check the config properties for each of the required parameters for that component, for example: MultiplexingChannelSelectorConfigurator would look for default channel etc. If a particular component does not have a configuration class then let the current mechanism continue.
> This will require implementation for each component, but it should not be too complex. One additional advantage we get from this is that we can separate out the config validation from the components into these stubs, but we will still need to read the values out of the Context as we do currently(else we end up making the configuration dependent on flume-ng-core itself which we wanted to avoid).
> A problem with this is making a change to the configuration would require changes in the configuration classes and in the components also(where the configuration is read and the component is actually configured). I could not figure out a way of avoiding this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-992) Create configuration stubs for
sources, channels, sinks etc
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225758#comment-13225758 ]
jiraposter@reviews.apache.org commented on FLUME-992:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4115/
-----------------------------------------------------------
(Updated 2012-03-09 01:04:54.371110)
Review request for Flume.
Changes
-------
Updated licenses.
Summary
-------
Currently the configuration of each component sits deep inside the component themselves. There is no way to verify if a configuration is valid before run time. In most systems like Flume, it is likely that these confs will be deployed from one single host, on to the machines where flume agents are running. Only when each agent starts, invalid confs are identified because the Agents would terminate by throwing exceptions. This is a first attempt to make a component-aware configuration system which is independent of the Flume, and does not require the Flume jar to be installed. Each component needs a configuration manager, which configures the components.
I have not completed the validation part yet, but have completed the configuration managers for all the components that come in the flume jar. I will add support for the other components, such as HDFS sink, JDBC channel etc soon. I will also add the validation support to the classes, but would like a review of the model being proposed here.
Note that I do know of a couple of bugs in the existing diff(like that if the name of a config key starts with an Upper case letter, an exception is thrown) - I will fix and upload the review here soon.
This addresses bug FLUME-992.
https://issues.apache.org/jira/browse/FLUME-992
Diffs (updated)
-----
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannel.java bca0c50
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProvider.java e445d61
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProviderFactory.java 6fbd6ef
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java 307ae89
flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestJdbcChannelProvider.java 7710d46
flume-ng-clients/flume-ng-log4jappender/src/test/java/org/apache/flume/clients/log4jappender/TestLog4jAppender.java 68d95fb
flume-ng-configuration/pom.xml PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfigurationFactory.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ConfigurationException.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/Context.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationError.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationErrorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MemoryChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MultiplexingChannelSelectorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/PseudoTxnMemoryChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/AvroSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/FailoverSinkProcessorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/RollingFileSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkGroupConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkProcessorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkProcessorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/AvroSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/ExecSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/NetcatSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceType.java PRE-CREATION
flume-ng-configuration/src/test/resources/flume-ng-error-test PRE-CREATION
flume-ng-configuration/src/test/resources/flume-ng-test.conf PRE-CREATION
flume-ng-core/pom.xml d753fa1
flume-ng-core/src/main/java/org/apache/flume/ChannelSelector.java fba2dcb
flume-ng-core/src/main/java/org/apache/flume/Context.java a6341a5
flume-ng-core/src/main/java/org/apache/flume/SinkProcessor.java 11651ad
flume-ng-core/src/main/java/org/apache/flume/SinkProcessorType.java be1891b
flume-ng-core/src/main/java/org/apache/flume/channel/AbstractChannel.java 352bf08
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorFactory.java 800f471
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorType.java 511fc65
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8
flume-ng-core/src/main/java/org/apache/flume/channel/DefaultChannelFactory.java 963a6a3
flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java bfa1fde
flume-ng-core/src/main/java/org/apache/flume/channel/MultiplexingChannelSelector.java 83928b7
flume-ng-core/src/main/java/org/apache/flume/channel/PseudoTxnMemoryChannel.java 489d3e5
flume-ng-core/src/main/java/org/apache/flume/channel/ReplicatingChannelSelector.java 8f22746
flume-ng-core/src/main/java/org/apache/flume/conf/Configurable.java 0fa4839
flume-ng-core/src/main/java/org/apache/flume/conf/ConfigurableComponent.java PRE-CREATION
flume-ng-core/src/main/java/org/apache/flume/conf/Configurables.java 84492e5
flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 7386d06
flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkFactory.java b89dfa0
flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkProcessor.java 257bab3
flume-ng-core/src/main/java/org/apache/flume/sink/FailoverSinkProcessor.java 9f5b856
flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java 7f1d3b3
flume-ng-core/src/main/java/org/apache/flume/sink/SinkGroup.java 2e80a56
flume-ng-core/src/main/java/org/apache/flume/sink/SinkProcessorFactory.java 10f9f4e
flume-ng-core/src/main/java/org/apache/flume/sink/SinkType.java 6b08c09
flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java a3f6640
flume-ng-core/src/main/java/org/apache/flume/source/DefaultSourceFactory.java a610e6f
flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java b6b1181
flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java 94245ac
flume-ng-core/src/main/java/org/apache/flume/source/SourceType.java 9082470
flume-ng-core/src/test/java/org/apache/flume/TestContext.java 51c350f
flume-ng-core/src/test/java/org/apache/flume/channel/AbstractBasicChannelSemanticsTest.java 6e71e46
flume-ng-core/src/test/java/org/apache/flume/channel/MockChannel.java 24b01e2
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannel.java e070864
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelConcurrency.java 8dad0b2
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelTransaction.java bc81f26
flume-ng-core/src/test/java/org/apache/flume/sink/TestAvroSink.java 467785f
flume-ng-core/src/test/java/org/apache/flume/sink/TestFailoverSinkProcessor.java 195c121
flume-ng-core/src/test/java/org/apache/flume/sink/TestLoggerSink.java 92ff6fe
flume-ng-core/src/test/java/org/apache/flume/sink/TestRollingFileSink.java 7e26e2a
flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java c5c3f2f
flume-ng-core/src/test/java/org/apache/flume/source/TestExecSource.java 6035270
flume-ng-core/src/test/java/org/apache/flume/source/TestPollableSourceRunner.java c27f82c
flume-ng-core/src/test/java/org/apache/flume/source/TestSequenceGeneratorSource.java 579b257
flume-ng-legacy-sources/flume-avro-source/src/main/java/org/apache/flume/source/avroLegacy/AvroLegacySource.java 65d9142
flume-ng-legacy-sources/flume-avro-source/src/test/java/org/apache/flume/source/avroLegacy/TestLegacyAvroSource.java 6e3eb53
flume-ng-legacy-sources/flume-thrift-source/src/main/java/org/apache/flume/source/thriftLegacy/ThriftLegacySource.java fbf7362
flume-ng-legacy-sources/flume-thrift-source/src/test/java/org/apache/flume/source/thriftLegacy/TestThriftLegacySource.java ddd9478
flume-ng-node/src/main/java/org/apache/flume/conf/properties/FlumeConfiguration.java d66f6d1
flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java 97f72e1
flume-ng-node/src/test/java/org/apache/flume/node/TestAbstractLogicalNodeManager.java 521b586
flume-ng-node/src/test/java/org/apache/flume/source/TestNetcatSource.java fb2a960
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 524b69c
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 7d8ee8a
flume-ng-sinks/flume-irc-sink/src/main/java/org/apache/flume/sink/irc/IRCSink.java 8e77218
pom.xml d785762
Diff: https://reviews.apache.org/r/4115/diff
Testing
-------
All existing unit tests for the components whose configuration has now been moved to these stubs pass.
Thanks,
Hari
> Create configuration stubs for sources, channels, sinks etc
> -----------------------------------------------------------
>
> Key: FLUME-992
> URL: https://issues.apache.org/jira/browse/FLUME-992
> Project: Flume
> Issue Type: Sub-task
> Affects Versions: v1.0.0
> Reporter: Hari Shreedharan
> Assignee: Hari Shreedharan
> Fix For: v1.1.0
>
>
> Currently the configuration of each component sits deep inside the component themselves. There is no way to verify if a configuration is valid before run time. In most systems like Flume, it is likely that these confs will be deployed from one single host, on to the machines where flume agents are running. Only when each agent starts, invalid confs are identified because the Agents would terminate by throwing exceptions. This is a first attempt to make a component-aware configuration system which is independent of the Flume, and does not require the Flume jar to be installed. Each component needs a configuration manager, which configures the components.
> Provide abstract Configuration stubs for each component type, sources, channels, sinks, selectors etc, which are in the new package, independent on ng-core. Now for each of the channels extend the abstract class and check the config properties for each of the required parameters for that component, for example: MultiplexingChannelSelectorConfigurator would look for default channel etc. If a particular component does not have a configuration class then let the current mechanism continue.
> This will require implementation for each component, but it should not be too complex. One additional advantage we get from this is that we can separate out the config validation from the components into these stubs, but we will still need to read the values out of the Context as we do currently(else we end up making the configuration dependent on flume-ng-core itself which we wanted to avoid).
> A problem with this is making a change to the configuration would require changes in the configuration classes and in the components also(where the configuration is read and the component is actually configured). I could not figure out a way of avoiding this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-992) Create configuration stubs for
sources, channels, sinks etc
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13220388#comment-13220388 ]
jiraposter@reviews.apache.org commented on FLUME-992:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4115/
-----------------------------------------------------------
(Updated 2012-03-01 21:35:41.679397)
Review request for Flume.
Changes
-------
Note that I do know of a couple of bugs in the existing diff(like that if the name of a config key starts with an Upper case letter, an exception is thrown) - I will fix and upload the review here soon.
Summary (updated)
-------
Currently the configuration of each component sits deep inside the component themselves. There is no way to verify if a configuration is valid before run time. In most systems like Flume, it is likely that these confs will be deployed from one single host, on to the machines where flume agents are running. Only when each agent starts, invalid confs are identified because the Agents would terminate by throwing exceptions. This is a first attempt to make a component-aware configuration system which is independent of the Flume, and does not require the Flume jar to be installed. Each component needs a configuration manager, which configures the components.
I have not completed the validation part yet, but have completed the configuration managers for all the components that come in the flume jar. I will add support for the other components, such as HDFS sink, JDBC channel etc soon. I will also add the validation support to the classes, but would like a review of the model being proposed here.
Note that I do know of a couple of bugs in the existing diff(like that if the name of a config key starts with an Upper case letter, an exception is thrown) - I will fix and upload the review here soon.
This addresses bug FLUME-992.
https://issues.apache.org/jira/browse/FLUME-992
Diffs
-----
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannel.java bca0c50
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProvider.java e445d61
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProviderFactory.java 6fbd6ef
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java 1cf1c0c
flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestJdbcChannelProvider.java 7710d46
flume-ng-clients/flume-ng-log4jappender/src/test/java/org/apache/flume/clients/log4jappender/TestLog4jAppender.java 68d95fb
flume-ng-configuration/pom.xml PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/AbstractComponentConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ConfigurationException.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/Context.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationError.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationErrorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MemoryChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MultiplexingChannelSelectorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/PseudoTxnMemoryChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/AvroSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/FailoverSinkProcessorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/RollingFileSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkGroupConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/AvroSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/ExecSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/NetcatSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/test/resources/flume-ng-error-test PRE-CREATION
flume-ng-configuration/src/test/resources/flume-ng-test.conf PRE-CREATION
flume-ng-core/pom.xml d753fa1
flume-ng-core/src/main/java/org/apache/flume/Context.java f1c8f85
flume-ng-core/src/main/java/org/apache/flume/channel/AbstractChannel.java 352bf08
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorFactory.java a7d5f94
flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java e79490e
flume-ng-core/src/main/java/org/apache/flume/channel/MultiplexingChannelSelector.java c63d0a1
flume-ng-core/src/main/java/org/apache/flume/channel/PseudoTxnMemoryChannel.java 1df580e
flume-ng-core/src/main/java/org/apache/flume/channel/ReplicatingChannelSelector.java 8f22746
flume-ng-core/src/main/java/org/apache/flume/conf/Configurable.java 0fa4839
flume-ng-core/src/main/java/org/apache/flume/conf/Configurables.java 84492e5
flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 5440631
flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkProcessor.java 257bab3
flume-ng-core/src/main/java/org/apache/flume/sink/FailoverSinkProcessor.java 9f5b856
flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java 45c031d
flume-ng-core/src/main/java/org/apache/flume/sink/SinkGroup.java 1adc5ff
flume-ng-core/src/main/java/org/apache/flume/sink/SinkProcessorFactory.java 859f4fd
flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java 7b079f9
flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java a96016c
flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java d205bbc
flume-ng-core/src/test/java/org/apache/flume/TestContext.java a5e6aa8
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannel.java 3392dff
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelConcurrency.java 8dad0b2
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelTransaction.java bc81f26
flume-ng-core/src/test/java/org/apache/flume/sink/TestAvroSink.java 467785f
flume-ng-core/src/test/java/org/apache/flume/sink/TestFailoverSinkProcessor.java 93ad3bf
flume-ng-core/src/test/java/org/apache/flume/sink/TestLoggerSink.java 92ff6fe
flume-ng-core/src/test/java/org/apache/flume/sink/TestRollingFileSink.java 7e26e2a
flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java c5c3f2f
flume-ng-core/src/test/java/org/apache/flume/source/TestExecSource.java 6035270
flume-ng-core/src/test/java/org/apache/flume/source/TestPollableSourceRunner.java c27f82c
flume-ng-core/src/test/java/org/apache/flume/source/TestSequenceGeneratorSource.java 579b257
flume-ng-legacy-sources/flume-avro-source/src/main/java/org/apache/flume/source/avroLegacy/AvroLegacySource.java b1e67f7
flume-ng-legacy-sources/flume-avro-source/src/test/java/org/apache/flume/source/avroLegacy/TestLegacyAvroSource.java 6e3eb53
flume-ng-legacy-sources/flume-thrift-source/src/main/java/org/apache/flume/source/thriftLegacy/ThriftLegacySource.java 5fe270a
flume-ng-legacy-sources/flume-thrift-source/src/test/java/org/apache/flume/source/thriftLegacy/TestThriftLegacySource.java ddd9478
flume-ng-node/pom.xml b9b062e
flume-ng-node/src/main/java/org/apache/flume/conf/properties/FlumeConfiguration.java d66f6d1
flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java 97f72e1
flume-ng-node/src/test/java/org/apache/flume/node/TestAbstractLogicalNodeManager.java 521b586
flume-ng-node/src/test/java/org/apache/flume/source/TestNetcatSource.java fb2a960
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 3da90a5
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 7d8ee8a
flume-ng-sinks/flume-irc-sink/src/main/java/org/apache/flume/sink/irc/IRCSink.java 0a5498f
pom.xml d785762
Diff: https://reviews.apache.org/r/4115/diff
Testing
-------
All existing unit tests for the components whose configuration has now been moved to these stubs pass.
Thanks,
Hari
> Create configuration stubs for sources, channels, sinks etc
> -----------------------------------------------------------
>
> Key: FLUME-992
> URL: https://issues.apache.org/jira/browse/FLUME-992
> Project: Flume
> Issue Type: Sub-task
> Affects Versions: v1.0.0
> Reporter: Hari Shreedharan
> Fix For: v1.1.0
>
>
> Currently the configuration of each component sits deep inside the component themselves. There is no way to verify if a configuration is valid before run time. In most systems like Flume, it is likely that these confs will be deployed from one single host, on to the machines where flume agents are running. Only when each agent starts, invalid confs are identified because the Agents would terminate by throwing exceptions. This is a first attempt to make a component-aware configuration system which is independent of the Flume, and does not require the Flume jar to be installed. Each component needs a configuration manager, which configures the components.
> Provide abstract Configuration stubs for each component type, sources, channels, sinks, selectors etc, which are in the new package, independent on ng-core. Now for each of the channels extend the abstract class and check the config properties for each of the required parameters for that component, for example: MultiplexingChannelSelectorConfigurator would look for default channel etc. If a particular component does not have a configuration class then let the current mechanism continue.
> This will require implementation for each component, but it should not be too complex. One additional advantage we get from this is that we can separate out the config validation from the components into these stubs, but we will still need to read the values out of the Context as we do currently(else we end up making the configuration dependent on flume-ng-core itself which we wanted to avoid).
> A problem with this is making a change to the configuration would require changes in the configuration classes and in the components also(where the configuration is read and the component is actually configured). I could not figure out a way of avoiding this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-992) Create configuration stubs for
sources, channels, sinks etc
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225742#comment-13225742 ]
jiraposter@reviews.apache.org commented on FLUME-992:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4115/
-----------------------------------------------------------
(Updated 2012-03-09 00:48:11.972265)
Review request for Flume.
Changes
-------
This is being submitted for early review. It builds, but is not based off of the latest commits. Currently it is not tested yet.
I am requesting a review of the new configuration model, rather than the code itself. I am sure there are a ton of fixes we can make to this.
Not all components have been updated(like JdbcChannel, HDFS and IRC sinks have not been updated yet). There probably are several bugs which I will iron out once I start testing the changes rigorously, but please do point out major logical bugs you may find.
Summary
-------
Currently the configuration of each component sits deep inside the component themselves. There is no way to verify if a configuration is valid before run time. In most systems like Flume, it is likely that these confs will be deployed from one single host, on to the machines where flume agents are running. Only when each agent starts, invalid confs are identified because the Agents would terminate by throwing exceptions. This is a first attempt to make a component-aware configuration system which is independent of the Flume, and does not require the Flume jar to be installed. Each component needs a configuration manager, which configures the components.
I have not completed the validation part yet, but have completed the configuration managers for all the components that come in the flume jar. I will add support for the other components, such as HDFS sink, JDBC channel etc soon. I will also add the validation support to the classes, but would like a review of the model being proposed here.
Note that I do know of a couple of bugs in the existing diff(like that if the name of a config key starts with an Upper case letter, an exception is thrown) - I will fix and upload the review here soon.
This addresses bug FLUME-992.
https://issues.apache.org/jira/browse/FLUME-992
Diffs (updated)
-----
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannel.java bca0c50
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProvider.java e445d61
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProviderFactory.java 6fbd6ef
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java 307ae89
flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestJdbcChannelProvider.java 7710d46
flume-ng-clients/flume-ng-log4jappender/src/test/java/org/apache/flume/clients/log4jappender/TestLog4jAppender.java 68d95fb
flume-ng-configuration/pom.xml PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfigurationFactory.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ConfigurationException.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/Context.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationError.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationErrorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MemoryChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MultiplexingChannelSelectorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/PseudoTxnMemoryChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/AvroSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/FailoverSinkProcessorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/RollingFileSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkGroupConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkProcessorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkProcessorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/AvroSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/ExecSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/NetcatSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceType.java PRE-CREATION
flume-ng-configuration/src/test/resources/flume-ng-error-test PRE-CREATION
flume-ng-configuration/src/test/resources/flume-ng-test.conf PRE-CREATION
flume-ng-core/pom.xml d753fa1
flume-ng-core/src/main/java/org/apache/flume/ChannelSelector.java fba2dcb
flume-ng-core/src/main/java/org/apache/flume/Context.java a6341a5
flume-ng-core/src/main/java/org/apache/flume/SinkProcessor.java 11651ad
flume-ng-core/src/main/java/org/apache/flume/SinkProcessorType.java be1891b
flume-ng-core/src/main/java/org/apache/flume/channel/AbstractChannel.java 352bf08
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorFactory.java 800f471
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8
flume-ng-core/src/main/java/org/apache/flume/channel/DefaultChannelFactory.java 963a6a3
flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java bfa1fde
flume-ng-core/src/main/java/org/apache/flume/channel/MultiplexingChannelSelector.java 83928b7
flume-ng-core/src/main/java/org/apache/flume/channel/PseudoTxnMemoryChannel.java 489d3e5
flume-ng-core/src/main/java/org/apache/flume/channel/ReplicatingChannelSelector.java 8f22746
flume-ng-core/src/main/java/org/apache/flume/conf/Configurable.java 0fa4839
flume-ng-core/src/main/java/org/apache/flume/conf/ConfigurableComponent.java PRE-CREATION
flume-ng-core/src/main/java/org/apache/flume/conf/Configurables.java 84492e5
flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 7386d06
flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkFactory.java b89dfa0
flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkProcessor.java 257bab3
flume-ng-core/src/main/java/org/apache/flume/sink/FailoverSinkProcessor.java 9f5b856
flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java 7f1d3b3
flume-ng-core/src/main/java/org/apache/flume/sink/SinkGroup.java 2e80a56
flume-ng-core/src/main/java/org/apache/flume/sink/SinkProcessorFactory.java 10f9f4e
flume-ng-core/src/main/java/org/apache/flume/sink/SinkType.java 6b08c09
flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java a3f6640
flume-ng-core/src/main/java/org/apache/flume/source/DefaultSourceFactory.java a610e6f
flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java b6b1181
flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java 94245ac
flume-ng-core/src/main/java/org/apache/flume/source/SourceType.java 9082470
flume-ng-core/src/test/java/org/apache/flume/TestContext.java 51c350f
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannel.java e070864
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelConcurrency.java 8dad0b2
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelTransaction.java bc81f26
flume-ng-core/src/test/java/org/apache/flume/sink/TestAvroSink.java 467785f
flume-ng-core/src/test/java/org/apache/flume/sink/TestFailoverSinkProcessor.java 195c121
flume-ng-core/src/test/java/org/apache/flume/sink/TestLoggerSink.java 92ff6fe
flume-ng-core/src/test/java/org/apache/flume/sink/TestRollingFileSink.java 7e26e2a
flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java c5c3f2f
flume-ng-core/src/test/java/org/apache/flume/source/TestExecSource.java 6035270
flume-ng-core/src/test/java/org/apache/flume/source/TestPollableSourceRunner.java c27f82c
flume-ng-core/src/test/java/org/apache/flume/source/TestSequenceGeneratorSource.java 579b257
flume-ng-legacy-sources/flume-avro-source/src/main/java/org/apache/flume/source/avroLegacy/AvroLegacySource.java 65d9142
flume-ng-legacy-sources/flume-avro-source/src/test/java/org/apache/flume/source/avroLegacy/TestLegacyAvroSource.java 6e3eb53
flume-ng-legacy-sources/flume-thrift-source/src/main/java/org/apache/flume/source/thriftLegacy/ThriftLegacySource.java fbf7362
flume-ng-legacy-sources/flume-thrift-source/src/test/java/org/apache/flume/source/thriftLegacy/TestThriftLegacySource.java ddd9478
flume-ng-node/src/main/java/org/apache/flume/conf/properties/FlumeConfiguration.java d66f6d1
flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java 97f72e1
flume-ng-node/src/test/java/org/apache/flume/node/TestAbstractLogicalNodeManager.java 521b586
flume-ng-node/src/test/java/org/apache/flume/source/TestNetcatSource.java fb2a960
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 524b69c
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 7d8ee8a
flume-ng-sinks/flume-irc-sink/src/main/java/org/apache/flume/sink/irc/IRCSink.java 8e77218
pom.xml d785762
Diff: https://reviews.apache.org/r/4115/diff
Testing
-------
All existing unit tests for the components whose configuration has now been moved to these stubs pass.
Thanks,
Hari
> Create configuration stubs for sources, channels, sinks etc
> -----------------------------------------------------------
>
> Key: FLUME-992
> URL: https://issues.apache.org/jira/browse/FLUME-992
> Project: Flume
> Issue Type: Sub-task
> Affects Versions: v1.0.0
> Reporter: Hari Shreedharan
> Assignee: Hari Shreedharan
> Fix For: v1.1.0
>
>
> Currently the configuration of each component sits deep inside the component themselves. There is no way to verify if a configuration is valid before run time. In most systems like Flume, it is likely that these confs will be deployed from one single host, on to the machines where flume agents are running. Only when each agent starts, invalid confs are identified because the Agents would terminate by throwing exceptions. This is a first attempt to make a component-aware configuration system which is independent of the Flume, and does not require the Flume jar to be installed. Each component needs a configuration manager, which configures the components.
> Provide abstract Configuration stubs for each component type, sources, channels, sinks, selectors etc, which are in the new package, independent on ng-core. Now for each of the channels extend the abstract class and check the config properties for each of the required parameters for that component, for example: MultiplexingChannelSelectorConfigurator would look for default channel etc. If a particular component does not have a configuration class then let the current mechanism continue.
> This will require implementation for each component, but it should not be too complex. One additional advantage we get from this is that we can separate out the config validation from the components into these stubs, but we will still need to read the values out of the Context as we do currently(else we end up making the configuration dependent on flume-ng-core itself which we wanted to avoid).
> A problem with this is making a change to the configuration would require changes in the configuration classes and in the components also(where the configuration is read and the component is actually configured). I could not figure out a way of avoiding this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-992) Create configuration stubs for
sources, channels, sinks etc
Posted by "Hari Shreedharan (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hari Shreedharan updated FLUME-992:
-----------------------------------
Description:
Currently the configuration of each component sits deep inside the component themselves. There is no way to verify if a configuration is valid before run time. In most systems like Flume, it is likely that these confs will be deployed from one single host, on to the machines where flume agents are running. Only when each agent starts, invalid confs are identified because the Agents would terminate by throwing exceptions. This is a first attempt to make a component-aware configuration system which is independent of the Flume, and does not require the Flume jar to be installed. Each component needs a configuration manager, which configures the components.
Provide abstract Configuration stubs for each component type, sources, channels, sinks, selectors etc, which are in the new package, independent on ng-core. Now for each of the channels extend the abstract class and check the config properties for each of the required parameters for that component, for example: MultiplexingChannelSelectorConfigurator would look for default channel etc. If a particular component does not have a configuration class then let the current mechanism continue.
This will require implementation for each component, but it should not be too complex. One additional advantage we get from this is that we can separate out the config validation from the components into these stubs, but we will still need to read the values out of the Context as we do currently(else we end up making the configuration dependent on flume-ng-core itself which we wanted to avoid).
A problem with this is making a change to the configuration would require changes in the configuration classes and in the components also(where the configuration is read and the component is actually configured). I could not figure out a way of avoiding this.
was:
Provide abstract Configuration stubs for each component type, sources, channels, sinks, selectors etc, which are in the new package, independent on ng-core. Now for each of the channels extend the abstract class and check the config properties for each of the required parameters for that component, for example: MultiplexingChannelSelectorConfigurator would look for default channel etc. If a particular component does not have a configuration class then let the current mechanism continue.
This will require implementation for each component, but it should not be too complex. One additional advantage we get from this is that we can separate out the config validation from the components into these stubs, but we will still need to read the values out of the Context as we do currently(else we end up making the configuration dependent on flume-ng-core itself which we wanted to avoid).
A problem with this is making a change to the configuration would require changes in the configuration classes and in the components also(where the configuration is read and the component is actually configured). I could not figure out a way of avoiding this.
> Create configuration stubs for sources, channels, sinks etc
> -----------------------------------------------------------
>
> Key: FLUME-992
> URL: https://issues.apache.org/jira/browse/FLUME-992
> Project: Flume
> Issue Type: Sub-task
> Affects Versions: v1.0.0
> Reporter: Hari Shreedharan
> Fix For: v1.1.0
>
>
> Currently the configuration of each component sits deep inside the component themselves. There is no way to verify if a configuration is valid before run time. In most systems like Flume, it is likely that these confs will be deployed from one single host, on to the machines where flume agents are running. Only when each agent starts, invalid confs are identified because the Agents would terminate by throwing exceptions. This is a first attempt to make a component-aware configuration system which is independent of the Flume, and does not require the Flume jar to be installed. Each component needs a configuration manager, which configures the components.
> Provide abstract Configuration stubs for each component type, sources, channels, sinks, selectors etc, which are in the new package, independent on ng-core. Now for each of the channels extend the abstract class and check the config properties for each of the required parameters for that component, for example: MultiplexingChannelSelectorConfigurator would look for default channel etc. If a particular component does not have a configuration class then let the current mechanism continue.
> This will require implementation for each component, but it should not be too complex. One additional advantage we get from this is that we can separate out the config validation from the components into these stubs, but we will still need to read the values out of the Context as we do currently(else we end up making the configuration dependent on flume-ng-core itself which we wanted to avoid).
> A problem with this is making a change to the configuration would require changes in the configuration classes and in the components also(where the configuration is read and the component is actually configured). I could not figure out a way of avoiding this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-992) Create configuration stubs for
sources, channels, sinks etc
Posted by "Hari Shreedharan (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hari Shreedharan updated FLUME-992:
-----------------------------------
Description:
Provide abstract Configuration stubs for each component type, sources, channels, sinks, selectors etc, which are in the new package, independent on ng-core. Now for each of the channels extend the abstract class and check the config properties for each of the required parameters for that component, for example: MultiplexingChannelSelectorConfigurator would look for default channel etc. If a particular component does not have a configuration class then let the current mechanism continue.
This will require implementation for each component, but it should not be too complex. One additional advantage we get from this is that we can separate out the config validation from the components into these stubs, but we will still need to read the values out of the Context as we do currently(else we end up making the configuration dependent on flume-ng-core itself which we wanted to avoid).
A problem with this is making a change to the configuration would require changes in the configuration classes and in the components also(where the configuration is read and the component is actually configured). I could not figure out a way of avoiding this.
was:Currently the configuration provider implementation encompasses all the syntactic and structural validation rules for loading the configuration. Externalizing this functionality to a library will allow external tools to easily operate on flume configuration files and be able to help parse and validate these files.
Summary: Create configuration stubs for sources, channels, sinks etc (was: Creat configuration stubs for sources, channels, sinks etc)
> Create configuration stubs for sources, channels, sinks etc
> -----------------------------------------------------------
>
> Key: FLUME-992
> URL: https://issues.apache.org/jira/browse/FLUME-992
> Project: Flume
> Issue Type: Sub-task
> Affects Versions: v1.0.0
> Reporter: Hari Shreedharan
> Fix For: v1.1.0
>
>
> Provide abstract Configuration stubs for each component type, sources, channels, sinks, selectors etc, which are in the new package, independent on ng-core. Now for each of the channels extend the abstract class and check the config properties for each of the required parameters for that component, for example: MultiplexingChannelSelectorConfigurator would look for default channel etc. If a particular component does not have a configuration class then let the current mechanism continue.
> This will require implementation for each component, but it should not be too complex. One additional advantage we get from this is that we can separate out the config validation from the components into these stubs, but we will still need to read the values out of the Context as we do currently(else we end up making the configuration dependent on flume-ng-core itself which we wanted to avoid).
> A problem with this is making a change to the configuration would require changes in the configuration classes and in the components also(where the configuration is read and the component is actually configured). I could not figure out a way of avoiding this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-992) Create configuration stubs for
sources, channels, sinks etc
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13241557#comment-13241557 ]
jiraposter@reviews.apache.org commented on FLUME-992:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4115/#review6535
-----------------------------------------------------------
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceType.java
<https://reviews.apache.org/r/4115/#comment14197>
Missing:
AVRO_LEGACY("org.apache.flume.source.avroLegacy.AvroLegacySource"),
THRIFT_LEGACY("org.apache.flume.source.thriftLegacy.ThriftLegacySource");
- Will
On 2012-03-27 01:05:49, Hari Shreedharan wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/4115/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-03-27 01:05:49)
bq.
bq.
bq. Review request for Flume.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Currently the configuration of each component sits deep inside the component themselves. There is no way to verify if a configuration is valid before run time. In most systems like Flume, it is likely that these confs will be deployed from one single host, on to the machines where flume agents are running. Only when each agent starts, invalid confs are identified because the Agents would terminate by throwing exceptions. This is a first attempt to make a component-aware configuration system which is independent of the Flume, and does not require the Flume jar to be installed. Each component needs a configuration manager, which configures the components.
bq.
bq. I have not completed the validation part yet, but have completed the configuration managers for all the components that come in the flume jar. I will add support for the other components, such as HDFS sink, JDBC channel etc soon. I will also add the validation support to the classes, but would like a review of the model being proposed here.
bq.
bq. Note that I do know of a couple of bugs in the existing diff(like that if the name of a config key starts with an Upper case letter, an exception is thrown) - I will fix and upload the review here soon.
bq.
bq.
bq. This addresses bug FLUME-992.
bq. https://issues.apache.org/jira/browse/FLUME-992
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453
bq. flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannel.java bca0c50
bq. flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProvider.java e445d61
bq. flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProviderFactory.java 6fbd6ef
bq. flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java 307ae89
bq. flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestJdbcChannelProvider.java 7710d46
bq. flume-ng-clients/flume-ng-log4jappender/src/test/java/org/apache/flume/clients/log4jappender/TestLog4jAppender.java 68d95fb
bq. flume-ng-configuration/pom.xml PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfiguration.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfigurationFactory.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/ConfigurationException.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/Context.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfiguration.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationError.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationErrorType.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelConfiguration.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorConfiguration.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorType.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelType.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/JdbcChannelConfiguration.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MemoryChannelConfiguration.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MultiplexingChannelSelectorConfiguration.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/PseudoTxnMemoryChannelConfiguration.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/AvroSinkConfiguration.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/FailoverSinkProcessorConfiguration.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/HDFSSinkConfiguration.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/RollingFileSinkConfiguration.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkConfiguration.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkGroupConfiguration.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkProcessorConfiguration.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkProcessorType.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkType.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/source/AvroSourceConfiguration.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/source/ExecSourceConfiguration.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/source/NetcatSourceConfiguration.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceConfiguration.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceType.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SyslogTcpSourceConfiguration.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SyslogUDPSourceConfiguration.java PRE-CREATION
bq. flume-ng-configuration/src/test/resources/flume-ng-error-test PRE-CREATION
bq. flume-ng-configuration/src/test/resources/flume-ng-test.conf PRE-CREATION
bq. flume-ng-core/pom.xml 37fb112
bq. flume-ng-core/src/main/java/org/apache/flume/ChannelSelector.java fba2dcb
bq. flume-ng-core/src/main/java/org/apache/flume/Context.java 5294e31
bq. flume-ng-core/src/main/java/org/apache/flume/SinkProcessor.java d863ed0
bq. flume-ng-core/src/main/java/org/apache/flume/SinkProcessorType.java be1891b
bq. flume-ng-core/src/main/java/org/apache/flume/channel/AbstractChannel.java 352bf08
bq. flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorFactory.java 800f471
bq. flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorType.java 511fc65
bq. flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8
bq. flume-ng-core/src/main/java/org/apache/flume/channel/DefaultChannelFactory.java 963a6a3
bq. flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java bfa1fde
bq. flume-ng-core/src/main/java/org/apache/flume/channel/MultiplexingChannelSelector.java 83928b7
bq. flume-ng-core/src/main/java/org/apache/flume/channel/PseudoTxnMemoryChannel.java 489d3e5
bq. flume-ng-core/src/main/java/org/apache/flume/channel/ReplicatingChannelSelector.java 8f22746
bq. flume-ng-core/src/main/java/org/apache/flume/conf/Configurable.java 1d69112
bq. flume-ng-core/src/main/java/org/apache/flume/conf/ConfigurableComponent.java PRE-CREATION
bq. flume-ng-core/src/main/java/org/apache/flume/conf/Configurables.java 84492e5
bq. flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 3edc563
bq. flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkFactory.java b89dfa0
bq. flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkProcessor.java 257bab3
bq. flume-ng-core/src/main/java/org/apache/flume/sink/FailoverSinkProcessor.java 066ee37
bq. flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java 7f1d3b3
bq. flume-ng-core/src/main/java/org/apache/flume/sink/SinkGroup.java 0dffd69
bq. flume-ng-core/src/main/java/org/apache/flume/sink/SinkProcessorFactory.java 6160a17
bq. flume-ng-core/src/main/java/org/apache/flume/sink/SinkType.java 6b08c09
bq. flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java 903889d
bq. flume-ng-core/src/main/java/org/apache/flume/source/DefaultSourceFactory.java a610e6f
bq. flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java dbf79e0
bq. flume-ng-core/src/main/java/org/apache/flume/source/ExecSourceConfigurationConstants.java 73c985e
bq. flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java a841b0e
bq. flume-ng-core/src/main/java/org/apache/flume/source/SequenceGeneratorSource.java 440c5a9
bq. flume-ng-core/src/main/java/org/apache/flume/source/SourceType.java 9082470
bq. flume-ng-core/src/main/java/org/apache/flume/source/SyslogTcpSource.java d78d27f
bq. flume-ng-core/src/main/java/org/apache/flume/source/SyslogUDPSource.java 34818f0
bq. flume-ng-core/src/test/java/org/apache/flume/TestContext.java 51c350f
bq. flume-ng-core/src/test/java/org/apache/flume/channel/AbstractBasicChannelSemanticsTest.java 6e71e46
bq. flume-ng-core/src/test/java/org/apache/flume/channel/MockChannel.java 24b01e2
bq. flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannel.java e070864
bq. flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelConcurrency.java 8dad0b2
bq. flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelTransaction.java bc81f26
bq. flume-ng-core/src/test/java/org/apache/flume/sink/TestAvroSink.java 3765924
bq. flume-ng-core/src/test/java/org/apache/flume/sink/TestFailoverSinkProcessor.java 3358cf4
bq. flume-ng-core/src/test/java/org/apache/flume/sink/TestLoggerSink.java 92ff6fe
bq. flume-ng-core/src/test/java/org/apache/flume/sink/TestRollingFileSink.java 7e26e2a
bq. flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java c5c3f2f
bq. flume-ng-core/src/test/java/org/apache/flume/source/TestExecSource.java 615f2a3
bq. flume-ng-core/src/test/java/org/apache/flume/source/TestPollableSourceRunner.java c27f82c
bq. flume-ng-core/src/test/java/org/apache/flume/source/TestSequenceGeneratorSource.java 579b257
bq. flume-ng-core/src/test/java/org/apache/flume/source/TestSyslogUdpSource.java d1db49d
bq. flume-ng-legacy-sources/flume-avro-source/src/main/java/org/apache/flume/source/avroLegacy/AvroLegacySource.java dde8f28
bq. flume-ng-legacy-sources/flume-avro-source/src/test/java/org/apache/flume/source/avroLegacy/TestLegacyAvroSource.java 6e3eb53
bq. flume-ng-legacy-sources/flume-thrift-source/src/main/java/org/apache/flume/source/thriftLegacy/ThriftLegacySource.java 00df7e4
bq. flume-ng-legacy-sources/flume-thrift-source/src/test/java/org/apache/flume/source/thriftLegacy/TestThriftLegacySource.java ddd9478
bq. flume-ng-node/src/main/java/org/apache/flume/conf/properties/FlumeConfiguration.java d66f6d1
bq. flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java 50b9f0c
bq. flume-ng-node/src/test/java/org/apache/flume/node/TestAbstractLogicalNodeManager.java 521b586
bq. flume-ng-node/src/test/java/org/apache/flume/source/TestNetcatSource.java fb2a960
bq. flume-ng-node/src/test/resources/flume-conf.properties 848caca
bq. flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java eee9221
bq. flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 6ff3737
bq. flume-ng-sinks/flume-irc-sink/src/main/java/org/apache/flume/sink/irc/IRCSink.java 8e77218
bq. pom.xml c91222f
bq.
bq. Diff: https://reviews.apache.org/r/4115/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. All existing unit tests for the components whose configuration has now been moved to these stubs pass.
bq.
bq.
bq. Thanks,
bq.
bq. Hari
bq.
bq.
> Create configuration stubs for sources, channels, sinks etc
> -----------------------------------------------------------
>
> Key: FLUME-992
> URL: https://issues.apache.org/jira/browse/FLUME-992
> Project: Flume
> Issue Type: Sub-task
> Affects Versions: v1.0.0
> Reporter: Hari Shreedharan
> Assignee: Hari Shreedharan
> Fix For: v1.2.0
>
>
> Currently the configuration of each component sits deep inside the component themselves. There is no way to verify if a configuration is valid before run time. In most systems like Flume, it is likely that these confs will be deployed from one single host, on to the machines where flume agents are running. Only when each agent starts, invalid confs are identified because the Agents would terminate by throwing exceptions. This is a first attempt to make a component-aware configuration system which is independent of the Flume, and does not require the Flume jar to be installed. Each component needs a configuration manager, which configures the components.
> Provide abstract Configuration stubs for each component type, sources, channels, sinks, selectors etc, which are in the new package, independent on ng-core. Now for each of the channels extend the abstract class and check the config properties for each of the required parameters for that component, for example: MultiplexingChannelSelectorConfigurator would look for default channel etc. If a particular component does not have a configuration class then let the current mechanism continue.
> This will require implementation for each component, but it should not be too complex. One additional advantage we get from this is that we can separate out the config validation from the components into these stubs, but we will still need to read the values out of the Context as we do currently(else we end up making the configuration dependent on flume-ng-core itself which we wanted to avoid).
> A problem with this is making a change to the configuration would require changes in the configuration classes and in the components also(where the configuration is read and the component is actually configured). I could not figure out a way of avoiding this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-992) Create configuration stubs for
sources, channels, sinks etc
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13225942#comment-13225942 ]
jiraposter@reviews.apache.org commented on FLUME-992:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4115/
-----------------------------------------------------------
(Updated 2012-03-09 08:47:00.765150)
Review request for Flume.
Changes
-------
Unit tests pass. End-to-end testing to be done. Will update patch if needed.
Summary
-------
Currently the configuration of each component sits deep inside the component themselves. There is no way to verify if a configuration is valid before run time. In most systems like Flume, it is likely that these confs will be deployed from one single host, on to the machines where flume agents are running. Only when each agent starts, invalid confs are identified because the Agents would terminate by throwing exceptions. This is a first attempt to make a component-aware configuration system which is independent of the Flume, and does not require the Flume jar to be installed. Each component needs a configuration manager, which configures the components.
I have not completed the validation part yet, but have completed the configuration managers for all the components that come in the flume jar. I will add support for the other components, such as HDFS sink, JDBC channel etc soon. I will also add the validation support to the classes, but would like a review of the model being proposed here.
Note that I do know of a couple of bugs in the existing diff(like that if the name of a config key starts with an Upper case letter, an exception is thrown) - I will fix and upload the review here soon.
This addresses bug FLUME-992.
https://issues.apache.org/jira/browse/FLUME-992
Diffs (updated)
-----
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/NetcatSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/ExecSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/AvroSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkProcessorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkProcessorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkGroupConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/RollingFileSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationError.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationErrorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MemoryChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MultiplexingChannelSelectorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/PseudoTxnMemoryChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/AvroSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/FailoverSinkProcessorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/Context.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ConfigurationException.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfigurationFactory.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfiguration.java PRE-CREATION
flume-ng-clients/flume-ng-log4jappender/src/test/java/org/apache/flume/clients/log4jappender/TestLog4jAppender.java 68d95fb
flume-ng-configuration/pom.xml PRE-CREATION
flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestJdbcChannelProvider.java 7710d46
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannel.java bca0c50
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProvider.java e445d61
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProviderFactory.java 6fbd6ef
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java 307ae89
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceType.java PRE-CREATION
flume-ng-configuration/src/test/resources/flume-ng-error-test PRE-CREATION
flume-ng-configuration/src/test/resources/flume-ng-test.conf PRE-CREATION
flume-ng-core/pom.xml d753fa1
flume-ng-core/src/main/java/org/apache/flume/ChannelSelector.java fba2dcb
flume-ng-core/src/main/java/org/apache/flume/Context.java a6341a5
flume-ng-core/src/main/java/org/apache/flume/SinkProcessor.java 11651ad
flume-ng-core/src/main/java/org/apache/flume/SinkProcessorType.java be1891b
flume-ng-core/src/main/java/org/apache/flume/channel/AbstractChannel.java 352bf08
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorFactory.java 800f471
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorType.java 511fc65
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8
flume-ng-core/src/main/java/org/apache/flume/channel/DefaultChannelFactory.java 963a6a3
flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java bfa1fde
flume-ng-core/src/main/java/org/apache/flume/channel/MultiplexingChannelSelector.java 83928b7
flume-ng-core/src/main/java/org/apache/flume/channel/PseudoTxnMemoryChannel.java 489d3e5
flume-ng-core/src/main/java/org/apache/flume/channel/ReplicatingChannelSelector.java 8f22746
flume-ng-core/src/main/java/org/apache/flume/conf/Configurable.java 0fa4839
flume-ng-core/src/main/java/org/apache/flume/conf/ConfigurableComponent.java PRE-CREATION
flume-ng-core/src/main/java/org/apache/flume/conf/Configurables.java 84492e5
flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 7386d06
flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkFactory.java b89dfa0
flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkProcessor.java 257bab3
flume-ng-core/src/main/java/org/apache/flume/sink/FailoverSinkProcessor.java 9f5b856
flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java 7f1d3b3
flume-ng-core/src/main/java/org/apache/flume/sink/SinkGroup.java 2e80a56
flume-ng-core/src/main/java/org/apache/flume/sink/SinkProcessorFactory.java 10f9f4e
flume-ng-core/src/main/java/org/apache/flume/sink/SinkType.java 6b08c09
flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java a3f6640
flume-ng-core/src/main/java/org/apache/flume/source/DefaultSourceFactory.java a610e6f
flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java b6b1181
flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java 94245ac
flume-ng-core/src/main/java/org/apache/flume/source/SourceType.java 9082470
flume-ng-core/src/test/java/org/apache/flume/TestContext.java 51c350f
flume-ng-core/src/test/java/org/apache/flume/channel/AbstractBasicChannelSemanticsTest.java 6e71e46
flume-ng-core/src/test/java/org/apache/flume/channel/MockChannel.java 24b01e2
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannel.java e070864
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelConcurrency.java 8dad0b2
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelTransaction.java bc81f26
flume-ng-core/src/test/java/org/apache/flume/sink/TestAvroSink.java 467785f
flume-ng-core/src/test/java/org/apache/flume/sink/TestFailoverSinkProcessor.java 195c121
flume-ng-core/src/test/java/org/apache/flume/sink/TestLoggerSink.java 92ff6fe
flume-ng-core/src/test/java/org/apache/flume/sink/TestRollingFileSink.java 7e26e2a
flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java c5c3f2f
flume-ng-core/src/test/java/org/apache/flume/source/TestExecSource.java 6035270
flume-ng-core/src/test/java/org/apache/flume/source/TestPollableSourceRunner.java c27f82c
flume-ng-core/src/test/java/org/apache/flume/source/TestSequenceGeneratorSource.java 579b257
flume-ng-legacy-sources/flume-avro-source/src/main/java/org/apache/flume/source/avroLegacy/AvroLegacySource.java 65d9142
flume-ng-legacy-sources/flume-avro-source/src/test/java/org/apache/flume/source/avroLegacy/TestLegacyAvroSource.java 6e3eb53
flume-ng-legacy-sources/flume-thrift-source/src/main/java/org/apache/flume/source/thriftLegacy/ThriftLegacySource.java fbf7362
flume-ng-legacy-sources/flume-thrift-source/src/test/java/org/apache/flume/source/thriftLegacy/TestThriftLegacySource.java ddd9478
flume-ng-node/src/main/java/org/apache/flume/conf/properties/FlumeConfiguration.java d66f6d1
flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java 97f72e1
flume-ng-node/src/test/java/org/apache/flume/node/TestAbstractLogicalNodeManager.java 521b586
flume-ng-node/src/test/java/org/apache/flume/source/TestNetcatSource.java fb2a960
flume-ng-node/src/test/resources/flume-conf.properties 848caca
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 524b69c
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 7d8ee8a
flume-ng-sinks/flume-irc-sink/src/main/java/org/apache/flume/sink/irc/IRCSink.java 8e77218
pom.xml d785762
Diff: https://reviews.apache.org/r/4115/diff
Testing
-------
All existing unit tests for the components whose configuration has now been moved to these stubs pass.
Thanks,
Hari
> Create configuration stubs for sources, channels, sinks etc
> -----------------------------------------------------------
>
> Key: FLUME-992
> URL: https://issues.apache.org/jira/browse/FLUME-992
> Project: Flume
> Issue Type: Sub-task
> Affects Versions: v1.0.0
> Reporter: Hari Shreedharan
> Assignee: Hari Shreedharan
> Fix For: v1.1.0
>
>
> Currently the configuration of each component sits deep inside the component themselves. There is no way to verify if a configuration is valid before run time. In most systems like Flume, it is likely that these confs will be deployed from one single host, on to the machines where flume agents are running. Only when each agent starts, invalid confs are identified because the Agents would terminate by throwing exceptions. This is a first attempt to make a component-aware configuration system which is independent of the Flume, and does not require the Flume jar to be installed. Each component needs a configuration manager, which configures the components.
> Provide abstract Configuration stubs for each component type, sources, channels, sinks, selectors etc, which are in the new package, independent on ng-core. Now for each of the channels extend the abstract class and check the config properties for each of the required parameters for that component, for example: MultiplexingChannelSelectorConfigurator would look for default channel etc. If a particular component does not have a configuration class then let the current mechanism continue.
> This will require implementation for each component, but it should not be too complex. One additional advantage we get from this is that we can separate out the config validation from the components into these stubs, but we will still need to read the values out of the Context as we do currently(else we end up making the configuration dependent on flume-ng-core itself which we wanted to avoid).
> A problem with this is making a change to the configuration would require changes in the configuration classes and in the components also(where the configuration is read and the component is actually configured). I could not figure out a way of avoiding this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-992) Create configuration stubs for
sources, channels, sinks etc
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13232998#comment-13232998 ]
jiraposter@reviews.apache.org commented on FLUME-992:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4115/
-----------------------------------------------------------
(Updated 2012-03-19 23:01:52.087904)
Review request for Flume.
Changes
-------
bug fixes
Summary
-------
Currently the configuration of each component sits deep inside the component themselves. There is no way to verify if a configuration is valid before run time. In most systems like Flume, it is likely that these confs will be deployed from one single host, on to the machines where flume agents are running. Only when each agent starts, invalid confs are identified because the Agents would terminate by throwing exceptions. This is a first attempt to make a component-aware configuration system which is independent of the Flume, and does not require the Flume jar to be installed. Each component needs a configuration manager, which configures the components.
I have not completed the validation part yet, but have completed the configuration managers for all the components that come in the flume jar. I will add support for the other components, such as HDFS sink, JDBC channel etc soon. I will also add the validation support to the classes, but would like a review of the model being proposed here.
Note that I do know of a couple of bugs in the existing diff(like that if the name of a config key starts with an Upper case letter, an exception is thrown) - I will fix and upload the review here soon.
This addresses bug FLUME-992.
https://issues.apache.org/jira/browse/FLUME-992
Diffs (updated)
-----
flume-ng-channels/flume-file-channel/pom.xml 48d1481
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453
flume-ng-channels/flume-jdbc-channel/pom.xml 8e82583
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannel.java bca0c50
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProvider.java e445d61
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProviderFactory.java 6fbd6ef
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java 307ae89
flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestJdbcChannelProvider.java 7710d46
flume-ng-channels/pom.xml 0bd8633
flume-ng-clients/flume-ng-log4jappender/pom.xml d1c443d
flume-ng-clients/flume-ng-log4jappender/src/test/java/org/apache/flume/clients/log4jappender/TestLog4jAppender.java 68d95fb
flume-ng-clients/pom.xml 0a0dc04
flume-ng-configuration/pom.xml PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfigurationFactory.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ConfigurationException.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/Context.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationError.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationErrorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/JdbcChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MemoryChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MultiplexingChannelSelectorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/PseudoTxnMemoryChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/AvroSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/FailoverSinkProcessorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/HDFSSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/RollingFileSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkGroupConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkProcessorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkProcessorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/AvroSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/ExecSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/NetcatSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceType.java PRE-CREATION
flume-ng-configuration/src/test/resources/flume-ng-error-test PRE-CREATION
flume-ng-configuration/src/test/resources/flume-ng-test.conf PRE-CREATION
flume-ng-core/pom.xml 37fb112
flume-ng-core/src/main/java/org/apache/flume/ChannelSelector.java fba2dcb
flume-ng-core/src/main/java/org/apache/flume/Context.java 5294e31
flume-ng-core/src/main/java/org/apache/flume/SinkProcessor.java ea6000b
flume-ng-core/src/main/java/org/apache/flume/SinkProcessorType.java be1891b
flume-ng-core/src/main/java/org/apache/flume/channel/AbstractChannel.java 352bf08
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorFactory.java 800f471
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorType.java 511fc65
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8
flume-ng-core/src/main/java/org/apache/flume/channel/DefaultChannelFactory.java 963a6a3
flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java bfa1fde
flume-ng-core/src/main/java/org/apache/flume/channel/MultiplexingChannelSelector.java 83928b7
flume-ng-core/src/main/java/org/apache/flume/channel/PseudoTxnMemoryChannel.java 489d3e5
flume-ng-core/src/main/java/org/apache/flume/channel/ReplicatingChannelSelector.java 8f22746
flume-ng-core/src/main/java/org/apache/flume/conf/Configurable.java 0fa4839
flume-ng-core/src/main/java/org/apache/flume/conf/ConfigurableComponent.java PRE-CREATION
flume-ng-core/src/main/java/org/apache/flume/conf/Configurables.java 84492e5
flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 3edc563
flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkFactory.java b89dfa0
flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkProcessor.java 257bab3
flume-ng-core/src/main/java/org/apache/flume/sink/FailoverSinkProcessor.java 9f5b856
flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java 7f1d3b3
flume-ng-core/src/main/java/org/apache/flume/sink/SinkGroup.java 0dffd69
flume-ng-core/src/main/java/org/apache/flume/sink/SinkProcessorFactory.java 6160a17
flume-ng-core/src/main/java/org/apache/flume/sink/SinkType.java 6b08c09
flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java 903889d
flume-ng-core/src/main/java/org/apache/flume/source/DefaultSourceFactory.java a610e6f
flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java b6b1181
flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java 94245ac
flume-ng-core/src/main/java/org/apache/flume/source/SequenceGeneratorSource.java 440c5a9
flume-ng-core/src/main/java/org/apache/flume/source/SourceType.java 9082470
flume-ng-core/src/main/java/org/apache/flume/source/SyslogTcpSource.java d78d27f
flume-ng-core/src/main/java/org/apache/flume/source/SyslogUDPSource.java 34818f0
flume-ng-core/src/test/java/org/apache/flume/TestContext.java 51c350f
flume-ng-core/src/test/java/org/apache/flume/channel/AbstractBasicChannelSemanticsTest.java 6e71e46
flume-ng-core/src/test/java/org/apache/flume/channel/MockChannel.java 24b01e2
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannel.java e070864
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelConcurrency.java 8dad0b2
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelTransaction.java bc81f26
flume-ng-core/src/test/java/org/apache/flume/sink/TestAvroSink.java 3765924
flume-ng-core/src/test/java/org/apache/flume/sink/TestFailoverSinkProcessor.java 195c121
flume-ng-core/src/test/java/org/apache/flume/sink/TestLoggerSink.java 92ff6fe
flume-ng-core/src/test/java/org/apache/flume/sink/TestRollingFileSink.java 7e26e2a
flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java c5c3f2f
flume-ng-core/src/test/java/org/apache/flume/source/TestExecSource.java 6035270
flume-ng-core/src/test/java/org/apache/flume/source/TestPollableSourceRunner.java c27f82c
flume-ng-core/src/test/java/org/apache/flume/source/TestSequenceGeneratorSource.java 579b257
flume-ng-core/src/test/java/org/apache/flume/source/TestSyslogUdpSource.java d1db49d
flume-ng-dist/pom.xml f7e0dd9
flume-ng-legacy-sources/flume-avro-source/pom.xml 216e479
flume-ng-legacy-sources/flume-avro-source/src/main/java/org/apache/flume/source/avroLegacy/AvroLegacySource.java dde8f28
flume-ng-legacy-sources/flume-avro-source/src/test/java/org/apache/flume/source/avroLegacy/TestLegacyAvroSource.java 6e3eb53
flume-ng-legacy-sources/flume-thrift-source/pom.xml 70b67a6
flume-ng-legacy-sources/flume-thrift-source/src/main/java/org/apache/flume/source/thriftLegacy/ThriftLegacySource.java fbf7362
flume-ng-legacy-sources/flume-thrift-source/src/test/java/org/apache/flume/source/thriftLegacy/TestThriftLegacySource.java ddd9478
flume-ng-legacy-sources/pom.xml 2e5b8dc
flume-ng-node/pom.xml ffa89fa
flume-ng-node/src/main/java/org/apache/flume/conf/properties/FlumeConfiguration.java d66f6d1
flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java 97f72e1
flume-ng-node/src/test/java/org/apache/flume/node/TestAbstractLogicalNodeManager.java 521b586
flume-ng-node/src/test/java/org/apache/flume/source/TestNetcatSource.java fb2a960
flume-ng-node/src/test/resources/flume-conf.properties 848caca
flume-ng-sdk/pom.xml d1c6adf
flume-ng-sinks/flume-hdfs-sink/pom.xml 76d69a3
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java da82f7e
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 8fa72a1
flume-ng-sinks/flume-irc-sink/pom.xml d935faf
flume-ng-sinks/flume-irc-sink/src/main/java/org/apache/flume/sink/irc/IRCSink.java 8e77218
flume-ng-sinks/pom.xml acb3087
pom.xml cf10727
Diff: https://reviews.apache.org/r/4115/diff
Testing
-------
All existing unit tests for the components whose configuration has now been moved to these stubs pass.
Thanks,
Hari
> Create configuration stubs for sources, channels, sinks etc
> -----------------------------------------------------------
>
> Key: FLUME-992
> URL: https://issues.apache.org/jira/browse/FLUME-992
> Project: Flume
> Issue Type: Sub-task
> Affects Versions: v1.0.0
> Reporter: Hari Shreedharan
> Assignee: Hari Shreedharan
> Fix For: v1.2.0
>
>
> Currently the configuration of each component sits deep inside the component themselves. There is no way to verify if a configuration is valid before run time. In most systems like Flume, it is likely that these confs will be deployed from one single host, on to the machines where flume agents are running. Only when each agent starts, invalid confs are identified because the Agents would terminate by throwing exceptions. This is a first attempt to make a component-aware configuration system which is independent of the Flume, and does not require the Flume jar to be installed. Each component needs a configuration manager, which configures the components.
> Provide abstract Configuration stubs for each component type, sources, channels, sinks, selectors etc, which are in the new package, independent on ng-core. Now for each of the channels extend the abstract class and check the config properties for each of the required parameters for that component, for example: MultiplexingChannelSelectorConfigurator would look for default channel etc. If a particular component does not have a configuration class then let the current mechanism continue.
> This will require implementation for each component, but it should not be too complex. One additional advantage we get from this is that we can separate out the config validation from the components into these stubs, but we will still need to read the values out of the Context as we do currently(else we end up making the configuration dependent on flume-ng-core itself which we wanted to avoid).
> A problem with this is making a change to the configuration would require changes in the configuration classes and in the components also(where the configuration is read and the component is actually configured). I could not figure out a way of avoiding this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-992) Create configuration stubs for
sources, channels, sinks etc
Posted by "Hari Shreedharan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13263341#comment-13263341 ]
Hari Shreedharan commented on FLUME-992:
----------------------------------------
Implemented as part of FLUME-1052.
> Create configuration stubs for sources, channels, sinks etc
> -----------------------------------------------------------
>
> Key: FLUME-992
> URL: https://issues.apache.org/jira/browse/FLUME-992
> Project: Flume
> Issue Type: Sub-task
> Affects Versions: v1.0.0
> Reporter: Hari Shreedharan
> Assignee: Hari Shreedharan
> Fix For: v1.2.0
>
> Original Estimate: 0h
> Remaining Estimate: 0h
>
> Currently the configuration of each component sits deep inside the component themselves. There is no way to verify if a configuration is valid before run time. In most systems like Flume, it is likely that these confs will be deployed from one single host, on to the machines where flume agents are running. Only when each agent starts, invalid confs are identified because the Agents would terminate by throwing exceptions. This is a first attempt to make a component-aware configuration system which is independent of the Flume, and does not require the Flume jar to be installed. Each component needs a configuration manager, which configures the components.
> Provide abstract Configuration stubs for each component type, sources, channels, sinks, selectors etc, which are in the new package, independent on ng-core. Now for each of the channels extend the abstract class and check the config properties for each of the required parameters for that component, for example: MultiplexingChannelSelectorConfigurator would look for default channel etc. If a particular component does not have a configuration class then let the current mechanism continue.
> This will require implementation for each component, but it should not be too complex. One additional advantage we get from this is that we can separate out the config validation from the components into these stubs, but we will still need to read the values out of the Context as we do currently(else we end up making the configuration dependent on flume-ng-core itself which we wanted to avoid).
> A problem with this is making a change to the configuration would require changes in the configuration classes and in the components also(where the configuration is read and the component is actually configured). I could not figure out a way of avoiding this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-992) Create configuration stubs for
sources, channels, sinks etc
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13236417#comment-13236417 ]
jiraposter@reviews.apache.org commented on FLUME-992:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4115/#review6199
-----------------------------------------------------------
Thanks for the patch Hari. The patch needs to be rebased on the latest sources and seems to have accidentally overlaid a lot of older code on top of the latest. For example, all the pom.xml files have their versions overwritten to 1.1.0-incubating. Not only the patch is very big, it actually has substantial formatting changes which are making it very hard to keep an eye on the actual change.
Therefore, for the benefit of the reviewers and for minimizing the risk, I suggest you update the patch to only have the configuration subsystem changes. This means that all components will get configured like custom components that do not have stubs exposed. Once we get that part reviewed and checked in, we can revive the rest of the changes component by component and get them committed.
I understand that you have gone through many painstaking revisions of the patch so far, and if you you so prefer - I will be happy to help out in breaking this patch into phased changes.
- Arvind
On 2012-03-19 23:01:52, Hari Shreedharan wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/4115/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-03-19 23:01:52)
bq.
bq.
bq. Review request for Flume.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Currently the configuration of each component sits deep inside the component themselves. There is no way to verify if a configuration is valid before run time. In most systems like Flume, it is likely that these confs will be deployed from one single host, on to the machines where flume agents are running. Only when each agent starts, invalid confs are identified because the Agents would terminate by throwing exceptions. This is a first attempt to make a component-aware configuration system which is independent of the Flume, and does not require the Flume jar to be installed. Each component needs a configuration manager, which configures the components.
bq.
bq. I have not completed the validation part yet, but have completed the configuration managers for all the components that come in the flume jar. I will add support for the other components, such as HDFS sink, JDBC channel etc soon. I will also add the validation support to the classes, but would like a review of the model being proposed here.
bq.
bq. Note that I do know of a couple of bugs in the existing diff(like that if the name of a config key starts with an Upper case letter, an exception is thrown) - I will fix and upload the review here soon.
bq.
bq.
bq. This addresses bug FLUME-992.
bq. https://issues.apache.org/jira/browse/FLUME-992
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. flume-ng-channels/flume-file-channel/pom.xml 48d1481
bq. flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453
bq. flume-ng-channels/flume-jdbc-channel/pom.xml 8e82583
bq. flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannel.java bca0c50
bq. flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProvider.java e445d61
bq. flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProviderFactory.java 6fbd6ef
bq. flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java 307ae89
bq. flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestJdbcChannelProvider.java 7710d46
bq. flume-ng-channels/pom.xml 0bd8633
bq. flume-ng-clients/flume-ng-log4jappender/pom.xml d1c443d
bq. flume-ng-clients/flume-ng-log4jappender/src/test/java/org/apache/flume/clients/log4jappender/TestLog4jAppender.java 68d95fb
bq. flume-ng-clients/pom.xml 0a0dc04
bq. flume-ng-configuration/pom.xml PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfiguration.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfigurationFactory.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/ConfigurationException.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/Context.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfiguration.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationError.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationErrorType.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelConfiguration.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorConfiguration.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorType.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelType.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/JdbcChannelConfiguration.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MemoryChannelConfiguration.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MultiplexingChannelSelectorConfiguration.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/PseudoTxnMemoryChannelConfiguration.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/AvroSinkConfiguration.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/FailoverSinkProcessorConfiguration.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/HDFSSinkConfiguration.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/RollingFileSinkConfiguration.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkConfiguration.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkGroupConfiguration.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkProcessorConfiguration.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkProcessorType.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkType.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/source/AvroSourceConfiguration.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/source/ExecSourceConfiguration.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/source/NetcatSourceConfiguration.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceConfiguration.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceType.java PRE-CREATION
bq. flume-ng-configuration/src/test/resources/flume-ng-error-test PRE-CREATION
bq. flume-ng-configuration/src/test/resources/flume-ng-test.conf PRE-CREATION
bq. flume-ng-core/pom.xml 37fb112
bq. flume-ng-core/src/main/java/org/apache/flume/ChannelSelector.java fba2dcb
bq. flume-ng-core/src/main/java/org/apache/flume/Context.java 5294e31
bq. flume-ng-core/src/main/java/org/apache/flume/SinkProcessor.java ea6000b
bq. flume-ng-core/src/main/java/org/apache/flume/SinkProcessorType.java be1891b
bq. flume-ng-core/src/main/java/org/apache/flume/channel/AbstractChannel.java 352bf08
bq. flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorFactory.java 800f471
bq. flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorType.java 511fc65
bq. flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8
bq. flume-ng-core/src/main/java/org/apache/flume/channel/DefaultChannelFactory.java 963a6a3
bq. flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java bfa1fde
bq. flume-ng-core/src/main/java/org/apache/flume/channel/MultiplexingChannelSelector.java 83928b7
bq. flume-ng-core/src/main/java/org/apache/flume/channel/PseudoTxnMemoryChannel.java 489d3e5
bq. flume-ng-core/src/main/java/org/apache/flume/channel/ReplicatingChannelSelector.java 8f22746
bq. flume-ng-core/src/main/java/org/apache/flume/conf/Configurable.java 0fa4839
bq. flume-ng-core/src/main/java/org/apache/flume/conf/ConfigurableComponent.java PRE-CREATION
bq. flume-ng-core/src/main/java/org/apache/flume/conf/Configurables.java 84492e5
bq. flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 3edc563
bq. flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkFactory.java b89dfa0
bq. flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkProcessor.java 257bab3
bq. flume-ng-core/src/main/java/org/apache/flume/sink/FailoverSinkProcessor.java 9f5b856
bq. flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java 7f1d3b3
bq. flume-ng-core/src/main/java/org/apache/flume/sink/SinkGroup.java 0dffd69
bq. flume-ng-core/src/main/java/org/apache/flume/sink/SinkProcessorFactory.java 6160a17
bq. flume-ng-core/src/main/java/org/apache/flume/sink/SinkType.java 6b08c09
bq. flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java 903889d
bq. flume-ng-core/src/main/java/org/apache/flume/source/DefaultSourceFactory.java a610e6f
bq. flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java b6b1181
bq. flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java 94245ac
bq. flume-ng-core/src/main/java/org/apache/flume/source/SequenceGeneratorSource.java 440c5a9
bq. flume-ng-core/src/main/java/org/apache/flume/source/SourceType.java 9082470
bq. flume-ng-core/src/main/java/org/apache/flume/source/SyslogTcpSource.java d78d27f
bq. flume-ng-core/src/main/java/org/apache/flume/source/SyslogUDPSource.java 34818f0
bq. flume-ng-core/src/test/java/org/apache/flume/TestContext.java 51c350f
bq. flume-ng-core/src/test/java/org/apache/flume/channel/AbstractBasicChannelSemanticsTest.java 6e71e46
bq. flume-ng-core/src/test/java/org/apache/flume/channel/MockChannel.java 24b01e2
bq. flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannel.java e070864
bq. flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelConcurrency.java 8dad0b2
bq. flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelTransaction.java bc81f26
bq. flume-ng-core/src/test/java/org/apache/flume/sink/TestAvroSink.java 3765924
bq. flume-ng-core/src/test/java/org/apache/flume/sink/TestFailoverSinkProcessor.java 195c121
bq. flume-ng-core/src/test/java/org/apache/flume/sink/TestLoggerSink.java 92ff6fe
bq. flume-ng-core/src/test/java/org/apache/flume/sink/TestRollingFileSink.java 7e26e2a
bq. flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java c5c3f2f
bq. flume-ng-core/src/test/java/org/apache/flume/source/TestExecSource.java 6035270
bq. flume-ng-core/src/test/java/org/apache/flume/source/TestPollableSourceRunner.java c27f82c
bq. flume-ng-core/src/test/java/org/apache/flume/source/TestSequenceGeneratorSource.java 579b257
bq. flume-ng-core/src/test/java/org/apache/flume/source/TestSyslogUdpSource.java d1db49d
bq. flume-ng-dist/pom.xml f7e0dd9
bq. flume-ng-legacy-sources/flume-avro-source/pom.xml 216e479
bq. flume-ng-legacy-sources/flume-avro-source/src/main/java/org/apache/flume/source/avroLegacy/AvroLegacySource.java dde8f28
bq. flume-ng-legacy-sources/flume-avro-source/src/test/java/org/apache/flume/source/avroLegacy/TestLegacyAvroSource.java 6e3eb53
bq. flume-ng-legacy-sources/flume-thrift-source/pom.xml 70b67a6
bq. flume-ng-legacy-sources/flume-thrift-source/src/main/java/org/apache/flume/source/thriftLegacy/ThriftLegacySource.java fbf7362
bq. flume-ng-legacy-sources/flume-thrift-source/src/test/java/org/apache/flume/source/thriftLegacy/TestThriftLegacySource.java ddd9478
bq. flume-ng-legacy-sources/pom.xml 2e5b8dc
bq. flume-ng-node/pom.xml ffa89fa
bq. flume-ng-node/src/main/java/org/apache/flume/conf/properties/FlumeConfiguration.java d66f6d1
bq. flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java 97f72e1
bq. flume-ng-node/src/test/java/org/apache/flume/node/TestAbstractLogicalNodeManager.java 521b586
bq. flume-ng-node/src/test/java/org/apache/flume/source/TestNetcatSource.java fb2a960
bq. flume-ng-node/src/test/resources/flume-conf.properties 848caca
bq. flume-ng-sdk/pom.xml d1c6adf
bq. flume-ng-sinks/flume-hdfs-sink/pom.xml 76d69a3
bq. flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java da82f7e
bq. flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 8fa72a1
bq. flume-ng-sinks/flume-irc-sink/pom.xml d935faf
bq. flume-ng-sinks/flume-irc-sink/src/main/java/org/apache/flume/sink/irc/IRCSink.java 8e77218
bq. flume-ng-sinks/pom.xml acb3087
bq. pom.xml cf10727
bq.
bq. Diff: https://reviews.apache.org/r/4115/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. All existing unit tests for the components whose configuration has now been moved to these stubs pass.
bq.
bq.
bq. Thanks,
bq.
bq. Hari
bq.
bq.
> Create configuration stubs for sources, channels, sinks etc
> -----------------------------------------------------------
>
> Key: FLUME-992
> URL: https://issues.apache.org/jira/browse/FLUME-992
> Project: Flume
> Issue Type: Sub-task
> Affects Versions: v1.0.0
> Reporter: Hari Shreedharan
> Assignee: Hari Shreedharan
> Fix For: v1.2.0
>
>
> Currently the configuration of each component sits deep inside the component themselves. There is no way to verify if a configuration is valid before run time. In most systems like Flume, it is likely that these confs will be deployed from one single host, on to the machines where flume agents are running. Only when each agent starts, invalid confs are identified because the Agents would terminate by throwing exceptions. This is a first attempt to make a component-aware configuration system which is independent of the Flume, and does not require the Flume jar to be installed. Each component needs a configuration manager, which configures the components.
> Provide abstract Configuration stubs for each component type, sources, channels, sinks, selectors etc, which are in the new package, independent on ng-core. Now for each of the channels extend the abstract class and check the config properties for each of the required parameters for that component, for example: MultiplexingChannelSelectorConfigurator would look for default channel etc. If a particular component does not have a configuration class then let the current mechanism continue.
> This will require implementation for each component, but it should not be too complex. One additional advantage we get from this is that we can separate out the config validation from the components into these stubs, but we will still need to read the values out of the Context as we do currently(else we end up making the configuration dependent on flume-ng-core itself which we wanted to avoid).
> A problem with this is making a change to the configuration would require changes in the configuration classes and in the components also(where the configuration is read and the component is actually configured). I could not figure out a way of avoiding this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (FLUME-992) Create configuration stubs for
sources, channels, sinks etc
Posted by "Arvind Prabhakar (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arvind Prabhakar reassigned FLUME-992:
--------------------------------------
Assignee: Hari Shreedharan
> Create configuration stubs for sources, channels, sinks etc
> -----------------------------------------------------------
>
> Key: FLUME-992
> URL: https://issues.apache.org/jira/browse/FLUME-992
> Project: Flume
> Issue Type: Sub-task
> Affects Versions: v1.0.0
> Reporter: Hari Shreedharan
> Assignee: Hari Shreedharan
> Fix For: v1.1.0
>
>
> Currently the configuration of each component sits deep inside the component themselves. There is no way to verify if a configuration is valid before run time. In most systems like Flume, it is likely that these confs will be deployed from one single host, on to the machines where flume agents are running. Only when each agent starts, invalid confs are identified because the Agents would terminate by throwing exceptions. This is a first attempt to make a component-aware configuration system which is independent of the Flume, and does not require the Flume jar to be installed. Each component needs a configuration manager, which configures the components.
> Provide abstract Configuration stubs for each component type, sources, channels, sinks, selectors etc, which are in the new package, independent on ng-core. Now for each of the channels extend the abstract class and check the config properties for each of the required parameters for that component, for example: MultiplexingChannelSelectorConfigurator would look for default channel etc. If a particular component does not have a configuration class then let the current mechanism continue.
> This will require implementation for each component, but it should not be too complex. One additional advantage we get from this is that we can separate out the config validation from the components into these stubs, but we will still need to read the values out of the Context as we do currently(else we end up making the configuration dependent on flume-ng-core itself which we wanted to avoid).
> A problem with this is making a change to the configuration would require changes in the configuration classes and in the components also(where the configuration is read and the component is actually configured). I could not figure out a way of avoiding this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-992) Create configuration stubs for
sources, channels, sinks etc
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13219680#comment-13219680 ]
jiraposter@reviews.apache.org commented on FLUME-992:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4115/
-----------------------------------------------------------
(Updated 2012-03-01 00:31:40.789559)
Review request for Flume.
Changes
-------
Last diff included changes not intended for this jira.
Summary
-------
Currently the configuration of each component sits deep inside the component themselves. There is no way to verify if a configuration is valid before run time. In most systems like Flume, it is likely that these confs will be deployed from one single host, on to the machines where flume agents are running. Only when each agent starts, invalid confs are identified because the Agents would terminate by throwing exceptions. This is a first attempt to make a component-aware configuration system which is independent of the Flume, and does not require the Flume jar to be installed. Each component needs a configuration manager, which configures the components.
I have not completed the validation part yet, but have completed the configuration managers for all the components that come in the flume jar. I will add support for the other components, such as HDFS sink, JDBC channel etc soon. I will also add the validation support to the classes, but would like a review of the model being proposed here.
This addresses bug FLUME-992.
https://issues.apache.org/jira/browse/FLUME-992
Diffs (updated)
-----
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannel.java bca0c50
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProvider.java e445d61
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProviderFactory.java 6fbd6ef
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java 1cf1c0c
flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestJdbcChannelProvider.java 7710d46
flume-ng-clients/flume-ng-log4jappender/src/test/java/org/apache/flume/clients/log4jappender/TestLog4jAppender.java 68d95fb
flume-ng-configuration/pom.xml PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/AbstractComponentConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ConfigurationException.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/Context.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationError.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationErrorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MemoryChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MultiplexingChannelSelectorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/PseudoTxnMemoryChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/AvroSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/FailoverSinkProcessorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/RollingFileSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkGroupConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/AvroSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/ExecSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/NetcatSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/test/resources/flume-ng-error-test PRE-CREATION
flume-ng-configuration/src/test/resources/flume-ng-test.conf PRE-CREATION
flume-ng-core/pom.xml d753fa1
flume-ng-core/src/main/java/org/apache/flume/Context.java f1c8f85
flume-ng-core/src/main/java/org/apache/flume/channel/AbstractChannel.java 352bf08
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorFactory.java a7d5f94
flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java e79490e
flume-ng-core/src/main/java/org/apache/flume/channel/MultiplexingChannelSelector.java c63d0a1
flume-ng-core/src/main/java/org/apache/flume/channel/PseudoTxnMemoryChannel.java 1df580e
flume-ng-core/src/main/java/org/apache/flume/channel/ReplicatingChannelSelector.java 8f22746
flume-ng-core/src/main/java/org/apache/flume/conf/Configurable.java 0fa4839
flume-ng-core/src/main/java/org/apache/flume/conf/Configurables.java 84492e5
flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 5440631
flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkProcessor.java 257bab3
flume-ng-core/src/main/java/org/apache/flume/sink/FailoverSinkProcessor.java 9f5b856
flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java 45c031d
flume-ng-core/src/main/java/org/apache/flume/sink/SinkGroup.java 1adc5ff
flume-ng-core/src/main/java/org/apache/flume/sink/SinkProcessorFactory.java 859f4fd
flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java 7b079f9
flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java a96016c
flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java d205bbc
flume-ng-core/src/test/java/org/apache/flume/TestContext.java a5e6aa8
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannel.java 3392dff
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelConcurrency.java 8dad0b2
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelTransaction.java bc81f26
flume-ng-core/src/test/java/org/apache/flume/sink/TestAvroSink.java 467785f
flume-ng-core/src/test/java/org/apache/flume/sink/TestFailoverSinkProcessor.java 93ad3bf
flume-ng-core/src/test/java/org/apache/flume/sink/TestLoggerSink.java 92ff6fe
flume-ng-core/src/test/java/org/apache/flume/sink/TestRollingFileSink.java 7e26e2a
flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java c5c3f2f
flume-ng-core/src/test/java/org/apache/flume/source/TestExecSource.java 6035270
flume-ng-core/src/test/java/org/apache/flume/source/TestPollableSourceRunner.java c27f82c
flume-ng-core/src/test/java/org/apache/flume/source/TestSequenceGeneratorSource.java 579b257
flume-ng-legacy-sources/flume-avro-source/src/main/java/org/apache/flume/source/avroLegacy/AvroLegacySource.java b1e67f7
flume-ng-legacy-sources/flume-avro-source/src/test/java/org/apache/flume/source/avroLegacy/TestLegacyAvroSource.java 6e3eb53
flume-ng-legacy-sources/flume-thrift-source/src/main/java/org/apache/flume/source/thriftLegacy/ThriftLegacySource.java 5fe270a
flume-ng-legacy-sources/flume-thrift-source/src/test/java/org/apache/flume/source/thriftLegacy/TestThriftLegacySource.java ddd9478
flume-ng-node/pom.xml b9b062e
flume-ng-node/src/main/java/org/apache/flume/conf/properties/FlumeConfiguration.java d66f6d1
flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java 97f72e1
flume-ng-node/src/test/java/org/apache/flume/node/TestAbstractLogicalNodeManager.java 521b586
flume-ng-node/src/test/java/org/apache/flume/source/TestNetcatSource.java fb2a960
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 3da90a5
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 7d8ee8a
flume-ng-sinks/flume-irc-sink/src/main/java/org/apache/flume/sink/irc/IRCSink.java 0a5498f
pom.xml d785762
Diff: https://reviews.apache.org/r/4115/diff
Testing
-------
All existing unit tests for the components whose configuration has now been moved to these stubs pass.
Thanks,
Hari
> Create configuration stubs for sources, channels, sinks etc
> -----------------------------------------------------------
>
> Key: FLUME-992
> URL: https://issues.apache.org/jira/browse/FLUME-992
> Project: Flume
> Issue Type: Sub-task
> Affects Versions: v1.0.0
> Reporter: Hari Shreedharan
> Fix For: v1.1.0
>
>
> Currently the configuration of each component sits deep inside the component themselves. There is no way to verify if a configuration is valid before run time. In most systems like Flume, it is likely that these confs will be deployed from one single host, on to the machines where flume agents are running. Only when each agent starts, invalid confs are identified because the Agents would terminate by throwing exceptions. This is a first attempt to make a component-aware configuration system which is independent of the Flume, and does not require the Flume jar to be installed. Each component needs a configuration manager, which configures the components.
> Provide abstract Configuration stubs for each component type, sources, channels, sinks, selectors etc, which are in the new package, independent on ng-core. Now for each of the channels extend the abstract class and check the config properties for each of the required parameters for that component, for example: MultiplexingChannelSelectorConfigurator would look for default channel etc. If a particular component does not have a configuration class then let the current mechanism continue.
> This will require implementation for each component, but it should not be too complex. One additional advantage we get from this is that we can separate out the config validation from the components into these stubs, but we will still need to read the values out of the Context as we do currently(else we end up making the configuration dependent on flume-ng-core itself which we wanted to avoid).
> A problem with this is making a change to the configuration would require changes in the configuration classes and in the components also(where the configuration is read and the component is actually configured). I could not figure out a way of avoiding this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-992) Create configuration stubs for
sources, channels, sinks etc
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13219662#comment-13219662 ]
jiraposter@reviews.apache.org commented on FLUME-992:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4115/
-----------------------------------------------------------
Review request for Flume.
Summary
-------
Currently the configuration of each component sits deep inside the component themselves. There is no way to verify if a configuration is valid before run time. In most systems like Flume, it is likely that these confs will be deployed from one single host, on to the machines where flume agents are running. Only when each agent starts, invalid confs are identified because the Agents would terminate by throwing exceptions. This is a first attempt to make a component-aware configuration system which is independent of the Flume, and does not require the Flume jar to be installed. Each component needs a configuration manager, which configures the components.
I have not completed the validation part yet, but have completed the configuration managers for all the components that come in the flume jar. I will add support for the other components, such as HDFS sink, JDBC channel etc soon. I will also add the validation support to the classes, but would like a review of the model being proposed here.
This addresses bug FLUME-992.
https://issues.apache.org/jira/browse/FLUME-992
Diffs
-----
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannel.java bca0c50
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProvider.java e445d61
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProviderFactory.java 6fbd6ef
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java 1cf1c0c
flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestJdbcChannelProvider.java 7710d46
flume-ng-clients/flume-ng-log4jappender/src/test/java/org/apache/flume/clients/log4jappender/TestLog4jAppender.java 68d95fb
flume-ng-configuration/pom.xml PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/AbstractComponentConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ConfigurationException.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/Context.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationError.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationErrorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MemoryChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MultiplexingChannelSelectorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/PseudoTxnMemoryChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/AvroSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/FailoverSinkProcessorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/RollingFileSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkGroupConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/AvroSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/ExecSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/NetcatSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/test/resources/flume-ng-error-test PRE-CREATION
flume-ng-configuration/src/test/resources/flume-ng-test.conf PRE-CREATION
flume-ng-core/pom.xml d753fa1
flume-ng-core/src/main/java/org/apache/flume/Context.java f1c8f85
flume-ng-core/src/main/java/org/apache/flume/channel/AbstractChannel.java 352bf08
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorFactory.java a7d5f94
flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java e79490e
flume-ng-core/src/main/java/org/apache/flume/channel/MultiplexingChannelSelector.java c63d0a1
flume-ng-core/src/main/java/org/apache/flume/channel/PseudoTxnMemoryChannel.java 1df580e
flume-ng-core/src/main/java/org/apache/flume/channel/ReplicatingChannelSelector.java 8f22746
flume-ng-core/src/main/java/org/apache/flume/conf/Configurable.java 0fa4839
flume-ng-core/src/main/java/org/apache/flume/conf/Configurables.java 84492e5
flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 5440631
flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkProcessor.java 257bab3
flume-ng-core/src/main/java/org/apache/flume/sink/FailoverSinkProcessor.java 9f5b856
flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java 45c031d
flume-ng-core/src/main/java/org/apache/flume/sink/SinkGroup.java 1adc5ff
flume-ng-core/src/main/java/org/apache/flume/sink/SinkProcessorFactory.java 859f4fd
flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java 7b079f9
flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java a96016c
flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java d205bbc
flume-ng-core/src/test/java/org/apache/flume/TestContext.java a5e6aa8
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannel.java 3392dff
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelConcurrency.java 8dad0b2
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelTransaction.java bc81f26
flume-ng-core/src/test/java/org/apache/flume/channel/TestMultiplexingChannelSelector.java 2626b20
flume-ng-core/src/test/java/org/apache/flume/sink/TestAvroSink.java 467785f
flume-ng-core/src/test/java/org/apache/flume/sink/TestFailoverSinkProcessor.java 93ad3bf
flume-ng-core/src/test/java/org/apache/flume/sink/TestLoggerSink.java 92ff6fe
flume-ng-core/src/test/java/org/apache/flume/sink/TestRollingFileSink.java 7e26e2a
flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java c5c3f2f
flume-ng-core/src/test/java/org/apache/flume/source/TestExecSource.java 6035270
flume-ng-core/src/test/java/org/apache/flume/source/TestPollableSourceRunner.java c27f82c
flume-ng-core/src/test/java/org/apache/flume/source/TestSequenceGeneratorSource.java 579b257
flume-ng-legacy-sources/flume-avro-source/src/main/java/org/apache/flume/source/avroLegacy/AvroLegacySource.java b1e67f7
flume-ng-legacy-sources/flume-avro-source/src/test/java/org/apache/flume/source/avroLegacy/TestLegacyAvroSource.java 6e3eb53
flume-ng-legacy-sources/flume-thrift-source/src/main/java/org/apache/flume/source/thriftLegacy/ThriftLegacySource.java 5fe270a
flume-ng-legacy-sources/flume-thrift-source/src/test/java/org/apache/flume/source/thriftLegacy/TestThriftLegacySource.java ddd9478
flume-ng-node/pom.xml b9b062e
flume-ng-node/src/main/java/org/apache/flume/conf/properties/FlumeConfiguration.java d66f6d1
flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java 97f72e1
flume-ng-node/src/test/java/org/apache/flume/node/TestAbstractLogicalNodeManager.java 521b586
flume-ng-node/src/test/java/org/apache/flume/source/TestNetcatSource.java fb2a960
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 3da90a5
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 7d8ee8a
flume-ng-sinks/flume-irc-sink/src/main/java/org/apache/flume/sink/irc/IRCSink.java 0a5498f
pom.xml d785762
Diff: https://reviews.apache.org/r/4115/diff
Testing
-------
All existing unit tests for the components whose configuration has now been moved to these stubs pass.
Thanks,
Hari
> Create configuration stubs for sources, channels, sinks etc
> -----------------------------------------------------------
>
> Key: FLUME-992
> URL: https://issues.apache.org/jira/browse/FLUME-992
> Project: Flume
> Issue Type: Sub-task
> Affects Versions: v1.0.0
> Reporter: Hari Shreedharan
> Fix For: v1.1.0
>
>
> Currently the configuration of each component sits deep inside the component themselves. There is no way to verify if a configuration is valid before run time. In most systems like Flume, it is likely that these confs will be deployed from one single host, on to the machines where flume agents are running. Only when each agent starts, invalid confs are identified because the Agents would terminate by throwing exceptions. This is a first attempt to make a component-aware configuration system which is independent of the Flume, and does not require the Flume jar to be installed. Each component needs a configuration manager, which configures the components.
> Provide abstract Configuration stubs for each component type, sources, channels, sinks, selectors etc, which are in the new package, independent on ng-core. Now for each of the channels extend the abstract class and check the config properties for each of the required parameters for that component, for example: MultiplexingChannelSelectorConfigurator would look for default channel etc. If a particular component does not have a configuration class then let the current mechanism continue.
> This will require implementation for each component, but it should not be too complex. One additional advantage we get from this is that we can separate out the config validation from the components into these stubs, but we will still need to read the values out of the Context as we do currently(else we end up making the configuration dependent on flume-ng-core itself which we wanted to avoid).
> A problem with this is making a change to the configuration would require changes in the configuration classes and in the components also(where the configuration is read and the component is actually configured). I could not figure out a way of avoiding this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-992) Create configuration stubs for
sources, channels, sinks etc
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13228569#comment-13228569 ]
jiraposter@reviews.apache.org commented on FLUME-992:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4115/
-----------------------------------------------------------
(Updated 2012-03-13 18:34:28.437390)
Review request for Flume.
Changes
-------
Adding support for context based configuration components
Summary
-------
Currently the configuration of each component sits deep inside the component themselves. There is no way to verify if a configuration is valid before run time. In most systems like Flume, it is likely that these confs will be deployed from one single host, on to the machines where flume agents are running. Only when each agent starts, invalid confs are identified because the Agents would terminate by throwing exceptions. This is a first attempt to make a component-aware configuration system which is independent of the Flume, and does not require the Flume jar to be installed. Each component needs a configuration manager, which configures the components.
I have not completed the validation part yet, but have completed the configuration managers for all the components that come in the flume jar. I will add support for the other components, such as HDFS sink, JDBC channel etc soon. I will also add the validation support to the classes, but would like a review of the model being proposed here.
Note that I do know of a couple of bugs in the existing diff(like that if the name of a config key starts with an Upper case letter, an exception is thrown) - I will fix and upload the review here soon.
This addresses bug FLUME-992.
https://issues.apache.org/jira/browse/FLUME-992
Diffs (updated)
-----
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannel.java bca0c50
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProvider.java e445d61
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProviderFactory.java 6fbd6ef
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java 307ae89
flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestJdbcChannelProvider.java 7710d46
flume-ng-clients/flume-ng-log4jappender/src/test/java/org/apache/flume/clients/log4jappender/TestLog4jAppender.java 68d95fb
flume-ng-configuration/pom.xml PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfigurationFactory.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ConfigurationException.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/Context.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationError.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationErrorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MemoryChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MultiplexingChannelSelectorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/PseudoTxnMemoryChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/AvroSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/FailoverSinkProcessorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/RollingFileSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkGroupConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkProcessorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkProcessorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/AvroSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/ExecSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/NetcatSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceType.java PRE-CREATION
flume-ng-configuration/src/test/resources/flume-ng-error-test PRE-CREATION
flume-ng-configuration/src/test/resources/flume-ng-test.conf PRE-CREATION
flume-ng-core/pom.xml d753fa1
flume-ng-core/src/main/java/org/apache/flume/ChannelSelector.java fba2dcb
flume-ng-core/src/main/java/org/apache/flume/Context.java a6341a5
flume-ng-core/src/main/java/org/apache/flume/SinkProcessor.java 11651ad
flume-ng-core/src/main/java/org/apache/flume/SinkProcessorType.java be1891b
flume-ng-core/src/main/java/org/apache/flume/channel/AbstractChannel.java 352bf08
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorFactory.java 800f471
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorType.java 511fc65
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8
flume-ng-core/src/main/java/org/apache/flume/channel/DefaultChannelFactory.java 963a6a3
flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java bfa1fde
flume-ng-core/src/main/java/org/apache/flume/channel/MultiplexingChannelSelector.java 83928b7
flume-ng-core/src/main/java/org/apache/flume/channel/PseudoTxnMemoryChannel.java 489d3e5
flume-ng-core/src/main/java/org/apache/flume/channel/ReplicatingChannelSelector.java 8f22746
flume-ng-core/src/main/java/org/apache/flume/conf/Configurable.java 0fa4839
flume-ng-core/src/main/java/org/apache/flume/conf/ConfigurableComponent.java PRE-CREATION
flume-ng-core/src/main/java/org/apache/flume/conf/Configurables.java 84492e5
flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 7386d06
flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkFactory.java b89dfa0
flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkProcessor.java 257bab3
flume-ng-core/src/main/java/org/apache/flume/sink/FailoverSinkProcessor.java 9f5b856
flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java 7f1d3b3
flume-ng-core/src/main/java/org/apache/flume/sink/SinkGroup.java 2e80a56
flume-ng-core/src/main/java/org/apache/flume/sink/SinkProcessorFactory.java 10f9f4e
flume-ng-core/src/main/java/org/apache/flume/sink/SinkType.java 6b08c09
flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java a3f6640
flume-ng-core/src/main/java/org/apache/flume/source/DefaultSourceFactory.java a610e6f
flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java b6b1181
flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java 94245ac
flume-ng-core/src/main/java/org/apache/flume/source/SourceType.java 9082470
flume-ng-core/src/test/java/org/apache/flume/TestContext.java 51c350f
flume-ng-core/src/test/java/org/apache/flume/channel/AbstractBasicChannelSemanticsTest.java 6e71e46
flume-ng-core/src/test/java/org/apache/flume/channel/MockChannel.java 24b01e2
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannel.java e070864
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelConcurrency.java 8dad0b2
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelTransaction.java bc81f26
flume-ng-core/src/test/java/org/apache/flume/sink/TestAvroSink.java 467785f
flume-ng-core/src/test/java/org/apache/flume/sink/TestFailoverSinkProcessor.java 195c121
flume-ng-core/src/test/java/org/apache/flume/sink/TestLoggerSink.java 92ff6fe
flume-ng-core/src/test/java/org/apache/flume/sink/TestRollingFileSink.java 7e26e2a
flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java c5c3f2f
flume-ng-core/src/test/java/org/apache/flume/source/TestExecSource.java 6035270
flume-ng-core/src/test/java/org/apache/flume/source/TestPollableSourceRunner.java c27f82c
flume-ng-core/src/test/java/org/apache/flume/source/TestSequenceGeneratorSource.java 579b257
flume-ng-legacy-sources/flume-avro-source/src/main/java/org/apache/flume/source/avroLegacy/AvroLegacySource.java 65d9142
flume-ng-legacy-sources/flume-avro-source/src/test/java/org/apache/flume/source/avroLegacy/TestLegacyAvroSource.java 6e3eb53
flume-ng-legacy-sources/flume-thrift-source/src/main/java/org/apache/flume/source/thriftLegacy/ThriftLegacySource.java fbf7362
flume-ng-legacy-sources/flume-thrift-source/src/test/java/org/apache/flume/source/thriftLegacy/TestThriftLegacySource.java ddd9478
flume-ng-node/src/main/java/org/apache/flume/conf/properties/FlumeConfiguration.java d66f6d1
flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java 97f72e1
flume-ng-node/src/test/java/org/apache/flume/node/TestAbstractLogicalNodeManager.java 521b586
flume-ng-node/src/test/java/org/apache/flume/source/TestNetcatSource.java fb2a960
flume-ng-node/src/test/resources/flume-conf.properties 848caca
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 524b69c
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 7d8ee8a
flume-ng-sinks/flume-irc-sink/src/main/java/org/apache/flume/sink/irc/IRCSink.java 8e77218
pom.xml d785762
Diff: https://reviews.apache.org/r/4115/diff
Testing
-------
All existing unit tests for the components whose configuration has now been moved to these stubs pass.
Thanks,
Hari
> Create configuration stubs for sources, channels, sinks etc
> -----------------------------------------------------------
>
> Key: FLUME-992
> URL: https://issues.apache.org/jira/browse/FLUME-992
> Project: Flume
> Issue Type: Sub-task
> Affects Versions: v1.0.0
> Reporter: Hari Shreedharan
> Assignee: Hari Shreedharan
> Fix For: v1.2.0
>
>
> Currently the configuration of each component sits deep inside the component themselves. There is no way to verify if a configuration is valid before run time. In most systems like Flume, it is likely that these confs will be deployed from one single host, on to the machines where flume agents are running. Only when each agent starts, invalid confs are identified because the Agents would terminate by throwing exceptions. This is a first attempt to make a component-aware configuration system which is independent of the Flume, and does not require the Flume jar to be installed. Each component needs a configuration manager, which configures the components.
> Provide abstract Configuration stubs for each component type, sources, channels, sinks, selectors etc, which are in the new package, independent on ng-core. Now for each of the channels extend the abstract class and check the config properties for each of the required parameters for that component, for example: MultiplexingChannelSelectorConfigurator would look for default channel etc. If a particular component does not have a configuration class then let the current mechanism continue.
> This will require implementation for each component, but it should not be too complex. One additional advantage we get from this is that we can separate out the config validation from the components into these stubs, but we will still need to read the values out of the Context as we do currently(else we end up making the configuration dependent on flume-ng-core itself which we wanted to avoid).
> A problem with this is making a change to the configuration would require changes in the configuration classes and in the components also(where the configuration is read and the component is actually configured). I could not figure out a way of avoiding this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-992) Create configuration stubs for
sources, channels, sinks etc
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13229895#comment-13229895 ]
jiraposter@reviews.apache.org commented on FLUME-992:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4115/
-----------------------------------------------------------
(Updated 2012-03-15 05:07:23.274303)
Review request for Flume.
Changes
-------
Bug fix.
Summary
-------
Currently the configuration of each component sits deep inside the component themselves. There is no way to verify if a configuration is valid before run time. In most systems like Flume, it is likely that these confs will be deployed from one single host, on to the machines where flume agents are running. Only when each agent starts, invalid confs are identified because the Agents would terminate by throwing exceptions. This is a first attempt to make a component-aware configuration system which is independent of the Flume, and does not require the Flume jar to be installed. Each component needs a configuration manager, which configures the components.
I have not completed the validation part yet, but have completed the configuration managers for all the components that come in the flume jar. I will add support for the other components, such as HDFS sink, JDBC channel etc soon. I will also add the validation support to the classes, but would like a review of the model being proposed here.
Note that I do know of a couple of bugs in the existing diff(like that if the name of a config key starts with an Upper case letter, an exception is thrown) - I will fix and upload the review here soon.
This addresses bug FLUME-992.
https://issues.apache.org/jira/browse/FLUME-992
Diffs (updated)
-----
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannel.java bca0c50
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProvider.java e445d61
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProviderFactory.java 6fbd6ef
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java 307ae89
flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestJdbcChannelProvider.java 7710d46
flume-ng-clients/flume-ng-log4jappender/src/test/java/org/apache/flume/clients/log4jappender/TestLog4jAppender.java 68d95fb
flume-ng-configuration/pom.xml PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfigurationFactory.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ConfigurationException.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/Context.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationError.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationErrorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MemoryChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MultiplexingChannelSelectorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/PseudoTxnMemoryChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/AvroSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/FailoverSinkProcessorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/RollingFileSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkGroupConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkProcessorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkProcessorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/AvroSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/ExecSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/NetcatSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceType.java PRE-CREATION
flume-ng-configuration/src/test/resources/flume-ng-error-test PRE-CREATION
flume-ng-configuration/src/test/resources/flume-ng-test.conf PRE-CREATION
flume-ng-core/pom.xml 247df37
flume-ng-core/src/main/java/org/apache/flume/ChannelSelector.java fba2dcb
flume-ng-core/src/main/java/org/apache/flume/Context.java 5294e31
flume-ng-core/src/main/java/org/apache/flume/SinkProcessor.java ea6000b
flume-ng-core/src/main/java/org/apache/flume/SinkProcessorType.java be1891b
flume-ng-core/src/main/java/org/apache/flume/channel/AbstractChannel.java 352bf08
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorFactory.java 800f471
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorType.java 511fc65
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8
flume-ng-core/src/main/java/org/apache/flume/channel/DefaultChannelFactory.java 963a6a3
flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java bfa1fde
flume-ng-core/src/main/java/org/apache/flume/channel/MultiplexingChannelSelector.java 83928b7
flume-ng-core/src/main/java/org/apache/flume/channel/PseudoTxnMemoryChannel.java 489d3e5
flume-ng-core/src/main/java/org/apache/flume/channel/ReplicatingChannelSelector.java 8f22746
flume-ng-core/src/main/java/org/apache/flume/conf/Configurable.java 0fa4839
flume-ng-core/src/main/java/org/apache/flume/conf/ConfigurableComponent.java PRE-CREATION
flume-ng-core/src/main/java/org/apache/flume/conf/Configurables.java 84492e5
flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 3edc563
flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkFactory.java b89dfa0
flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkProcessor.java 257bab3
flume-ng-core/src/main/java/org/apache/flume/sink/FailoverSinkProcessor.java 9f5b856
flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java 7f1d3b3
flume-ng-core/src/main/java/org/apache/flume/sink/SinkGroup.java 0dffd69
flume-ng-core/src/main/java/org/apache/flume/sink/SinkProcessorFactory.java 6160a17
flume-ng-core/src/main/java/org/apache/flume/sink/SinkType.java 6b08c09
flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java 903889d
flume-ng-core/src/main/java/org/apache/flume/source/DefaultSourceFactory.java a610e6f
flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java b6b1181
flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java 94245ac
flume-ng-core/src/main/java/org/apache/flume/source/SequenceGeneratorSource.java 440c5a9
flume-ng-core/src/main/java/org/apache/flume/source/SourceType.java 9082470
flume-ng-core/src/main/java/org/apache/flume/source/SyslogTcpSource.java d78d27f
flume-ng-core/src/main/java/org/apache/flume/source/SyslogUDPSource.java 34818f0
flume-ng-core/src/test/java/org/apache/flume/TestContext.java 51c350f
flume-ng-core/src/test/java/org/apache/flume/channel/AbstractBasicChannelSemanticsTest.java 6e71e46
flume-ng-core/src/test/java/org/apache/flume/channel/MockChannel.java 24b01e2
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannel.java e070864
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelConcurrency.java 8dad0b2
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelTransaction.java bc81f26
flume-ng-core/src/test/java/org/apache/flume/sink/TestAvroSink.java 3765924
flume-ng-core/src/test/java/org/apache/flume/sink/TestFailoverSinkProcessor.java 195c121
flume-ng-core/src/test/java/org/apache/flume/sink/TestLoggerSink.java 92ff6fe
flume-ng-core/src/test/java/org/apache/flume/sink/TestRollingFileSink.java 7e26e2a
flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java c5c3f2f
flume-ng-core/src/test/java/org/apache/flume/source/TestExecSource.java 6035270
flume-ng-core/src/test/java/org/apache/flume/source/TestPollableSourceRunner.java c27f82c
flume-ng-core/src/test/java/org/apache/flume/source/TestSequenceGeneratorSource.java 579b257
flume-ng-core/src/test/java/org/apache/flume/source/TestSyslogUdpSource.java d1db49d
flume-ng-legacy-sources/flume-avro-source/src/main/java/org/apache/flume/source/avroLegacy/AvroLegacySource.java dde8f28
flume-ng-legacy-sources/flume-avro-source/src/test/java/org/apache/flume/source/avroLegacy/TestLegacyAvroSource.java 6e3eb53
flume-ng-legacy-sources/flume-thrift-source/src/main/java/org/apache/flume/source/thriftLegacy/ThriftLegacySource.java fbf7362
flume-ng-legacy-sources/flume-thrift-source/src/test/java/org/apache/flume/source/thriftLegacy/TestThriftLegacySource.java ddd9478
flume-ng-node/src/main/java/org/apache/flume/conf/properties/FlumeConfiguration.java d66f6d1
flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java 97f72e1
flume-ng-node/src/test/java/org/apache/flume/node/TestAbstractLogicalNodeManager.java 521b586
flume-ng-node/src/test/java/org/apache/flume/source/TestNetcatSource.java fb2a960
flume-ng-node/src/test/resources/flume-conf.properties 848caca
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java da82f7e
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 8fa72a1
flume-ng-sinks/flume-irc-sink/src/main/java/org/apache/flume/sink/irc/IRCSink.java 8e77218
pom.xml 794ebe3
Diff: https://reviews.apache.org/r/4115/diff
Testing
-------
All existing unit tests for the components whose configuration has now been moved to these stubs pass.
Thanks,
Hari
> Create configuration stubs for sources, channels, sinks etc
> -----------------------------------------------------------
>
> Key: FLUME-992
> URL: https://issues.apache.org/jira/browse/FLUME-992
> Project: Flume
> Issue Type: Sub-task
> Affects Versions: v1.0.0
> Reporter: Hari Shreedharan
> Assignee: Hari Shreedharan
> Fix For: v1.2.0
>
>
> Currently the configuration of each component sits deep inside the component themselves. There is no way to verify if a configuration is valid before run time. In most systems like Flume, it is likely that these confs will be deployed from one single host, on to the machines where flume agents are running. Only when each agent starts, invalid confs are identified because the Agents would terminate by throwing exceptions. This is a first attempt to make a component-aware configuration system which is independent of the Flume, and does not require the Flume jar to be installed. Each component needs a configuration manager, which configures the components.
> Provide abstract Configuration stubs for each component type, sources, channels, sinks, selectors etc, which are in the new package, independent on ng-core. Now for each of the channels extend the abstract class and check the config properties for each of the required parameters for that component, for example: MultiplexingChannelSelectorConfigurator would look for default channel etc. If a particular component does not have a configuration class then let the current mechanism continue.
> This will require implementation for each component, but it should not be too complex. One additional advantage we get from this is that we can separate out the config validation from the components into these stubs, but we will still need to read the values out of the Context as we do currently(else we end up making the configuration dependent on flume-ng-core itself which we wanted to avoid).
> A problem with this is making a change to the configuration would require changes in the configuration classes and in the components also(where the configuration is read and the component is actually configured). I could not figure out a way of avoiding this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-992) Create configuration stubs for
sources, channels, sinks etc
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13229833#comment-13229833 ]
jiraposter@reviews.apache.org commented on FLUME-992:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4115/
-----------------------------------------------------------
(Updated 2012-03-15 01:55:45.546999)
Review request for Flume.
Changes
-------
Added headers to Sequence generator source to test multiplexing channel selector.
Summary
-------
Currently the configuration of each component sits deep inside the component themselves. There is no way to verify if a configuration is valid before run time. In most systems like Flume, it is likely that these confs will be deployed from one single host, on to the machines where flume agents are running. Only when each agent starts, invalid confs are identified because the Agents would terminate by throwing exceptions. This is a first attempt to make a component-aware configuration system which is independent of the Flume, and does not require the Flume jar to be installed. Each component needs a configuration manager, which configures the components.
I have not completed the validation part yet, but have completed the configuration managers for all the components that come in the flume jar. I will add support for the other components, such as HDFS sink, JDBC channel etc soon. I will also add the validation support to the classes, but would like a review of the model being proposed here.
Note that I do know of a couple of bugs in the existing diff(like that if the name of a config key starts with an Upper case letter, an exception is thrown) - I will fix and upload the review here soon.
This addresses bug FLUME-992.
https://issues.apache.org/jira/browse/FLUME-992
Diffs (updated)
-----
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannel.java bca0c50
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProvider.java e445d61
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProviderFactory.java 6fbd6ef
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java 307ae89
flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestJdbcChannelProvider.java 7710d46
flume-ng-clients/flume-ng-log4jappender/src/test/java/org/apache/flume/clients/log4jappender/TestLog4jAppender.java 68d95fb
flume-ng-configuration/pom.xml PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfigurationFactory.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ConfigurationException.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/Context.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationError.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationErrorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MemoryChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MultiplexingChannelSelectorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/PseudoTxnMemoryChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/AvroSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/FailoverSinkProcessorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/RollingFileSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkGroupConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkProcessorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkProcessorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/AvroSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/ExecSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/NetcatSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceType.java PRE-CREATION
flume-ng-configuration/src/test/resources/flume-ng-error-test PRE-CREATION
flume-ng-configuration/src/test/resources/flume-ng-test.conf PRE-CREATION
flume-ng-core/pom.xml 247df37
flume-ng-core/src/main/java/org/apache/flume/ChannelSelector.java fba2dcb
flume-ng-core/src/main/java/org/apache/flume/Context.java 5294e31
flume-ng-core/src/main/java/org/apache/flume/SinkProcessor.java ea6000b
flume-ng-core/src/main/java/org/apache/flume/SinkProcessorType.java be1891b
flume-ng-core/src/main/java/org/apache/flume/channel/AbstractChannel.java 352bf08
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorFactory.java 800f471
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorType.java 511fc65
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8
flume-ng-core/src/main/java/org/apache/flume/channel/DefaultChannelFactory.java 963a6a3
flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java bfa1fde
flume-ng-core/src/main/java/org/apache/flume/channel/MultiplexingChannelSelector.java 83928b7
flume-ng-core/src/main/java/org/apache/flume/channel/PseudoTxnMemoryChannel.java 489d3e5
flume-ng-core/src/main/java/org/apache/flume/channel/ReplicatingChannelSelector.java 8f22746
flume-ng-core/src/main/java/org/apache/flume/conf/Configurable.java 0fa4839
flume-ng-core/src/main/java/org/apache/flume/conf/ConfigurableComponent.java PRE-CREATION
flume-ng-core/src/main/java/org/apache/flume/conf/Configurables.java 84492e5
flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 3edc563
flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkFactory.java b89dfa0
flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkProcessor.java 257bab3
flume-ng-core/src/main/java/org/apache/flume/sink/FailoverSinkProcessor.java 9f5b856
flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java 7f1d3b3
flume-ng-core/src/main/java/org/apache/flume/sink/SinkGroup.java 0dffd69
flume-ng-core/src/main/java/org/apache/flume/sink/SinkProcessorFactory.java 6160a17
flume-ng-core/src/main/java/org/apache/flume/sink/SinkType.java 6b08c09
flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java 903889d
flume-ng-core/src/main/java/org/apache/flume/source/DefaultSourceFactory.java a610e6f
flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java b6b1181
flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java 94245ac
flume-ng-core/src/main/java/org/apache/flume/source/SequenceGeneratorSource.java 440c5a9
flume-ng-core/src/main/java/org/apache/flume/source/SourceType.java 9082470
flume-ng-core/src/main/java/org/apache/flume/source/SyslogTcpSource.java d78d27f
flume-ng-core/src/main/java/org/apache/flume/source/SyslogUDPSource.java 34818f0
flume-ng-core/src/test/java/org/apache/flume/TestContext.java 51c350f
flume-ng-core/src/test/java/org/apache/flume/channel/AbstractBasicChannelSemanticsTest.java 6e71e46
flume-ng-core/src/test/java/org/apache/flume/channel/MockChannel.java 24b01e2
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannel.java e070864
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelConcurrency.java 8dad0b2
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelTransaction.java bc81f26
flume-ng-core/src/test/java/org/apache/flume/sink/TestAvroSink.java 3765924
flume-ng-core/src/test/java/org/apache/flume/sink/TestFailoverSinkProcessor.java 195c121
flume-ng-core/src/test/java/org/apache/flume/sink/TestLoggerSink.java 92ff6fe
flume-ng-core/src/test/java/org/apache/flume/sink/TestRollingFileSink.java 7e26e2a
flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java c5c3f2f
flume-ng-core/src/test/java/org/apache/flume/source/TestExecSource.java 6035270
flume-ng-core/src/test/java/org/apache/flume/source/TestPollableSourceRunner.java c27f82c
flume-ng-core/src/test/java/org/apache/flume/source/TestSequenceGeneratorSource.java 579b257
flume-ng-core/src/test/java/org/apache/flume/source/TestSyslogUdpSource.java d1db49d
flume-ng-legacy-sources/flume-avro-source/src/main/java/org/apache/flume/source/avroLegacy/AvroLegacySource.java dde8f28
flume-ng-legacy-sources/flume-avro-source/src/test/java/org/apache/flume/source/avroLegacy/TestLegacyAvroSource.java 6e3eb53
flume-ng-legacy-sources/flume-thrift-source/src/main/java/org/apache/flume/source/thriftLegacy/ThriftLegacySource.java fbf7362
flume-ng-legacy-sources/flume-thrift-source/src/test/java/org/apache/flume/source/thriftLegacy/TestThriftLegacySource.java ddd9478
flume-ng-node/src/main/java/org/apache/flume/conf/properties/FlumeConfiguration.java d66f6d1
flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java 97f72e1
flume-ng-node/src/test/java/org/apache/flume/node/TestAbstractLogicalNodeManager.java 521b586
flume-ng-node/src/test/java/org/apache/flume/source/TestNetcatSource.java fb2a960
flume-ng-node/src/test/resources/flume-conf.properties 848caca
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java da82f7e
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 8fa72a1
flume-ng-sinks/flume-irc-sink/src/main/java/org/apache/flume/sink/irc/IRCSink.java 8e77218
pom.xml 794ebe3
Diff: https://reviews.apache.org/r/4115/diff
Testing
-------
All existing unit tests for the components whose configuration has now been moved to these stubs pass.
Thanks,
Hari
> Create configuration stubs for sources, channels, sinks etc
> -----------------------------------------------------------
>
> Key: FLUME-992
> URL: https://issues.apache.org/jira/browse/FLUME-992
> Project: Flume
> Issue Type: Sub-task
> Affects Versions: v1.0.0
> Reporter: Hari Shreedharan
> Assignee: Hari Shreedharan
> Fix For: v1.2.0
>
>
> Currently the configuration of each component sits deep inside the component themselves. There is no way to verify if a configuration is valid before run time. In most systems like Flume, it is likely that these confs will be deployed from one single host, on to the machines where flume agents are running. Only when each agent starts, invalid confs are identified because the Agents would terminate by throwing exceptions. This is a first attempt to make a component-aware configuration system which is independent of the Flume, and does not require the Flume jar to be installed. Each component needs a configuration manager, which configures the components.
> Provide abstract Configuration stubs for each component type, sources, channels, sinks, selectors etc, which are in the new package, independent on ng-core. Now for each of the channels extend the abstract class and check the config properties for each of the required parameters for that component, for example: MultiplexingChannelSelectorConfigurator would look for default channel etc. If a particular component does not have a configuration class then let the current mechanism continue.
> This will require implementation for each component, but it should not be too complex. One additional advantage we get from this is that we can separate out the config validation from the components into these stubs, but we will still need to read the values out of the Context as we do currently(else we end up making the configuration dependent on flume-ng-core itself which we wanted to avoid).
> A problem with this is making a change to the configuration would require changes in the configuration classes and in the components also(where the configuration is read and the component is actually configured). I could not figure out a way of avoiding this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-992) Create configuration stubs for
sources, channels, sinks etc
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13238940#comment-13238940 ]
jiraposter@reviews.apache.org commented on FLUME-992:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4115/
-----------------------------------------------------------
(Updated 2012-03-26 22:50:32.784108)
Review request for Flume.
Changes
-------
missed a couple of files.
Summary
-------
Currently the configuration of each component sits deep inside the component themselves. There is no way to verify if a configuration is valid before run time. In most systems like Flume, it is likely that these confs will be deployed from one single host, on to the machines where flume agents are running. Only when each agent starts, invalid confs are identified because the Agents would terminate by throwing exceptions. This is a first attempt to make a component-aware configuration system which is independent of the Flume, and does not require the Flume jar to be installed. Each component needs a configuration manager, which configures the components.
I have not completed the validation part yet, but have completed the configuration managers for all the components that come in the flume jar. I will add support for the other components, such as HDFS sink, JDBC channel etc soon. I will also add the validation support to the classes, but would like a review of the model being proposed here.
Note that I do know of a couple of bugs in the existing diff(like that if the name of a config key starts with an Upper case letter, an exception is thrown) - I will fix and upload the review here soon.
This addresses bug FLUME-992.
https://issues.apache.org/jira/browse/FLUME-992
Diffs (updated)
-----
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannel.java bca0c50
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProvider.java e445d61
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProviderFactory.java 6fbd6ef
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java 307ae89
flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestJdbcChannelProvider.java 7710d46
flume-ng-clients/flume-ng-log4jappender/src/test/java/org/apache/flume/clients/log4jappender/TestLog4jAppender.java 68d95fb
flume-ng-configuration/pom.xml PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfigurationFactory.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ConfigurationException.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/Context.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationError.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationErrorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/JdbcChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MemoryChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MultiplexingChannelSelectorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/PseudoTxnMemoryChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/AvroSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/FailoverSinkProcessorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/HDFSSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/RollingFileSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkGroupConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkProcessorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkProcessorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/AvroSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/ExecSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/NetcatSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SyslogTcpSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SyslogUDPSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/test/resources/flume-ng-error-test PRE-CREATION
flume-ng-configuration/src/test/resources/flume-ng-test.conf PRE-CREATION
flume-ng-core/pom.xml 37fb112
flume-ng-core/src/main/java/org/apache/flume/ChannelSelector.java fba2dcb
flume-ng-core/src/main/java/org/apache/flume/Context.java 5294e31
flume-ng-core/src/main/java/org/apache/flume/SinkProcessor.java d863ed0
flume-ng-core/src/main/java/org/apache/flume/SinkProcessorType.java be1891b
flume-ng-core/src/main/java/org/apache/flume/channel/AbstractChannel.java 352bf08
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorFactory.java 800f471
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorType.java 511fc65
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8
flume-ng-core/src/main/java/org/apache/flume/channel/DefaultChannelFactory.java 963a6a3
flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java bfa1fde
flume-ng-core/src/main/java/org/apache/flume/channel/MultiplexingChannelSelector.java 83928b7
flume-ng-core/src/main/java/org/apache/flume/channel/PseudoTxnMemoryChannel.java 489d3e5
flume-ng-core/src/main/java/org/apache/flume/channel/ReplicatingChannelSelector.java 8f22746
flume-ng-core/src/main/java/org/apache/flume/conf/Configurable.java 1d69112
flume-ng-core/src/main/java/org/apache/flume/conf/ConfigurableComponent.java PRE-CREATION
flume-ng-core/src/main/java/org/apache/flume/conf/Configurables.java 84492e5
flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 3edc563
flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkFactory.java b89dfa0
flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkProcessor.java 257bab3
flume-ng-core/src/main/java/org/apache/flume/sink/FailoverSinkProcessor.java 066ee37
flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java 7f1d3b3
flume-ng-core/src/main/java/org/apache/flume/sink/SinkGroup.java 0dffd69
flume-ng-core/src/main/java/org/apache/flume/sink/SinkProcessorFactory.java 6160a17
flume-ng-core/src/main/java/org/apache/flume/sink/SinkType.java 6b08c09
flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java 903889d
flume-ng-core/src/main/java/org/apache/flume/source/DefaultSourceFactory.java a610e6f
flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java dbf79e0
flume-ng-core/src/main/java/org/apache/flume/source/ExecSourceConfigurationConstants.java 73c985e
flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java a841b0e
flume-ng-core/src/main/java/org/apache/flume/source/SequenceGeneratorSource.java 440c5a9
flume-ng-core/src/main/java/org/apache/flume/source/SourceType.java 9082470
flume-ng-core/src/main/java/org/apache/flume/source/SyslogTcpSource.java d78d27f
flume-ng-core/src/main/java/org/apache/flume/source/SyslogUDPSource.java 34818f0
flume-ng-core/src/test/java/org/apache/flume/TestContext.java 51c350f
flume-ng-core/src/test/java/org/apache/flume/channel/AbstractBasicChannelSemanticsTest.java 6e71e46
flume-ng-core/src/test/java/org/apache/flume/channel/MockChannel.java 24b01e2
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannel.java e070864
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelConcurrency.java 8dad0b2
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelTransaction.java bc81f26
flume-ng-core/src/test/java/org/apache/flume/sink/TestAvroSink.java 3765924
flume-ng-core/src/test/java/org/apache/flume/sink/TestFailoverSinkProcessor.java 3358cf4
flume-ng-core/src/test/java/org/apache/flume/sink/TestLoggerSink.java 92ff6fe
flume-ng-core/src/test/java/org/apache/flume/sink/TestRollingFileSink.java 7e26e2a
flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java c5c3f2f
flume-ng-core/src/test/java/org/apache/flume/source/TestExecSource.java 615f2a3
flume-ng-core/src/test/java/org/apache/flume/source/TestPollableSourceRunner.java c27f82c
flume-ng-core/src/test/java/org/apache/flume/source/TestSequenceGeneratorSource.java 579b257
flume-ng-core/src/test/java/org/apache/flume/source/TestSyslogUdpSource.java d1db49d
flume-ng-legacy-sources/flume-avro-source/src/main/java/org/apache/flume/source/avroLegacy/AvroLegacySource.java dde8f28
flume-ng-legacy-sources/flume-avro-source/src/test/java/org/apache/flume/source/avroLegacy/TestLegacyAvroSource.java 6e3eb53
flume-ng-legacy-sources/flume-thrift-source/src/main/java/org/apache/flume/source/thriftLegacy/ThriftLegacySource.java 00df7e4
flume-ng-legacy-sources/flume-thrift-source/src/test/java/org/apache/flume/source/thriftLegacy/TestThriftLegacySource.java ddd9478
flume-ng-node/src/main/java/org/apache/flume/conf/properties/FlumeConfiguration.java d66f6d1
flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java 50b9f0c
flume-ng-node/src/test/java/org/apache/flume/node/TestAbstractLogicalNodeManager.java 521b586
flume-ng-node/src/test/java/org/apache/flume/source/TestNetcatSource.java fb2a960
flume-ng-node/src/test/resources/flume-conf.properties 848caca
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java eee9221
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 6ff3737
flume-ng-sinks/flume-irc-sink/src/main/java/org/apache/flume/sink/irc/IRCSink.java 8e77218
pom.xml c91222f
Diff: https://reviews.apache.org/r/4115/diff
Testing
-------
All existing unit tests for the components whose configuration has now been moved to these stubs pass.
Thanks,
Hari
> Create configuration stubs for sources, channels, sinks etc
> -----------------------------------------------------------
>
> Key: FLUME-992
> URL: https://issues.apache.org/jira/browse/FLUME-992
> Project: Flume
> Issue Type: Sub-task
> Affects Versions: v1.0.0
> Reporter: Hari Shreedharan
> Assignee: Hari Shreedharan
> Fix For: v1.2.0
>
>
> Currently the configuration of each component sits deep inside the component themselves. There is no way to verify if a configuration is valid before run time. In most systems like Flume, it is likely that these confs will be deployed from one single host, on to the machines where flume agents are running. Only when each agent starts, invalid confs are identified because the Agents would terminate by throwing exceptions. This is a first attempt to make a component-aware configuration system which is independent of the Flume, and does not require the Flume jar to be installed. Each component needs a configuration manager, which configures the components.
> Provide abstract Configuration stubs for each component type, sources, channels, sinks, selectors etc, which are in the new package, independent on ng-core. Now for each of the channels extend the abstract class and check the config properties for each of the required parameters for that component, for example: MultiplexingChannelSelectorConfigurator would look for default channel etc. If a particular component does not have a configuration class then let the current mechanism continue.
> This will require implementation for each component, but it should not be too complex. One additional advantage we get from this is that we can separate out the config validation from the components into these stubs, but we will still need to read the values out of the Context as we do currently(else we end up making the configuration dependent on flume-ng-core itself which we wanted to avoid).
> A problem with this is making a change to the configuration would require changes in the configuration classes and in the components also(where the configuration is read and the component is actually configured). I could not figure out a way of avoiding this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-992) Create configuration stubs for
sources, channels, sinks etc
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13226302#comment-13226302 ]
jiraposter@reviews.apache.org commented on FLUME-992:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4115/#review5783
-----------------------------------------------------------
Thanks for the interim patch Hari. I did a quick walk through of these changes and have some feedback that follows. Some general issues:
* The component configuration stubs do not report any configuration errors at this stage. While we cannot expect these stubs to do runtime validation, we should at least check all the static rules that may be being violated. For example a non-numeric port value is something that does not get caught/reported by the current implementation.
* As implemented, the configuration validation mechanism only reports errors. It should also report warnings where default values override the specified values etc.
More specific feedback below:
flume-ng-configuration/pom.xml
<https://reviews.apache.org/r/4115/#comment12598>
Better to call it Flume NG Configuration only.
flume-ng-configuration/pom.xml
<https://reviews.apache.org/r/4115/#comment12600>
Duplicate
flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfiguration.java
<https://reviews.apache.org/r/4115/#comment12623>
should be private
flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfiguration.java
<https://reviews.apache.org/r/4115/#comment12624>
Does not look like its being used, in which case it should be removed considering the risk it introduces.
flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfiguration.java
<https://reviews.apache.org/r/4115/#comment12625>
This method is not returning any configuration values. Assuming it is work in progress - you should have a way by which the specialized component classes can use this without having to copy/rewrite this method.
flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfigurationFactory.java
<https://reviews.apache.org/r/4115/#comment12614>
Please make a ComponentTypeEnum that has SOURCE, CHANNEL, SINKPROCESSOR, CHANNELSELECTOR etc in it.
flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfigurationFactory.java
<https://reviews.apache.org/r/4115/#comment12615>
Shouldn't this be Class<SinkConfiguration> ?
Ideally, you want to have an enum that can return the actual class instance directly rather than doing a Class.forName() invocation.
flume-ng-configuration/src/main/java/org/apache/flume/conf/ConfigurationException.java
<https://reviews.apache.org/r/4115/#comment12597>
ConfigurationException should extend FlumeException.
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfiguration.java
<https://reviews.apache.org/r/4115/#comment12602>
Actually creates a fully populated configuration.
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfiguration.java
<https://reviews.apache.org/r/4115/#comment12607>
Need a few test cases to make sure this is working as expected.
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfiguration.java
<https://reviews.apache.org/r/4115/#comment12603>
Misleading error. When value is null, the error should not be agent name missing.
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfiguration.java
<https://reviews.apache.org/r/4115/#comment12621>
This exception is currently raised only if the instantiation fails. Ideally though, you want the conf object to be created and invoke a .validate() method on it, and then be able to extract any errors for reporting upwards.
Same comment applies to validateSources etc.
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfiguration.java
<https://reviews.apache.org/r/4115/#comment12612>
Please use generics to ensure type compliance instead of casts.
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationError.java
<https://reviews.apache.org/r/4115/#comment12604>
This should have provision for value that is in question (since some errors are value specific), and also the raw property that triggered this. Otherwise the error could be ambiguous
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationErrorType.java
<https://reviews.apache.org/r/4115/#comment12605>
These enums should have a descriptive message that tell the user what this problem is. Something like:
PROPERTY_NAME_NULL("A null or empty value was specified for the given property")
etc.
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkConfiguration.java
<https://reviews.apache.org/r/4115/#comment12617>
I don't think this is enum should have sink group and sink processor information. They belong to separate category.
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkConfiguration.java
<https://reviews.apache.org/r/4115/#comment12616>
Please define an interface that this enum implements to provide the class name instead of overloading the toString() method. Same for other enums too.
flume-ng-core/pom.xml
<https://reviews.apache.org/r/4115/#comment12599>
Indentation.
- Arvind
On 2012-03-09 08:47:00, Hari Shreedharan wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/4115/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-03-09 08:47:00)
bq.
bq.
bq. Review request for Flume.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Currently the configuration of each component sits deep inside the component themselves. There is no way to verify if a configuration is valid before run time. In most systems like Flume, it is likely that these confs will be deployed from one single host, on to the machines where flume agents are running. Only when each agent starts, invalid confs are identified because the Agents would terminate by throwing exceptions. This is a first attempt to make a component-aware configuration system which is independent of the Flume, and does not require the Flume jar to be installed. Each component needs a configuration manager, which configures the components.
bq.
bq. I have not completed the validation part yet, but have completed the configuration managers for all the components that come in the flume jar. I will add support for the other components, such as HDFS sink, JDBC channel etc soon. I will also add the validation support to the classes, but would like a review of the model being proposed here.
bq.
bq. Note that I do know of a couple of bugs in the existing diff(like that if the name of a config key starts with an Upper case letter, an exception is thrown) - I will fix and upload the review here soon.
bq.
bq.
bq. This addresses bug FLUME-992.
bq. https://issues.apache.org/jira/browse/FLUME-992
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceConfiguration.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/source/NetcatSourceConfiguration.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/source/ExecSourceConfiguration.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/source/AvroSourceConfiguration.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkProcessorType.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkType.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkProcessorConfiguration.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkGroupConfiguration.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/RollingFileSinkConfiguration.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkConfiguration.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationError.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationErrorType.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelConfiguration.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorConfiguration.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorType.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelType.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MemoryChannelConfiguration.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MultiplexingChannelSelectorConfiguration.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/PseudoTxnMemoryChannelConfiguration.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/AvroSinkConfiguration.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/FailoverSinkProcessorConfiguration.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfiguration.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/Context.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/ConfigurationException.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfigurationFactory.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfiguration.java PRE-CREATION
bq. flume-ng-clients/flume-ng-log4jappender/src/test/java/org/apache/flume/clients/log4jappender/TestLog4jAppender.java 68d95fb
bq. flume-ng-configuration/pom.xml PRE-CREATION
bq. flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestJdbcChannelProvider.java 7710d46
bq. flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453
bq. flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannel.java bca0c50
bq. flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProvider.java e445d61
bq. flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProviderFactory.java 6fbd6ef
bq. flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java 307ae89
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceType.java PRE-CREATION
bq. flume-ng-configuration/src/test/resources/flume-ng-error-test PRE-CREATION
bq. flume-ng-configuration/src/test/resources/flume-ng-test.conf PRE-CREATION
bq. flume-ng-core/pom.xml d753fa1
bq. flume-ng-core/src/main/java/org/apache/flume/ChannelSelector.java fba2dcb
bq. flume-ng-core/src/main/java/org/apache/flume/Context.java a6341a5
bq. flume-ng-core/src/main/java/org/apache/flume/SinkProcessor.java 11651ad
bq. flume-ng-core/src/main/java/org/apache/flume/SinkProcessorType.java be1891b
bq. flume-ng-core/src/main/java/org/apache/flume/channel/AbstractChannel.java 352bf08
bq. flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorFactory.java 800f471
bq. flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorType.java 511fc65
bq. flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8
bq. flume-ng-core/src/main/java/org/apache/flume/channel/DefaultChannelFactory.java 963a6a3
bq. flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java bfa1fde
bq. flume-ng-core/src/main/java/org/apache/flume/channel/MultiplexingChannelSelector.java 83928b7
bq. flume-ng-core/src/main/java/org/apache/flume/channel/PseudoTxnMemoryChannel.java 489d3e5
bq. flume-ng-core/src/main/java/org/apache/flume/channel/ReplicatingChannelSelector.java 8f22746
bq. flume-ng-core/src/main/java/org/apache/flume/conf/Configurable.java 0fa4839
bq. flume-ng-core/src/main/java/org/apache/flume/conf/ConfigurableComponent.java PRE-CREATION
bq. flume-ng-core/src/main/java/org/apache/flume/conf/Configurables.java 84492e5
bq. flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 7386d06
bq. flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkFactory.java b89dfa0
bq. flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkProcessor.java 257bab3
bq. flume-ng-core/src/main/java/org/apache/flume/sink/FailoverSinkProcessor.java 9f5b856
bq. flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java 7f1d3b3
bq. flume-ng-core/src/main/java/org/apache/flume/sink/SinkGroup.java 2e80a56
bq. flume-ng-core/src/main/java/org/apache/flume/sink/SinkProcessorFactory.java 10f9f4e
bq. flume-ng-core/src/main/java/org/apache/flume/sink/SinkType.java 6b08c09
bq. flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java a3f6640
bq. flume-ng-core/src/main/java/org/apache/flume/source/DefaultSourceFactory.java a610e6f
bq. flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java b6b1181
bq. flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java 94245ac
bq. flume-ng-core/src/main/java/org/apache/flume/source/SourceType.java 9082470
bq. flume-ng-core/src/test/java/org/apache/flume/TestContext.java 51c350f
bq. flume-ng-core/src/test/java/org/apache/flume/channel/AbstractBasicChannelSemanticsTest.java 6e71e46
bq. flume-ng-core/src/test/java/org/apache/flume/channel/MockChannel.java 24b01e2
bq. flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannel.java e070864
bq. flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelConcurrency.java 8dad0b2
bq. flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelTransaction.java bc81f26
bq. flume-ng-core/src/test/java/org/apache/flume/sink/TestAvroSink.java 467785f
bq. flume-ng-core/src/test/java/org/apache/flume/sink/TestFailoverSinkProcessor.java 195c121
bq. flume-ng-core/src/test/java/org/apache/flume/sink/TestLoggerSink.java 92ff6fe
bq. flume-ng-core/src/test/java/org/apache/flume/sink/TestRollingFileSink.java 7e26e2a
bq. flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java c5c3f2f
bq. flume-ng-core/src/test/java/org/apache/flume/source/TestExecSource.java 6035270
bq. flume-ng-core/src/test/java/org/apache/flume/source/TestPollableSourceRunner.java c27f82c
bq. flume-ng-core/src/test/java/org/apache/flume/source/TestSequenceGeneratorSource.java 579b257
bq. flume-ng-legacy-sources/flume-avro-source/src/main/java/org/apache/flume/source/avroLegacy/AvroLegacySource.java 65d9142
bq. flume-ng-legacy-sources/flume-avro-source/src/test/java/org/apache/flume/source/avroLegacy/TestLegacyAvroSource.java 6e3eb53
bq. flume-ng-legacy-sources/flume-thrift-source/src/main/java/org/apache/flume/source/thriftLegacy/ThriftLegacySource.java fbf7362
bq. flume-ng-legacy-sources/flume-thrift-source/src/test/java/org/apache/flume/source/thriftLegacy/TestThriftLegacySource.java ddd9478
bq. flume-ng-node/src/main/java/org/apache/flume/conf/properties/FlumeConfiguration.java d66f6d1
bq. flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java 97f72e1
bq. flume-ng-node/src/test/java/org/apache/flume/node/TestAbstractLogicalNodeManager.java 521b586
bq. flume-ng-node/src/test/java/org/apache/flume/source/TestNetcatSource.java fb2a960
bq. flume-ng-node/src/test/resources/flume-conf.properties 848caca
bq. flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 524b69c
bq. flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 7d8ee8a
bq. flume-ng-sinks/flume-irc-sink/src/main/java/org/apache/flume/sink/irc/IRCSink.java 8e77218
bq. pom.xml d785762
bq.
bq. Diff: https://reviews.apache.org/r/4115/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. All existing unit tests for the components whose configuration has now been moved to these stubs pass.
bq.
bq.
bq. Thanks,
bq.
bq. Hari
bq.
bq.
> Create configuration stubs for sources, channels, sinks etc
> -----------------------------------------------------------
>
> Key: FLUME-992
> URL: https://issues.apache.org/jira/browse/FLUME-992
> Project: Flume
> Issue Type: Sub-task
> Affects Versions: v1.0.0
> Reporter: Hari Shreedharan
> Assignee: Hari Shreedharan
> Fix For: v1.1.0
>
>
> Currently the configuration of each component sits deep inside the component themselves. There is no way to verify if a configuration is valid before run time. In most systems like Flume, it is likely that these confs will be deployed from one single host, on to the machines where flume agents are running. Only when each agent starts, invalid confs are identified because the Agents would terminate by throwing exceptions. This is a first attempt to make a component-aware configuration system which is independent of the Flume, and does not require the Flume jar to be installed. Each component needs a configuration manager, which configures the components.
> Provide abstract Configuration stubs for each component type, sources, channels, sinks, selectors etc, which are in the new package, independent on ng-core. Now for each of the channels extend the abstract class and check the config properties for each of the required parameters for that component, for example: MultiplexingChannelSelectorConfigurator would look for default channel etc. If a particular component does not have a configuration class then let the current mechanism continue.
> This will require implementation for each component, but it should not be too complex. One additional advantage we get from this is that we can separate out the config validation from the components into these stubs, but we will still need to read the values out of the Context as we do currently(else we end up making the configuration dependent on flume-ng-core itself which we wanted to avoid).
> A problem with this is making a change to the configuration would require changes in the configuration classes and in the components also(where the configuration is read and the component is actually configured). I could not figure out a way of avoiding this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-992) Create configuration stubs for
sources, channels, sinks etc
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13228688#comment-13228688 ]
jiraposter@reviews.apache.org commented on FLUME-992:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4115/
-----------------------------------------------------------
(Updated 2012-03-13 20:49:17.095984)
Review request for Flume.
Changes
-------
Minor fixes, more logging.
Summary
-------
Currently the configuration of each component sits deep inside the component themselves. There is no way to verify if a configuration is valid before run time. In most systems like Flume, it is likely that these confs will be deployed from one single host, on to the machines where flume agents are running. Only when each agent starts, invalid confs are identified because the Agents would terminate by throwing exceptions. This is a first attempt to make a component-aware configuration system which is independent of the Flume, and does not require the Flume jar to be installed. Each component needs a configuration manager, which configures the components.
I have not completed the validation part yet, but have completed the configuration managers for all the components that come in the flume jar. I will add support for the other components, such as HDFS sink, JDBC channel etc soon. I will also add the validation support to the classes, but would like a review of the model being proposed here.
Note that I do know of a couple of bugs in the existing diff(like that if the name of a config key starts with an Upper case letter, an exception is thrown) - I will fix and upload the review here soon.
This addresses bug FLUME-992.
https://issues.apache.org/jira/browse/FLUME-992
Diffs (updated)
-----
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannel.java bca0c50
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProvider.java e445d61
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProviderFactory.java 6fbd6ef
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java 307ae89
flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestJdbcChannelProvider.java 7710d46
flume-ng-clients/flume-ng-log4jappender/src/test/java/org/apache/flume/clients/log4jappender/TestLog4jAppender.java 68d95fb
flume-ng-configuration/pom.xml PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfigurationFactory.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ConfigurationException.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/Context.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationError.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationErrorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MemoryChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MultiplexingChannelSelectorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/PseudoTxnMemoryChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/AvroSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/FailoverSinkProcessorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/RollingFileSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkGroupConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkProcessorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkProcessorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/AvroSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/ExecSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/NetcatSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceType.java PRE-CREATION
flume-ng-configuration/src/test/resources/flume-ng-error-test PRE-CREATION
flume-ng-configuration/src/test/resources/flume-ng-test.conf PRE-CREATION
flume-ng-core/pom.xml d753fa1
flume-ng-core/src/main/java/org/apache/flume/ChannelSelector.java fba2dcb
flume-ng-core/src/main/java/org/apache/flume/Context.java a6341a5
flume-ng-core/src/main/java/org/apache/flume/SinkProcessor.java 11651ad
flume-ng-core/src/main/java/org/apache/flume/SinkProcessorType.java be1891b
flume-ng-core/src/main/java/org/apache/flume/channel/AbstractChannel.java 352bf08
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorFactory.java 800f471
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorType.java 511fc65
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8
flume-ng-core/src/main/java/org/apache/flume/channel/DefaultChannelFactory.java 963a6a3
flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java bfa1fde
flume-ng-core/src/main/java/org/apache/flume/channel/MultiplexingChannelSelector.java 83928b7
flume-ng-core/src/main/java/org/apache/flume/channel/PseudoTxnMemoryChannel.java 489d3e5
flume-ng-core/src/main/java/org/apache/flume/channel/ReplicatingChannelSelector.java 8f22746
flume-ng-core/src/main/java/org/apache/flume/conf/Configurable.java 0fa4839
flume-ng-core/src/main/java/org/apache/flume/conf/ConfigurableComponent.java PRE-CREATION
flume-ng-core/src/main/java/org/apache/flume/conf/Configurables.java 84492e5
flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 7386d06
flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkFactory.java b89dfa0
flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkProcessor.java 257bab3
flume-ng-core/src/main/java/org/apache/flume/sink/FailoverSinkProcessor.java 9f5b856
flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java 7f1d3b3
flume-ng-core/src/main/java/org/apache/flume/sink/SinkGroup.java 2e80a56
flume-ng-core/src/main/java/org/apache/flume/sink/SinkProcessorFactory.java 10f9f4e
flume-ng-core/src/main/java/org/apache/flume/sink/SinkType.java 6b08c09
flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java a3f6640
flume-ng-core/src/main/java/org/apache/flume/source/DefaultSourceFactory.java a610e6f
flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java b6b1181
flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java 94245ac
flume-ng-core/src/main/java/org/apache/flume/source/SourceType.java 9082470
flume-ng-core/src/test/java/org/apache/flume/TestContext.java 51c350f
flume-ng-core/src/test/java/org/apache/flume/channel/AbstractBasicChannelSemanticsTest.java 6e71e46
flume-ng-core/src/test/java/org/apache/flume/channel/MockChannel.java 24b01e2
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannel.java e070864
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelConcurrency.java 8dad0b2
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelTransaction.java bc81f26
flume-ng-core/src/test/java/org/apache/flume/sink/TestAvroSink.java 467785f
flume-ng-core/src/test/java/org/apache/flume/sink/TestFailoverSinkProcessor.java 195c121
flume-ng-core/src/test/java/org/apache/flume/sink/TestLoggerSink.java 92ff6fe
flume-ng-core/src/test/java/org/apache/flume/sink/TestRollingFileSink.java 7e26e2a
flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java c5c3f2f
flume-ng-core/src/test/java/org/apache/flume/source/TestExecSource.java 6035270
flume-ng-core/src/test/java/org/apache/flume/source/TestPollableSourceRunner.java c27f82c
flume-ng-core/src/test/java/org/apache/flume/source/TestSequenceGeneratorSource.java 579b257
flume-ng-legacy-sources/flume-avro-source/src/main/java/org/apache/flume/source/avroLegacy/AvroLegacySource.java 65d9142
flume-ng-legacy-sources/flume-avro-source/src/test/java/org/apache/flume/source/avroLegacy/TestLegacyAvroSource.java 6e3eb53
flume-ng-legacy-sources/flume-thrift-source/src/main/java/org/apache/flume/source/thriftLegacy/ThriftLegacySource.java fbf7362
flume-ng-legacy-sources/flume-thrift-source/src/test/java/org/apache/flume/source/thriftLegacy/TestThriftLegacySource.java ddd9478
flume-ng-node/src/main/java/org/apache/flume/conf/properties/FlumeConfiguration.java d66f6d1
flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java 97f72e1
flume-ng-node/src/test/java/org/apache/flume/node/TestAbstractLogicalNodeManager.java 521b586
flume-ng-node/src/test/java/org/apache/flume/source/TestNetcatSource.java fb2a960
flume-ng-node/src/test/resources/flume-conf.properties 848caca
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 524b69c
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 7d8ee8a
flume-ng-sinks/flume-irc-sink/src/main/java/org/apache/flume/sink/irc/IRCSink.java 8e77218
pom.xml d785762
Diff: https://reviews.apache.org/r/4115/diff
Testing
-------
All existing unit tests for the components whose configuration has now been moved to these stubs pass.
Thanks,
Hari
> Create configuration stubs for sources, channels, sinks etc
> -----------------------------------------------------------
>
> Key: FLUME-992
> URL: https://issues.apache.org/jira/browse/FLUME-992
> Project: Flume
> Issue Type: Sub-task
> Affects Versions: v1.0.0
> Reporter: Hari Shreedharan
> Assignee: Hari Shreedharan
> Fix For: v1.2.0
>
>
> Currently the configuration of each component sits deep inside the component themselves. There is no way to verify if a configuration is valid before run time. In most systems like Flume, it is likely that these confs will be deployed from one single host, on to the machines where flume agents are running. Only when each agent starts, invalid confs are identified because the Agents would terminate by throwing exceptions. This is a first attempt to make a component-aware configuration system which is independent of the Flume, and does not require the Flume jar to be installed. Each component needs a configuration manager, which configures the components.
> Provide abstract Configuration stubs for each component type, sources, channels, sinks, selectors etc, which are in the new package, independent on ng-core. Now for each of the channels extend the abstract class and check the config properties for each of the required parameters for that component, for example: MultiplexingChannelSelectorConfigurator would look for default channel etc. If a particular component does not have a configuration class then let the current mechanism continue.
> This will require implementation for each component, but it should not be too complex. One additional advantage we get from this is that we can separate out the config validation from the components into these stubs, but we will still need to read the values out of the Context as we do currently(else we end up making the configuration dependent on flume-ng-core itself which we wanted to avoid).
> A problem with this is making a change to the configuration would require changes in the configuration classes and in the components also(where the configuration is read and the component is actually configured). I could not figure out a way of avoiding this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-992) Create configuration stubs for
sources, channels, sinks etc
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13229633#comment-13229633 ]
jiraposter@reviews.apache.org commented on FLUME-992:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4115/
-----------------------------------------------------------
(Updated 2012-03-14 21:18:41.095856)
Review request for Flume.
Changes
-------
Added stubs for Jdbc channel and Hdfs sink which do not have any implementation. I am leaving it for the original implementer to update it.
Summary
-------
Currently the configuration of each component sits deep inside the component themselves. There is no way to verify if a configuration is valid before run time. In most systems like Flume, it is likely that these confs will be deployed from one single host, on to the machines where flume agents are running. Only when each agent starts, invalid confs are identified because the Agents would terminate by throwing exceptions. This is a first attempt to make a component-aware configuration system which is independent of the Flume, and does not require the Flume jar to be installed. Each component needs a configuration manager, which configures the components.
I have not completed the validation part yet, but have completed the configuration managers for all the components that come in the flume jar. I will add support for the other components, such as HDFS sink, JDBC channel etc soon. I will also add the validation support to the classes, but would like a review of the model being proposed here.
Note that I do know of a couple of bugs in the existing diff(like that if the name of a config key starts with an Upper case letter, an exception is thrown) - I will fix and upload the review here soon.
This addresses bug FLUME-992.
https://issues.apache.org/jira/browse/FLUME-992
Diffs (updated)
-----
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannel.java bca0c50
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProvider.java e445d61
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProviderFactory.java 6fbd6ef
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java 307ae89
flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestJdbcChannelProvider.java 7710d46
flume-ng-clients/flume-ng-log4jappender/src/test/java/org/apache/flume/clients/log4jappender/TestLog4jAppender.java 68d95fb
flume-ng-configuration/pom.xml PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfigurationFactory.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ConfigurationException.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/Context.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationError.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationErrorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MemoryChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MultiplexingChannelSelectorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/PseudoTxnMemoryChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/AvroSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/FailoverSinkProcessorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/RollingFileSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkGroupConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkProcessorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkProcessorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/AvroSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/ExecSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/NetcatSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceType.java PRE-CREATION
flume-ng-configuration/src/test/resources/flume-ng-error-test PRE-CREATION
flume-ng-configuration/src/test/resources/flume-ng-test.conf PRE-CREATION
flume-ng-core/pom.xml 247df37
flume-ng-core/src/main/java/org/apache/flume/ChannelSelector.java fba2dcb
flume-ng-core/src/main/java/org/apache/flume/Context.java 5294e31
flume-ng-core/src/main/java/org/apache/flume/SinkProcessor.java ea6000b
flume-ng-core/src/main/java/org/apache/flume/SinkProcessorType.java be1891b
flume-ng-core/src/main/java/org/apache/flume/channel/AbstractChannel.java 352bf08
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorFactory.java 800f471
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorType.java 511fc65
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8
flume-ng-core/src/main/java/org/apache/flume/channel/DefaultChannelFactory.java 963a6a3
flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java bfa1fde
flume-ng-core/src/main/java/org/apache/flume/channel/MultiplexingChannelSelector.java 83928b7
flume-ng-core/src/main/java/org/apache/flume/channel/PseudoTxnMemoryChannel.java 489d3e5
flume-ng-core/src/main/java/org/apache/flume/channel/ReplicatingChannelSelector.java 8f22746
flume-ng-core/src/main/java/org/apache/flume/conf/Configurable.java 0fa4839
flume-ng-core/src/main/java/org/apache/flume/conf/ConfigurableComponent.java PRE-CREATION
flume-ng-core/src/main/java/org/apache/flume/conf/Configurables.java 84492e5
flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 3edc563
flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkFactory.java b89dfa0
flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkProcessor.java 257bab3
flume-ng-core/src/main/java/org/apache/flume/sink/FailoverSinkProcessor.java 9f5b856
flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java 7f1d3b3
flume-ng-core/src/main/java/org/apache/flume/sink/SinkGroup.java 0dffd69
flume-ng-core/src/main/java/org/apache/flume/sink/SinkProcessorFactory.java 6160a17
flume-ng-core/src/main/java/org/apache/flume/sink/SinkType.java 6b08c09
flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java 903889d
flume-ng-core/src/main/java/org/apache/flume/source/DefaultSourceFactory.java a610e6f
flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java b6b1181
flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java 94245ac
flume-ng-core/src/main/java/org/apache/flume/source/SourceType.java 9082470
flume-ng-core/src/main/java/org/apache/flume/source/SyslogTcpSource.java d78d27f
flume-ng-core/src/main/java/org/apache/flume/source/SyslogUDPSource.java 34818f0
flume-ng-core/src/test/java/org/apache/flume/TestContext.java 51c350f
flume-ng-core/src/test/java/org/apache/flume/channel/AbstractBasicChannelSemanticsTest.java 6e71e46
flume-ng-core/src/test/java/org/apache/flume/channel/MockChannel.java 24b01e2
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannel.java e070864
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelConcurrency.java 8dad0b2
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelTransaction.java bc81f26
flume-ng-core/src/test/java/org/apache/flume/sink/TestAvroSink.java 3765924
flume-ng-core/src/test/java/org/apache/flume/sink/TestFailoverSinkProcessor.java 195c121
flume-ng-core/src/test/java/org/apache/flume/sink/TestLoggerSink.java 92ff6fe
flume-ng-core/src/test/java/org/apache/flume/sink/TestRollingFileSink.java 7e26e2a
flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java c5c3f2f
flume-ng-core/src/test/java/org/apache/flume/source/TestExecSource.java 6035270
flume-ng-core/src/test/java/org/apache/flume/source/TestPollableSourceRunner.java c27f82c
flume-ng-core/src/test/java/org/apache/flume/source/TestSequenceGeneratorSource.java 579b257
flume-ng-core/src/test/java/org/apache/flume/source/TestSyslogUdpSource.java d1db49d
flume-ng-legacy-sources/flume-avro-source/src/main/java/org/apache/flume/source/avroLegacy/AvroLegacySource.java dde8f28
flume-ng-legacy-sources/flume-avro-source/src/test/java/org/apache/flume/source/avroLegacy/TestLegacyAvroSource.java 6e3eb53
flume-ng-legacy-sources/flume-thrift-source/src/main/java/org/apache/flume/source/thriftLegacy/ThriftLegacySource.java fbf7362
flume-ng-legacy-sources/flume-thrift-source/src/test/java/org/apache/flume/source/thriftLegacy/TestThriftLegacySource.java ddd9478
flume-ng-node/src/main/java/org/apache/flume/conf/properties/FlumeConfiguration.java d66f6d1
flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java 97f72e1
flume-ng-node/src/test/java/org/apache/flume/node/TestAbstractLogicalNodeManager.java 521b586
flume-ng-node/src/test/java/org/apache/flume/source/TestNetcatSource.java fb2a960
flume-ng-node/src/test/resources/flume-conf.properties 848caca
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java da82f7e
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 8fa72a1
flume-ng-sinks/flume-irc-sink/src/main/java/org/apache/flume/sink/irc/IRCSink.java 8e77218
pom.xml 794ebe3
Diff: https://reviews.apache.org/r/4115/diff
Testing
-------
All existing unit tests for the components whose configuration has now been moved to these stubs pass.
Thanks,
Hari
> Create configuration stubs for sources, channels, sinks etc
> -----------------------------------------------------------
>
> Key: FLUME-992
> URL: https://issues.apache.org/jira/browse/FLUME-992
> Project: Flume
> Issue Type: Sub-task
> Affects Versions: v1.0.0
> Reporter: Hari Shreedharan
> Assignee: Hari Shreedharan
> Fix For: v1.2.0
>
>
> Currently the configuration of each component sits deep inside the component themselves. There is no way to verify if a configuration is valid before run time. In most systems like Flume, it is likely that these confs will be deployed from one single host, on to the machines where flume agents are running. Only when each agent starts, invalid confs are identified because the Agents would terminate by throwing exceptions. This is a first attempt to make a component-aware configuration system which is independent of the Flume, and does not require the Flume jar to be installed. Each component needs a configuration manager, which configures the components.
> Provide abstract Configuration stubs for each component type, sources, channels, sinks, selectors etc, which are in the new package, independent on ng-core. Now for each of the channels extend the abstract class and check the config properties for each of the required parameters for that component, for example: MultiplexingChannelSelectorConfigurator would look for default channel etc. If a particular component does not have a configuration class then let the current mechanism continue.
> This will require implementation for each component, but it should not be too complex. One additional advantage we get from this is that we can separate out the config validation from the components into these stubs, but we will still need to read the values out of the Context as we do currently(else we end up making the configuration dependent on flume-ng-core itself which we wanted to avoid).
> A problem with this is making a change to the configuration would require changes in the configuration classes and in the components also(where the configuration is read and the component is actually configured). I could not figure out a way of avoiding this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-992) Create configuration stubs for
sources, channels, sinks etc
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13229550#comment-13229550 ]
jiraposter@reviews.apache.org commented on FLUME-992:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4115/#review5961
-----------------------------------------------------------
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorConfiguration.java
<https://reviews.apache.org/r/4115/#comment12930>
In general I think errors like this should be thrown up.
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorConfiguration.java
<https://reviews.apache.org/r/4115/#comment12929>
We should be doing something with this exception. Else the only way to debug is a debugger.
- Brock
On 2012-03-14 19:21:20, Hari Shreedharan wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/4115/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-03-14 19:21:20)
bq.
bq.
bq. Review request for Flume.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Currently the configuration of each component sits deep inside the component themselves. There is no way to verify if a configuration is valid before run time. In most systems like Flume, it is likely that these confs will be deployed from one single host, on to the machines where flume agents are running. Only when each agent starts, invalid confs are identified because the Agents would terminate by throwing exceptions. This is a first attempt to make a component-aware configuration system which is independent of the Flume, and does not require the Flume jar to be installed. Each component needs a configuration manager, which configures the components.
bq.
bq. I have not completed the validation part yet, but have completed the configuration managers for all the components that come in the flume jar. I will add support for the other components, such as HDFS sink, JDBC channel etc soon. I will also add the validation support to the classes, but would like a review of the model being proposed here.
bq.
bq. Note that I do know of a couple of bugs in the existing diff(like that if the name of a config key starts with an Upper case letter, an exception is thrown) - I will fix and upload the review here soon.
bq.
bq.
bq. This addresses bug FLUME-992.
bq. https://issues.apache.org/jira/browse/FLUME-992
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationErrorType.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelConfiguration.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorConfiguration.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationError.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfiguration.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/Context.java PRE-CREATION
bq. flume-ng-clients/flume-ng-log4jappender/src/test/java/org/apache/flume/clients/log4jappender/TestLog4jAppender.java 68d95fb
bq. flume-ng-configuration/pom.xml PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfiguration.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfigurationFactory.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/ConfigurationException.java PRE-CREATION
bq. flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestJdbcChannelProvider.java 7710d46
bq. flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453
bq. flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannel.java bca0c50
bq. flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProvider.java e445d61
bq. flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProviderFactory.java 6fbd6ef
bq. flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java 307ae89
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorType.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelType.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MemoryChannelConfiguration.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MultiplexingChannelSelectorConfiguration.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/PseudoTxnMemoryChannelConfiguration.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/AvroSinkConfiguration.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/FailoverSinkProcessorConfiguration.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/RollingFileSinkConfiguration.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkConfiguration.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkGroupConfiguration.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkProcessorConfiguration.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkProcessorType.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkType.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/source/AvroSourceConfiguration.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/source/ExecSourceConfiguration.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/source/NetcatSourceConfiguration.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceConfiguration.java PRE-CREATION
bq. flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceType.java PRE-CREATION
bq. flume-ng-configuration/src/test/resources/flume-ng-error-test PRE-CREATION
bq. flume-ng-configuration/src/test/resources/flume-ng-test.conf PRE-CREATION
bq. flume-ng-core/pom.xml 247df37
bq. flume-ng-core/src/main/java/org/apache/flume/ChannelSelector.java fba2dcb
bq. flume-ng-core/src/main/java/org/apache/flume/Context.java 5294e31
bq. flume-ng-core/src/main/java/org/apache/flume/SinkProcessor.java ea6000b
bq. flume-ng-core/src/main/java/org/apache/flume/SinkProcessorType.java be1891b
bq. flume-ng-core/src/main/java/org/apache/flume/channel/AbstractChannel.java 352bf08
bq. flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorFactory.java 800f471
bq. flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorType.java 511fc65
bq. flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8
bq. flume-ng-core/src/main/java/org/apache/flume/channel/DefaultChannelFactory.java 963a6a3
bq. flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java bfa1fde
bq. flume-ng-core/src/main/java/org/apache/flume/channel/MultiplexingChannelSelector.java 83928b7
bq. flume-ng-core/src/main/java/org/apache/flume/channel/PseudoTxnMemoryChannel.java 489d3e5
bq. flume-ng-core/src/main/java/org/apache/flume/channel/ReplicatingChannelSelector.java 8f22746
bq. flume-ng-core/src/main/java/org/apache/flume/conf/Configurable.java 0fa4839
bq. flume-ng-core/src/main/java/org/apache/flume/conf/ConfigurableComponent.java PRE-CREATION
bq. flume-ng-core/src/main/java/org/apache/flume/conf/Configurables.java 84492e5
bq. flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 3edc563
bq. flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkFactory.java b89dfa0
bq. flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkProcessor.java 257bab3
bq. flume-ng-core/src/main/java/org/apache/flume/sink/FailoverSinkProcessor.java 9f5b856
bq. flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java 7f1d3b3
bq. flume-ng-core/src/main/java/org/apache/flume/sink/SinkGroup.java 0dffd69
bq. flume-ng-core/src/main/java/org/apache/flume/sink/SinkProcessorFactory.java 6160a17
bq. flume-ng-core/src/main/java/org/apache/flume/sink/SinkType.java 6b08c09
bq. flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java 903889d
bq. flume-ng-core/src/main/java/org/apache/flume/source/DefaultSourceFactory.java a610e6f
bq. flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java b6b1181
bq. flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java 94245ac
bq. flume-ng-core/src/main/java/org/apache/flume/source/SourceType.java 9082470
bq. flume-ng-core/src/main/java/org/apache/flume/source/SyslogTcpSource.java d78d27f
bq. flume-ng-core/src/main/java/org/apache/flume/source/SyslogUDPSource.java 34818f0
bq. flume-ng-core/src/test/java/org/apache/flume/TestContext.java 51c350f
bq. flume-ng-core/src/test/java/org/apache/flume/channel/AbstractBasicChannelSemanticsTest.java 6e71e46
bq. flume-ng-core/src/test/java/org/apache/flume/channel/MockChannel.java 24b01e2
bq. flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannel.java e070864
bq. flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelConcurrency.java 8dad0b2
bq. flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelTransaction.java bc81f26
bq. flume-ng-core/src/test/java/org/apache/flume/sink/TestAvroSink.java 3765924
bq. flume-ng-core/src/test/java/org/apache/flume/sink/TestFailoverSinkProcessor.java 195c121
bq. flume-ng-core/src/test/java/org/apache/flume/sink/TestLoggerSink.java 92ff6fe
bq. flume-ng-core/src/test/java/org/apache/flume/sink/TestRollingFileSink.java 7e26e2a
bq. flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java c5c3f2f
bq. flume-ng-core/src/test/java/org/apache/flume/source/TestExecSource.java 6035270
bq. flume-ng-core/src/test/java/org/apache/flume/source/TestPollableSourceRunner.java c27f82c
bq. flume-ng-core/src/test/java/org/apache/flume/source/TestSequenceGeneratorSource.java 579b257
bq. flume-ng-core/src/test/java/org/apache/flume/source/TestSyslogUdpSource.java d1db49d
bq. flume-ng-legacy-sources/flume-avro-source/src/main/java/org/apache/flume/source/avroLegacy/AvroLegacySource.java dde8f28
bq. flume-ng-legacy-sources/flume-avro-source/src/test/java/org/apache/flume/source/avroLegacy/TestLegacyAvroSource.java 6e3eb53
bq. flume-ng-legacy-sources/flume-thrift-source/src/main/java/org/apache/flume/source/thriftLegacy/ThriftLegacySource.java fbf7362
bq. flume-ng-legacy-sources/flume-thrift-source/src/test/java/org/apache/flume/source/thriftLegacy/TestThriftLegacySource.java ddd9478
bq. flume-ng-node/src/main/java/org/apache/flume/conf/properties/FlumeConfiguration.java d66f6d1
bq. flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java 97f72e1
bq. flume-ng-node/src/test/java/org/apache/flume/node/TestAbstractLogicalNodeManager.java 521b586
bq. flume-ng-node/src/test/java/org/apache/flume/source/TestNetcatSource.java fb2a960
bq. flume-ng-node/src/test/resources/flume-conf.properties 848caca
bq. flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java da82f7e
bq. flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 8fa72a1
bq. flume-ng-sinks/flume-irc-sink/src/main/java/org/apache/flume/sink/irc/IRCSink.java 8e77218
bq. pom.xml 794ebe3
bq.
bq. Diff: https://reviews.apache.org/r/4115/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. All existing unit tests for the components whose configuration has now been moved to these stubs pass.
bq.
bq.
bq. Thanks,
bq.
bq. Hari
bq.
bq.
> Create configuration stubs for sources, channels, sinks etc
> -----------------------------------------------------------
>
> Key: FLUME-992
> URL: https://issues.apache.org/jira/browse/FLUME-992
> Project: Flume
> Issue Type: Sub-task
> Affects Versions: v1.0.0
> Reporter: Hari Shreedharan
> Assignee: Hari Shreedharan
> Fix For: v1.2.0
>
>
> Currently the configuration of each component sits deep inside the component themselves. There is no way to verify if a configuration is valid before run time. In most systems like Flume, it is likely that these confs will be deployed from one single host, on to the machines where flume agents are running. Only when each agent starts, invalid confs are identified because the Agents would terminate by throwing exceptions. This is a first attempt to make a component-aware configuration system which is independent of the Flume, and does not require the Flume jar to be installed. Each component needs a configuration manager, which configures the components.
> Provide abstract Configuration stubs for each component type, sources, channels, sinks, selectors etc, which are in the new package, independent on ng-core. Now for each of the channels extend the abstract class and check the config properties for each of the required parameters for that component, for example: MultiplexingChannelSelectorConfigurator would look for default channel etc. If a particular component does not have a configuration class then let the current mechanism continue.
> This will require implementation for each component, but it should not be too complex. One additional advantage we get from this is that we can separate out the config validation from the components into these stubs, but we will still need to read the values out of the Context as we do currently(else we end up making the configuration dependent on flume-ng-core itself which we wanted to avoid).
> A problem with this is making a change to the configuration would require changes in the configuration classes and in the components also(where the configuration is read and the component is actually configured). I could not figure out a way of avoiding this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-992) Create configuration stubs for
sources, channels, sinks etc
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13239090#comment-13239090 ]
jiraposter@reviews.apache.org commented on FLUME-992:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4115/
-----------------------------------------------------------
(Updated 2012-03-27 01:05:49.485603)
Review request for Flume.
Changes
-------
<<---No longer for review - for keeping track of diffs only. I am splitting this up into smaller, easier to review patches. --->>
Summary
-------
Currently the configuration of each component sits deep inside the component themselves. There is no way to verify if a configuration is valid before run time. In most systems like Flume, it is likely that these confs will be deployed from one single host, on to the machines where flume agents are running. Only when each agent starts, invalid confs are identified because the Agents would terminate by throwing exceptions. This is a first attempt to make a component-aware configuration system which is independent of the Flume, and does not require the Flume jar to be installed. Each component needs a configuration manager, which configures the components.
I have not completed the validation part yet, but have completed the configuration managers for all the components that come in the flume jar. I will add support for the other components, such as HDFS sink, JDBC channel etc soon. I will also add the validation support to the classes, but would like a review of the model being proposed here.
Note that I do know of a couple of bugs in the existing diff(like that if the name of a config key starts with an Upper case letter, an exception is thrown) - I will fix and upload the review here soon.
This addresses bug FLUME-992.
https://issues.apache.org/jira/browse/FLUME-992
Diffs (updated)
-----
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java a279453
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannel.java bca0c50
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProvider.java e445d61
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/JdbcChannelProviderFactory.java 6fbd6ef
flume-ng-channels/flume-jdbc-channel/src/main/java/org/apache/flume/channel/jdbc/impl/JdbcChannelProviderImpl.java 307ae89
flume-ng-channels/flume-jdbc-channel/src/test/java/org/apache/flume/channel/jdbc/TestJdbcChannelProvider.java 7710d46
flume-ng-clients/flume-ng-log4jappender/src/test/java/org/apache/flume/clients/log4jappender/TestLog4jAppender.java 68d95fb
flume-ng-configuration/pom.xml PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfigurationFactory.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/ConfigurationException.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/Context.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationError.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfigurationErrorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelSelectorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/ChannelType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/JdbcChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MemoryChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/MultiplexingChannelSelectorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/channel/PseudoTxnMemoryChannelConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/AvroSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/FailoverSinkProcessorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/HDFSSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/RollingFileSinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkGroupConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkProcessorConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkProcessorType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/AvroSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/ExecSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/NetcatSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceType.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SyslogTcpSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SyslogUDPSourceConfiguration.java PRE-CREATION
flume-ng-configuration/src/test/resources/flume-ng-error-test PRE-CREATION
flume-ng-configuration/src/test/resources/flume-ng-test.conf PRE-CREATION
flume-ng-core/pom.xml 37fb112
flume-ng-core/src/main/java/org/apache/flume/ChannelSelector.java fba2dcb
flume-ng-core/src/main/java/org/apache/flume/Context.java 5294e31
flume-ng-core/src/main/java/org/apache/flume/SinkProcessor.java d863ed0
flume-ng-core/src/main/java/org/apache/flume/SinkProcessorType.java be1891b
flume-ng-core/src/main/java/org/apache/flume/channel/AbstractChannel.java 352bf08
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorFactory.java 800f471
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelSelectorType.java 511fc65
flume-ng-core/src/main/java/org/apache/flume/channel/ChannelType.java d8419e8
flume-ng-core/src/main/java/org/apache/flume/channel/DefaultChannelFactory.java 963a6a3
flume-ng-core/src/main/java/org/apache/flume/channel/MemoryChannel.java bfa1fde
flume-ng-core/src/main/java/org/apache/flume/channel/MultiplexingChannelSelector.java 83928b7
flume-ng-core/src/main/java/org/apache/flume/channel/PseudoTxnMemoryChannel.java 489d3e5
flume-ng-core/src/main/java/org/apache/flume/channel/ReplicatingChannelSelector.java 8f22746
flume-ng-core/src/main/java/org/apache/flume/conf/Configurable.java 1d69112
flume-ng-core/src/main/java/org/apache/flume/conf/ConfigurableComponent.java PRE-CREATION
flume-ng-core/src/main/java/org/apache/flume/conf/Configurables.java 84492e5
flume-ng-core/src/main/java/org/apache/flume/sink/AvroSink.java 3edc563
flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkFactory.java b89dfa0
flume-ng-core/src/main/java/org/apache/flume/sink/DefaultSinkProcessor.java 257bab3
flume-ng-core/src/main/java/org/apache/flume/sink/FailoverSinkProcessor.java 066ee37
flume-ng-core/src/main/java/org/apache/flume/sink/RollingFileSink.java 7f1d3b3
flume-ng-core/src/main/java/org/apache/flume/sink/SinkGroup.java 0dffd69
flume-ng-core/src/main/java/org/apache/flume/sink/SinkProcessorFactory.java 6160a17
flume-ng-core/src/main/java/org/apache/flume/sink/SinkType.java 6b08c09
flume-ng-core/src/main/java/org/apache/flume/source/AvroSource.java 903889d
flume-ng-core/src/main/java/org/apache/flume/source/DefaultSourceFactory.java a610e6f
flume-ng-core/src/main/java/org/apache/flume/source/ExecSource.java dbf79e0
flume-ng-core/src/main/java/org/apache/flume/source/ExecSourceConfigurationConstants.java 73c985e
flume-ng-core/src/main/java/org/apache/flume/source/NetcatSource.java a841b0e
flume-ng-core/src/main/java/org/apache/flume/source/SequenceGeneratorSource.java 440c5a9
flume-ng-core/src/main/java/org/apache/flume/source/SourceType.java 9082470
flume-ng-core/src/main/java/org/apache/flume/source/SyslogTcpSource.java d78d27f
flume-ng-core/src/main/java/org/apache/flume/source/SyslogUDPSource.java 34818f0
flume-ng-core/src/test/java/org/apache/flume/TestContext.java 51c350f
flume-ng-core/src/test/java/org/apache/flume/channel/AbstractBasicChannelSemanticsTest.java 6e71e46
flume-ng-core/src/test/java/org/apache/flume/channel/MockChannel.java 24b01e2
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannel.java e070864
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelConcurrency.java 8dad0b2
flume-ng-core/src/test/java/org/apache/flume/channel/TestMemoryChannelTransaction.java bc81f26
flume-ng-core/src/test/java/org/apache/flume/sink/TestAvroSink.java 3765924
flume-ng-core/src/test/java/org/apache/flume/sink/TestFailoverSinkProcessor.java 3358cf4
flume-ng-core/src/test/java/org/apache/flume/sink/TestLoggerSink.java 92ff6fe
flume-ng-core/src/test/java/org/apache/flume/sink/TestRollingFileSink.java 7e26e2a
flume-ng-core/src/test/java/org/apache/flume/source/TestAvroSource.java c5c3f2f
flume-ng-core/src/test/java/org/apache/flume/source/TestExecSource.java 615f2a3
flume-ng-core/src/test/java/org/apache/flume/source/TestPollableSourceRunner.java c27f82c
flume-ng-core/src/test/java/org/apache/flume/source/TestSequenceGeneratorSource.java 579b257
flume-ng-core/src/test/java/org/apache/flume/source/TestSyslogUdpSource.java d1db49d
flume-ng-legacy-sources/flume-avro-source/src/main/java/org/apache/flume/source/avroLegacy/AvroLegacySource.java dde8f28
flume-ng-legacy-sources/flume-avro-source/src/test/java/org/apache/flume/source/avroLegacy/TestLegacyAvroSource.java 6e3eb53
flume-ng-legacy-sources/flume-thrift-source/src/main/java/org/apache/flume/source/thriftLegacy/ThriftLegacySource.java 00df7e4
flume-ng-legacy-sources/flume-thrift-source/src/test/java/org/apache/flume/source/thriftLegacy/TestThriftLegacySource.java ddd9478
flume-ng-node/src/main/java/org/apache/flume/conf/properties/FlumeConfiguration.java d66f6d1
flume-ng-node/src/main/java/org/apache/flume/conf/properties/PropertiesFileConfigurationProvider.java 50b9f0c
flume-ng-node/src/test/java/org/apache/flume/node/TestAbstractLogicalNodeManager.java 521b586
flume-ng-node/src/test/java/org/apache/flume/source/TestNetcatSource.java fb2a960
flume-ng-node/src/test/resources/flume-conf.properties 848caca
flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java eee9221
flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSink.java 6ff3737
flume-ng-sinks/flume-irc-sink/src/main/java/org/apache/flume/sink/irc/IRCSink.java 8e77218
pom.xml c91222f
Diff: https://reviews.apache.org/r/4115/diff
Testing
-------
All existing unit tests for the components whose configuration has now been moved to these stubs pass.
Thanks,
Hari
> Create configuration stubs for sources, channels, sinks etc
> -----------------------------------------------------------
>
> Key: FLUME-992
> URL: https://issues.apache.org/jira/browse/FLUME-992
> Project: Flume
> Issue Type: Sub-task
> Affects Versions: v1.0.0
> Reporter: Hari Shreedharan
> Assignee: Hari Shreedharan
> Fix For: v1.2.0
>
>
> Currently the configuration of each component sits deep inside the component themselves. There is no way to verify if a configuration is valid before run time. In most systems like Flume, it is likely that these confs will be deployed from one single host, on to the machines where flume agents are running. Only when each agent starts, invalid confs are identified because the Agents would terminate by throwing exceptions. This is a first attempt to make a component-aware configuration system which is independent of the Flume, and does not require the Flume jar to be installed. Each component needs a configuration manager, which configures the components.
> Provide abstract Configuration stubs for each component type, sources, channels, sinks, selectors etc, which are in the new package, independent on ng-core. Now for each of the channels extend the abstract class and check the config properties for each of the required parameters for that component, for example: MultiplexingChannelSelectorConfigurator would look for default channel etc. If a particular component does not have a configuration class then let the current mechanism continue.
> This will require implementation for each component, but it should not be too complex. One additional advantage we get from this is that we can separate out the config validation from the components into these stubs, but we will still need to read the values out of the Context as we do currently(else we end up making the configuration dependent on flume-ng-core itself which we wanted to avoid).
> A problem with this is making a change to the configuration would require changes in the configuration classes and in the components also(where the configuration is read and the component is actually configured). I could not figure out a way of avoiding this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira