You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by "Mike Percy (JIRA)" <ji...@apache.org> on 2012/06/12 02:50:42 UTC
[jira] [Created] (FLUME-1273) Components left in
partially-configured state if RuntimeException thrown in configure()
Mike Percy created FLUME-1273:
---------------------------------
Summary: Components left in partially-configured state if RuntimeException thrown in configure()
Key: FLUME-1273
URL: https://issues.apache.org/jira/browse/FLUME-1273
Project: Flume
Issue Type: Bug
Affects Versions: v1.1.0
Reporter: Mike Percy
Fix For: v1.2.0
If a component, such as a Channel or a Sink, throws a RuntimeException (such as IllegalArgumentException or IllegalStateException) in its configure() method to indicate an error, some strange, non-deterministic stuff happens to the rest of the components in the system according to the ordering of the components in the configuration properties hash.
Example:
1. Channel throws IllegalArgumentException in configure() method
2. Throw propagates to PropertiesFileConfigurationProvider.loadChannels(), not caught inside loop (some components not configured)
3. Propagates to load() method in same class, Sources/Sinks not configured, RuntimeException not caught
4. Ends up getting all the way to the AbstractFileConfigurationProvider$FileWatcherRunnable.run() method, which simply logs an error message("Failed to load configuration data. Exception follows.") with a stack trace and continues. So the components will get started, etc.
The behavior should probably be changed to something like the following:
1. Exception should be caught and logged at the PropertiesFileConfigurationProvider.loadChannels() level (along with each other component type such as Source, Sink, etc).
2. At the beginning of PropertiesFileConfigurationProvider.loadChannels(), we should call conf.getChannels.clear() and only add the components to the conf hashes if they were indeed configured successfully. Either that, or at least set them to an unconfigured state that the start routine needs to respect so avoid starting unconfigured or misconfigured components.
This work needs to be done carefully to avoid any unintended consequences of changing this core logic.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1273) Components left in
partially-configured state if RuntimeException thrown in configure()
Posted by "Hari Shreedharan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13293258#comment-13293258 ]
Hari Shreedharan commented on FLUME-1273:
-----------------------------------------
I am actually wondering whether we should fix it. We do not actually start any components anyway. So I guess we are ok, if the agent does not start up. Maybe we can make the logs clearer?
> Components left in partially-configured state if RuntimeException thrown in configure()
> ---------------------------------------------------------------------------------------
>
> Key: FLUME-1273
> URL: https://issues.apache.org/jira/browse/FLUME-1273
> Project: Flume
> Issue Type: Bug
> Affects Versions: v1.1.0
> Reporter: Mike Percy
> Fix For: v1.2.0
>
>
> If a component, such as a Channel or a Sink, throws a RuntimeException (such as IllegalArgumentException or IllegalStateException) in its configure() method to indicate an error, some strange, non-deterministic stuff happens to the rest of the components in the system according to the ordering of the components in the configuration properties hash.
> Example:
> 1. Channel throws IllegalArgumentException in configure() method
> 2. Throw propagates to PropertiesFileConfigurationProvider.loadChannels(), not caught inside loop (some components not configured)
> 3. Propagates to load() method in same class, Sources/Sinks not configured, RuntimeException not caught
> 4. Ends up getting all the way to the AbstractFileConfigurationProvider$FileWatcherRunnable.run() method, which simply logs an error message("Failed to load configuration data. Exception follows.") with a stack trace and continues. So the components will get started, etc.
> The behavior should probably be changed to something like the following:
> 1. Exception should be caught and logged at the PropertiesFileConfigurationProvider.loadChannels() level (along with each other component type such as Source, Sink, etc).
> 2. At the beginning of PropertiesFileConfigurationProvider.loadChannels(), we should call conf.getChannels.clear() and only add the components to the conf hashes if they were indeed configured successfully. Either that, or at least set them to an unconfigured state that the start routine needs to respect so avoid starting unconfigured or misconfigured components.
> This work needs to be done carefully to avoid any unintended consequences of changing this core logic.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1273) Components left in
partially-configured state if RuntimeException thrown in configure()
Posted by "Mike Percy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13293266#comment-13293266 ]
Mike Percy commented on FLUME-1273:
-----------------------------------
Hmm... yeah I was wrong about starting the components... they are actually not started in this case. Actually none of them are, which is good.
I agree that this is basically desired behavior and since it's already like that let's not change it.
I'll file another JIRA to make the error messages more clear.
> Components left in partially-configured state if RuntimeException thrown in configure()
> ---------------------------------------------------------------------------------------
>
> Key: FLUME-1273
> URL: https://issues.apache.org/jira/browse/FLUME-1273
> Project: Flume
> Issue Type: Bug
> Reporter: Mike Percy
>
> If a component, such as a Channel or a Sink, throws a RuntimeException (such as IllegalArgumentException or IllegalStateException) in its configure() method to indicate an error, some strange, non-deterministic stuff happens to the rest of the components in the system according to the ordering of the components in the configuration properties hash.
> Example:
> 1. Channel throws IllegalArgumentException in configure() method
> 2. Throw propagates to PropertiesFileConfigurationProvider.loadChannels(), not caught inside loop (some components not configured)
> 3. Propagates to load() method in same class, Sources/Sinks not configured, RuntimeException not caught
> 4. Ends up getting all the way to the AbstractFileConfigurationProvider$FileWatcherRunnable.run() method, which simply logs an error message("Failed to load configuration data. Exception follows.") with a stack trace and continues. So the components will get started, etc.
> The behavior should probably be changed to something like the following:
> 1. Exception should be caught and logged at the PropertiesFileConfigurationProvider.loadChannels() level (along with each other component type such as Source, Sink, etc).
> 2. At the beginning of PropertiesFileConfigurationProvider.loadChannels(), we should call conf.getChannels.clear() and only add the components to the conf hashes if they were indeed configured successfully. Either that, or at least set them to an unconfigured state that the start routine needs to respect so avoid starting unconfigured or misconfigured components.
> This work needs to be done carefully to avoid any unintended consequences of changing this core logic.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-1273) Components left in
partially-configured state if RuntimeException thrown in configure()
Posted by "Mike Percy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mike Percy updated FLUME-1273:
------------------------------
Affects Version/s: (was: v1.1.0)
> Components left in partially-configured state if RuntimeException thrown in configure()
> ---------------------------------------------------------------------------------------
>
> Key: FLUME-1273
> URL: https://issues.apache.org/jira/browse/FLUME-1273
> Project: Flume
> Issue Type: Bug
> Reporter: Mike Percy
>
> If a component, such as a Channel or a Sink, throws a RuntimeException (such as IllegalArgumentException or IllegalStateException) in its configure() method to indicate an error, some strange, non-deterministic stuff happens to the rest of the components in the system according to the ordering of the components in the configuration properties hash.
> Example:
> 1. Channel throws IllegalArgumentException in configure() method
> 2. Throw propagates to PropertiesFileConfigurationProvider.loadChannels(), not caught inside loop (some components not configured)
> 3. Propagates to load() method in same class, Sources/Sinks not configured, RuntimeException not caught
> 4. Ends up getting all the way to the AbstractFileConfigurationProvider$FileWatcherRunnable.run() method, which simply logs an error message("Failed to load configuration data. Exception follows.") with a stack trace and continues. So the components will get started, etc.
> The behavior should probably be changed to something like the following:
> 1. Exception should be caught and logged at the PropertiesFileConfigurationProvider.loadChannels() level (along with each other component type such as Source, Sink, etc).
> 2. At the beginning of PropertiesFileConfigurationProvider.loadChannels(), we should call conf.getChannels.clear() and only add the components to the conf hashes if they were indeed configured successfully. Either that, or at least set them to an unconfigured state that the start routine needs to respect so avoid starting unconfigured or misconfigured components.
> This work needs to be done carefully to avoid any unintended consequences of changing this core logic.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1273) Components left in
partially-configured state if RuntimeException thrown in configure()
Posted by "Hari Shreedharan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13293252#comment-13293252 ]
Hari Shreedharan commented on FLUME-1273:
-----------------------------------------
Mike - thanks. This has been an issue since the beginning, I guess. configure() throwing was sort of a gray area. I will ignore the component which throws. For now, I will not clear the components(it is a new NodeConfiguration - so we should not have to clear + I don't have the time to investigate if it will break any current behavior.)
> Components left in partially-configured state if RuntimeException thrown in configure()
> ---------------------------------------------------------------------------------------
>
> Key: FLUME-1273
> URL: https://issues.apache.org/jira/browse/FLUME-1273
> Project: Flume
> Issue Type: Bug
> Affects Versions: v1.1.0
> Reporter: Mike Percy
> Fix For: v1.2.0
>
>
> If a component, such as a Channel or a Sink, throws a RuntimeException (such as IllegalArgumentException or IllegalStateException) in its configure() method to indicate an error, some strange, non-deterministic stuff happens to the rest of the components in the system according to the ordering of the components in the configuration properties hash.
> Example:
> 1. Channel throws IllegalArgumentException in configure() method
> 2. Throw propagates to PropertiesFileConfigurationProvider.loadChannels(), not caught inside loop (some components not configured)
> 3. Propagates to load() method in same class, Sources/Sinks not configured, RuntimeException not caught
> 4. Ends up getting all the way to the AbstractFileConfigurationProvider$FileWatcherRunnable.run() method, which simply logs an error message("Failed to load configuration data. Exception follows.") with a stack trace and continues. So the components will get started, etc.
> The behavior should probably be changed to something like the following:
> 1. Exception should be caught and logged at the PropertiesFileConfigurationProvider.loadChannels() level (along with each other component type such as Source, Sink, etc).
> 2. At the beginning of PropertiesFileConfigurationProvider.loadChannels(), we should call conf.getChannels.clear() and only add the components to the conf hashes if they were indeed configured successfully. Either that, or at least set them to an unconfigured state that the start routine needs to respect so avoid starting unconfigured or misconfigured components.
> This work needs to be done carefully to avoid any unintended consequences of changing this core logic.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (FLUME-1273) Components left in
partially-configured state if RuntimeException thrown in configure()
Posted by "Mike Percy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mike Percy resolved FLUME-1273.
-------------------------------
Resolution: Invalid
Fix Version/s: (was: v1.2.0)
> Components left in partially-configured state if RuntimeException thrown in configure()
> ---------------------------------------------------------------------------------------
>
> Key: FLUME-1273
> URL: https://issues.apache.org/jira/browse/FLUME-1273
> Project: Flume
> Issue Type: Bug
> Reporter: Mike Percy
>
> If a component, such as a Channel or a Sink, throws a RuntimeException (such as IllegalArgumentException or IllegalStateException) in its configure() method to indicate an error, some strange, non-deterministic stuff happens to the rest of the components in the system according to the ordering of the components in the configuration properties hash.
> Example:
> 1. Channel throws IllegalArgumentException in configure() method
> 2. Throw propagates to PropertiesFileConfigurationProvider.loadChannels(), not caught inside loop (some components not configured)
> 3. Propagates to load() method in same class, Sources/Sinks not configured, RuntimeException not caught
> 4. Ends up getting all the way to the AbstractFileConfigurationProvider$FileWatcherRunnable.run() method, which simply logs an error message("Failed to load configuration data. Exception follows.") with a stack trace and continues. So the components will get started, etc.
> The behavior should probably be changed to something like the following:
> 1. Exception should be caught and logged at the PropertiesFileConfigurationProvider.loadChannels() level (along with each other component type such as Source, Sink, etc).
> 2. At the beginning of PropertiesFileConfigurationProvider.loadChannels(), we should call conf.getChannels.clear() and only add the components to the conf hashes if they were indeed configured successfully. Either that, or at least set them to an unconfigured state that the start routine needs to respect so avoid starting unconfigured or misconfigured components.
> This work needs to be done carefully to avoid any unintended consequences of changing this core logic.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira