You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by "Arvind Prabhakar (Created) (JIRA)" <ji...@apache.org> on 2012/02/01 04:05:03 UTC
[jira] [Created] (FLUME-946) Allow multiplexing channel selector to
specify optional channels.
Allow multiplexing channel selector to specify optional channels.
-----------------------------------------------------------------
Key: FLUME-946
URL: https://issues.apache.org/jira/browse/FLUME-946
Project: Flume
Issue Type: Improvement
Reporter: Arvind Prabhakar
Right now the multiplexing channel selector treats all channels as required channels. There should be a simple provision where some channel can be marked optional so that a failure in publishing the event to that particular channel does not cause the entire request to fail. For example, the following configuration:
agent.sources.foo.selector.mapping.foo = ch1 ch2
agent.sources.foo.selector.mapping.bar = ch2 ch3
agent.sources.foo.selector.optional.channels = ch2
Would imply that for mapping value "foo", ch1 is a required channel whereas ch2 is an optional channel. Similarly, for mapping "bar', ch3 is a required channel but ch2 is optional.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-946) Allow multiplexing channel selector to
specify optional channels.
Posted by "Hari Shreedharan (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hari Shreedharan updated FLUME-946:
-----------------------------------
Description:
Right now the multiplexing channel selector treats all channels as required channels. There should be a simple provision where some channel can be marked optional so that a failure in publishing the event to that particular channel does not cause the entire request to fail. For example, the following configuration:
agent.sources.foo.selector.mapping.foo = ch1 ch2
agent.sources.foo.selector.mapping.bar = ch2 ch3
agent.sources.foo.selector.optional.channels = ch2
Would imply that for mapping value "foo", ch1 is a required channel whereas ch2 is an optional channel. Similarly, for mapping "bar', ch3 is a required channel but ch2 is optional.
The implementation in the patch is:
I did not implement it exactly as defined in the Jira, instead:
agent.sources.foo.selector.mapping.zebra = ch2
agent.sources.foo.selector.optional.zebra = ch1
Therefore the behavior of "mapping" keyword does not change, instead the optional keyword simply adds more channels as optional channels. If "mapping" keyword does not exist for the header or has no values, then the event is written to default channel and optional channels. If optional is empty then even is simply written to channels specificied by "mapping"(if empty, then to default channels). Basically mapping and optional means: write to channels specified by "mapping" and if available, to the ones specified by "optional." If mapping is empty write to default, and optionally to optional channels.
was:
Right now the multiplexing channel selector treats all channels as required channels. There should be a simple provision where some channel can be marked optional so that a failure in publishing the event to that particular channel does not cause the entire request to fail. For example, the following configuration:
agent.sources.foo.selector.mapping.foo = ch1 ch2
agent.sources.foo.selector.mapping.bar = ch2 ch3
agent.sources.foo.selector.optional.channels = ch2
Would imply that for mapping value "foo", ch1 is a required channel whereas ch2 is an optional channel. Similarly, for mapping "bar', ch3 is a required channel but ch2 is optional.
> Allow multiplexing channel selector to specify optional channels.
> -----------------------------------------------------------------
>
> Key: FLUME-946
> URL: https://issues.apache.org/jira/browse/FLUME-946
> Project: Flume
> Issue Type: Improvement
> Affects Versions: v1.0.0
> Reporter: Arvind Prabhakar
> Fix For: v1.1.0
>
> Attachments: FLUME-946-2.patch
>
>
> Right now the multiplexing channel selector treats all channels as required channels. There should be a simple provision where some channel can be marked optional so that a failure in publishing the event to that particular channel does not cause the entire request to fail. For example, the following configuration:
> agent.sources.foo.selector.mapping.foo = ch1 ch2
> agent.sources.foo.selector.mapping.bar = ch2 ch3
> agent.sources.foo.selector.optional.channels = ch2
> Would imply that for mapping value "foo", ch1 is a required channel whereas ch2 is an optional channel. Similarly, for mapping "bar', ch3 is a required channel but ch2 is optional.
> The implementation in the patch is:
> I did not implement it exactly as defined in the Jira, instead:
> agent.sources.foo.selector.mapping.zebra = ch2
> agent.sources.foo.selector.optional.zebra = ch1
> Therefore the behavior of "mapping" keyword does not change, instead the optional keyword simply adds more channels as optional channels. If "mapping" keyword does not exist for the header or has no values, then the event is written to default channel and optional channels. If optional is empty then even is simply written to channels specificied by "mapping"(if empty, then to default channels). Basically mapping and optional means: write to channels specified by "mapping" and if available, to the ones specified by "optional." If mapping is empty write to default, and optionally to optional channels.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (FLUME-946) Allow multiplexing channel selector
to specify optional channels.
Posted by "Hari Shreedharan (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hari Shreedharan reassigned FLUME-946:
--------------------------------------
Assignee: Hari Shreedharan
> Allow multiplexing channel selector to specify optional channels.
> -----------------------------------------------------------------
>
> Key: FLUME-946
> URL: https://issues.apache.org/jira/browse/FLUME-946
> Project: Flume
> Issue Type: Improvement
> Affects Versions: v1.0.0
> Reporter: Arvind Prabhakar
> Assignee: Hari Shreedharan
> Fix For: v1.1.0
>
> Attachments: FLUME-946-2.patch
>
>
> Right now the multiplexing channel selector treats all channels as required channels. There should be a simple provision where some channel can be marked optional so that a failure in publishing the event to that particular channel does not cause the entire request to fail. For example, the following configuration:
> agent.sources.foo.selector.mapping.foo = ch1 ch2
> agent.sources.foo.selector.mapping.bar = ch2 ch3
> agent.sources.foo.selector.optional.channels = ch2
> Would imply that for mapping value "foo", ch1 is a required channel whereas ch2 is an optional channel. Similarly, for mapping "bar', ch3 is a required channel but ch2 is optional.
> The implementation in the patch is:
> I did not implement it exactly as defined in the Jira, instead:
> agent.sources.foo.selector.mapping.zebra = ch2
> agent.sources.foo.selector.optional.zebra = ch1
> Therefore the behavior of "mapping" keyword does not change, instead the optional keyword simply adds more channels as optional channels. If "mapping" keyword does not exist for the header or has no values, then the event is written to default channel and optional channels. If optional is empty then even is simply written to channels specificied by "mapping"(if empty, then to default channels). Basically mapping and optional means: write to channels specified by "mapping" and if available, to the ones specified by "optional." If mapping is empty write to default, and optionally to optional channels.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-946) Allow multiplexing channel selector to
specify optional channels.
Posted by "Hari Shreedharan (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hari Shreedharan updated FLUME-946:
-----------------------------------
Attachment: FLUME-946-2.patch
> Allow multiplexing channel selector to specify optional channels.
> -----------------------------------------------------------------
>
> Key: FLUME-946
> URL: https://issues.apache.org/jira/browse/FLUME-946
> Project: Flume
> Issue Type: Improvement
> Affects Versions: v1.0.0
> Reporter: Arvind Prabhakar
> Fix For: v1.1.0
>
> Attachments: FLUME-946-2.patch
>
>
> Right now the multiplexing channel selector treats all channels as required channels. There should be a simple provision where some channel can be marked optional so that a failure in publishing the event to that particular channel does not cause the entire request to fail. For example, the following configuration:
> agent.sources.foo.selector.mapping.foo = ch1 ch2
> agent.sources.foo.selector.mapping.bar = ch2 ch3
> agent.sources.foo.selector.optional.channels = ch2
> Would imply that for mapping value "foo", ch1 is a required channel whereas ch2 is an optional channel. Similarly, for mapping "bar', ch3 is a required channel but ch2 is optional.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-946) Allow multiplexing channel selector to
specify optional channels.
Posted by "Hari Shreedharan (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hari Shreedharan updated FLUME-946:
-----------------------------------
Status: Patch Available (was: Open)
> Allow multiplexing channel selector to specify optional channels.
> -----------------------------------------------------------------
>
> Key: FLUME-946
> URL: https://issues.apache.org/jira/browse/FLUME-946
> Project: Flume
> Issue Type: Improvement
> Affects Versions: v1.0.0
> Reporter: Arvind Prabhakar
> Attachments: FLUME-946-2.patch
>
>
> Right now the multiplexing channel selector treats all channels as required channels. There should be a simple provision where some channel can be marked optional so that a failure in publishing the event to that particular channel does not cause the entire request to fail. For example, the following configuration:
> agent.sources.foo.selector.mapping.foo = ch1 ch2
> agent.sources.foo.selector.mapping.bar = ch2 ch3
> agent.sources.foo.selector.optional.channels = ch2
> Would imply that for mapping value "foo", ch1 is a required channel whereas ch2 is an optional channel. Similarly, for mapping "bar', ch3 is a required channel but ch2 is optional.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-946) Allow multiplexing channel selector to
specify optional channels.
Posted by "Hari Shreedharan (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hari Shreedharan updated FLUME-946:
-----------------------------------
Priority: Minor (was: Major)
Affects Version/s: v1.1.0
Remaining Estimate: 0h
Original Estimate: 0h
> Allow multiplexing channel selector to specify optional channels.
> -----------------------------------------------------------------
>
> Key: FLUME-946
> URL: https://issues.apache.org/jira/browse/FLUME-946
> Project: Flume
> Issue Type: Improvement
> Affects Versions: v1.0.0, v1.1.0
> Reporter: Arvind Prabhakar
> Assignee: Hari Shreedharan
> Priority: Minor
> Fix For: v1.2.0
>
> Attachments: FLUME-946-2.patch
>
> Original Estimate: 0h
> Remaining Estimate: 0h
>
> Right now the multiplexing channel selector treats all channels as required channels. There should be a simple provision where some channel can be marked optional so that a failure in publishing the event to that particular channel does not cause the entire request to fail. For example, the following configuration:
> agent.sources.foo.selector.mapping.foo = ch1 ch2
> agent.sources.foo.selector.mapping.bar = ch2 ch3
> agent.sources.foo.selector.optional.channels = ch2
> Would imply that for mapping value "foo", ch1 is a required channel whereas ch2 is an optional channel. Similarly, for mapping "bar', ch3 is a required channel but ch2 is optional.
> The implementation in the patch is:
> I did not implement it exactly as defined in the Jira, instead:
> agent.sources.foo.selector.mapping.zebra = ch2
> agent.sources.foo.selector.optional.zebra = ch1
> Therefore the behavior of "mapping" keyword does not change, instead the optional keyword simply adds more channels as optional channels. If "mapping" keyword does not exist for the header or has no values, then the event is written to default channel and optional channels. If optional is empty then even is simply written to channels specificied by "mapping"(if empty, then to default channels). Basically mapping and optional means: write to channels specified by "mapping" and if available, to the ones specified by "optional." If mapping is empty write to default, and optionally to optional channels.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-946) Allow multiplexing channel selector to
specify optional channels.
Posted by "Hari Shreedharan (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hari Shreedharan updated FLUME-946:
-----------------------------------
Fix Version/s: v1.1.0
> Allow multiplexing channel selector to specify optional channels.
> -----------------------------------------------------------------
>
> Key: FLUME-946
> URL: https://issues.apache.org/jira/browse/FLUME-946
> Project: Flume
> Issue Type: Improvement
> Affects Versions: v1.0.0
> Reporter: Arvind Prabhakar
> Fix For: v1.1.0
>
> Attachments: FLUME-946-2.patch
>
>
> Right now the multiplexing channel selector treats all channels as required channels. There should be a simple provision where some channel can be marked optional so that a failure in publishing the event to that particular channel does not cause the entire request to fail. For example, the following configuration:
> agent.sources.foo.selector.mapping.foo = ch1 ch2
> agent.sources.foo.selector.mapping.bar = ch2 ch3
> agent.sources.foo.selector.optional.channels = ch2
> Would imply that for mapping value "foo", ch1 is a required channel whereas ch2 is an optional channel. Similarly, for mapping "bar', ch3 is a required channel but ch2 is optional.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-946) Allow multiplexing channel selector to
specify optional channels.
Posted by "Arvind Prabhakar (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arvind Prabhakar updated FLUME-946:
-----------------------------------
Affects Version/s: v1.0.0
> Allow multiplexing channel selector to specify optional channels.
> -----------------------------------------------------------------
>
> Key: FLUME-946
> URL: https://issues.apache.org/jira/browse/FLUME-946
> Project: Flume
> Issue Type: Improvement
> Affects Versions: v1.0.0
> Reporter: Arvind Prabhakar
>
> Right now the multiplexing channel selector treats all channels as required channels. There should be a simple provision where some channel can be marked optional so that a failure in publishing the event to that particular channel does not cause the entire request to fail. For example, the following configuration:
> agent.sources.foo.selector.mapping.foo = ch1 ch2
> agent.sources.foo.selector.mapping.bar = ch2 ch3
> agent.sources.foo.selector.optional.channels = ch2
> Would imply that for mapping value "foo", ch1 is a required channel whereas ch2 is an optional channel. Similarly, for mapping "bar', ch3 is a required channel but ch2 is optional.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-946) Allow multiplexing channel selector
to specify optional channels.
Posted by "Hari Shreedharan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13215835#comment-13215835 ]
Hari Shreedharan commented on FLUME-946:
----------------------------------------
I did not implement it exactly as defined in the Jira, instead:
agent.sources.foo.selector.mapping.zebra = ch2
agent.sources.foo.selector.optional.zebra = ch1
Therefore the behavior of "mapping" keyword does not change, instead the optional keyword simply adds more channels as optional channels. If "mapping" keyword does not exist for the header or has no values, then the event is written to default channel and optional channels. If optional is empty then even is simply written to channels specificied by "mapping"(if empty, then to default channels). Basically mapping and optional means: write to channels specified by "mapping" and if available, to the ones specified by "optional." If mapping is empty write to default, and optionally to optional channels.
> Allow multiplexing channel selector to specify optional channels.
> -----------------------------------------------------------------
>
> Key: FLUME-946
> URL: https://issues.apache.org/jira/browse/FLUME-946
> Project: Flume
> Issue Type: Improvement
> Affects Versions: v1.0.0
> Reporter: Arvind Prabhakar
> Fix For: v1.1.0
>
> Attachments: FLUME-946-2.patch
>
>
> Right now the multiplexing channel selector treats all channels as required channels. There should be a simple provision where some channel can be marked optional so that a failure in publishing the event to that particular channel does not cause the entire request to fail. For example, the following configuration:
> agent.sources.foo.selector.mapping.foo = ch1 ch2
> agent.sources.foo.selector.mapping.bar = ch2 ch3
> agent.sources.foo.selector.optional.channels = ch2
> Would imply that for mapping value "foo", ch1 is a required channel whereas ch2 is an optional channel. Similarly, for mapping "bar', ch3 is a required channel but ch2 is optional.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-946) Allow multiplexing channel selector
to specify optional channels.
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13215804#comment-13215804 ]
jiraposter@reviews.apache.org commented on FLUME-946:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4034/
-----------------------------------------------------------
Review request for Flume.
Summary
-------
Support for optional channels in multiplexing channel selector. If no required channels configured, but there are optional channels then it means, write to default channels and optionally write to optional channels if they are available.
This addresses bug FLUME-946.
https://issues.apache.org/jira/browse/FLUME-946
Diffs
-----
flume-ng-core/src/main/java/org/apache/flume/channel/MultiplexingChannelSelector.java c63d0a1
flume-ng-core/src/test/java/org/apache/flume/channel/TestMultiplexingChannelSelector.java 2626b20
Diff: https://reviews.apache.org/r/4034/diff
Testing
-------
Updated unit tests to support these cases
Thanks,
Hari
> Allow multiplexing channel selector to specify optional channels.
> -----------------------------------------------------------------
>
> Key: FLUME-946
> URL: https://issues.apache.org/jira/browse/FLUME-946
> Project: Flume
> Issue Type: Improvement
> Affects Versions: v1.0.0
> Reporter: Arvind Prabhakar
>
> Right now the multiplexing channel selector treats all channels as required channels. There should be a simple provision where some channel can be marked optional so that a failure in publishing the event to that particular channel does not cause the entire request to fail. For example, the following configuration:
> agent.sources.foo.selector.mapping.foo = ch1 ch2
> agent.sources.foo.selector.mapping.bar = ch2 ch3
> agent.sources.foo.selector.optional.channels = ch2
> Would imply that for mapping value "foo", ch1 is a required channel whereas ch2 is an optional channel. Similarly, for mapping "bar', ch3 is a required channel but ch2 is optional.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-946) Allow multiplexing channel selector
to specify optional channels.
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13215822#comment-13215822 ]
jiraposter@reviews.apache.org commented on FLUME-946:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4034/
-----------------------------------------------------------
(Updated 2012-02-24 19:00:14.512832)
Review request for Flume.
Changes
-------
Updated unit test to get optional channel from an event which does not have required channels.
Summary
-------
Support for optional channels in multiplexing channel selector. If no required channels configured, but there are optional channels then it means, write to default channels and optionally write to optional channels if they are available.
This addresses bug FLUME-946.
https://issues.apache.org/jira/browse/FLUME-946
Diffs (updated)
-----
flume-ng-core/src/main/java/org/apache/flume/channel/MultiplexingChannelSelector.java c63d0a1
flume-ng-core/src/test/java/org/apache/flume/channel/TestMultiplexingChannelSelector.java 2626b20
Diff: https://reviews.apache.org/r/4034/diff
Testing
-------
Updated unit tests to support these cases
Thanks,
Hari
> Allow multiplexing channel selector to specify optional channels.
> -----------------------------------------------------------------
>
> Key: FLUME-946
> URL: https://issues.apache.org/jira/browse/FLUME-946
> Project: Flume
> Issue Type: Improvement
> Affects Versions: v1.0.0
> Reporter: Arvind Prabhakar
> Fix For: v1.1.0
>
> Attachments: FLUME-946-2.patch
>
>
> Right now the multiplexing channel selector treats all channels as required channels. There should be a simple provision where some channel can be marked optional so that a failure in publishing the event to that particular channel does not cause the entire request to fail. For example, the following configuration:
> agent.sources.foo.selector.mapping.foo = ch1 ch2
> agent.sources.foo.selector.mapping.bar = ch2 ch3
> agent.sources.foo.selector.optional.channels = ch2
> Would imply that for mapping value "foo", ch1 is a required channel whereas ch2 is an optional channel. Similarly, for mapping "bar', ch3 is a required channel but ch2 is optional.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira