You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by "Will McQueen (Created) (JIRA)" <ji...@apache.org> on 2012/02/27 19:06:49 UTC
[jira] [Created] (FLUME-1002) FailoverSinkProcessor replaces sinks
with same priority
FailoverSinkProcessor replaces sinks with same priority
-------------------------------------------------------
Key: FLUME-1002
URL: https://issues.apache.org/jira/browse/FLUME-1002
Project: Flume
Issue Type: Bug
Components: Sinks+Sources
Affects Versions: v1.1.0
Environment: RHEL 6.2 64-bit
Reporter: Will McQueen
Fix For: v1.1.0
In FailoverSinkProcessor, sinks are indexed in a SortedMap according to their priority. This means that specifying 2 failover sinks with the same priority will actually cause only one of them to be included in the map since the last sink replaces all previous sinks of the same priority ("last one wins"). For example:
a1.sinkgroups.g1.sinks = k1 k2 k3 k4
a1.sinkgroups.g1.processor.type = failover
a1.sinkgroups.g1.processor.priority.k1 = 5
a1.sinkgroups.g1.processor.priority.k2 = 10
a1.sinkgroups.g1.processor.priority.k3 = 10
a1.sinkgroups.g1.processor.priority.k4 = 15
Here, the intent of the user is to configure sinks k2 and k3 to have equal priority, but in fact only 1 of these sinks will be used... so the user is left with a total of 3 working failover sinks instead of 4.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-1002) FailoverSinkProcessor replaces sinks
with same priority
Posted by "Arvind Prabhakar (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arvind Prabhakar updated FLUME-1002:
------------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
Patch committed. Thanks Juhani!
> FailoverSinkProcessor replaces sinks with same priority
> -------------------------------------------------------
>
> Key: FLUME-1002
> URL: https://issues.apache.org/jira/browse/FLUME-1002
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Affects Versions: v1.1.0
> Environment: RHEL 6.2 64-bit
> Reporter: Will McQueen
> Assignee: Juhani Connolly
> Fix For: v1.2.0
>
> Attachments: FLUME-1002.patch
>
>
> In FailoverSinkProcessor, sinks are indexed in a SortedMap according to their priority. This means that specifying 2 failover sinks with the same priority will actually cause only one of them to be included in the map since the last sink replaces all previous sinks of the same priority ("last one wins"). For example:
> a1.sinkgroups.g1.sinks = k1 k2 k3 k4
> a1.sinkgroups.g1.processor.type = failover
> a1.sinkgroups.g1.processor.priority.k1 = 5
> a1.sinkgroups.g1.processor.priority.k2 = 10
> a1.sinkgroups.g1.processor.priority.k3 = 10
> a1.sinkgroups.g1.processor.priority.k4 = 15
> Here, the intent of the user is to configure sinks k2 and k3 to have equal priority, but in fact only 1 of these sinks will be used... so the user is left with a total of 3 working failover sinks instead of 4.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1002) FailoverSinkProcessor replaces
sinks with same priority
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13235293#comment-13235293 ]
Hudson commented on FLUME-1002:
-------------------------------
Integrated in flume-trunk #134 (See [https://builds.apache.org/job/flume-trunk/134/])
FLUME-1002. FailoverSinkProcessor replaces sinks with same priority.
(Juhani Connolly via Arvind Prabhakar) (Revision 1303635)
Result = SUCCESS
arvind : http://svn.apache.org/viewvc/?view=rev&rev=1303635
Files :
* /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/sink/FailoverSinkProcessor.java
> FailoverSinkProcessor replaces sinks with same priority
> -------------------------------------------------------
>
> Key: FLUME-1002
> URL: https://issues.apache.org/jira/browse/FLUME-1002
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Affects Versions: v1.1.0
> Environment: RHEL 6.2 64-bit
> Reporter: Will McQueen
> Assignee: Juhani Connolly
> Fix For: v1.2.0
>
> Attachments: FLUME-1002.patch
>
>
> In FailoverSinkProcessor, sinks are indexed in a SortedMap according to their priority. This means that specifying 2 failover sinks with the same priority will actually cause only one of them to be included in the map since the last sink replaces all previous sinks of the same priority ("last one wins"). For example:
> a1.sinkgroups.g1.sinks = k1 k2 k3 k4
> a1.sinkgroups.g1.processor.type = failover
> a1.sinkgroups.g1.processor.priority.k1 = 5
> a1.sinkgroups.g1.processor.priority.k2 = 10
> a1.sinkgroups.g1.processor.priority.k3 = 10
> a1.sinkgroups.g1.processor.priority.k4 = 15
> Here, the intent of the user is to configure sinks k2 and k3 to have equal priority, but in fact only 1 of these sinks will be used... so the user is left with a total of 3 working failover sinks instead of 4.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-1002) FailoverSinkProcessor replaces sinks
with same priority
Posted by "Juhani Connolly (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Juhani Connolly updated FLUME-1002:
-----------------------------------
Status: Patch Available (was: Open)
> FailoverSinkProcessor replaces sinks with same priority
> -------------------------------------------------------
>
> Key: FLUME-1002
> URL: https://issues.apache.org/jira/browse/FLUME-1002
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Affects Versions: v1.1.0
> Environment: RHEL 6.2 64-bit
> Reporter: Will McQueen
> Fix For: v1.1.0
>
> Attachments: FLUME-1002.patch
>
>
> In FailoverSinkProcessor, sinks are indexed in a SortedMap according to their priority. This means that specifying 2 failover sinks with the same priority will actually cause only one of them to be included in the map since the last sink replaces all previous sinks of the same priority ("last one wins"). For example:
> a1.sinkgroups.g1.sinks = k1 k2 k3 k4
> a1.sinkgroups.g1.processor.type = failover
> a1.sinkgroups.g1.processor.priority.k1 = 5
> a1.sinkgroups.g1.processor.priority.k2 = 10
> a1.sinkgroups.g1.processor.priority.k3 = 10
> a1.sinkgroups.g1.processor.priority.k4 = 15
> Here, the intent of the user is to configure sinks k2 and k3 to have equal priority, but in fact only 1 of these sinks will be used... so the user is left with a total of 3 working failover sinks instead of 4.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (FLUME-1002) FailoverSinkProcessor replaces sinks
with same priority
Posted by "Arvind Prabhakar (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arvind Prabhakar reassigned FLUME-1002:
---------------------------------------
Assignee: Juhani Connolly
> FailoverSinkProcessor replaces sinks with same priority
> -------------------------------------------------------
>
> Key: FLUME-1002
> URL: https://issues.apache.org/jira/browse/FLUME-1002
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Affects Versions: v1.1.0
> Environment: RHEL 6.2 64-bit
> Reporter: Will McQueen
> Assignee: Juhani Connolly
> Fix For: v1.2.0
>
> Attachments: FLUME-1002.patch
>
>
> In FailoverSinkProcessor, sinks are indexed in a SortedMap according to their priority. This means that specifying 2 failover sinks with the same priority will actually cause only one of them to be included in the map since the last sink replaces all previous sinks of the same priority ("last one wins"). For example:
> a1.sinkgroups.g1.sinks = k1 k2 k3 k4
> a1.sinkgroups.g1.processor.type = failover
> a1.sinkgroups.g1.processor.priority.k1 = 5
> a1.sinkgroups.g1.processor.priority.k2 = 10
> a1.sinkgroups.g1.processor.priority.k3 = 10
> a1.sinkgroups.g1.processor.priority.k4 = 15
> Here, the intent of the user is to configure sinks k2 and k3 to have equal priority, but in fact only 1 of these sinks will be used... so the user is left with a total of 3 working failover sinks instead of 4.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1002) FailoverSinkProcessor replaces
sinks with same priority
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222158#comment-13222158 ]
jiraposter@reviews.apache.org commented on FLUME-1002:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4173/
-----------------------------------------------------------
Review request for Flume.
Summary
-------
Fixed documentation to make it clear that priorities must be unique.
Also added a logging warning when a non-unique priority is encountered, ignoring the sink that would overwrite.
This addresses bug FLUME-1002.
https://issues.apache.org/jira/browse/FLUME-1002
Diffs
-----
flume-ng-core/src/main/java/org/apache/flume/sink/FailoverSinkProcessor.java 9f5b856
Diff: https://reviews.apache.org/r/4173/diff
Testing
-------
Run full test suite, all clear
Thanks,
Juhani
> FailoverSinkProcessor replaces sinks with same priority
> -------------------------------------------------------
>
> Key: FLUME-1002
> URL: https://issues.apache.org/jira/browse/FLUME-1002
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Affects Versions: v1.1.0
> Environment: RHEL 6.2 64-bit
> Reporter: Will McQueen
> Fix For: v1.1.0
>
>
> In FailoverSinkProcessor, sinks are indexed in a SortedMap according to their priority. This means that specifying 2 failover sinks with the same priority will actually cause only one of them to be included in the map since the last sink replaces all previous sinks of the same priority ("last one wins"). For example:
> a1.sinkgroups.g1.sinks = k1 k2 k3 k4
> a1.sinkgroups.g1.processor.type = failover
> a1.sinkgroups.g1.processor.priority.k1 = 5
> a1.sinkgroups.g1.processor.priority.k2 = 10
> a1.sinkgroups.g1.processor.priority.k3 = 10
> a1.sinkgroups.g1.processor.priority.k4 = 15
> Here, the intent of the user is to configure sinks k2 and k3 to have equal priority, but in fact only 1 of these sinks will be used... so the user is left with a total of 3 working failover sinks instead of 4.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1002) FailoverSinkProcessor replaces
sinks with same priority
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13235283#comment-13235283 ]
jiraposter@reviews.apache.org commented on FLUME-1002:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4173/#review6200
-----------------------------------------------------------
Ship it!
+1
(ignoring the whitespace issue that I will plan to run a sweep for soon).
- Arvind
On 2012-03-05 03:51:43, Juhani Connolly wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/4173/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-03-05 03:51:43)
bq.
bq.
bq. Review request for Flume.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Fixed documentation to make it clear that priorities must be unique.
bq.
bq. Also added a logging warning when a non-unique priority is encountered, ignoring the sink that would overwrite.
bq.
bq.
bq. This addresses bug FLUME-1002.
bq. https://issues.apache.org/jira/browse/FLUME-1002
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. flume-ng-core/src/main/java/org/apache/flume/sink/FailoverSinkProcessor.java 9f5b856
bq.
bq. Diff: https://reviews.apache.org/r/4173/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Run full test suite, all clear
bq.
bq.
bq. Thanks,
bq.
bq. Juhani
bq.
bq.
> FailoverSinkProcessor replaces sinks with same priority
> -------------------------------------------------------
>
> Key: FLUME-1002
> URL: https://issues.apache.org/jira/browse/FLUME-1002
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Affects Versions: v1.1.0
> Environment: RHEL 6.2 64-bit
> Reporter: Will McQueen
> Fix For: v1.2.0
>
> Attachments: FLUME-1002.patch
>
>
> In FailoverSinkProcessor, sinks are indexed in a SortedMap according to their priority. This means that specifying 2 failover sinks with the same priority will actually cause only one of them to be included in the map since the last sink replaces all previous sinks of the same priority ("last one wins"). For example:
> a1.sinkgroups.g1.sinks = k1 k2 k3 k4
> a1.sinkgroups.g1.processor.type = failover
> a1.sinkgroups.g1.processor.priority.k1 = 5
> a1.sinkgroups.g1.processor.priority.k2 = 10
> a1.sinkgroups.g1.processor.priority.k3 = 10
> a1.sinkgroups.g1.processor.priority.k4 = 15
> Here, the intent of the user is to configure sinks k2 and k3 to have equal priority, but in fact only 1 of these sinks will be used... so the user is left with a total of 3 working failover sinks instead of 4.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1002) FailoverSinkProcessor replaces
sinks with same priority
Posted by "Juhani Connolly (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13217788#comment-13217788 ]
Juhani Connolly commented on FLUME-1002:
----------------------------------------
I could've sworn I documented this as a requirement, but I may missed it when I rewrote the implementation.
As far as I'm concerned I think that having identical priorities is a bad idea as it makes failover order unpredictable, and that things are fine as they are. However it should be documented(sorry I must have missed that)
> FailoverSinkProcessor replaces sinks with same priority
> -------------------------------------------------------
>
> Key: FLUME-1002
> URL: https://issues.apache.org/jira/browse/FLUME-1002
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Affects Versions: v1.1.0
> Environment: RHEL 6.2 64-bit
> Reporter: Will McQueen
> Fix For: v1.1.0
>
>
> In FailoverSinkProcessor, sinks are indexed in a SortedMap according to their priority. This means that specifying 2 failover sinks with the same priority will actually cause only one of them to be included in the map since the last sink replaces all previous sinks of the same priority ("last one wins"). For example:
> a1.sinkgroups.g1.sinks = k1 k2 k3 k4
> a1.sinkgroups.g1.processor.type = failover
> a1.sinkgroups.g1.processor.priority.k1 = 5
> a1.sinkgroups.g1.processor.priority.k2 = 10
> a1.sinkgroups.g1.processor.priority.k3 = 10
> a1.sinkgroups.g1.processor.priority.k4 = 15
> Here, the intent of the user is to configure sinks k2 and k3 to have equal priority, but in fact only 1 of these sinks will be used... so the user is left with a total of 3 working failover sinks instead of 4.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-1002) FailoverSinkProcessor replaces sinks
with same priority
Posted by "Juhani Connolly (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Juhani Connolly updated FLUME-1002:
-----------------------------------
Attachment: FLUME-1002.patch
> FailoverSinkProcessor replaces sinks with same priority
> -------------------------------------------------------
>
> Key: FLUME-1002
> URL: https://issues.apache.org/jira/browse/FLUME-1002
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Affects Versions: v1.1.0
> Environment: RHEL 6.2 64-bit
> Reporter: Will McQueen
> Fix For: v1.1.0
>
> Attachments: FLUME-1002.patch
>
>
> In FailoverSinkProcessor, sinks are indexed in a SortedMap according to their priority. This means that specifying 2 failover sinks with the same priority will actually cause only one of them to be included in the map since the last sink replaces all previous sinks of the same priority ("last one wins"). For example:
> a1.sinkgroups.g1.sinks = k1 k2 k3 k4
> a1.sinkgroups.g1.processor.type = failover
> a1.sinkgroups.g1.processor.priority.k1 = 5
> a1.sinkgroups.g1.processor.priority.k2 = 10
> a1.sinkgroups.g1.processor.priority.k3 = 10
> a1.sinkgroups.g1.processor.priority.k4 = 15
> Here, the intent of the user is to configure sinks k2 and k3 to have equal priority, but in fact only 1 of these sinks will be used... so the user is left with a total of 3 working failover sinks instead of 4.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira