You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by Tristan Stevens <tr...@cloudera.com> on 2017/02/01 08:09:37 UTC
REST API Proposal
Hi all,
I'd like to put forward a proposal that I've been considering based on
conversations with users and on observations of some threads on this
mailing list.
My proposal is that we build into Flume a REST API that would give
administrators greater control over a running instance of Flume. Basically
I'm thinking of the following features:
- Allow browsing of status and configuration of all components (Sources,
Channels, Sinks).
- Allow starting and stopping of individual components.
- Allow deployment of new components (Sources, Channels, Sinks) into a
running agent.
- Allow modification of configuration of deployed components (Sources,
Channels, Sinks).
- Allow modification of log4j configuration of a running instance
(FLUME-3038).
Overall long-term goal: Eliminate the need for routine administration of
Flume via command-line during a dev-cycle and increase the
supportability/administerability of Flume in general.
In terms of benefits, my thinking is as follows:
- Granular visibility of component statuses.
- Graceful shutdown of agent (e.g. shut down Sources, allow Channels to
drain, and then shut down Sinks) (I think there's a JIRA kicking around for
this)
- Failure scenario management:
- Enable re-pointing of Sinks (e.g. because of downstream issues)
without interrupting Sources or losing events in Channels.
- Re-configuring channels or sinks in order to improve performance.
- Add sinks to running instance in order to relieve pressure on
over-full channel.
- Improve developer experience by allowing for dynamic (re)configuration
of agent without using the command-line and without needing to restart the
whole process.
- Significantly lower the barrier to adoption for both developers and
administrators.
There is also the possibility that we could then support third party
tooling for building interactive web UIs on top of Flume, which would
greatly improve usability for both administrators and also developers (e.g.
configurators).
I've knocked together a bit of a design proposal which I've made available
at:
https://docs.google.com/document/d/1OKrX__YVfMMSInezgIOj53j6JYPKfI8mNF7RP1wIhA8/edit?usp=sharing
Please
add specific comments inline in the doc and general comments back to this
thread.
My question to the group is threefold:
1. Is this something that we think is a) worthwhile and b) achievable?
2. I'm happy to lead the development, but is there a committer who can
offer time to review and support?
3. Would anyone else be interested in contributing features or testing?
Many thanks,
Tristan
Re: REST API Proposal
Posted by Mike Percy <mp...@apache.org>.
Thanks for sending this out, Tristan. Sorry for my late response. I added a
couple of comments to the doc. One missing piece is how do the
configuration changes get persisted (or do they?)
Due to other commitments it would be difficult for me to help out very
much. However, it may be possible for the PMC to grant access for a feature
branch to use for collaboration.
Mike
On Wed, Feb 1, 2017 at 12:09 AM, Tristan Stevens <tr...@cloudera.com>
wrote:
> Hi all,
> I'd like to put forward a proposal that I've been considering based on
> conversations with users and on observations of some threads on this
> mailing list.
>
> My proposal is that we build into Flume a REST API that would give
> administrators greater control over a running instance of Flume. Basically
> I'm thinking of the following features:
> - Allow browsing of status and configuration of all components (Sources,
> Channels, Sinks).
> - Allow starting and stopping of individual components.
> - Allow deployment of new components (Sources, Channels, Sinks) into a
> running agent.
> - Allow modification of configuration of deployed components (Sources,
> Channels, Sinks).
> - Allow modification of log4j configuration of a running instance
> (FLUME-3038).
>
> Overall long-term goal: Eliminate the need for routine administration of
> Flume via command-line during a dev-cycle and increase the
> supportability/administerability of Flume in general.
>
> In terms of benefits, my thinking is as follows:
>
> - Granular visibility of component statuses.
> - Graceful shutdown of agent (e.g. shut down Sources, allow Channels to
> drain, and then shut down Sinks) (I think there's a JIRA kicking around for
> this)
> - Failure scenario management:
> - Enable re-pointing of Sinks (e.g. because of downstream issues)
> without interrupting Sources or losing events in Channels.
> - Re-configuring channels or sinks in order to improve performance.
> - Add sinks to running instance in order to relieve pressure on
> over-full channel.
> - Improve developer experience by allowing for dynamic (re)configuration
> of agent without using the command-line and without needing to restart the
> whole process.
> - Significantly lower the barrier to adoption for both developers and
> administrators.
>
> There is also the possibility that we could then support third party
> tooling for building interactive web UIs on top of Flume, which would
> greatly improve usability for both administrators and also developers (e.g.
> configurators).
>
> I've knocked together a bit of a design proposal which I've made available
> at:
> https://docs.google.com/document/d/1OKrX__YVfMMSInezgIOj53j6JYPKfI8mNF7R
> P1wIhA8/edit?usp=sharing
> Please
> add specific comments inline in the doc and general comments back to this
> thread.
>
> My question to the group is threefold:
>
> 1. Is this something that we think is a) worthwhile and b) achievable?
> 2. I'm happy to lead the development, but is there a committer who can
> offer time to review and support?
> 3. Would anyone else be interested in contributing features or testing?
>
> Many thanks,
> Tristan
>
Re: REST API Proposal
Posted by Shang Wu <Sh...@symantec.com>.
+1 for idea, +1 for feature contribution, and +1 for design proposal reviewing. Thanks.
BR,
Shang
On 2/1/17, 7:38 AM, "Saikat Kanjilal" <sx...@hotmail.com> wrote:
I like this idea and would be interested in contributing features and testing, am guessing the first step is to review the design proposal if I'm not mistaken, let me know.
________________________________
From: Tristan Stevens <tr...@cloudera.com>
Sent: Wednesday, February 1, 2017 12:09 AM
To: dev@flume.apache.org
Subject: REST API Proposal
Hi all,
I'd like to put forward a proposal that I've been considering based on
conversations with users and on observations of some threads on this
mailing list.
My proposal is that we build into Flume a REST API that would give
administrators greater control over a running instance of Flume. Basically
I'm thinking of the following features:
- Allow browsing of status and configuration of all components (Sources,
Channels, Sinks).
- Allow starting and stopping of individual components.
- Allow deployment of new components (Sources, Channels, Sinks) into a
running agent.
- Allow modification of configuration of deployed components (Sources,
Channels, Sinks).
- Allow modification of log4j configuration of a running instance
(FLUME-3038).
Overall long-term goal: Eliminate the need for routine administration of
Flume via command-line during a dev-cycle and increase the
supportability/administerability of Flume in general.
In terms of benefits, my thinking is as follows:
- Granular visibility of component statuses.
- Graceful shutdown of agent (e.g. shut down Sources, allow Channels to
drain, and then shut down Sinks) (I think there's a JIRA kicking around for
this)
- Failure scenario management:
- Enable re-pointing of Sinks (e.g. because of downstream issues)
without interrupting Sources or losing events in Channels.
- Re-configuring channels or sinks in order to improve performance.
- Add sinks to running instance in order to relieve pressure on
over-full channel.
- Improve developer experience by allowing for dynamic (re)configuration
of agent without using the command-line and without needing to restart the
whole process.
- Significantly lower the barrier to adoption for both developers and
administrators.
There is also the possibility that we could then support third party
tooling for building interactive web UIs on top of Flume, which would
greatly improve usability for both administrators and also developers (e.g.
configurators).
I've knocked together a bit of a design proposal which I've made available
at:
https://docs.google.com/document/d/1OKrX__YVfMMSInezgIOj53j6JYPKfI8mNF7RP1wIhA8/edit?usp=sharing
[https://lh3.googleusercontent.com/mBCEYs5pz2OADkkd_YGpFpQYlg8uwAF02FJ59dnq70DgqwoBhf4JL7H3PR_ZS-jHQ71VRg=w1200-h630-p]<https://docs.google.com/document/d/1OKrX__YVfMMSInezgIOj53j6JYPKfI8mNF7RP1wIhA8/edit?usp=sharing>
flume-rest-proposal<https://docs.google.com/document/d/1OKrX__YVfMMSInezgIOj53j6JYPKfI8mNF7RP1wIhA8/edit?usp=sharing>
docs.google.com
Design Proposal: Flume REST API This proposal details the design rationale for a REST API onto Flume. Current Architecture As currently implemented, the ability for an administrator to interact with a running instance of Flume is very limited The overall scope of a REST API is to provide a mech...
Please
add specific comments inline in the doc and general comments back to this
thread.
My question to the group is threefold:
1. Is this something that we think is a) worthwhile and b) achievable?
2. I'm happy to lead the development, but is there a committer who can
offer time to review and support?
3. Would anyone else be interested in contributing features or testing?
Many thanks,
Tristan
Re: REST API Proposal
Posted by Saikat Kanjilal <sx...@hotmail.com>.
I like this idea and would be interested in contributing features and testing, am guessing the first step is to review the design proposal if I'm not mistaken, let me know.
________________________________
From: Tristan Stevens <tr...@cloudera.com>
Sent: Wednesday, February 1, 2017 12:09 AM
To: dev@flume.apache.org
Subject: REST API Proposal
Hi all,
I'd like to put forward a proposal that I've been considering based on
conversations with users and on observations of some threads on this
mailing list.
My proposal is that we build into Flume a REST API that would give
administrators greater control over a running instance of Flume. Basically
I'm thinking of the following features:
- Allow browsing of status and configuration of all components (Sources,
Channels, Sinks).
- Allow starting and stopping of individual components.
- Allow deployment of new components (Sources, Channels, Sinks) into a
running agent.
- Allow modification of configuration of deployed components (Sources,
Channels, Sinks).
- Allow modification of log4j configuration of a running instance
(FLUME-3038).
Overall long-term goal: Eliminate the need for routine administration of
Flume via command-line during a dev-cycle and increase the
supportability/administerability of Flume in general.
In terms of benefits, my thinking is as follows:
- Granular visibility of component statuses.
- Graceful shutdown of agent (e.g. shut down Sources, allow Channels to
drain, and then shut down Sinks) (I think there's a JIRA kicking around for
this)
- Failure scenario management:
- Enable re-pointing of Sinks (e.g. because of downstream issues)
without interrupting Sources or losing events in Channels.
- Re-configuring channels or sinks in order to improve performance.
- Add sinks to running instance in order to relieve pressure on
over-full channel.
- Improve developer experience by allowing for dynamic (re)configuration
of agent without using the command-line and without needing to restart the
whole process.
- Significantly lower the barrier to adoption for both developers and
administrators.
There is also the possibility that we could then support third party
tooling for building interactive web UIs on top of Flume, which would
greatly improve usability for both administrators and also developers (e.g.
configurators).
I've knocked together a bit of a design proposal which I've made available
at:
https://docs.google.com/document/d/1OKrX__YVfMMSInezgIOj53j6JYPKfI8mNF7RP1wIhA8/edit?usp=sharing
[https://lh3.googleusercontent.com/mBCEYs5pz2OADkkd_YGpFpQYlg8uwAF02FJ59dnq70DgqwoBhf4JL7H3PR_ZS-jHQ71VRg=w1200-h630-p]<https://docs.google.com/document/d/1OKrX__YVfMMSInezgIOj53j6JYPKfI8mNF7RP1wIhA8/edit?usp=sharing>
flume-rest-proposal<https://docs.google.com/document/d/1OKrX__YVfMMSInezgIOj53j6JYPKfI8mNF7RP1wIhA8/edit?usp=sharing>
docs.google.com
Design Proposal: Flume REST API This proposal details the design rationale for a REST API onto Flume. Current Architecture As currently implemented, the ability for an administrator to interact with a running instance of Flume is very limited The overall scope of a REST API is to provide a mech...
Please
add specific comments inline in the doc and general comments back to this
thread.
My question to the group is threefold:
1. Is this something that we think is a) worthwhile and b) achievable?
2. I'm happy to lead the development, but is there a committer who can
offer time to review and support?
3. Would anyone else be interested in contributing features or testing?
Many thanks,
Tristan
Re: REST API Proposal
Posted by Apache <ra...@dslextreme.com>.
I love this. I have a situation where I need this capability and the only way to do it is to change the configuration and restart. Being able to enable/disable a channel and sink dynamically would be great.
FWIW, I would expect that this could also be done through the existing JMX support.
Ralph
> On Feb 1, 2017, at 1:09 AM, Tristan Stevens <tr...@cloudera.com> wrote:
>
> Hi all,
> I'd like to put forward a proposal that I've been considering based on
> conversations with users and on observations of some threads on this
> mailing list.
>
> My proposal is that we build into Flume a REST API that would give
> administrators greater control over a running instance of Flume. Basically
> I'm thinking of the following features:
> - Allow browsing of status and configuration of all components (Sources,
> Channels, Sinks).
> - Allow starting and stopping of individual components.
> - Allow deployment of new components (Sources, Channels, Sinks) into a
> running agent.
> - Allow modification of configuration of deployed components (Sources,
> Channels, Sinks).
> - Allow modification of log4j configuration of a running instance
> (FLUME-3038).
>
> Overall long-term goal: Eliminate the need for routine administration of
> Flume via command-line during a dev-cycle and increase the
> supportability/administerability of Flume in general.
>
> In terms of benefits, my thinking is as follows:
>
> - Granular visibility of component statuses.
> - Graceful shutdown of agent (e.g. shut down Sources, allow Channels to
> drain, and then shut down Sinks) (I think there's a JIRA kicking around for
> this)
> - Failure scenario management:
> - Enable re-pointing of Sinks (e.g. because of downstream issues)
> without interrupting Sources or losing events in Channels.
> - Re-configuring channels or sinks in order to improve performance.
> - Add sinks to running instance in order to relieve pressure on
> over-full channel.
> - Improve developer experience by allowing for dynamic (re)configuration
> of agent without using the command-line and without needing to restart the
> whole process.
> - Significantly lower the barrier to adoption for both developers and
> administrators.
>
> There is also the possibility that we could then support third party
> tooling for building interactive web UIs on top of Flume, which would
> greatly improve usability for both administrators and also developers (e.g.
> configurators).
>
> I've knocked together a bit of a design proposal which I've made available
> at:
> https://docs.google.com/document/d/1OKrX__YVfMMSInezgIOj53j6JYPKfI8mNF7RP1wIhA8/edit?usp=sharing
> Please
> add specific comments inline in the doc and general comments back to this
> thread.
>
> My question to the group is threefold:
>
> 1. Is this something that we think is a) worthwhile and b) achievable?
> 2. I'm happy to lead the development, but is there a committer who can
> offer time to review and support?
> 3. Would anyone else be interested in contributing features or testing?
>
> Many thanks,
> Tristan