Proposal for tools to generate duplicated configuration

[Denny Ye <de...@gmail.com>]

hi all,
    In the progress of maintaining Flume NG, it's hard to manage
configuration in too many flows.
    Meanwhile, most of such configurations are similar. Maybe tiny
difference in component name;

    Thus, I would like to generate configuration with Java code in simple
method.
    I created and tested tool to generate configuration. That runs
regularly.

    Could anyone give your tips or advice? If tool like this is useful for
you, I will submit it to community.
    Wish your attention, thanks!


     Java Code Example:
                Agent agent = new Agent().setName("agent");
 Source source = new Source("scribe")
.setProperty("port", "1499")
.setType(SourceType.OTHER, "org.apache.flume.source.scribe.ScribeSource");
agent.registerSource(source);
 for (int i = 0; i < 5; i++) {
Channel channel = new Channel("mc" + i)
      .setProperty("capacity", "1000000")
      .setType(ChannelType.MEMORY, null);
agent.registerChannel(channel);
 Sink sink = new Sink("hfds" + i)
      .setType(SinkType.HDFS, null)
      .setProperty("hfds.txtEventMax", "1000")
       .setProperty("hdfs.path", "/flume_test/data" + i)
       .connectChannel(channel);
agent.registerSink(sink);
 source.connectChannel(channel);
}

                ConfGenerator.generateConf(agent, new
File("d:\\flume.conf"));

-Regards
Denny Ye

Re: Proposal for tools to generate duplicated configuration

[Brock Noland <br...@cloudera.com>]

Hi,

I think a tool like this could be useful. Do you want to create a JIRA and
attach it so we can have a look?

Brock

On Thu, Jul 26, 2012 at 1:12 AM, Denny Ye <de...@gmail.com> wrote:

> hi all,
>     In the progress of maintaining Flume NG, it's hard to manage
> configuration in too many flows.
>     Meanwhile, most of such configurations are similar. Maybe tiny
> difference in component name;
>
>     Thus, I would like to generate configuration with Java code in simple
> method.
>     I created and tested tool to generate configuration. That runs
> regularly.
>
>     Could anyone give your tips or advice? If tool like this is useful for
> you, I will submit it to community.
>     Wish your attention, thanks!
>
>
>      Java Code Example:
>                 Agent agent = new Agent().setName("agent");
>  Source source = new Source("scribe")
> .setProperty("port", "1499")
> .setType(SourceType.OTHER, "org.apache.flume.source.scribe.ScribeSource");
> agent.registerSource(source);
>  for (int i = 0; i < 5; i++) {
> Channel channel = new Channel("mc" + i)
>       .setProperty("capacity", "1000000")
>       .setType(ChannelType.MEMORY, null);
> agent.registerChannel(channel);
>  Sink sink = new Sink("hfds" + i)
>       .setType(SinkType.HDFS, null)
>       .setProperty("hfds.txtEventMax", "1000")
>        .setProperty("hdfs.path", "/flume_test/data" + i)
>        .connectChannel(channel);
> agent.registerSink(sink);
>  source.connectChannel(channel);
> }
>
>                 ConfGenerator.generateConf(agent, new
> File("d:\\flume.conf"));
>
> -Regards
> Denny Ye
>



-- 
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/