You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by Denny Ye <de...@gmail.com> on 2012/07/30 12:17:32 UTC
Review Request: Tools to generate configuration file
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/6206/
-----------------------------------------------------------
Review request for Flume and Brock Noland.
Description
-------
Here is the tool for generating Flume configuration file with Java code.
This addresses bug https://issues.apache.org/jira/browse/FLUME-1405.
https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/FLUME-1405
Diffs
-----
flume/trunk/flume-ng-configuration/src/main/java/org/apache/flume/conf/gen/AbstractComponent.java PRE-CREATION
flume/trunk/flume-ng-configuration/src/main/java/org/apache/flume/conf/gen/Agent.java PRE-CREATION
flume/trunk/flume-ng-configuration/src/main/java/org/apache/flume/conf/gen/Channel.java PRE-CREATION
flume/trunk/flume-ng-configuration/src/main/java/org/apache/flume/conf/gen/ConfGenerator.java PRE-CREATION
flume/trunk/flume-ng-configuration/src/main/java/org/apache/flume/conf/gen/Interceptor.java PRE-CREATION
flume/trunk/flume-ng-configuration/src/main/java/org/apache/flume/conf/gen/NameAvailable.java PRE-CREATION
flume/trunk/flume-ng-configuration/src/main/java/org/apache/flume/conf/gen/Recorder.java PRE-CREATION
flume/trunk/flume-ng-configuration/src/main/java/org/apache/flume/conf/gen/Sink.java PRE-CREATION
flume/trunk/flume-ng-configuration/src/main/java/org/apache/flume/conf/gen/SinkGroup.java PRE-CREATION
flume/trunk/flume-ng-configuration/src/main/java/org/apache/flume/conf/gen/Source.java PRE-CREATION
flume/trunk/flume-ng-configuration/src/main/java/org/apache/flume/conf/gen/Validation.java PRE-CREATION
Diff: https://reviews.apache.org/r/6206/diff/
Testing
-------
Test example:
Agent agent = new Agent().setName("agent");
Source source = new Source("scribe")
.setProperty("port", "1499")
.setType(SourceType.OTHER,"org.apache.flume.source.scribe.ScribeSource");
agent.registerSource(source);
for (int i = 0; i < 5; i++) {
Channel channel = new Channel("mc" + i)
.setProperty("capacity", "1000000")
.setType(ChannelType.MEMORY, null);
agent.registerChannel(channel);
Sink sink = new Sink("hfds" + i)
.setType(SinkType.HDFS, null)
.setProperty("hfds.txtEventMax", "1000")
.setProperty("hdfs.path", "/flume_test/data" + i)
.connectChannel(channel);
agent.registerSink(sink);
source.connectChannel(channel);
}
ConfGenerator.generateConf(agent, new File("XXXXX"));
Thanks,
Denny Ye
Re: Review Request: Tools to generate configuration file
Posted by Brock Noland <br...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/6206/#review10070
-----------------------------------------------------------
The formatting is off, can you reformat the patch? I'd like to get some input from other devs to see if this is useful. Personally I had writing the configuration files so I am inclined to have something like this.
flume/trunk/flume-ng-configuration/src/main/java/org/apache/flume/conf/gen/ConfGenerator.java
<https://reviews.apache.org/r/6206/#comment21330>
probably should be thrown
- Brock Noland
On Aug. 6, 2012, 6:56 a.m., Denny Ye wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/6206/
> -----------------------------------------------------------
>
> (Updated Aug. 6, 2012, 6:56 a.m.)
>
>
> Review request for Flume and Brock Noland.
>
>
> Description
> -------
>
> Here is the tool for generating Flume configuration file with Java code.
>
>
> This addresses bug FLUME-1405.
> https://issues.apache.org/jira/browse/FLUME-1405
>
>
> Diffs
> -----
>
> flume/trunk/flume-ng-configuration/src/main/java/org/apache/flume/conf/gen/AbstractComponent.java PRE-CREATION
> flume/trunk/flume-ng-configuration/src/main/java/org/apache/flume/conf/gen/Agent.java PRE-CREATION
> flume/trunk/flume-ng-configuration/src/main/java/org/apache/flume/conf/gen/Channel.java PRE-CREATION
> flume/trunk/flume-ng-configuration/src/main/java/org/apache/flume/conf/gen/ConfGenerator.java PRE-CREATION
> flume/trunk/flume-ng-configuration/src/main/java/org/apache/flume/conf/gen/Interceptor.java PRE-CREATION
> flume/trunk/flume-ng-configuration/src/main/java/org/apache/flume/conf/gen/NameAvailable.java PRE-CREATION
> flume/trunk/flume-ng-configuration/src/main/java/org/apache/flume/conf/gen/Recorder.java PRE-CREATION
> flume/trunk/flume-ng-configuration/src/main/java/org/apache/flume/conf/gen/Sink.java PRE-CREATION
> flume/trunk/flume-ng-configuration/src/main/java/org/apache/flume/conf/gen/SinkGroup.java PRE-CREATION
> flume/trunk/flume-ng-configuration/src/main/java/org/apache/flume/conf/gen/Source.java PRE-CREATION
> flume/trunk/flume-ng-configuration/src/main/java/org/apache/flume/conf/gen/Validation.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/6206/diff/
>
>
> Testing
> -------
>
> Test example:
>
> Agent agent = new Agent().setName("agent");
>
> Source source = new Source("scribe")
> .setProperty("port", "1499")
> .setType(SourceType.OTHER,"org.apache.flume.source.scribe.ScribeSource");
>
> agent.registerSource(source);
>
> for (int i = 0; i < 5; i++) {
> Channel channel = new Channel("mc" + i)
> .setProperty("capacity", "1000000")
> .setType(ChannelType.MEMORY, null);
>
> agent.registerChannel(channel);
>
> Sink sink = new Sink("hfds" + i)
> .setType(SinkType.HDFS, null)
> .setProperty("hfds.txtEventMax", "1000")
> .setProperty("hdfs.path", "/flume_test/data" + i)
> .connectChannel(channel);
>
> agent.registerSink(sink);
> source.connectChannel(channel);
> }
>
> ConfGenerator.generateConf(agent, new File("XXXXX"));
>
>
> Thanks,
>
> Denny Ye
>
>
Re: Review Request: Tools to generate configuration file
Posted by Denny Ye <de...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/6206/
-----------------------------------------------------------
(Updated Aug. 6, 2012, 6:56 a.m.)
Review request for Flume and Brock Noland.
Changes
-------
Find the original bug
Description
-------
Here is the tool for generating Flume configuration file with Java code.
This addresses bug FLUME-1405.
https://issues.apache.org/jira/browse/FLUME-1405
Diffs
-----
flume/trunk/flume-ng-configuration/src/main/java/org/apache/flume/conf/gen/AbstractComponent.java PRE-CREATION
flume/trunk/flume-ng-configuration/src/main/java/org/apache/flume/conf/gen/Agent.java PRE-CREATION
flume/trunk/flume-ng-configuration/src/main/java/org/apache/flume/conf/gen/Channel.java PRE-CREATION
flume/trunk/flume-ng-configuration/src/main/java/org/apache/flume/conf/gen/ConfGenerator.java PRE-CREATION
flume/trunk/flume-ng-configuration/src/main/java/org/apache/flume/conf/gen/Interceptor.java PRE-CREATION
flume/trunk/flume-ng-configuration/src/main/java/org/apache/flume/conf/gen/NameAvailable.java PRE-CREATION
flume/trunk/flume-ng-configuration/src/main/java/org/apache/flume/conf/gen/Recorder.java PRE-CREATION
flume/trunk/flume-ng-configuration/src/main/java/org/apache/flume/conf/gen/Sink.java PRE-CREATION
flume/trunk/flume-ng-configuration/src/main/java/org/apache/flume/conf/gen/SinkGroup.java PRE-CREATION
flume/trunk/flume-ng-configuration/src/main/java/org/apache/flume/conf/gen/Source.java PRE-CREATION
flume/trunk/flume-ng-configuration/src/main/java/org/apache/flume/conf/gen/Validation.java PRE-CREATION
Diff: https://reviews.apache.org/r/6206/diff/
Testing
-------
Test example:
Agent agent = new Agent().setName("agent");
Source source = new Source("scribe")
.setProperty("port", "1499")
.setType(SourceType.OTHER,"org.apache.flume.source.scribe.ScribeSource");
agent.registerSource(source);
for (int i = 0; i < 5; i++) {
Channel channel = new Channel("mc" + i)
.setProperty("capacity", "1000000")
.setType(ChannelType.MEMORY, null);
agent.registerChannel(channel);
Sink sink = new Sink("hfds" + i)
.setType(SinkType.HDFS, null)
.setProperty("hfds.txtEventMax", "1000")
.setProperty("hdfs.path", "/flume_test/data" + i)
.connectChannel(channel);
agent.registerSink(sink);
source.connectChannel(channel);
}
ConfGenerator.generateConf(agent, new File("XXXXX"));
Thanks,
Denny Ye
Re: Review Request: Tools to generate configuration file
Posted by Denny Ye <de...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/6206/
-----------------------------------------------------------
(Updated July 31, 2012, 1:08 a.m.)
Review request for Flume and Brock Noland.
Changes
-------
Updated two classes structures to avoid invalid source code description
Description
-------
Here is the tool for generating Flume configuration file with Java code.
This addresses bug https://issues.apache.org/jira/browse/FLUME-1405.
https://issues.apache.org/jira/browse/https://issues.apache.org/jira/browse/FLUME-1405
Diffs (updated)
-----
flume/trunk/flume-ng-configuration/src/main/java/org/apache/flume/conf/gen/AbstractComponent.java PRE-CREATION
flume/trunk/flume-ng-configuration/src/main/java/org/apache/flume/conf/gen/Agent.java PRE-CREATION
flume/trunk/flume-ng-configuration/src/main/java/org/apache/flume/conf/gen/Channel.java PRE-CREATION
flume/trunk/flume-ng-configuration/src/main/java/org/apache/flume/conf/gen/ConfGenerator.java PRE-CREATION
flume/trunk/flume-ng-configuration/src/main/java/org/apache/flume/conf/gen/Interceptor.java PRE-CREATION
flume/trunk/flume-ng-configuration/src/main/java/org/apache/flume/conf/gen/NameAvailable.java PRE-CREATION
flume/trunk/flume-ng-configuration/src/main/java/org/apache/flume/conf/gen/Recorder.java PRE-CREATION
flume/trunk/flume-ng-configuration/src/main/java/org/apache/flume/conf/gen/Sink.java PRE-CREATION
flume/trunk/flume-ng-configuration/src/main/java/org/apache/flume/conf/gen/SinkGroup.java PRE-CREATION
flume/trunk/flume-ng-configuration/src/main/java/org/apache/flume/conf/gen/Source.java PRE-CREATION
flume/trunk/flume-ng-configuration/src/main/java/org/apache/flume/conf/gen/Validation.java PRE-CREATION
Diff: https://reviews.apache.org/r/6206/diff/
Testing
-------
Test example:
Agent agent = new Agent().setName("agent");
Source source = new Source("scribe")
.setProperty("port", "1499")
.setType(SourceType.OTHER,"org.apache.flume.source.scribe.ScribeSource");
agent.registerSource(source);
for (int i = 0; i < 5; i++) {
Channel channel = new Channel("mc" + i)
.setProperty("capacity", "1000000")
.setType(ChannelType.MEMORY, null);
agent.registerChannel(channel);
Sink sink = new Sink("hfds" + i)
.setType(SinkType.HDFS, null)
.setProperty("hfds.txtEventMax", "1000")
.setProperty("hdfs.path", "/flume_test/data" + i)
.connectChannel(channel);
agent.registerSink(sink);
source.connectChannel(channel);
}
ConfGenerator.generateConf(agent, new File("XXXXX"));
Thanks,
Denny Ye