You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by "Denny Ye (JIRA)" <ji...@apache.org> on 2012/07/30 03:23:35 UTC
[jira] [Created] (FLUME-1405) Tools to generate configuration file
Denny Ye created FLUME-1405:
-------------------------------
Summary: Tools to generate configuration file
Key: FLUME-1405
URL: https://issues.apache.org/jira/browse/FLUME-1405
Project: Flume
Issue Type: Improvement
Components: Configuration
Affects Versions: v1.2.0
Reporter: Denny Ye
Assignee: Denny Ye
Priority: Minor
Fix For: v1.3.0
If there are several flows in Flume, the configuration file may be hard to maintain. Even there is tiny difference.
Under this condition, I would like to create configuration file with Java code. It's easy to build concrete file with loop and combination.
Code example :
Agent agent = new Agent().setName("agent");
Source source = new Source("scribe")
.setProperty("port", "1499")
.setType(SourceType.OTHER,"org.apache.flume.source.scribe.ScribeSource");
agent.registerSource(source);
for (int i = 0; i < 5; i++) {
Channel channel = new Channel("mc" + i)
.setProperty("capacity", "1000000")
.setType(ChannelType.MEMORY, null);
agent.registerChannel(channel);
Sink sink = new Sink("hfds" + i)
.setType(SinkType.HDFS, null)
.setProperty("hfds.txtEventMax", "1000")
.setProperty("hdfs.path", "/flume_test/data" + i)
.connectChannel(channel);
agent.registerSink(sink);
source.connectChannel(channel);
}
ConfGenerator.generateConf(agent, new File("XXXXX"));
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-1405) Tools to generate configuration file
Posted by "Brock Noland (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brock Noland updated FLUME-1405:
--------------------------------
Fix Version/s: (was: v1.3.0)
v1.4.0
> Tools to generate configuration file
> ------------------------------------
>
> Key: FLUME-1405
> URL: https://issues.apache.org/jira/browse/FLUME-1405
> Project: Flume
> Issue Type: Improvement
> Components: Configuration
> Affects Versions: v1.2.0
> Reporter: Denny Ye
> Assignee: Denny Ye
> Priority: Minor
> Fix For: v1.4.0
>
> Attachments: FLUME-1405.patch
>
>
> If there are several flows in Flume, the configuration file may be hard to maintain. Even there is tiny difference.
> Under this condition, I would like to create configuration file with Java code. It's easy to build concrete file with loop and combination.
> Code example :
> Agent agent = new Agent().setName("agent");
>
> Source source = new Source("scribe")
> .setProperty("port", "1499")
> .setType(SourceType.OTHER,"org.apache.flume.source.scribe.ScribeSource");
>
> agent.registerSource(source);
>
> for (int i = 0; i < 5; i++) {
> Channel channel = new Channel("mc" + i)
> .setProperty("capacity", "1000000")
> .setType(ChannelType.MEMORY, null);
>
> agent.registerChannel(channel);
>
> Sink sink = new Sink("hfds" + i)
> .setType(SinkType.HDFS, null)
> .setProperty("hfds.txtEventMax", "1000")
> .setProperty("hdfs.path", "/flume_test/data" + i)
> .connectChannel(channel);
> agent.registerSink(sink);
> source.connectChannel(channel);
> }
> ConfGenerator.generateConf(agent, new File("XXXXX"));
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1405) Tools to generate configuration
file
Posted by "Brock Noland (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13431953#comment-13431953 ]
Brock Noland commented on FLUME-1405:
-------------------------------------
Does anyone else have an opinion on this? Personally I had writing the configuration files so I am inclined to have something like this but if no one else is going to use it, we might as well keep the source out of the main tree just use the tool amongst ourselves.
> Tools to generate configuration file
> ------------------------------------
>
> Key: FLUME-1405
> URL: https://issues.apache.org/jira/browse/FLUME-1405
> Project: Flume
> Issue Type: Improvement
> Components: Configuration
> Affects Versions: v1.2.0
> Reporter: Denny Ye
> Assignee: Denny Ye
> Priority: Minor
> Fix For: v1.3.0
>
> Attachments: FLUME-1405.patch
>
>
> If there are several flows in Flume, the configuration file may be hard to maintain. Even there is tiny difference.
> Under this condition, I would like to create configuration file with Java code. It's easy to build concrete file with loop and combination.
> Code example :
> Agent agent = new Agent().setName("agent");
>
> Source source = new Source("scribe")
> .setProperty("port", "1499")
> .setType(SourceType.OTHER,"org.apache.flume.source.scribe.ScribeSource");
>
> agent.registerSource(source);
>
> for (int i = 0; i < 5; i++) {
> Channel channel = new Channel("mc" + i)
> .setProperty("capacity", "1000000")
> .setType(ChannelType.MEMORY, null);
>
> agent.registerChannel(channel);
>
> Sink sink = new Sink("hfds" + i)
> .setType(SinkType.HDFS, null)
> .setProperty("hfds.txtEventMax", "1000")
> .setProperty("hdfs.path", "/flume_test/data" + i)
> .connectChannel(channel);
> agent.registerSink(sink);
> source.connectChannel(channel);
> }
> ConfGenerator.generateConf(agent, new File("XXXXX"));
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (FLUME-1405) Tools to generate configuration file
Posted by "Denny Ye (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FLUME-1405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Denny Ye updated FLUME-1405:
----------------------------
Attachment: FLUME-1405.patch
> Tools to generate configuration file
> ------------------------------------
>
> Key: FLUME-1405
> URL: https://issues.apache.org/jira/browse/FLUME-1405
> Project: Flume
> Issue Type: Improvement
> Components: Configuration
> Affects Versions: v1.2.0
> Reporter: Denny Ye
> Assignee: Denny Ye
> Priority: Minor
> Fix For: v1.3.0
>
> Attachments: FLUME-1405.patch
>
>
> If there are several flows in Flume, the configuration file may be hard to maintain. Even there is tiny difference.
> Under this condition, I would like to create configuration file with Java code. It's easy to build concrete file with loop and combination.
> Code example :
> Agent agent = new Agent().setName("agent");
>
> Source source = new Source("scribe")
> .setProperty("port", "1499")
> .setType(SourceType.OTHER,"org.apache.flume.source.scribe.ScribeSource");
>
> agent.registerSource(source);
>
> for (int i = 0; i < 5; i++) {
> Channel channel = new Channel("mc" + i)
> .setProperty("capacity", "1000000")
> .setType(ChannelType.MEMORY, null);
>
> agent.registerChannel(channel);
>
> Sink sink = new Sink("hfds" + i)
> .setType(SinkType.HDFS, null)
> .setProperty("hfds.txtEventMax", "1000")
> .setProperty("hdfs.path", "/flume_test/data" + i)
> .connectChannel(channel);
> agent.registerSink(sink);
> source.connectChannel(channel);
> }
> ConfGenerator.generateConf(agent, new File("XXXXX"));
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira