You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by John Fang <xi...@alibaba-inc.com> on 2015/11/16 06:03:41 UTC

答复: load configurations to topology

As we know, the priority order of the configuration is :
CommandLine > user stormconf > storm.yaml > defaults.yaml

-----邮件原件-----
发件人: jinhong lu [mailto:lujinhong2@gmail.com] 
发送时间: 2015年11月16日 12:57
收件人: dev@storm.apache.org
主题: load configurations to topology


Hi, I do not understand the order of the configuration loading in
submitToplogy:

public static void submitTopologyAs(String name, Map stormConf,
StormTopology topology, SubmitOptions opts, ProgressListener
progressListener, String asUser)
            throws AlreadyAliveException, InvalidTopologyException,
AuthorizationException {
        if(!Utils.isValidConf(stormConf)) {
            throw new IllegalArgumentException("Storm conf is not valid.
Must be json-serializable");
        }
        stormConf = new HashMap(stormConf);
        stormConf.putAll(Utils.readCommandLineOpts());
        Map conf = Utils.readStormConfig();
        conf.putAll(stormConf);
 
stormConf.putAll(prepareZookeeperAuthentication(conf));readStormConfig();
        conf.putAll(stormConf);

It seems stormConf load the configuration from user stormConf, then
commandLine, after that, defaults.yaml and storm.yaml.
Isn’t the yaml files will overwrite the configuration in user stormConf and
commandLine?

Thanks.

Re: 答复: load configurations to topology

Posted by Bobby Evans <ev...@yahoo-inc.com.INVALID>.
first of all the code looks like it was copied and pasted oddly
https://github.com/apache/storm/blob/master/storm-core/src/jvm/backtype/storm/StormSubmitter.java#L188-L199
```
public static void submitTopologyAs(String name, Map stormConf, StormTopology topology, SubmitOptions opts, ProgressListener progressListener, String asUser)
        throws AlreadyAliveException, InvalidTopologyException, AuthorizationException, IllegalArgumentException {
    if(!Utils.isValidConf(stormConf)) {
        throw new IllegalArgumentException("Storm conf is not valid. Must be json-serializable");
    }
    stormConf = new HashMap(stormConf);
    stormConf.putAll(Utils.readCommandLineOpts());
    Map conf = Utils.readStormConfig();//storm.yaml + defaults.yaml
    conf.putAll(stormConf);
    stormConf.putAll(prepareZookeeperAuthentication(conf));

    validateConfs(conf, topology);```

There are two configs here.  stormConf and conf.  conf is a full configuration object with CommandLine + user-conf + storm.yaml + defaults.yaml.  This is used for establishing the connection to nimbus.  stormConf is just (ZK Auth Token) + CommandLine + user-conf.  It does not include storm.yaml or defaults.yaml because the assumption is that they are the same everywhere in the cluster, or if they are not, the one configured on nimbus/supervisors should override the values configured on whatever gateway you are running on.
- Bobby 


     On Sunday, November 15, 2015 11:04 PM, John Fang <xi...@alibaba-inc.com> wrote:
   

 As we know, the priority order of the configuration is :
CommandLine > user stormconf > storm.yaml > defaults.yaml

-----邮件原件-----
发件人: jinhong lu [mailto:lujinhong2@gmail.com] 
发送时间: 2015年11月16日 12:57
收件人: dev@storm.apache.org
主题: load configurations to topology


Hi, I do not understand the order of the configuration loading in
submitToplogy:

public static void submitTopologyAs(String name, Map stormConf,
StormTopology topology, SubmitOptions opts, ProgressListener
progressListener, String asUser)
            throws AlreadyAliveException, InvalidTopologyException,
AuthorizationException {
        if(!Utils.isValidConf(stormConf)) {
            throw new IllegalArgumentException("Storm conf is not valid.
Must be json-serializable");
        }
        stormConf = new HashMap(stormConf);
        stormConf.putAll(Utils.readCommandLineOpts());
        Map conf = Utils.readStormConfig();
        conf.putAll(stormConf);
 
stormConf.putAll(prepareZookeeperAuthentication(conf));readStormConfig();
        conf.putAll(stormConf);

It seems stormConf load the configuration from user stormConf, then
commandLine, after that, defaults.yaml and storm.yaml.
Isn’t the yaml files will overwrite the configuration in user stormConf and
commandLine?

Thanks.