You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@eagle.apache.org by wujinhu <gi...@git.apache.org> on 2016/09/09 06:51:45 UTC

[GitHub] incubator-eagle pull request #430: [EAGLE-526] aggregation framework for his...

GitHub user wujinhu opened a pull request:

    https://github.com/apache/incubator-eagle/pull/430

    [EAGLE-526] aggregation framework for history jobs

    

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/wujinhu/incubator-eagle EAGLE-526

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-eagle/pull/430.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #430
    
----
commit 9f37a6a8f65b3c0a5a2c543831a2b0463f060d98
Author: wujinhu <wu...@126.com>
Date:   2016-09-09T02:22:02Z

    aggregation framework

commit f438bef6cf8710b226a14ea78f4111d4338b2dda
Author: wujinhu <wu...@126.com>
Date:   2016-09-09T02:53:46Z

    Merge branch 'master' of https://github.com/apache/incubator-eagle into EAGLE-526
    
    # Conflicts:
    #	eagle-core/eagle-app/eagle-app-base/src/main/java/org/apache/eagle/app/environment/impl/StormExecutionRuntime.java

commit eb3ccf60e100e5e1ef0e62a0091d7aa535713474
Author: wujinhu <wu...@126.com>
Date:   2016-09-09T06:50:54Z

    Merge branch 'master' of https://github.com/apache/incubator-eagle into EAGLE-526
    
    # Conflicts:
    #	eagle-core/eagle-app/eagle-app-base/src/main/java/org/apache/eagle/app/environment/impl/StormExecutionRuntime.java

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-eagle pull request #430: [EAGLE-526] aggregation framework for his...

Posted by haoch <gi...@git.apache.org>.
Github user haoch commented on a diff in the pull request:

    https://github.com/apache/incubator-eagle/pull/430#discussion_r78320618
  
    --- Diff: eagle-jpm/eagle-jpm-aggregation/src/main/java/org/apache/eagle/jpm/aggregation/storm/AggregationSpout.java ---
    @@ -64,7 +64,7 @@ public void open(Map conf, TopologyContext context, final SpoutOutputCollector c
         @Override
         public void nextTuple() {
             Utils.sleep(5);
    -        LOG.info("start to run");
    +        LOG.debug("start to run");
    --- End diff --
    
    if(LOG.isDebugEnabled())


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-eagle pull request #430: [EAGLE-526] aggregation framework for his...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/incubator-eagle/pull/430


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-eagle pull request #430: [EAGLE-526] aggregation framework for his...

Posted by haoch <gi...@git.apache.org>.
Github user haoch commented on a diff in the pull request:

    https://github.com/apache/incubator-eagle/pull/430#discussion_r78320310
  
    --- Diff: eagle-jpm/eagle-jpm-aggregation/src/main/java/org/apache/eagle/jpm/aggregation/storm/AggregationSpout.java ---
    @@ -0,0 +1,137 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    +*/
    +
    +package org.apache.eagle.jpm.aggregation.storm;
    +
    +import org.apache.eagle.jpm.aggregation.AggregationConfig;
    +import org.apache.eagle.jpm.aggregation.common.MetricsAggregateContainer;
    +import org.apache.eagle.jpm.aggregation.state.AggregationTimeManager;
    +import org.apache.eagle.jpm.util.Utils;
    +import backtype.storm.spout.SpoutOutputCollector;
    +import backtype.storm.task.TopologyContext;
    +import backtype.storm.topology.OutputFieldsDeclarer;
    +import backtype.storm.topology.base.BaseRichSpout;
    +import backtype.storm.tuple.Fields;
    +import backtype.storm.tuple.Values;
    +import com.typesafe.config.Config;
    +import org.slf4j.Logger;
    +import org.slf4j.LoggerFactory;
    +
    +import java.util.HashSet;
    +import java.util.Map;
    +import java.util.Set;
    +
    +public class AggregationSpout extends BaseRichSpout {
    +    private static final Logger LOG = LoggerFactory.getLogger(AggregationSpout.class);
    +    private static final Long MAX_WAIT_TIME = 12 * 60 * 60000L;//12 hours
    +    private static final Long MAX_SAFE_TIME = 6 * 60 * 60000L;//6 hours
    +
    +    private Config config;
    +    MetricsAggregateContainer jobProcessTime;
    +
    +    private SpoutOutputCollector collector;
    +    private Set<Long> processStartTime;
    +    private Long lastUpdateTime;
    +
    +    public AggregationSpout(Config config, MetricsAggregateContainer jobProcessTime) {
    +        this.config = config;
    +        this.jobProcessTime = jobProcessTime;
    +        this.processStartTime = new HashSet<>();
    +    }
    +
    +    @Override
    +    public void open(Map conf, TopologyContext context, final SpoutOutputCollector collector) {
    +        AggregationConfig.getInstance(config);
    +        AggregationTimeManager.instance().init(AggregationConfig.get().getZkStateConfig());
    +        this.collector = collector;
    +    }
    +
    +    @Override
    +    public void nextTuple() {
    +        Utils.sleep(5);
    +        LOG.info("start to run");
    --- End diff --
    
    LOG.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-eagle pull request #430: [EAGLE-526] aggregation framework for his...

Posted by haoch <gi...@git.apache.org>.
Github user haoch commented on a diff in the pull request:

    https://github.com/apache/incubator-eagle/pull/430#discussion_r78320342
  
    --- Diff: eagle-jpm/eagle-jpm-aggregation/src/main/java/org/apache/eagle/jpm/aggregation/storm/AggregationSpout.java ---
    @@ -0,0 +1,137 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    +*/
    +
    +package org.apache.eagle.jpm.aggregation.storm;
    +
    +import org.apache.eagle.jpm.aggregation.AggregationConfig;
    +import org.apache.eagle.jpm.aggregation.common.MetricsAggregateContainer;
    +import org.apache.eagle.jpm.aggregation.state.AggregationTimeManager;
    +import org.apache.eagle.jpm.util.Utils;
    +import backtype.storm.spout.SpoutOutputCollector;
    +import backtype.storm.task.TopologyContext;
    +import backtype.storm.topology.OutputFieldsDeclarer;
    +import backtype.storm.topology.base.BaseRichSpout;
    +import backtype.storm.tuple.Fields;
    +import backtype.storm.tuple.Values;
    +import com.typesafe.config.Config;
    +import org.slf4j.Logger;
    +import org.slf4j.LoggerFactory;
    +
    +import java.util.HashSet;
    +import java.util.Map;
    +import java.util.Set;
    +
    +public class AggregationSpout extends BaseRichSpout {
    +    private static final Logger LOG = LoggerFactory.getLogger(AggregationSpout.class);
    +    private static final Long MAX_WAIT_TIME = 12 * 60 * 60000L;//12 hours
    +    private static final Long MAX_SAFE_TIME = 6 * 60 * 60000L;//6 hours
    +
    +    private Config config;
    +    MetricsAggregateContainer jobProcessTime;
    +
    +    private SpoutOutputCollector collector;
    +    private Set<Long> processStartTime;
    +    private Long lastUpdateTime;
    +
    +    public AggregationSpout(Config config, MetricsAggregateContainer jobProcessTime) {
    +        this.config = config;
    +        this.jobProcessTime = jobProcessTime;
    +        this.processStartTime = new HashSet<>();
    +    }
    +
    +    @Override
    +    public void open(Map conf, TopologyContext context, final SpoutOutputCollector collector) {
    +        AggregationConfig.getInstance(config);
    +        AggregationTimeManager.instance().init(AggregationConfig.get().getZkStateConfig());
    +        this.collector = collector;
    +    }
    +
    +    @Override
    +    public void nextTuple() {
    +        Utils.sleep(5);
    +        LOG.info("start to run");
    +        try {
    +            //0, check processStartTime size
    +            if (this.processStartTime.size() > 0) {
    +                //bolts are processing
    +                return;
    +            }
    +
    +            long currentJobTime = System.currentTimeMillis();//this.jobProcessTime.fetchLatestJobProcessTime();
    +            //1, get last updateTime;
    +            lastUpdateTime = AggregationTimeManager.instance().readLastFinishTime();
    +            if (lastUpdateTime == 0L) {
    +                //init state, just set to currentTime - 18 hours
    +                lastUpdateTime = (currentJobTime - (MAX_SAFE_TIME + MAX_WAIT_TIME)) / 3600000 * 3600000;
    +            }
    +
    +            if (currentJobTime - lastUpdateTime < MAX_WAIT_TIME) {
    +                LOG.info("wait for max wait time, remaining {}ms", MAX_WAIT_TIME - currentJobTime + lastUpdateTime);
    +                return;
    +            }
    +
    +            for (Long startTime = lastUpdateTime; startTime < lastUpdateTime + MAX_SAFE_TIME;) {
    +                collector.emit(new Values(startTime), startTime);
    +                this.processStartTime.add(startTime);
    +                startTime += AggregationConfig.get().getJobExtractorConfig().aggregationDuration * 1000;
    +            }
    +        } catch (Exception e) {
    +            LOG.warn("{}", e);
    +        }
    +    }
    +
    +    @Override
    +    public void declareOutputFields(OutputFieldsDeclarer declarer) {
    +        declarer.declare(new Fields("startTime"));
    +    }
    +
    +    @Override
    +    public void ack(Object messageId) {
    +        //remove from processStartTime, if size == 0, then update lastUpdateTime
    +        Long startTime = (Long)messageId;
    +        LOG.info("succeed startTime {}", startTime);
    --- End diff --
    
    debug log


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-eagle pull request #430: [EAGLE-526] aggregation framework for his...

Posted by haoch <gi...@git.apache.org>.
Github user haoch commented on a diff in the pull request:

    https://github.com/apache/incubator-eagle/pull/430#discussion_r78320255
  
    --- Diff: eagle-jpm/eagle-jpm-aggregation/src/main/java/org/apache/eagle/jpm/aggregation/storm/AggregationBolt.java ---
    @@ -0,0 +1,69 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one
    + * or more contributor license agreements.  See the NOTICE file
    + * distributed with this work for additional information
    + * regarding copyright ownership.  The ASF licenses this file
    + * to you under the Apache License, Version 2.0 (the
    + * "License"); you may not use this file except in compliance
    + * with the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    +*/
    +
    +package org.apache.eagle.jpm.aggregation.storm;
    +
    +import org.apache.eagle.jpm.aggregation.AggregationConfig;
    +import org.apache.eagle.jpm.aggregation.common.MetricsAggregateContainer;
    +import backtype.storm.task.OutputCollector;
    +import backtype.storm.task.TopologyContext;
    +import backtype.storm.topology.OutputFieldsDeclarer;
    +import backtype.storm.topology.base.BaseRichBolt;
    +import backtype.storm.tuple.Tuple;
    +import com.typesafe.config.Config;
    +import org.slf4j.Logger;
    +import org.slf4j.LoggerFactory;
    +
    +import java.util.Map;
    +
    +public class AggregationBolt extends BaseRichBolt {
    +    private static final Logger LOG = LoggerFactory.getLogger(AggregationBolt.class);
    +    private Config config;
    +    private OutputCollector collector;
    +    private MetricsAggregateContainer metricsAggregateContainer;
    +
    +    public AggregationBolt(Config config, MetricsAggregateContainer metricsAggregateContainer) {
    +        this.config = config;
    +        this.metricsAggregateContainer = metricsAggregateContainer;
    +    }
    +
    +    @Override
    +    public void prepare(Map map, TopologyContext topologyContext, OutputCollector outputCollector) {
    +        AggregationConfig.getInstance(config);
    +        this.collector = outputCollector;
    +    }
    +
    +    @Override
    +    public void execute(Tuple tuple) {
    +        Long startTime = tuple.getLongByField("startTime");
    +        LOG.info("get startTime {}", startTime);
    --- End diff --
    
    Don't log per event.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-eagle issue #430: [EAGLE-526] aggregation framework for history jo...

Posted by yonzhang <gi...@git.apache.org>.
Github user yonzhang commented on the issue:

    https://github.com/apache/incubator-eagle/pull/430
  
    @wujinhu can you please add description for people to understand the changes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-eagle pull request #430: [EAGLE-526] aggregation framework for his...

Posted by haoch <gi...@git.apache.org>.
Github user haoch commented on a diff in the pull request:

    https://github.com/apache/incubator-eagle/pull/430#discussion_r78320622
  
    --- Diff: eagle-jpm/eagle-jpm-aggregation/src/main/java/org/apache/eagle/jpm/aggregation/storm/AggregationSpout.java ---
    @@ -104,7 +104,7 @@ public void declareOutputFields(OutputFieldsDeclarer declarer) {
         public void ack(Object messageId) {
             //remove from processStartTime, if size == 0, then update lastUpdateTime
             Long startTime = (Long)messageId;
    -        LOG.info("succeed startTime {}", startTime);
    +        LOG.debug("succeed startTime {}", startTime);
    --- End diff --
    
    if(LOG.isDebugEnabled())


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-eagle pull request #430: [EAGLE-526] aggregation framework for his...

Posted by haoch <gi...@git.apache.org>.
Github user haoch commented on a diff in the pull request:

    https://github.com/apache/incubator-eagle/pull/430#discussion_r78320627
  
    --- Diff: eagle-jpm/eagle-jpm-aggregation/src/main/java/org/apache/eagle/jpm/aggregation/storm/AggregationSpout.java ---
    @@ -123,7 +123,7 @@ public void ack(Object messageId) {
         @Override
         public void fail(Object messageId) {
             Long startTime = (Long)messageId;
    -        LOG.info("failed startTime {}", startTime);
    +        LOG.warn("failed startTime {}", startTime);
    --- End diff --
    
    if(LOG.isDebugEnabled())


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-eagle issue #430: [EAGLE-526] aggregation framework for history jo...

Posted by haoch <gi...@git.apache.org>.
Github user haoch commented on the issue:

    https://github.com/apache/incubator-eagle/pull/430
  
    LGTM


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-eagle issue #430: [EAGLE-526] aggregation framework for history jo...

Posted by wujinhu <gi...@git.apache.org>.
Github user wujinhu commented on the issue:

    https://github.com/apache/incubator-eagle/pull/430
  
    Please review https://github.com/apache/incubator-eagle/pull/419 first.
    
    This is the second step of the aggregation framework. This step uses results of the job metrics by each minute that generated by the first step. 
    
    The framework is a storm topology. The spout decides which time range of job metrics needs to be aggregated, divide the time range into several pieces(For example, each piece is one hour) and emits these pieces to aggregate bolts.
    The aggregate bolts aggregate the time range piece of job metrics that the spout assigns to them.
    In aggregate bolts, they fetch job metrics by eagle server api and write the aggregate results back to eagle server.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---