You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@eagle.apache.org by pkuwm <gi...@git.apache.org> on 2016/08/23 01:25:41 UTC
[GitHub] incubator-eagle pull request #375: [EAGLE-492] Fix negative total execution ...
GitHub user pkuwm opened a pull request:
https://github.com/apache/incubator-eagle/pull/375
[EAGLE-492] Fix negative total execution time and make some code opti\u2026
\u2026mization for spark history job.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/pkuwm/incubator-eagle EAGLE-492
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-eagle/pull/375.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #375
----
commit cbf4cf7585f66d9cdb5cbd26b9832b529d2852fa
Author: pkuwm <ih...@gmail.com>
Date: 2016-08-23T01:21:31Z
[EAGLE-492] Fix negative total execution time and make some code optimization for spark history job.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-eagle issue #375: [EAGLE-492] Fix negative total execution time an...
Posted by DadanielZ <gi...@git.apache.org>.
Github user DadanielZ commented on the issue:
https://github.com/apache/incubator-eagle/pull/375
@haoch Thank you, this is very useful \U0001f44d
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-eagle issue #375: [EAGLE-492] Fix negative total execution time an...
Posted by pkuwm <gi...@git.apache.org>.
Github user pkuwm commented on the issue:
https://github.com/apache/incubator-eagle/pull/375
It was because of the conflicts with the previous check style commit. I have fixed it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-eagle issue #375: [EAGLE-492] Fix negative total execution time an...
Posted by wujinhu <gi...@git.apache.org>.
Github user wujinhu commented on the issue:
https://github.com/apache/incubator-eagle/pull/375
Looks good to me. Please do more testing and merge then.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-eagle issue #375: [EAGLE-492] Fix negative total execution time an...
Posted by DadanielZ <gi...@git.apache.org>.
Github user DadanielZ commented on the issue:
https://github.com/apache/incubator-eagle/pull/375
merged, plz close this PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-eagle issue #375: [EAGLE-492] Fix negative total execution time an...
Posted by haoch <gi...@git.apache.org>.
Github user haoch commented on the issue:
https://github.com/apache/incubator-eagle/pull/375
@DadanielZ very glad to see you to start to review and merge contributed PR, just remind that you could also use [`eagle-dev/merge-eagle-pr.py`](https://github.com/apache/incubator-eagle/blob/develop/eagle-dev/merge-eagle-pr.py) besides manually doing it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-eagle issue #375: [EAGLE-492] Fix negative total execution time an...
Posted by DadanielZ <gi...@git.apache.org>.
Github user DadanielZ commented on the issue:
https://github.com/apache/incubator-eagle/pull/375
Patch apply failed, please fix it first.
error: patch failed: eagle-jpm/eagle-jpm-entity/src/main/java/org/apache/eagle/jpm/spark/crawl/JHFSparkEventReader.java:21
error: eagle-jpm/eagle-jpm-entity/src/main/java/org/apache/eagle/jpm/spark/crawl/JHFSparkEventReader.java: patch does not apply
error: patch failed: eagle-jpm/eagle-jpm-entity/src/main/java/org/apache/eagle/jpm/spark/crawl/JHFSparkParser.java:31
error: eagle-jpm/eagle-jpm-entity/src/main/java/org/apache/eagle/jpm/spark/crawl/JHFSparkParser.java: patch does not apply
error: patch failed: eagle-jpm/eagle-jpm-entity/src/main/java/org/apache/eagle/jpm/spark/entity/SparkStage.java:36
error: eagle-jpm/eagle-jpm-entity/src/main/java/org/apache/eagle/jpm/spark/entity/SparkStage.java: patch does not apply
error: patch failed: eagle-jpm/eagle-jpm-util/src/main/java/org/apache/eagle/jpm/util/Utils.java:68
error: eagle-jpm/eagle-jpm-util/src/main/java/org/apache/eagle/jpm/util/Utils.java: patch does not apply
error: patch failed: eagle-jpm/eagle-jpm-util/src/main/java/org/apache/eagle/jpm/util/resourceFetch/RMResourceFetcher.java:128
error: eagle-jpm/eagle-jpm-util/src/main/java/org/apache/eagle/jpm/util/resourceFetch/RMResourceFetcher.java: patch does not apply
error: patch failed: eagle-jpm/eagle-jpm-entity/src/main/java/org/apache/eagle/jpm/spark/crawl/JHFSparkParser.java:34
error: eagle-jpm/eagle-jpm-entity/src/main/java/org/apache/eagle/jpm/spark/crawl/JHFSparkParser.java: patch does not apply
error: patch failed: eagle-jpm/eagle-jpm-util/src/main/java/org/apache/eagle/jpm/util/resourceFetch/RMResourceFetcher.java:107
error: eagle-jpm/eagle-jpm-util/src/main/java/org/apache/eagle/jpm/util/resourceFetch/RMResourceFetcher.java: patch does not apply
Das-MacBook-Pro:incubator-eagle dazhou$ git apply -v 375.patch
Checking patch eagle-jpm/eagle-jpm-entity/src/main/java/org/apache/eagle/jpm/spark/crawl/JHFSparkEventReader.java...
error: while searching for:
import com.typesafe.config.ConfigFactory;
import org.apache.eagle.jpm.spark.entity.JobConfig;
import org.apache.eagle.jpm.spark.entity.*;
import org.apache.eagle.jpm.util.JSONUtil;
import org.apache.eagle.jpm.util.JobNameNormalization;
import org.apache.eagle.jpm.util.SparkEntityConstant;
import org.apache.eagle.jpm.util.SparkJobTagName;
import org.apache.eagle.log.base.taggedlog.TaggedLogAPIEntity;
import org.apache.eagle.service.client.impl.EagleServiceBaseClient;
import org.apache.eagle.service.client.impl.EagleServiceClientImpl;
error: patch failed: eagle-jpm/eagle-jpm-entity/src/main/java/org/apache/eagle/jpm/spark/crawl/JHFSparkEventReader.java:21
error: eagle-jpm/eagle-jpm-entity/src/main/java/org/apache/eagle/jpm/spark/crawl/JHFSparkEventReader.java: patch does not apply
Checking patch eagle-jpm/eagle-jpm-entity/src/main/java/org/apache/eagle/jpm/spark/crawl/JHFSparkParser.java...
error: while searching for:
private static final Logger logger = LoggerFactory.getLogger(JHFSparkParser.class);
JHFSparkEventReader eventReader;
public JHFSparkParser(JHFSparkEventReader reader){
error: patch failed: eagle-jpm/eagle-jpm-entity/src/main/java/org/apache/eagle/jpm/spark/crawl/JHFSparkParser.java:31
error: eagle-jpm/eagle-jpm-entity/src/main/java/org/apache/eagle/jpm/spark/crawl/JHFSparkParser.java: patch does not apply
Checking patch eagle-jpm/eagle-jpm-entity/src/main/java/org/apache/eagle/jpm/spark/entity/SparkStage.java...
error: while searching for:
@Column("a")
private String status;
@Column("b")
private int numActiveTasks=0;
@Column("c")
private int numCompletedTasks=0;
@Column("d")
private int numFailedTasks=0;
@Column("e")
private long executorRunTime=0l;
@Column("f")
private long inputBytes=0l;
@Column("g")
private long inputRecords=0l;
@Column("h")
private long outputBytes=0l;
@Column("i")
private long outputRecords=0l;
@Column("j")
private long shuffleReadBytes=0l;
@Column("k")
private long shuffleReadRecords=0l;
@Column("l")
private long shuffleWriteBytes=0l;
@Column("m")
private long shuffleWriteRecords=0l;
@Column("n")
private long memoryBytesSpilled=0l;
@Column("o")
private long diskBytesSpilled=0l;
@Column("p")
private String name;
@Column("q")
error: patch failed: eagle-jpm/eagle-jpm-entity/src/main/java/org/apache/eagle/jpm/spark/entity/SparkStage.java:36
error: eagle-jpm/eagle-jpm-entity/src/main/java/org/apache/eagle/jpm/spark/entity/SparkStage.java: patch does not apply
Checking patch eagle-jpm/eagle-jpm-spark-history/src/main/java/org/apache/eagle/jpm/spark/history/storm/FinishedSparkJobSpout.java...
Hunk #1 succeeded at 76 (offset -2 lines).
Checking patch eagle-jpm/eagle-jpm-spark-history/src/main/java/org/apache/eagle/jpm/spark/history/storm/SparkJobParseBolt.java...
Checking patch eagle-jpm/eagle-jpm-util/src/main/java/org/apache/eagle/jpm/util/Utils.java...
error: while searching for:
public static long parseMemory(String memory) {
if (memory.endsWith("g") || memory.endsWith("G")) {
int executorGB = Integer.parseInt(memory.substring(0, memory.length() - 1));
return 1024l * 1024 * 1024 * executorGB;
} else if (memory.endsWith("m") || memory.endsWith("M")) {
int executorMB = Integer.parseInt(memory.substring(0, memory.length() - 1));
return 1024l * 1024 * executorMB;
} else if (memory.endsWith("k") || memory.endsWith("K")) {
int executorKB = Integer.parseInt(memory.substring(0, memory.length() - 1));
return 1024l * executorKB;
} else if (memory.endsWith("t") || memory.endsWith("T")) {
int executorTB = Integer.parseInt(memory.substring(0, memory.length() - 1));
return 1024l * 1024 * 1024 * 1024 * executorTB;
} else if (memory.endsWith("p") || memory.endsWith("P")) {
int executorPB = Integer.parseInt(memory.substring(0, memory.length() - 1));
return 1024l * 1024 * 1024 * 1024 * 1024 * executorPB;
}
LOG.info("Cannot parse memory info " + memory);
return 0l;
}
}
error: patch failed: eagle-jpm/eagle-jpm-util/src/main/java/org/apache/eagle/jpm/util/Utils.java:68
error: eagle-jpm/eagle-jpm-util/src/main/java/org/apache/eagle/jpm/util/Utils.java: patch does not apply
Checking patch eagle-jpm/eagle-jpm-util/src/main/java/org/apache/eagle/jpm/util/resourceFetch/RMResourceFetcher.java...
error: while searching for:
}
}
private List<AppInfo> getResource(Constants.ResourceType resoureType, Constants.CompressionType compressionType, Object... parameter) throws Exception {
switch (resoureType) {
case COMPLETE_SPARK_JOB:
final String urlString = sparkCompleteJobServiceURLBuilder.build((String) parameter[0]);
return doFetchFinishApplicationsList(urlString, compressionType);
case RUNNING_SPARK_JOB:
return doFetchRunningApplicationsList(getSparkRunningJobURL(), compressionType);
error: patch failed: eagle-jpm/eagle-jpm-util/src/main/java/org/apache/eagle/jpm/util/resourceFetch/RMResourceFetcher.java:128
error: eagle-jpm/eagle-jpm-util/src/main/java/org/apache/eagle/jpm/util/resourceFetch/RMResourceFetcher.java: patch does not apply
Checking patch eagle-jpm/eagle-jpm-entity/src/main/java/org/apache/eagle/jpm/spark/crawl/JHFSparkParser.java...
error: while searching for:
private boolean isValidJson;
JHFSparkEventReader eventReader;
public JHFSparkParser(JHFSparkEventReader reader) {
this.eventReader = reader;
error: patch failed: eagle-jpm/eagle-jpm-entity/src/main/java/org/apache/eagle/jpm/spark/crawl/JHFSparkParser.java:34
error: eagle-jpm/eagle-jpm-entity/src/main/java/org/apache/eagle/jpm/spark/crawl/JHFSparkParser.java: patch does not apply
Checking patch eagle-jpm/eagle-jpm-util/src/main/java/org/apache/eagle/jpm/util/resourceFetch/RMResourceFetcher.java...
error: while searching for:
private String getMRFinishedJobURL(String lastFinishedTime) {
String url = URLUtil.removeTrailingSlash(selector.getSelectedUrl());
StringBuilder sb = new StringBuilder();
sb.append(url).append("/").append(Constants.V2_APPS_URL);
sb.append("?applicationTypes=MAPREDUCE&state=FINISHED&finishedTimeBegin=");
sb.append(lastFinishedTime).append("&").append(Constants.ANONYMOUS_PARAMETER);
return sb.toString();
}
private List<AppInfo> doFetchRunningApplicationsList(String urlString, Constants.CompressionType compressionType) throws Exception {
error: patch failed: eagle-jpm/eagle-jpm-util/src/main/java/org/apache/eagle/jpm/util/resourceFetch/RMResourceFetcher.java:107
error: eagle-jpm/eagle-jpm-util/src/main/java/org/apache/eagle/jpm/util/resourceFetch/RMResourceFetcher.java: patch does not apply
Checking patch eagle-jpm/eagle-jpm-util/src/main/java/org/apache/eagle/jpm/util/resourceFetch/ha/HAURLSelectorImpl.java...
Checking patch eagle-jpm/eagle-jpm-util/src/main/java/org/apache/eagle/jpm/util/resourceFetch/url/SparkCompleteJobServiceURLBuilderImpl.java...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-eagle issue #375: [EAGLE-492] Fix negative total execution time an...
Posted by pkuwm <gi...@git.apache.org>.
Github user pkuwm commented on the issue:
https://github.com/apache/incubator-eagle/pull/375
@qingwen220
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-eagle pull request #375: [EAGLE-492] Fix negative total execution ...
Posted by pkuwm <gi...@git.apache.org>.
Github user pkuwm closed the pull request at:
https://github.com/apache/incubator-eagle/pull/375
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---