You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@phoenix.apache.org by mujtabachohan <gi...@git.apache.org> on 2015/09/01 21:35:46 UTC
[GitHub] phoenix pull request: PHOENIX-2182
GitHub user mujtabachohan opened a pull request:
https://github.com/apache/phoenix/pull/115
PHOENIX-2182
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mujtabachohan/phoenix-1 master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/phoenix/pull/115.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #115
----
commit 6d732b42b6df4bba4ea7e890c85e8270ed087646
Author: Mujtaba <mc...@salesforce.com>
Date: 2015-09-01T19:32:25Z
PHOENIX-2182
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] phoenix pull request: PHOENIX-2182
Posted by codymarcel <gi...@git.apache.org>.
Github user codymarcel commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/115#discussion_r38472365
--- Diff: phoenix-pherf/src/main/java/org/apache/phoenix/pherf/result/ResultUtil.java ---
@@ -147,9 +147,9 @@ public String getSuffix() {
if (null == FILE_SUFFIX) {
Date date = new Date();
Format formatter = new SimpleDateFormat("YYYY-MM-dd_hh-mm-ss");
- FILE_SUFFIX = "_" + formatter.format(date);
+ FILE_SUFFIX = formatter.format(date);
--- End diff --
Should you call setFileSuffix()?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] phoenix pull request: PHOENIX-2182
Posted by codymarcel <gi...@git.apache.org>.
Github user codymarcel commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/115#discussion_r38472004
--- Diff: phoenix-pherf/src/main/java/org/apache/phoenix/pherf/PherfConstants.java ---
@@ -62,6 +62,8 @@
public static final int MONITOR_FREQUENCY = 5000;
public static final String MONITOR_FILE_NAME = "STATS_MONITOR";
+ public static final String SUMMARY_HTML_FILE_NAME = "summary.html";
--- End diff --
I am trying to make everything here configurable from properties.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] phoenix pull request: PHOENIX-2182
Posted by codymarcel <gi...@git.apache.org>.
Github user codymarcel commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/115#discussion_r38471394
--- Diff: phoenix-pherf/src/main/java/org/apache/phoenix/pherf/Pherf.java ---
@@ -136,6 +142,8 @@ public Pherf(String[] args) throws Exception {
command.getOptionValue("writerThreadSize",
properties.getProperty("pherf.default.dataloader.threadpool"));
properties.setProperty("pherf. default.dataloader.threadpool", writerThreadPoolSize);
+ label = command.getOptionValue("label", null);
+ compareResults = command.getOptionValue("compare", null);
--- End diff --
Use boolean and set to false as default
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] phoenix pull request: PHOENIX-2182
Posted by codymarcel <gi...@git.apache.org>.
Github user codymarcel commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/115#discussion_r38470777
--- Diff: phoenix-pherf/src/main/java/org/apache/phoenix/pherf/Pherf.java ---
@@ -136,6 +142,8 @@ public Pherf(String[] args) throws Exception {
command.getOptionValue("writerThreadSize",
properties.getProperty("pherf.default.dataloader.threadpool"));
properties.setProperty("pherf. default.dataloader.threadpool", writerThreadPoolSize);
+ label = command.getOptionValue("label", null);
+ compareResults = command.getOptionValue("compare", null);
--- End diff --
default to false instead of null
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] phoenix pull request: PHOENIX-2182
Posted by codymarcel <gi...@git.apache.org>.
Github user codymarcel commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/115#discussion_r38470844
--- Diff: phoenix-pherf/src/main/java/org/apache/phoenix/pherf/Pherf.java ---
@@ -179,6 +188,15 @@ public void run() throws Exception {
}
return;
}
+
+ // Compare results and exit
+ if (null != compareResults) {
--- End diff --
if(!compareResults){}
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] phoenix pull request: PHOENIX-2182
Posted by codymarcel <gi...@git.apache.org>.
Github user codymarcel commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/115#discussion_r38477433
--- Diff: phoenix-pherf/src/main/java/org/apache/phoenix/pherf/workload/MultithreadedDiffer.java ---
@@ -51,7 +53,7 @@ private void diffQuery() throws Exception {
Date startDate = Calendar.getInstance().getTime();
String newCSV = queryVerifier.exportCSV(query);
boolean verifyResult = queryVerifier.doDiff(query, newCSV);
- String explainPlan = queryVerifier.getExplainPlan(query);
--- End diff --
Could you add the explainPlan stuff as a different Jira/CL? It might get checked in faster if it's detached from this CL. This one might need a little more love.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] phoenix pull request: PHOENIX-2182
Posted by codymarcel <gi...@git.apache.org>.
Github user codymarcel commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/115#discussion_r38474158
--- Diff: phoenix-pherf/src/main/java/org/apache/phoenix/pherf/util/ResultsComparator.java ---
@@ -0,0 +1,353 @@
+package org.apache.phoenix.pherf.util;
+
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.PrintWriter;
+import java.io.UnsupportedEncodingException;
+import java.text.DecimalFormat;
+import java.text.SimpleDateFormat;
+import java.util.Date;
+import java.util.LinkedHashMap;
+import java.util.Map;
+import java.util.TreeMap;
+
+import org.apache.commons.csv.CSVFormat;
+import org.apache.commons.csv.CSVParser;
+import org.apache.commons.csv.CSVRecord;
+import org.apache.phoenix.pherf.PherfConstants;
+import org.apache.phoenix.pherf.result.file.ResultFileDetails;
+
+/**
+ * Compare results based on set threshold and render results as Google Charts
+ */
+public class ResultsComparator {
+
+ private String[] labels;
+ private final Map<String, DataNode> datanodes = new TreeMap<String, DataNode>();
+ private final PherfConstants constants = PherfConstants.create();
+ private final String resultDir = constants.getProperty("pherf.default.results.dir");
+ private final double threshold = Double.parseDouble(constants.getProperty("pherf.default.comparison.threshold"));
+
+ public ResultsComparator(String labels) {
+ this.setLabelsAsString(labels);
+ }
+
+ String[] getLabels() {
+ return labels;
+ }
+
+ void setLabels(String[] labels) {
+ this.labels = labels;
+ }
+
+ void setLabelsAsString(String labels) {
+ this.labels = labels.split(",");
+ }
+
+ public void readAndRender() {
+ try {
+ for (String label : labels) {
+ read(label);
+ }
+ renderAsGoogleChartsHTML();
+
+ } catch (Exception e) {
+ e.printStackTrace();
+ }
+ }
+
+ /**
+ * Reads aggregate file and convert it to DataNode
+ * @param label
+ * @throws Exception
+ */
+ private void read(String label) throws Exception {
+ String resultFileName = resultDir
+ + PherfConstants.PATH_SEPARATOR
+ + PherfConstants.RESULT_PREFIX
+ + label
+ + ResultFileDetails.CSV_AGGREGATE_PERFORMANCE.getExtension();
+
+ FileReader in = new FileReader(resultFileName);
+ final CSVParser parser = new CSVParser(in, CSVFormat.DEFAULT.withHeader());
+
+ for (CSVRecord record : parser) {
+ String group = record.get("QUERY_GROUP");
+ String query = record.get("QUERY");
+ String explain = record.get("EXPLAIN_PLAN");
+ String tenantId = record.get("TENANT_ID");
+ long avgTime = Long.parseLong(record.get("AVG_TIME_MS"));
+ long minTime = Long.parseLong(record.get("AVG_MIN_TIME_MS"));
+ long numRuns = Long.parseLong(record.get("RUN_COUNT"));
+ long rowCount = Long.parseLong(record.get("RESULT_ROW_COUNT"));
+ Node node = new Node(minTime, avgTime, numRuns, explain, query, tenantId, label, rowCount);
+
+ if (datanodes.containsKey(group)) {
+ datanodes.get(group).getDataSet().put(label, node);
+ } else {
+ datanodes.put(group, new DataNode(label, node));
+ }
+ }
+ parser.close();
+ }
+
+ /**
+ * Verifies if the first result is within the set
+ * threshold of pherf.default.comparison.threshold
+ * set in pherf.properties files
+ * @param threshold
+ * @return
+ */
+ private boolean verifyWithinThreshold(double threshold) {
+ long resetTimeToCompare = -1;
+ long timeToCompare = resetTimeToCompare;
+ for (Map.Entry<String, DataNode> dn : datanodes.entrySet()) {
+ for (Map.Entry<String, Node> node : dn.getValue().getDataSet().entrySet()) {
+ if (timeToCompare == -1) {
+ timeToCompare = node.getValue().getMinTime();
+ }
+ if ((((double) (timeToCompare - node.getValue().getMinTime())) / (double) node
+ .getValue().getMinTime()) > threshold) {
+ return false;
+ }
+ }
+ timeToCompare = resetTimeToCompare;
+ }
+ return true;
+ }
+
+ /**
+ * Render results as Google charts
+ * @throws FileNotFoundException
+ * @throws UnsupportedEncodingException
+ */
+ private void renderAsGoogleChartsHTML() throws FileNotFoundException, UnsupportedEncodingException {
--- End diff --
Should all this chart stuff go in a util object? It seems unrelated to a Result comparison.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] phoenix pull request: PHOENIX-2182
Posted by codymarcel <gi...@git.apache.org>.
Github user codymarcel commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/115#discussion_r38473049
--- Diff: phoenix-pherf/src/main/java/org/apache/phoenix/pherf/util/ResultsComparator.java ---
@@ -0,0 +1,353 @@
+package org.apache.phoenix.pherf.util;
+
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.PrintWriter;
+import java.io.UnsupportedEncodingException;
+import java.text.DecimalFormat;
+import java.text.SimpleDateFormat;
+import java.util.Date;
+import java.util.LinkedHashMap;
+import java.util.Map;
+import java.util.TreeMap;
+
+import org.apache.commons.csv.CSVFormat;
+import org.apache.commons.csv.CSVParser;
+import org.apache.commons.csv.CSVRecord;
+import org.apache.phoenix.pherf.PherfConstants;
+import org.apache.phoenix.pherf.result.file.ResultFileDetails;
+
+/**
+ * Compare results based on set threshold and render results as Google Charts
+ */
+public class ResultsComparator {
--- End diff --
Should this implement Comparable? Calling it Comparator is misleading if it's not doing that.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] phoenix pull request: PHOENIX-2182
Posted by codymarcel <gi...@git.apache.org>.
Github user codymarcel commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/115#discussion_r38477085
--- Diff: phoenix-pherf/src/main/java/org/apache/phoenix/pherf/util/ResultsComparator.java ---
@@ -0,0 +1,353 @@
+package org.apache.phoenix.pherf.util;
+
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.PrintWriter;
+import java.io.UnsupportedEncodingException;
+import java.text.DecimalFormat;
+import java.text.SimpleDateFormat;
+import java.util.Date;
+import java.util.LinkedHashMap;
+import java.util.Map;
+import java.util.TreeMap;
+
+import org.apache.commons.csv.CSVFormat;
+import org.apache.commons.csv.CSVParser;
+import org.apache.commons.csv.CSVRecord;
+import org.apache.phoenix.pherf.PherfConstants;
+import org.apache.phoenix.pherf.result.file.ResultFileDetails;
+
+/**
+ * Compare results based on set threshold and render results as Google Charts
+ */
+public class ResultsComparator {
+
+ private String[] labels;
+ private final Map<String, DataNode> datanodes = new TreeMap<String, DataNode>();
+ private final PherfConstants constants = PherfConstants.create();
+ private final String resultDir = constants.getProperty("pherf.default.results.dir");
+ private final double threshold = Double.parseDouble(constants.getProperty("pherf.default.comparison.threshold"));
+
+ public ResultsComparator(String labels) {
+ this.setLabelsAsString(labels);
+ }
+
+ String[] getLabels() {
+ return labels;
+ }
+
+ void setLabels(String[] labels) {
+ this.labels = labels;
+ }
+
+ void setLabelsAsString(String labels) {
+ this.labels = labels.split(",");
+ }
+
+ public void readAndRender() {
+ try {
+ for (String label : labels) {
+ read(label);
+ }
+ renderAsGoogleChartsHTML();
+
+ } catch (Exception e) {
+ e.printStackTrace();
+ }
+ }
+
+ /**
+ * Reads aggregate file and convert it to DataNode
+ * @param label
+ * @throws Exception
+ */
+ private void read(String label) throws Exception {
+ String resultFileName = resultDir
+ + PherfConstants.PATH_SEPARATOR
+ + PherfConstants.RESULT_PREFIX
+ + label
+ + ResultFileDetails.CSV_AGGREGATE_PERFORMANCE.getExtension();
+
+ FileReader in = new FileReader(resultFileName);
+ final CSVParser parser = new CSVParser(in, CSVFormat.DEFAULT.withHeader());
+
+ for (CSVRecord record : parser) {
+ String group = record.get("QUERY_GROUP");
+ String query = record.get("QUERY");
+ String explain = record.get("EXPLAIN_PLAN");
+ String tenantId = record.get("TENANT_ID");
+ long avgTime = Long.parseLong(record.get("AVG_TIME_MS"));
+ long minTime = Long.parseLong(record.get("AVG_MIN_TIME_MS"));
+ long numRuns = Long.parseLong(record.get("RUN_COUNT"));
+ long rowCount = Long.parseLong(record.get("RESULT_ROW_COUNT"));
+ Node node = new Node(minTime, avgTime, numRuns, explain, query, tenantId, label, rowCount);
+
+ if (datanodes.containsKey(group)) {
+ datanodes.get(group).getDataSet().put(label, node);
+ } else {
+ datanodes.put(group, new DataNode(label, node));
+ }
+ }
+ parser.close();
+ }
+
+ /**
+ * Verifies if the first result is within the set
+ * threshold of pherf.default.comparison.threshold
+ * set in pherf.properties files
+ * @param threshold
+ * @return
+ */
+ private boolean verifyWithinThreshold(double threshold) {
+ long resetTimeToCompare = -1;
+ long timeToCompare = resetTimeToCompare;
+ for (Map.Entry<String, DataNode> dn : datanodes.entrySet()) {
+ for (Map.Entry<String, Node> node : dn.getValue().getDataSet().entrySet()) {
+ if (timeToCompare == -1) {
+ timeToCompare = node.getValue().getMinTime();
+ }
+ if ((((double) (timeToCompare - node.getValue().getMinTime())) / (double) node
+ .getValue().getMinTime()) > threshold) {
+ return false;
+ }
+ }
+ timeToCompare = resetTimeToCompare;
+ }
+ return true;
+ }
+
+ /**
+ * Render results as Google charts
+ * @throws FileNotFoundException
+ * @throws UnsupportedEncodingException
+ */
+ private void renderAsGoogleChartsHTML() throws FileNotFoundException, UnsupportedEncodingException {
+ String lastKeyPrefix = "";
+ StringBuffer sb = new StringBuffer();
+ for (String label : labels) {
+ sb.append("dataTable.addColumn('number', '" + label + "');\n");
+ sb.append("dataTable.addColumn({type: 'string', role: 'tooltip', 'p': {'html': true}});\n");
+ }
+ sb.append("dataTable.addRows([\n");
+ for (Map.Entry<String, DataNode> dn : datanodes.entrySet()) {
+ String currentKeyPrefix = dn.getKey().substring(0, dn.getKey().indexOf('|'));
+ if (!lastKeyPrefix.equalsIgnoreCase(currentKeyPrefix) && lastKeyPrefix != "") {
+ sb.append(getBlankRow());
+ }
+ lastKeyPrefix = currentKeyPrefix;
+ sb.append("['" + dn.getKey() + "'");
+ for (Map.Entry<String, Node> nodeSet : dn.getValue().getDataSet().entrySet()) {
+ sb.append (", " + nodeSet.getValue().getMinTime());
+ sb.append (",'" + getToolTipAsHTML(dn.getValue().getDataSet()) + "'");
+ }
+ sb.append("],\n");
+ }
+ String summaryDir = PherfConstants.create().getProperty("pherf.default.summary.dir");
+ PrintWriter writer = new PrintWriter(summaryDir + "/" + PherfConstants.SUMMARY_HTML_FILE_NAME, "UTF-8");
+ writer.println(StaticGoogleChartsRenderingData.HEADER);
+ writer.println(sb.substring(0, sb.length() - 2) + "\n]);");
+ String thresholdString = Math.round((threshold*100)) + "%";
+ String footer = StaticGoogleChartsRenderingData.FOOTER
+ .replace("[summary]",
+ ((verifyWithinThreshold(threshold) == true ? "<font color=green>PASSED | Results are within ":
+ "<font color=red>FAILED | Results are outside "))
+ + "set threshold of " + thresholdString + "</font><br>"
+ + new SimpleDateFormat("yyyy/MM/dd ha z").format(new Date()));
+ footer = footer.replace("[title]", labels[0]);
+ writer.println(footer);
+ writer.close();
+ }
+
+ /**
+ * Render a blank Google charts row
+ * @return
+ */
+ private String getBlankRow() {
+ String ret = "['" + new String(new char[60]).replace("\0", ".") + "'";
+ for (int i=0; i<labels.length; i++)
+ ret += ",0,''";
+ ret += "],";
+ return ret;
+ }
+
+ /**
+ * Render tooltip as HTML table
+ * @param nodeDataSet
+ * @return
+ */
+ private String getToolTipAsHTML(Map<String, Node> nodeDataSet) {
+ String ret = "<table width=1000 cellpadding=1 cellspacing=0 border=0 bgcolor=#F4F4F4><tr>";
+ for (Map.Entry<String, Node> nodeSet : nodeDataSet.entrySet())
+ ret += "<td>" + getToolText(nodeSet.getValue()) + "</td>";
+ return ret + "</tr></table>";
+ }
+
+ /**
+ * Get tooltip for node
+ * @param node
+ * @return
+ */
+ private String getToolText(Node node) {
+ return node.getLabelAsHTML()
+ + node.getAvgTimeAsHTML()
+ + node.getMinTimeAsHTML()
+ + node.getNumRunsAsHTML()
+ + node.getRowCountAsHTML()
+ + node.getExplainPlanAsHTML()
+ + node.getQueryAsHTML();
+ }
+
+ /**
+ * DataNode to store results to render and compare
+ */
+ class DataNode {
+ private Map<String, Node> dataSet = new LinkedHashMap<String, Node>();
+
+ public DataNode(String label, Node node) {
+ this.getDataSet().put(label, node);
+ }
+
+ public Map<String, Node> getDataSet() {
+ return dataSet;
+ }
+ public void setDataSet(Map<String, Node> dataSet) {
+ this.dataSet = dataSet;
+ }
+ }
+
+ class Node {
+ private String explainPlan;
+ private String query;
+ private String tenantId;
+ private long minTime;
+ private long avgTime;
+ private long numRuns;
+ private long rowCount;
+ private String label;
+ private DecimalFormat df = new DecimalFormat("#.#");
+
+ public Node(long minTime, long avgTime, long numRuns, String explainPlan, String query, String tenantId, String label, long rowCount) {
+ this.setMinTime(minTime);
+ this.setAvgTime(avgTime);
+ this.setNumRuns(numRuns);
+ this.setExplainPlan(explainPlan);
+ this.setQuery(query);
+ this.setTenantId(tenantId);
+ this.setLabel(label);
+ this.setRowCount(rowCount);
+ }
+
+ String getExplainPlan() {
+ return explainPlan;
+ }
+ String getExplainPlanAsHTML() {
+ return "</br><font face=arial size=1><b>EXPLAIN PLAN </b>"
--- End diff --
Perhaps we should use an html generation library instead of hard coding all this in strings?
Never tried this one, bit looks like it has god template support.
http://freemarker.org/docs/dgui_quickstart_basics.html
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] phoenix pull request: PHOENIX-2182
Posted by codymarcel <gi...@git.apache.org>.
Github user codymarcel commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/115#discussion_r38471433
--- Diff: phoenix-pherf/src/main/java/org/apache/phoenix/pherf/Pherf.java ---
@@ -179,6 +188,15 @@ public void run() throws Exception {
}
return;
}
+
+ // Compare results and exit
+ if (null != compareResults) {
--- End diff --
if(!compareResults){}
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] phoenix pull request: PHOENIX-2182
Posted by codymarcel <gi...@git.apache.org>.
Github user codymarcel commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/115#discussion_r38471909
--- Diff: phoenix-pherf/src/main/java/org/apache/phoenix/pherf/PherfConstants.java ---
@@ -62,6 +62,8 @@
public static final int MONITOR_FREQUENCY = 5000;
public static final String MONITOR_FILE_NAME = "STATS_MONITOR";
+ public static final String SUMMARY_HTML_FILE_NAME = "summary.html";
--- End diff --
Can you put this in the properties file?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] phoenix pull request: PHOENIX-2182
Posted by codymarcel <gi...@git.apache.org>.
Github user codymarcel commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/115#discussion_r38472866
--- Diff: phoenix-pherf/src/main/java/org/apache/phoenix/pherf/util/PhoenixUtil.java ---
@@ -274,4 +274,34 @@ public void updatePhoenixStats(String tableName, Scenario scenario) throws Excep
logger.info("Updating stats for " + tableName);
executeStatement("UPDATE STATISTICS " + tableName, scenario);
}
+
+ /**
+ * Get explain plan for a query
+ *
+ * @param query
+ * @return
+ * @throws SQLException
+ */
+ public String getExplainPlan(Query query) throws SQLException {
+ Connection conn = null;
+ ResultSet rs = null;
+ PreparedStatement statement = null;
+ StringBuilder buf = new StringBuilder();
+ try {
--- End diff --
You might want to consider using the JdbcSession() to handle all the boiler plate try/catch/finally stuff on the connection.
Check out testRWWorkload()
https://github.com/apache/phoenix/blob/master/phoenix-pherf/src/it/java/org/apache/phoenix/pherf/DataIngestIT.java
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] phoenix pull request: PHOENIX-2182
Posted by codymarcel <gi...@git.apache.org>.
Github user codymarcel commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/115#discussion_r38473156
--- Diff: phoenix-pherf/src/main/java/org/apache/phoenix/pherf/util/ResultsComparator.java ---
@@ -0,0 +1,353 @@
+package org.apache.phoenix.pherf.util;
+
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.PrintWriter;
+import java.io.UnsupportedEncodingException;
+import java.text.DecimalFormat;
+import java.text.SimpleDateFormat;
+import java.util.Date;
+import java.util.LinkedHashMap;
+import java.util.Map;
+import java.util.TreeMap;
+
+import org.apache.commons.csv.CSVFormat;
+import org.apache.commons.csv.CSVParser;
+import org.apache.commons.csv.CSVRecord;
+import org.apache.phoenix.pherf.PherfConstants;
+import org.apache.phoenix.pherf.result.file.ResultFileDetails;
+
+/**
+ * Compare results based on set threshold and render results as Google Charts
+ */
+public class ResultsComparator {
+
+ private String[] labels;
+ private final Map<String, DataNode> datanodes = new TreeMap<String, DataNode>();
+ private final PherfConstants constants = PherfConstants.create();
+ private final String resultDir = constants.getProperty("pherf.default.results.dir");
+ private final double threshold = Double.parseDouble(constants.getProperty("pherf.default.comparison.threshold"));
+
+ public ResultsComparator(String labels) {
--- End diff --
nit: label should be singular ;)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] phoenix pull request: PHOENIX-2182
Posted by codymarcel <gi...@git.apache.org>.
Github user codymarcel commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/115#discussion_r38473654
--- Diff: phoenix-pherf/src/main/java/org/apache/phoenix/pherf/util/ResultsComparator.java ---
@@ -0,0 +1,353 @@
+package org.apache.phoenix.pherf.util;
+
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.PrintWriter;
+import java.io.UnsupportedEncodingException;
+import java.text.DecimalFormat;
+import java.text.SimpleDateFormat;
+import java.util.Date;
+import java.util.LinkedHashMap;
+import java.util.Map;
+import java.util.TreeMap;
+
+import org.apache.commons.csv.CSVFormat;
+import org.apache.commons.csv.CSVParser;
+import org.apache.commons.csv.CSVRecord;
+import org.apache.phoenix.pherf.PherfConstants;
+import org.apache.phoenix.pherf.result.file.ResultFileDetails;
+
+/**
+ * Compare results based on set threshold and render results as Google Charts
+ */
+public class ResultsComparator {
+
+ private String[] labels;
+ private final Map<String, DataNode> datanodes = new TreeMap<String, DataNode>();
+ private final PherfConstants constants = PherfConstants.create();
+ private final String resultDir = constants.getProperty("pherf.default.results.dir");
+ private final double threshold = Double.parseDouble(constants.getProperty("pherf.default.comparison.threshold"));
+
+ public ResultsComparator(String labels) {
+ this.setLabelsAsString(labels);
+ }
+
+ String[] getLabels() {
+ return labels;
+ }
+
+ void setLabels(String[] labels) {
+ this.labels = labels;
+ }
+
+ void setLabelsAsString(String labels) {
--- End diff --
This is setting as array. Why not just call it setLabels() and let overloading do it's thing?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] phoenix pull request: PHOENIX-2182
Posted by codymarcel <gi...@git.apache.org>.
Github user codymarcel commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/115#discussion_r38470727
--- Diff: phoenix-pherf/src/main/java/org/apache/phoenix/pherf/Pherf.java ---
@@ -76,6 +78,8 @@
options.addOption("d", "debug", false, "Put tool in debug mode");
options.addOption("stats", false,
"Update Phoenix Statistics after data is loaded with -l argument");
+ options.addOption("label", true, "Label a run. Result file name will be suffixed with specified label");
--- End diff --
Is this an override to then dynamically generated label? Also, it would be good if the label was applied as a column value in the result data. That way we can distinguish between runs on analysis.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] phoenix pull request: PHOENIX-2182
Posted by codymarcel <gi...@git.apache.org>.
Github user codymarcel commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/115#discussion_r38476254
--- Diff: phoenix-pherf/src/main/java/org/apache/phoenix/pherf/util/ResultsComparator.java ---
@@ -0,0 +1,353 @@
+package org.apache.phoenix.pherf.util;
+
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.PrintWriter;
+import java.io.UnsupportedEncodingException;
+import java.text.DecimalFormat;
+import java.text.SimpleDateFormat;
+import java.util.Date;
+import java.util.LinkedHashMap;
+import java.util.Map;
+import java.util.TreeMap;
+
+import org.apache.commons.csv.CSVFormat;
+import org.apache.commons.csv.CSVParser;
+import org.apache.commons.csv.CSVRecord;
+import org.apache.phoenix.pherf.PherfConstants;
+import org.apache.phoenix.pherf.result.file.ResultFileDetails;
+
+/**
+ * Compare results based on set threshold and render results as Google Charts
+ */
+public class ResultsComparator {
+
+ private String[] labels;
+ private final Map<String, DataNode> datanodes = new TreeMap<String, DataNode>();
+ private final PherfConstants constants = PherfConstants.create();
+ private final String resultDir = constants.getProperty("pherf.default.results.dir");
+ private final double threshold = Double.parseDouble(constants.getProperty("pherf.default.comparison.threshold"));
+
+ public ResultsComparator(String labels) {
+ this.setLabelsAsString(labels);
+ }
+
+ String[] getLabels() {
+ return labels;
+ }
+
+ void setLabels(String[] labels) {
+ this.labels = labels;
+ }
+
+ void setLabelsAsString(String labels) {
+ this.labels = labels.split(",");
+ }
+
+ public void readAndRender() {
+ try {
+ for (String label : labels) {
+ read(label);
+ }
+ renderAsGoogleChartsHTML();
+
+ } catch (Exception e) {
+ e.printStackTrace();
+ }
+ }
+
+ /**
+ * Reads aggregate file and convert it to DataNode
+ * @param label
+ * @throws Exception
+ */
+ private void read(String label) throws Exception {
--- End diff --
Can we use a CSVResultHandler with a ResultFileDetails.CSV_AGGREGATE_PERFORMANCE? It looks like there is a fair amount of redundancy with that and the ResultWriter. It makes sense that you need to wrap the compare logic into this, but result reading and writing should use the existing framework for that.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] phoenix pull request: PHOENIX-2182
Posted by codymarcel <gi...@git.apache.org>.
Github user codymarcel commented on a diff in the pull request:
https://github.com/apache/phoenix/pull/115#discussion_r38471489
--- Diff: phoenix-pherf/src/main/java/org/apache/phoenix/pherf/Pherf.java ---
@@ -179,6 +188,15 @@ public void run() throws Exception {
}
return;
}
+
+ // Compare results and exit
+ if (null != compareResults) {
+ logger.info("\nStarting to compare results and exiting for " + compareResults);
+ ResultsComparator rc = new ResultsComparator(compareResults);
--- End diff --
Should this be the label?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---