You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@systemml.apache.org by ni...@apache.org on 2017/07/21 17:09:21 UTC
[3/3] systemml git commit: [SYSTEMML-1796] Provide fine-grained
statistics for SystemML
[SYSTEMML-1796] Provide fine-grained statistics for SystemML
Though SystemML provides explain and statistics tooling functionality to
understand the execution plan and the performance, most users (even
committers) are not able to explain why SystemML's performance is so good
or bad. This is likely because SystemML's optimizer performs sophisticated
dataflow analysis, cost optimization as well as static and dynamic
rewrites. New features/backends such as code generation, automatic fusion,
compression and GPU backend adds additional level of complexity to the
optimizer. This commit is a small step in the direction of explaining
SystemML's behavior.
Instead of examining statistics output and then
trying to correlate with the explain plan, this commit provides
instruction-level statistics with the corresponding DML snippet (file
beginLine:beginCol-endLine-endCol).
Output with the configuration property systemml.stats.finegrained set to true:
```
Heavy hitter instructions:
# Instruction Time(s) Count Misc Timers
1 solve [148:0-148:0] 0.027 1 aqmd[0.000s,1]
2 tsmm [137:4-137:13] 0.008 1 rlsi[0.000s,1], aqmd[0.000s,1], aqrd[0.000s,1]
3 uak+ [161:10-161:10] 0.007 1 aqrd[0.000s,1], rlsi[0.000s,1]
4 ba+* [138:4-138:13] 0.005 1 aqmd[0.000s,1], aqrd[0.000s,2], rlsi[0.000s,2]
5 rand [89:0-89:0] 0.004 1 aqmd[0.000s,1]
6 * [133:9-133:24] 0.002 1 aqmd[0.000s,1], aqrd[0.000s,1], rlsi[0.000s,1]
7 r' [138:4-138:13] 0.001 2 aqmd[0.000s,2], aqrd[0.000s,2], rlsi[0.000s,2]
8 append [97:4-97:4] 0.001 1 rlswr[0.000s,1], aqmd[0.000s,1], rlsev[0.000s,0], aqrd[0.000s,2], rlsi[0.000s,2]
9 + [144:4-144:21] 0.001 1 aqmd[0.000s,1], aqrd[0.000s,2], rlsi[0.000s,2]
10 leftIndex [104:4-104:26] 0.001 1 aqmd[0.000s,1], rlsi[0.000s,1], aqrd[0.000s,1]
```
aqrd and aqmd refers to dense input and outputs respectively. Similarly, aqrs and aqms refers to sparse input and outputs. It can also show both:
```
1 leftIndex [106:4-106:42] 141.928 28552 aqrs[0.024s,23165], rls[0.066s,57104], csrlix[116.652s,22649], s2d[9.297s,258], aqrd[14.097s,33939], lixcp[1.410s,5903]
```
Closes #577.
Project: http://git-wip-us.apache.org/repos/asf/systemml/repo
Commit: http://git-wip-us.apache.org/repos/asf/systemml/commit/648eb21d
Tree: http://git-wip-us.apache.org/repos/asf/systemml/tree/648eb21d
Diff: http://git-wip-us.apache.org/repos/asf/systemml/diff/648eb21d
Branch: refs/heads/master
Commit: 648eb21d66f9cd8727090cdf950986765a7e6ee8
Parents: 6133be2
Author: Niketan Pansare <np...@us.ibm.com>
Authored: Fri Jul 21 09:09:03 2017 -0800
Committer: Niketan Pansare <np...@us.ibm.com>
Committed: Fri Jul 21 10:09:03 2017 -0700
----------------------------------------------------------------------
conf/SystemML-config.xml.template | 3 +
.../java/org/apache/sysml/api/DMLScript.java | 1 +
.../apache/sysml/api/ScriptExecutorUtils.java | 1 +
.../java/org/apache/sysml/conf/DMLConfig.java | 4 +-
.../sysml/debug/DMLDebuggerProgramInfo.java | 6 +-
.../java/org/apache/sysml/hops/BinaryOp.java | 36 +++++------
src/main/java/org/apache/sysml/hops/Hop.java | 15 +++--
.../java/org/apache/sysml/hops/TernaryOp.java | 16 ++---
.../java/org/apache/sysml/hops/UnaryOp.java | 2 +-
.../sysml/hops/rewrite/HopRewriteUtils.java | 2 +-
.../org/apache/sysml/lops/CombineBinary.java | 2 +-
.../org/apache/sysml/lops/CombineTernary.java | 2 +-
.../org/apache/sysml/lops/CombineUnary.java | 2 +-
src/main/java/org/apache/sysml/lops/Lop.java | 7 ++-
.../java/org/apache/sysml/lops/SortKeys.java | 4 +-
.../java/org/apache/sysml/lops/compile/Dag.java | 2 +-
.../org/apache/sysml/parser/DMLProgram.java | 10 +--
.../org/apache/sysml/parser/DMLTranslator.java | 52 +++++++--------
.../ExternalFunctionProgramBlock.java | 2 +-
.../runtime/controlprogram/ProgramBlock.java | 6 +-
.../controlprogram/caching/CacheableData.java | 38 +++++++++--
.../controlprogram/caching/LazyWriteBuffer.java | 21 ++++++-
.../context/ExecutionContext.java | 49 +++++++++++----
.../parfor/opt/OptimizerRuleBased.java | 2 +-
.../sysml/runtime/instructions/Instruction.java | 41 +++++++++---
.../cp/AggregateBinaryCPInstruction.java | 10 +--
.../cp/AggregateTernaryCPInstruction.java | 14 ++---
.../cp/AggregateUnaryCPInstruction.java | 6 +-
.../runtime/instructions/cp/CPInstruction.java | 16 +++++
.../cp/CentralMomentCPInstruction.java | 8 +--
.../cp/CompressionCPInstruction.java | 6 +-
.../cp/ConvolutionCPInstruction.java | 60 +++++++++---------
.../cp/CovarianceCPInstruction.java | 16 ++---
.../instructions/cp/DataGenCPInstruction.java | 2 +-
.../cp/DataPartitionCPInstruction.java | 2 +-
.../instructions/cp/MMChainCPInstruction.java | 14 ++---
.../instructions/cp/MMTSJCPInstruction.java | 6 +-
.../cp/MatrixAppendCPInstruction.java | 10 +--
.../cp/MatrixBuiltinCPInstruction.java | 8 +--
.../cp/MatrixIndexingCPInstruction.java | 22 +++----
.../cp/MatrixMatrixArithmeticCPInstruction.java | 10 +--
.../cp/MatrixMatrixBuiltinCPInstruction.java | 12 ++--
.../cp/MatrixMatrixRelationalCPInstruction.java | 10 +--
.../cp/MatrixReshapeCPInstruction.java | 6 +-
.../cp/MatrixScalarBuiltinCPInstruction.java | 6 +-
.../cp/MultiReturnBuiltinCPInstruction.java | 2 +-
...ReturnParameterizedBuiltinCPInstruction.java | 2 +-
.../instructions/cp/PMMJCPInstruction.java | 10 +--
.../cp/ParameterizedBuiltinCPInstruction.java | 42 ++++++-------
.../instructions/cp/PlusMultCPInstruction.java | 10 +--
.../cp/QuantilePickCPInstruction.java | 18 +++---
.../cp/QuantileSortCPInstruction.java | 10 +--
.../cp/QuaternaryCPInstruction.java | 20 +++---
.../instructions/cp/ReorgCPInstruction.java | 6 +-
.../cp/ScalarMatrixArithmeticCPInstruction.java | 6 +-
.../cp/ScalarMatrixRelationalCPInstruction.java | 6 +-
.../instructions/cp/SpoofCPInstruction.java | 6 +-
.../cp/StringInitCPInstruction.java | 2 +-
.../instructions/cp/TernaryCPInstruction.java | 20 +++---
.../cp/UaggOuterChainCPInstruction.java | 14 ++---
.../instructions/cp/VariableCPInstruction.java | 10 +--
.../cpfile/MatrixIndexingCPFileInstruction.java | 2 +-
.../spark/AggregateTernarySPInstruction.java | 2 +-
.../spark/AggregateUnarySPInstruction.java | 2 +-
.../spark/ConvolutionSPInstruction.java | 4 +-
.../instructions/spark/CpmmSPInstruction.java | 2 +-
.../spark/MapmmChainSPInstruction.java | 2 +-
.../instructions/spark/MapmmSPInstruction.java | 2 +-
.../spark/MatrixIndexingSPInstruction.java | 4 +-
.../ParameterizedBuiltinSPInstruction.java | 4 +-
.../instructions/spark/RandSPInstruction.java | 2 +-
.../instructions/spark/SpoofSPInstruction.java | 4 +-
.../instructions/spark/Tsmm2SPInstruction.java | 2 +-
.../instructions/spark/TsmmSPInstruction.java | 2 +-
.../spark/UaggOuterChainSPInstruction.java | 6 +-
.../instructions/spark/ZipmmSPInstruction.java | 2 +-
.../sysml/runtime/matrix/data/MatrixBlock.java | 66 +++++++++++++++++---
.../org/apache/sysml/utils/GPUStatistics.java | 4 +-
.../java/org/apache/sysml/utils/Statistics.java | 6 +-
79 files changed, 526 insertions(+), 346 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/systemml/blob/648eb21d/conf/SystemML-config.xml.template
----------------------------------------------------------------------
diff --git a/conf/SystemML-config.xml.template b/conf/SystemML-config.xml.template
index ff724b4..11e86ed 100644
--- a/conf/SystemML-config.xml.template
+++ b/conf/SystemML-config.xml.template
@@ -69,6 +69,9 @@
<!-- enables native blas for matrix multiplication and convolution, experimental feature (options: auto, mkl, openblas, none) -->
<native.blas>none</native.blas>
+ <!-- prints finegrained statistics information -->
+ <systemml.stats.finegrained>false</systemml.stats.finegrained>
+
<!-- prints extra statistics information for GPU -->
<systemml.stats.extraGPU>false</systemml.stats.extraGPU>
http://git-wip-us.apache.org/repos/asf/systemml/blob/648eb21d/src/main/java/org/apache/sysml/api/DMLScript.java
----------------------------------------------------------------------
diff --git a/src/main/java/org/apache/sysml/api/DMLScript.java b/src/main/java/org/apache/sysml/api/DMLScript.java
index 515d632..2674af4 100644
--- a/src/main/java/org/apache/sysml/api/DMLScript.java
+++ b/src/main/java/org/apache/sysml/api/DMLScript.java
@@ -160,6 +160,7 @@ public class DMLScript
public static RUNTIME_PLATFORM rtplatform = DMLOptions.defaultOptions.execMode; // the execution mode
public static boolean STATISTICS = DMLOptions.defaultOptions.stats; // whether to print statistics
+ public static boolean FINEGRAINED_STATISTICS = DMLOptions.defaultOptions.stats; // whether to print statistics
public static int STATISTICS_COUNT = DMLOptions.defaultOptions.statsCount; // statistics maximum heavy hitter count
public static boolean ENABLE_DEBUG_MODE = DMLOptions.defaultOptions.debug; // debug mode
public static ExplainType EXPLAIN = DMLOptions.defaultOptions.explainType; // explain type
http://git-wip-us.apache.org/repos/asf/systemml/blob/648eb21d/src/main/java/org/apache/sysml/api/ScriptExecutorUtils.java
----------------------------------------------------------------------
diff --git a/src/main/java/org/apache/sysml/api/ScriptExecutorUtils.java b/src/main/java/org/apache/sysml/api/ScriptExecutorUtils.java
index 2895aa4..0e0950e 100644
--- a/src/main/java/org/apache/sysml/api/ScriptExecutorUtils.java
+++ b/src/main/java/org/apache/sysml/api/ScriptExecutorUtils.java
@@ -76,6 +76,7 @@ public class ScriptExecutorUtils {
// in digging into performance problems are recorded and displayed
GPUStatistics.DISPLAY_STATISTICS = dmlconf.getBooleanValue(DMLConfig.EXTRA_GPU_STATS);
LibMatrixDNN.DISPLAY_STATISTICS = dmlconf.getBooleanValue(DMLConfig.EXTRA_DNN_STATS);
+ DMLScript.FINEGRAINED_STATISTICS = dmlconf.getBooleanValue(DMLConfig.EXTRA_FINEGRAINED_STATS);
// Sets the maximum number of GPUs per process, -1 for all available
// GPUs
http://git-wip-us.apache.org/repos/asf/systemml/blob/648eb21d/src/main/java/org/apache/sysml/conf/DMLConfig.java
----------------------------------------------------------------------
diff --git a/src/main/java/org/apache/sysml/conf/DMLConfig.java b/src/main/java/org/apache/sysml/conf/DMLConfig.java
index 5b5b8ea..c248098 100644
--- a/src/main/java/org/apache/sysml/conf/DMLConfig.java
+++ b/src/main/java/org/apache/sysml/conf/DMLConfig.java
@@ -75,6 +75,7 @@ public class DMLConfig
public static final String CODEGEN = "codegen.enabled"; //boolean
public static final String CODEGEN_PLANCACHE = "codegen.plancache"; //boolean
public static final String CODEGEN_LITERALS = "codegen.literals"; //1..heuristic, 2..always
+ public static final String EXTRA_FINEGRAINED_STATS = "systemml.stats.finegrained"; //boolean
public static final String EXTRA_GPU_STATS = "systemml.stats.extraGPU"; //boolean
public static final String EXTRA_DNN_STATS = "systemml.stats.extraDNN"; //boolean
public static final String MAX_GPUS_PER_PROCESS = "systemml.gpu.perProcessMax"; // boolean, maximum number of gpus to use, -1 for all
@@ -117,6 +118,7 @@ public class DMLConfig
_defaultVals.put(CODEGEN_PLANCACHE, "true" );
_defaultVals.put(CODEGEN_LITERALS, "1" );
_defaultVals.put(NATIVE_BLAS, "none" );
+ _defaultVals.put(EXTRA_FINEGRAINED_STATS,"false" );
_defaultVals.put(EXTRA_GPU_STATS, "false" );
_defaultVals.put(EXTRA_DNN_STATS, "false" );
@@ -403,7 +405,7 @@ public class DMLConfig
YARN_APPMASTER, YARN_APPMASTERMEM, YARN_MAPREDUCEMEM,
CP_PARALLEL_OPS, CP_PARALLEL_IO, NATIVE_BLAS,
COMPRESSED_LINALG, CODEGEN, CODEGEN_LITERALS, CODEGEN_PLANCACHE,
- EXTRA_GPU_STATS, EXTRA_DNN_STATS
+ EXTRA_GPU_STATS, EXTRA_DNN_STATS, EXTRA_FINEGRAINED_STATS
};
StringBuilder sb = new StringBuilder();
http://git-wip-us.apache.org/repos/asf/systemml/blob/648eb21d/src/main/java/org/apache/sysml/debug/DMLDebuggerProgramInfo.java
----------------------------------------------------------------------
diff --git a/src/main/java/org/apache/sysml/debug/DMLDebuggerProgramInfo.java b/src/main/java/org/apache/sysml/debug/DMLDebuggerProgramInfo.java
index d943a69..92a9495 100644
--- a/src/main/java/org/apache/sysml/debug/DMLDebuggerProgramInfo.java
+++ b/src/main/java/org/apache/sysml/debug/DMLDebuggerProgramInfo.java
@@ -336,9 +336,9 @@ public class DMLDebuggerProgramInfo
}
//set MR job line number
if (min == 0 || min == Integer.MAX_VALUE)
- currMRInst.setLocation(prevLineNum, prevLineNum, -1, -1); //last seen instruction line number
+ currMRInst.setLocation(null, prevLineNum, prevLineNum, -1, -1); //last seen instruction line number
else
- currMRInst.setLocation(min, min, -1, -1); //minimum instruction line number for this MR job
+ currMRInst.setLocation(null, min, min, -1, -1); //minimum instruction line number for this MR job
//insert current MR instruction into corresponding source code line
if (!disassembler.containsKey(currMRInst.getLineNum()))
disassembler.put(currMRInst.getLineNum(), new ArrayList<Instruction>());
@@ -348,7 +348,7 @@ public class DMLDebuggerProgramInfo
{
//if CP instruction line number is not set, then approximate to last seen line number
if (currInst.getLineNum() == 0)
- currInst.setLocation(prevLineNum, prevLineNum, -1, -1);
+ currInst.setLocation(null, prevLineNum, prevLineNum, -1, -1);
//insert current CP instruction into corresponding source code line
if (!disassembler.containsKey(currInst.getLineNum()))
disassembler.put(currInst.getLineNum(), new ArrayList<Instruction>());
http://git-wip-us.apache.org/repos/asf/systemml/blob/648eb21d/src/main/java/org/apache/sysml/hops/BinaryOp.java
----------------------------------------------------------------------
diff --git a/src/main/java/org/apache/sysml/hops/BinaryOp.java b/src/main/java/org/apache/sysml/hops/BinaryOp.java
index 2c88a9e..d072afd 100644
--- a/src/main/java/org/apache/sysml/hops/BinaryOp.java
+++ b/src/main/java/org/apache/sysml/hops/BinaryOp.java
@@ -318,7 +318,7 @@ public class BinaryOp extends Hop
pick.getOutputParameters().setDimensions(getDim1(),
getDim2(), getRowsInBlock(), getColsInBlock(), getNnz());
- pick.setAllPositions(this.getBeginLine(), this.getBeginColumn(), this.getEndLine(), this.getEndColumn());
+ pick.setAllPositions(this.getFilename(), this.getBeginLine(), this.getBeginColumn(), this.getEndLine(), this.getEndColumn());
setLops(pick);
}
@@ -344,7 +344,7 @@ public class BinaryOp extends Hop
pick.getOutputParameters().setDimensions(getDim1(),
getDim2(), getRowsInBlock(), getColsInBlock(), getNnz());
- pick.setAllPositions(this.getBeginLine(), this.getBeginColumn(), this.getEndLine(), this.getEndColumn());
+ pick.setAllPositions(this.getFilename(), this.getBeginLine(), this.getBeginColumn(), this.getEndLine(), this.getEndColumn());
setLops(pick);
}
@@ -465,7 +465,7 @@ public class BinaryOp extends Hop
pick.getOutputParameters().setDimensions(getDim1(),
getDim2(), getRowsInBlock(), getColsInBlock(), getNnz());
- pick.setAllPositions(this.getBeginLine(), this.getBeginColumn(), this.getEndLine(), this.getEndColumn());
+ pick.setAllPositions(this.getFilename(), this.getBeginLine(), this.getBeginColumn(), this.getEndLine(), this.getEndColumn());
setLops(pick);
}
@@ -683,7 +683,7 @@ public class BinaryOp extends Hop
< OptimizerUtils.getLocalMemBudget()) ? ExecType.CP : ExecType.MR; //operator selection
dcInput = new DataPartition(dcInput, DataType.MATRIX, ValueType.DOUBLE, etPart, (right.getDim2()==1)?PDataPartitionFormat.ROW_BLOCK_WISE_N:PDataPartitionFormat.COLUMN_BLOCK_WISE_N);
dcInput.getOutputParameters().setDimensions(right.getDim1(), right.getDim2(), right.getRowsInBlock(), right.getColsInBlock(), right.getNnz());
- dcInput.setAllPositions(right.getBeginLine(), right.getBeginColumn(), right.getEndLine(), right.getEndColumn());
+ dcInput.setAllPositions(right.getFilename(), right.getBeginLine(), right.getBeginColumn(), right.getEndLine(), right.getEndColumn());
}
BinaryM binary = new BinaryM(left.constructLops(), dcInput, HopsOpOp2LopsB.get(op),
@@ -1083,11 +1083,11 @@ public class BinaryOp extends Hop
< OptimizerUtils.getLocalMemBudget()) ? ExecType.CP : ExecType.MR; //operator selection
dcInput = new DataPartition(dcInput, DataType.MATRIX, ValueType.DOUBLE, etPart, PDataPartitionFormat.ROW_BLOCK_WISE_N);
dcInput.getOutputParameters().setDimensions(right.getDim1(), right.getDim2(), right.getRowsInBlock(), right.getColsInBlock(), right.getNnz());
- dcInput.setAllPositions(right.getBeginLine(), right.getBeginColumn(), right.getEndLine(), right.getEndColumn());
+ dcInput.setAllPositions(right.getFilename(), right.getBeginLine(), right.getBeginColumn(), right.getEndLine(), right.getEndColumn());
}
AppendM appM = new AppendM(left.constructLops(), dcInput, offset, dt, vt, cbind, needPart, ExecType.MR);
- appM.setAllPositions(current.getBeginLine(), current.getBeginColumn(), current.getEndLine(), current.getEndColumn());
+ appM.setAllPositions(current.getFilename(), current.getBeginLine(), current.getBeginColumn(), current.getEndLine(), current.getEndColumn());
appM.getOutputParameters().setDimensions(m3_dim1, m3_dim2, brlen, bclen, m3_nnz);
ret = appM;
break;
@@ -1097,15 +1097,15 @@ public class BinaryOp extends Hop
//group
Group group1 = new Group(left.constructLops(), Group.OperationTypes.Sort, DataType.MATRIX, vt);
group1.getOutputParameters().setDimensions(m1_dim1, m1_dim2, brlen, bclen, left.getNnz());
- group1.setAllPositions(left.getBeginLine(), left.getBeginColumn(), left.getEndLine(), left.getEndColumn());
+ group1.setAllPositions(left.getFilename(), left.getBeginLine(), left.getBeginColumn(), left.getEndLine(), left.getEndColumn());
Group group2 = new Group(right.constructLops(), Group.OperationTypes.Sort, DataType.MATRIX, vt);
group1.getOutputParameters().setDimensions(m2_dim1, m2_dim2, brlen, bclen, right.getNnz());
- group1.setAllPositions(right.getBeginLine(), right.getBeginColumn(), right.getEndLine(), right.getEndColumn());
+ group1.setAllPositions(right.getFilename(), right.getBeginLine(), right.getBeginColumn(), right.getEndLine(), right.getEndColumn());
AppendR appR = new AppendR(group1, group2, dt, vt, cbind, ExecType.MR);
appR.getOutputParameters().setDimensions(m3_dim1, m3_dim2, brlen, bclen, m3_nnz);
- appR.setAllPositions(current.getBeginLine(), current.getBeginColumn(), current.getEndLine(), current.getEndColumn());
+ appR.setAllPositions(current.getFilename(), current.getBeginLine(), current.getBeginColumn(), current.getEndLine(), current.getEndColumn());
ret = appR;
break;
@@ -1117,17 +1117,17 @@ public class BinaryOp extends Hop
AppendG appG = new AppendG(left.constructLops(), right.constructLops(), offset, offset2, dt, vt, cbind, ExecType.MR);
appG.getOutputParameters().setDimensions(m3_dim1, m3_dim2, brlen, bclen, m3_nnz);
- appG.setAllPositions(current.getBeginLine(), current.getBeginColumn(), current.getEndLine(), current.getEndColumn());
+ appG.setAllPositions(current.getFilename(), current.getBeginLine(), current.getBeginColumn(), current.getEndLine(), current.getEndColumn());
//group
Group group1 = new Group(appG, Group.OperationTypes.Sort, DataType.MATRIX, vt);
group1.getOutputParameters().setDimensions(m3_dim1, m3_dim2, brlen, bclen, m3_nnz);
- group1.setAllPositions(current.getBeginLine(), current.getBeginColumn(), current.getEndLine(), current.getEndColumn());
+ group1.setAllPositions(current.getFilename(), current.getBeginLine(), current.getBeginColumn(), current.getEndLine(), current.getEndColumn());
//aggregate
Aggregate agg1 = new Aggregate(group1, Aggregate.OperationTypes.Sum, DataType.MATRIX, vt, ExecType.MR);
agg1.getOutputParameters().setDimensions(m3_dim1, m3_dim2, brlen, bclen, m3_nnz);
- agg1.setAllPositions(current.getBeginLine(), current.getBeginColumn(), current.getEndLine(), current.getEndColumn());
+ agg1.setAllPositions(current.getFilename(), current.getBeginLine(), current.getBeginColumn(), current.getEndLine(), current.getEndColumn());
ret = agg1;
break;
}
@@ -1178,7 +1178,7 @@ public class BinaryOp extends Hop
throw new HopsException("Invalid SP append method: "+am);
}
- ret.setAllPositions(current.getBeginLine(), current.getBeginColumn(), current.getEndLine(), current.getEndColumn());
+ ret.setAllPositions(current.getFilename(), current.getBeginLine(), current.getBeginColumn(), current.getEndLine(), current.getEndColumn());
return ret;
@@ -1225,23 +1225,23 @@ public class BinaryOp extends Hop
Group group1 = new Group(left.constructLops(), Group.OperationTypes.Sort, DataType.MATRIX, vt);
group1.getOutputParameters().setDimensions(m1_dim1, m1_dim2, brlen, bclen, left.getNnz());
- group1.setAllPositions(left.getBeginLine(), left.getBeginColumn(), left.getEndLine(), left.getEndColumn());
+ group1.setAllPositions(left.getFilename(), left.getBeginLine(), left.getBeginColumn(), left.getEndLine(), left.getEndColumn());
Group group2 = new Group(right1.constructLops(), Group.OperationTypes.Sort, DataType.MATRIX, vt);
group1.getOutputParameters().setDimensions(m2_dim1, m2_dim2, brlen, bclen, right1.getNnz());
- group1.setAllPositions(right1.getBeginLine(), right1.getBeginColumn(), right1.getEndLine(), right1.getEndColumn());
+ group1.setAllPositions(right1.getFilename(), right1.getBeginLine(), right1.getBeginColumn(), right1.getEndLine(), right1.getEndColumn());
Group group3 = new Group(right2.constructLops(), Group.OperationTypes.Sort, DataType.MATRIX, vt);
group1.getOutputParameters().setDimensions(m3_dim1, m3_dim2, brlen, bclen, right2.getNnz());
- group1.setAllPositions(right2.getBeginLine(), right2.getBeginColumn(), right2.getEndLine(), right2.getEndColumn());
+ group1.setAllPositions(right2.getFilename(), right2.getBeginLine(), right2.getBeginColumn(), right2.getEndLine(), right2.getEndColumn());
AppendR appR1 = new AppendR(group1, group2, dt, vt, cbind, ExecType.MR);
appR1.getOutputParameters().setDimensions(m1_dim1, m41_dim2, brlen, bclen, m41_nnz);
- appR1.setAllPositions(current.getBeginLine(), current.getBeginColumn(), current.getEndLine(), current.getEndColumn());
+ appR1.setAllPositions(current.getFilename(), current.getBeginLine(), current.getBeginColumn(), current.getEndLine(), current.getEndColumn());
AppendR appR2 = new AppendR(appR1, group3, dt, vt, cbind, ExecType.MR);
appR1.getOutputParameters().setDimensions(m1_dim1, m42_dim2, brlen, bclen, m42_nnz);
- appR1.setAllPositions(current.getBeginLine(), current.getBeginColumn(), current.getEndLine(), current.getEndColumn());
+ appR1.setAllPositions(current.getFilename(), current.getBeginLine(), current.getBeginColumn(), current.getEndLine(), current.getEndColumn());
return appR2;
}
http://git-wip-us.apache.org/repos/asf/systemml/blob/648eb21d/src/main/java/org/apache/sysml/hops/Hop.java
----------------------------------------------------------------------
diff --git a/src/main/java/org/apache/sysml/hops/Hop.java b/src/main/java/org/apache/sysml/hops/Hop.java
index 4529d04..7aa58b7 100644
--- a/src/main/java/org/apache/sysml/hops/Hop.java
+++ b/src/main/java/org/apache/sysml/hops/Hop.java
@@ -444,7 +444,7 @@ public abstract class Hop
}
offset.getOutputParameters().setDimensions(0, 0, 0, 0, -1);
- offset.setAllPositions(hop.getBeginLine(), hop.getBeginColumn(), hop.getEndLine(), hop.getEndColumn());
+ offset.setAllPositions(hop.getFilename(), hop.getBeginLine(), hop.getBeginColumn(), hop.getEndLine(), hop.getEndColumn());
return offset;
}
@@ -1802,13 +1802,16 @@ public abstract class Hop
///////////////////////////////////////////////////////////////////////////
public int _beginLine, _beginColumn;
public int _endLine, _endColumn;
+ public String _filename;
public void setBeginLine(int passed) { _beginLine = passed; }
public void setBeginColumn(int passed) { _beginColumn = passed; }
public void setEndLine(int passed) { _endLine = passed; }
public void setEndColumn(int passed) { _endColumn = passed; }
+ public void setFilename(String passed) { _filename = passed; }
- public void setAllPositions(int blp, int bcp, int elp, int ecp){
+ public void setAllPositions(String filename, int blp, int bcp, int elp, int ecp){
+ _filename = filename;
_beginLine = blp;
_beginColumn = bcp;
_endLine = elp;
@@ -1819,9 +1822,13 @@ public abstract class Hop
public int getBeginColumn() { return _beginColumn; }
public int getEndLine() { return _endLine; }
public int getEndColumn() { return _endColumn; }
+ public String getFilename() { return _filename; }
public String printErrorLocation(){
- return "ERROR: line " + _beginLine + ", column " + _beginColumn + " -- ";
+ if(_filename != null)
+ return "ERROR: " + _filename + " line " + _beginLine + ", column " + _beginColumn + " -- ";
+ else
+ return "ERROR: line " + _beginLine + ", column " + _beginColumn + " -- ";
}
/**
@@ -1831,7 +1838,7 @@ public abstract class Hop
*/
protected void setLineNumbers(Lop lop)
{
- lop.setAllPositions(this.getBeginLine(), this.getBeginColumn(), this.getEndLine(), this.getEndColumn());
+ lop.setAllPositions(this.getFilename(), this.getBeginLine(), this.getBeginColumn(), this.getEndLine(), this.getEndColumn());
}
} // end class
http://git-wip-us.apache.org/repos/asf/systemml/blob/648eb21d/src/main/java/org/apache/sysml/hops/TernaryOp.java
----------------------------------------------------------------------
diff --git a/src/main/java/org/apache/sysml/hops/TernaryOp.java b/src/main/java/org/apache/sysml/hops/TernaryOp.java
index e1bef3e..22895e5 100644
--- a/src/main/java/org/apache/sysml/hops/TernaryOp.java
+++ b/src/main/java/org/apache/sysml/hops/TernaryOp.java
@@ -422,7 +422,7 @@ public class TernaryOp extends Hop
Ternary tertiary = new Ternary(inputLops, tertiaryOp, getDataType(), getValueType(), ignoreZeros, et);
tertiary.getOutputParameters().setDimensions(_dim1, _dim2, getRowsInBlock(), getColsInBlock(), -1);
- tertiary.setAllPositions(this.getBeginLine(), this.getBeginColumn(), this.getEndLine(), this.getEndColumn());
+ tertiary.setAllPositions(this.getFilename(), this.getBeginLine(), this.getBeginColumn(), this.getEndLine(), this.getEndColumn());
//force blocked output in CP (see below), otherwise binarycell
if ( et == ExecType.SPARK ) {
@@ -447,7 +447,7 @@ public class TernaryOp extends Hop
group1.getOutputParameters().setDimensions(getDim1(),
getDim2(), getRowsInBlock(), getColsInBlock(), getNnz());
- group1.setAllPositions(this.getBeginLine(), this.getBeginColumn(), this.getEndLine(), this.getEndColumn());
+ group1.setAllPositions(this.getFilename(), this.getBeginLine(), this.getBeginColumn(), this.getEndLine(), this.getEndColumn());
Ternary tertiary = null;
// create "group" lops for MATRIX inputs
@@ -462,7 +462,7 @@ public class TernaryOp extends Hop
group2.getOutputParameters().setDimensions(getDim1(),
getDim2(), getRowsInBlock(),
getColsInBlock(), getNnz());
- group2.setAllPositions(this.getBeginLine(), this.getBeginColumn(), this.getEndLine(), this.getEndColumn());
+ group2.setAllPositions(this.getFilename(), this.getBeginLine(), this.getBeginColumn(), this.getEndLine(), this.getEndColumn());
group3 = new Group(
inputLops[2],
@@ -471,7 +471,7 @@ public class TernaryOp extends Hop
group3.getOutputParameters().setDimensions(getDim1(),
getDim2(), getRowsInBlock(),
getColsInBlock(), getNnz());
- group3.setAllPositions(this.getBeginLine(), this.getBeginColumn(), this.getEndLine(), this.getEndColumn());
+ group3.setAllPositions(this.getFilename(), this.getBeginLine(), this.getBeginColumn(), this.getEndLine(), this.getEndColumn());
if ( inputLops.length == 3 )
tertiary = new Ternary(
@@ -495,7 +495,7 @@ public class TernaryOp extends Hop
group2.getOutputParameters().setDimensions(getDim1(),
getDim2(), getRowsInBlock(),
getColsInBlock(), getNnz());
- group2.setAllPositions(this.getBeginLine(), this.getBeginColumn(), this.getEndLine(), this.getEndColumn());
+ group2.setAllPositions(this.getFilename(), this.getBeginLine(), this.getBeginColumn(), this.getEndLine(), this.getEndColumn());
if ( inputLops.length == 3)
tertiary = new Ternary(
@@ -579,7 +579,7 @@ public class TernaryOp extends Hop
group3.getOutputParameters().setDimensions(getDim1(),
getDim2(), getRowsInBlock(),
getColsInBlock(), getNnz());
- group3.setAllPositions(this.getBeginLine(), this.getBeginColumn(), this.getEndLine(), this.getEndColumn());
+ group3.setAllPositions(this.getFilename(), this.getBeginLine(), this.getBeginColumn(), this.getEndLine(), this.getEndColumn());
if ( inputLops.length == 3)
tertiary = new Ternary(
@@ -618,14 +618,14 @@ public class TernaryOp extends Hop
tertiary, Group.OperationTypes.Sort, getDataType(),
getValueType());
group4.getOutputParameters().setDimensions(_dim1, _dim2, ( _dimInputsPresent ? getRowsInBlock() : -1), ( _dimInputsPresent ? getColsInBlock() : -1), -1);
- group4.setAllPositions(this.getBeginLine(), this.getBeginColumn(), this.getEndLine(), this.getEndColumn());
+ group4.setAllPositions(this.getFilename(), this.getBeginLine(), this.getBeginColumn(), this.getEndLine(), this.getEndColumn());
Aggregate agg1 = new Aggregate(
group4, HopsAgg2Lops.get(AggOp.SUM), getDataType(),
getValueType(), ExecType.MR);
agg1.getOutputParameters().setDimensions(_dim1, _dim2, ( _dimInputsPresent ? getRowsInBlock() : -1), ( _dimInputsPresent ? getColsInBlock() : -1), -1);
- agg1.setAllPositions(this.getBeginLine(), this.getBeginColumn(), this.getEndLine(), this.getEndColumn());
+ agg1.setAllPositions(this.getFilename(), this.getBeginLine(), this.getBeginColumn(), this.getEndLine(), this.getEndColumn());
// kahamSum is used for aggregation but inputs do not have
// correction values
http://git-wip-us.apache.org/repos/asf/systemml/blob/648eb21d/src/main/java/org/apache/sysml/hops/UnaryOp.java
----------------------------------------------------------------------
diff --git a/src/main/java/org/apache/sysml/hops/UnaryOp.java b/src/main/java/org/apache/sysml/hops/UnaryOp.java
index 61ebedf..35902b7 100644
--- a/src/main/java/org/apache/sysml/hops/UnaryOp.java
+++ b/src/main/java/org/apache/sysml/hops/UnaryOp.java
@@ -290,7 +290,7 @@ public class UnaryOp extends Hop implements MultiThreadedHop
Data lit = Data.createLiteralLop(ValueType.DOUBLE, Double.toString(0.25));
- lit.setAllPositions(this.getBeginLine(), this.getBeginColumn(), this.getEndLine(), this.getEndColumn());
+ lit.setAllPositions(this.getFilename(), this.getBeginLine(), this.getBeginColumn(), this.getEndLine(), this.getEndColumn());
PickByCount pick = new PickByCount(
sort, lit, DataType.MATRIX, getValueType(),
http://git-wip-us.apache.org/repos/asf/systemml/blob/648eb21d/src/main/java/org/apache/sysml/hops/rewrite/HopRewriteUtils.java
----------------------------------------------------------------------
diff --git a/src/main/java/org/apache/sysml/hops/rewrite/HopRewriteUtils.java b/src/main/java/org/apache/sysml/hops/rewrite/HopRewriteUtils.java
index ec8e20d..14bcc1e 100644
--- a/src/main/java/org/apache/sysml/hops/rewrite/HopRewriteUtils.java
+++ b/src/main/java/org/apache/sysml/hops/rewrite/HopRewriteUtils.java
@@ -645,7 +645,7 @@ public class HopRewriteUtils
}
public static void copyLineNumbers( Hop src, Hop dest ) {
- dest.setAllPositions(src.getBeginLine(), src.getBeginColumn(), src.getEndLine(), src.getEndColumn());
+ dest.setAllPositions(src.getFilename(), src.getBeginLine(), src.getBeginColumn(), src.getEndLine(), src.getEndColumn());
}
public static void updateHopCharacteristics( Hop hop, long brlen, long bclen, Hop src )
http://git-wip-us.apache.org/repos/asf/systemml/blob/648eb21d/src/main/java/org/apache/sysml/lops/CombineBinary.java
----------------------------------------------------------------------
diff --git a/src/main/java/org/apache/sysml/lops/CombineBinary.java b/src/main/java/org/apache/sysml/lops/CombineBinary.java
index 1c98b11..f3c757e 100644
--- a/src/main/java/org/apache/sysml/lops/CombineBinary.java
+++ b/src/main/java/org/apache/sysml/lops/CombineBinary.java
@@ -124,7 +124,7 @@ public class CombineBinary extends Lop
}
CombineBinary comn = new CombineBinary(op, input1, input2, dt, vt);
- comn.setAllPositions(input1.getBeginLine(), input1.getBeginColumn(), input1.getEndLine(), input1.getEndColumn());
+ comn.setAllPositions(input1.getFilename(), input1.getBeginLine(), input1.getBeginColumn(), input1.getEndLine(), input1.getEndColumn());
return comn;
}
http://git-wip-us.apache.org/repos/asf/systemml/blob/648eb21d/src/main/java/org/apache/sysml/lops/CombineTernary.java
----------------------------------------------------------------------
diff --git a/src/main/java/org/apache/sysml/lops/CombineTernary.java b/src/main/java/org/apache/sysml/lops/CombineTernary.java
index e580045..9200f5c 100644
--- a/src/main/java/org/apache/sysml/lops/CombineTernary.java
+++ b/src/main/java/org/apache/sysml/lops/CombineTernary.java
@@ -115,7 +115,7 @@ public class CombineTernary extends Lop
}
CombineTernary comn = new CombineTernary(op, input1, input2, input3, dt, vt);
- comn.setAllPositions(input1.getBeginLine(), input1.getBeginColumn(), input1.getEndLine(), input1.getEndColumn());
+ comn.setAllPositions(input1.getFilename(), input1.getBeginLine(), input1.getBeginColumn(), input1.getEndLine(), input1.getEndColumn());
return comn;
}
http://git-wip-us.apache.org/repos/asf/systemml/blob/648eb21d/src/main/java/org/apache/sysml/lops/CombineUnary.java
----------------------------------------------------------------------
diff --git a/src/main/java/org/apache/sysml/lops/CombineUnary.java b/src/main/java/org/apache/sysml/lops/CombineUnary.java
index 98f0a99..4882e3c 100644
--- a/src/main/java/org/apache/sysml/lops/CombineUnary.java
+++ b/src/main/java/org/apache/sysml/lops/CombineUnary.java
@@ -90,7 +90,7 @@ public class CombineUnary extends Lop
}
CombineUnary comn = new CombineUnary(input1, dt, vt);
- comn.setAllPositions(input1.getBeginLine(), input1.getBeginColumn(), input1.getEndLine(), input1.getEndLine());
+ comn.setAllPositions(input1.getFilename(), input1.getBeginLine(), input1.getBeginColumn(), input1.getEndLine(), input1.getEndLine());
return comn;
}
http://git-wip-us.apache.org/repos/asf/systemml/blob/648eb21d/src/main/java/org/apache/sysml/lops/Lop.java
----------------------------------------------------------------------
diff --git a/src/main/java/org/apache/sysml/lops/Lop.java b/src/main/java/org/apache/sysml/lops/Lop.java
index c92fb57..1e88561 100644
--- a/src/main/java/org/apache/sysml/lops/Lop.java
+++ b/src/main/java/org/apache/sysml/lops/Lop.java
@@ -23,7 +23,6 @@ import java.util.ArrayList;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
-
import org.apache.sysml.lops.LopProperties.ExecLocation;
import org.apache.sysml.lops.LopProperties.ExecType;
import org.apache.sysml.lops.compile.Dag;
@@ -620,13 +619,16 @@ public abstract class Lop
///////////////////////////////////////////////////////////////////////////
public int _beginLine, _beginColumn;
public int _endLine, _endColumn;
+ public String _filename;
public void setBeginLine(int passed) { _beginLine = passed; }
public void setBeginColumn(int passed) { _beginColumn = passed; }
public void setEndLine(int passed) { _endLine = passed; }
public void setEndColumn(int passed) { _endColumn = passed; }
+ public void setFilename(String passed) { _filename = passed; }
- public void setAllPositions(int blp, int bcp, int elp, int ecp){
+ public void setAllPositions(String filename, int blp, int bcp, int elp, int ecp){
+ _filename = filename;
_beginLine = blp;
_beginColumn = bcp;
_endLine = elp;
@@ -637,6 +639,7 @@ public abstract class Lop
public int getBeginColumn() { return _beginColumn; }
public int getEndLine() { return _endLine; }
public int getEndColumn() { return _endColumn; }
+ public String getFilename() { return _filename; }
public String printErrorLocation(){
return "ERROR: line " + _beginLine + ", column " + _beginColumn + " -- ";
http://git-wip-us.apache.org/repos/asf/systemml/blob/648eb21d/src/main/java/org/apache/sysml/lops/SortKeys.java
----------------------------------------------------------------------
diff --git a/src/main/java/org/apache/sysml/lops/SortKeys.java b/src/main/java/org/apache/sysml/lops/SortKeys.java
index e2c6b6a..6344eda 100644
--- a/src/main/java/org/apache/sysml/lops/SortKeys.java
+++ b/src/main/java/org/apache/sysml/lops/SortKeys.java
@@ -152,7 +152,7 @@ public class SortKeys extends Lop
}
SortKeys retVal = new SortKeys(input1, op, dt, vt, et);
- retVal.setAllPositions(input1.getBeginLine(), input1.getBeginColumn(), input1.getEndLine(), input1.getEndColumn());
+ retVal.setAllPositions(input1.getFilename(), input1.getBeginLine(), input1.getBeginColumn(), input1.getEndLine(), input1.getEndColumn());
return retVal;
}
@@ -172,7 +172,7 @@ public class SortKeys extends Lop
}
SortKeys retVal = new SortKeys(input1, input2, op, dt, vt, et);
- retVal.setAllPositions(input1.getBeginLine(), input1.getBeginColumn(), input1.getEndLine(), input1.getEndColumn());
+ retVal.setAllPositions(input1.getFilename(), input1.getBeginLine(), input1.getBeginColumn(), input1.getEndLine(), input1.getEndColumn());
return retVal;
}
http://git-wip-us.apache.org/repos/asf/systemml/blob/648eb21d/src/main/java/org/apache/sysml/lops/compile/Dag.java
----------------------------------------------------------------------
diff --git a/src/main/java/org/apache/sysml/lops/compile/Dag.java b/src/main/java/org/apache/sysml/lops/compile/Dag.java
index ddee17d..1da1ba6 100644
--- a/src/main/java/org/apache/sysml/lops/compile/Dag.java
+++ b/src/main/java/org/apache/sysml/lops/compile/Dag.java
@@ -366,7 +366,7 @@ public class Dag<N extends Lop>
for (String varName : sb.liveIn().getVariableNames()) {
if (!sb.liveOut().containsVariable(varName)) {
inst = VariableCPInstruction.prepareRemoveInstruction(varName);
- inst.setLocation(sb.getEndLine(), sb.getEndLine(), -1, -1);
+ inst.setLocation(sb.getFilename(), sb.getEndLine(), sb.getEndLine(), -1, -1);
deleteInst.add(inst);
http://git-wip-us.apache.org/repos/asf/systemml/blob/648eb21d/src/main/java/org/apache/sysml/parser/DMLProgram.java
----------------------------------------------------------------------
diff --git a/src/main/java/org/apache/sysml/parser/DMLProgram.java b/src/main/java/org/apache/sysml/parser/DMLProgram.java
index 2f0d682..15d46c7 100644
--- a/src/main/java/org/apache/sysml/parser/DMLProgram.java
+++ b/src/main/java/org/apache/sysml/parser/DMLProgram.java
@@ -284,7 +284,7 @@ public class DMLProgram
retPB.setStatementBlock(sb);
// add location information
- retPB.setAllPositions(sb.getBeginLine(), sb.getBeginColumn(), sb.getEndLine(), sb.getEndColumn());
+ retPB.setAllPositions(sb.getFilename(), sb.getBeginLine(), sb.getBeginColumn(), sb.getEndLine(), sb.getEndColumn());
}
// process If Statement - add runtime program blocks to program
@@ -351,7 +351,7 @@ public class DMLProgram
retPB.setStatementBlock(sb);
// add location information
- retPB.setAllPositions(sb.getBeginLine(), sb.getBeginColumn(), sb.getEndLine(), sb.getEndColumn());
+ retPB.setAllPositions(sb.getFilename(), sb.getBeginLine(), sb.getBeginColumn(), sb.getEndLine(), sb.getEndColumn());
}
// process For Statement - add runtime program blocks to program
@@ -429,7 +429,7 @@ public class DMLProgram
retPB.setStatementBlock(sb);
// add location information
- retPB.setAllPositions(sb.getBeginLine(), sb.getBeginColumn(), sb.getEndLine(), sb.getEndColumn());
+ retPB.setAllPositions(sb.getFilename(), sb.getBeginLine(), sb.getBeginColumn(), sb.getEndLine(), sb.getEndColumn());
}
// process function statement block - add runtime program blocks to program
@@ -511,7 +511,7 @@ public class DMLProgram
retPB = rtpb;
// add location information
- retPB.setAllPositions(sb.getBeginLine(), sb.getBeginColumn(), sb.getEndLine(), sb.getEndColumn());
+ retPB.setAllPositions(sb.getFilename(), sb.getBeginLine(), sb.getBeginColumn(), sb.getEndLine(), sb.getEndColumn());
}
else {
@@ -548,7 +548,7 @@ public class DMLProgram
retPB.setStatementBlock(sb);
// add location information
- retPB.setAllPositions(sb.getBeginLine(), sb.getBeginColumn(), sb.getEndLine(), sb.getEndColumn());
+ retPB.setAllPositions(sb.getFilename(), sb.getBeginLine(), sb.getBeginColumn(), sb.getEndLine(), sb.getEndColumn());
}
return retPB;
http://git-wip-us.apache.org/repos/asf/systemml/blob/648eb21d/src/main/java/org/apache/sysml/parser/DMLTranslator.java
----------------------------------------------------------------------
diff --git a/src/main/java/org/apache/sysml/parser/DMLTranslator.java b/src/main/java/org/apache/sysml/parser/DMLTranslator.java
index 42ab12e..61f7217 100644
--- a/src/main/java/org/apache/sysml/parser/DMLTranslator.java
+++ b/src/main/java/org/apache/sysml/parser/DMLTranslator.java
@@ -877,7 +877,7 @@ public class DMLTranslator
long actualDim1 = (var instanceof IndexedIdentifier) ? ((IndexedIdentifier)var).getOrigDim1() : var.getDim1();
long actualDim2 = (var instanceof IndexedIdentifier) ? ((IndexedIdentifier)var).getOrigDim2() : var.getDim2();
DataOp read = new DataOp(var.getName(), var.getDataType(), var.getValueType(), DataOpTypes.TRANSIENTREAD, null, actualDim1, actualDim2, var.getNnz(), var.getRowsInBlock(), var.getColumnsInBlock());
- read.setAllPositions(var.getBeginLine(), var.getBeginColumn(), var.getEndLine(), var.getEndColumn());
+ read.setAllPositions(var.getFilename(), var.getBeginLine(), var.getBeginColumn(), var.getEndLine(), var.getEndColumn());
ids.put(varName, read);
}
}
@@ -946,7 +946,7 @@ public class DMLTranslator
Hop ae = processExpression(source, target, ids);
Hop printHop = new UnaryOp(target.getName(), target.getDataType(), target.getValueType(), op,
ae);
- printHop.setAllPositions(current.getBeginLine(), current.getBeginColumn(), current.getEndLine(),
+ printHop.setAllPositions(current.getFilename(), current.getBeginLine(), current.getBeginColumn(), current.getEndLine(),
current.getEndColumn());
output.add(printHop);
} else if (ptype == PRINTTYPE.STOP) {
@@ -955,7 +955,7 @@ public class DMLTranslator
Hop ae = processExpression(source, target, ids);
Hop stopHop = new UnaryOp(target.getName(), target.getDataType(), target.getValueType(), op,
ae);
- stopHop.setAllPositions(current.getBeginLine(), current.getBeginColumn(), current.getEndLine(),
+ stopHop.setAllPositions(current.getFilename(), current.getBeginLine(), current.getBeginColumn(), current.getEndLine(),
current.getEndColumn());
output.add(stopHop);
} else if (ptype == PRINTTYPE.PRINTF) {
@@ -1000,7 +1000,7 @@ public class DMLTranslator
if ((statementId != null) && (statementId.intValue() == i)) {
DataOp transientwrite = new DataOp(target.getName(), target.getDataType(), target.getValueType(), ae, DataOpTypes.TRANSIENTWRITE, null);
transientwrite.setOutputParams(ae.getDim1(), ae.getDim2(), ae.getNnz(), ae.getUpdateType(), ae.getRowsInBlock(), ae.getColsInBlock());
- transientwrite.setAllPositions(target.getBeginLine(), target.getBeginColumn(), target.getEndLine(), target.getEndLine());
+ transientwrite.setAllPositions(target.getFilename(), target.getBeginLine(), target.getBeginColumn(), target.getEndLine(), target.getEndLine());
updatedLiveOut.addVariable(target.getName(), target);
output.add(transientwrite);
}
@@ -1031,7 +1031,7 @@ public class DMLTranslator
if ((statementId != null) && (statementId.intValue() == i)) {
DataOp transientwrite = new DataOp(target.getName(), target.getDataType(), target.getValueType(), ae, DataOpTypes.TRANSIENTWRITE, null);
transientwrite.setOutputParams(origDim1, origDim2, ae.getNnz(), ae.getUpdateType(), ae.getRowsInBlock(), ae.getColsInBlock());
- transientwrite.setAllPositions(target.getBeginLine(), target.getBeginColumn(), target.getEndLine(), target.getEndColumn());
+ transientwrite.setAllPositions(target.getFilename(), target.getBeginLine(), target.getBeginColumn(), target.getEndLine(), target.getEndColumn());
updatedLiveOut.addVariable(target.getName(), target);
output.add(transientwrite);
}
@@ -1241,7 +1241,7 @@ public class DMLTranslator
read = new DataOp(var.getName(), var.getDataType(), var.getValueType(), DataOpTypes.TRANSIENTREAD,
null, actualDim1, actualDim2, var.getNnz(), var.getRowsInBlock(), var.getColumnsInBlock());
- read.setAllPositions(var.getBeginLine(), var.getBeginColumn(), var.getEndLine(), var.getEndColumn());
+ read.setAllPositions(var.getFilename(), var.getBeginLine(), var.getBeginColumn(), var.getEndLine(), var.getEndColumn());
}
_ids.put(varName, read);
}
@@ -1339,7 +1339,7 @@ public class DMLTranslator
long actualDim2 = (var instanceof IndexedIdentifier) ? ((IndexedIdentifier)var).getOrigDim2() : var.getDim2();
read = new DataOp(var.getName(), var.getDataType(), var.getValueType(), DataOpTypes.TRANSIENTREAD,
null, actualDim1, actualDim2, var.getNnz(), var.getRowsInBlock(), var.getColumnsInBlock());
- read.setAllPositions(var.getBeginLine(), var.getBeginColumn(), var.getEndLine(), var.getEndColumn());
+ read.setAllPositions(var.getFilename(), var.getBeginLine(), var.getBeginColumn(), var.getEndLine(), var.getEndColumn());
}
_ids.put(varName, read);
}
@@ -1417,28 +1417,28 @@ public class DMLTranslator
else if (source instanceof IntIdentifier) {
IntIdentifier sourceInt = (IntIdentifier) source;
LiteralOp litop = new LiteralOp(sourceInt.getValue());
- litop.setAllPositions(sourceInt.getBeginLine(), sourceInt.getBeginColumn(), sourceInt.getEndLine(), sourceInt.getEndColumn());
+ litop.setAllPositions(sourceInt.getFilename(), sourceInt.getBeginLine(), sourceInt.getBeginColumn(), sourceInt.getEndLine(), sourceInt.getEndColumn());
setIdentifierParams(litop, sourceInt);
return litop;
}
else if (source instanceof DoubleIdentifier) {
DoubleIdentifier sourceDouble = (DoubleIdentifier) source;
LiteralOp litop = new LiteralOp(sourceDouble.getValue());
- litop.setAllPositions(sourceDouble.getBeginLine(), sourceDouble.getBeginColumn(), sourceDouble.getEndLine(), sourceDouble.getEndColumn());
+ litop.setAllPositions(source.getFilename(), sourceDouble.getBeginLine(), sourceDouble.getBeginColumn(), sourceDouble.getEndLine(), sourceDouble.getEndColumn());
setIdentifierParams(litop, sourceDouble);
return litop;
}
else if (source instanceof BooleanIdentifier) {
BooleanIdentifier sourceBoolean = (BooleanIdentifier) source;
LiteralOp litop = new LiteralOp(sourceBoolean.getValue());
- litop.setAllPositions(sourceBoolean.getBeginLine(), sourceBoolean.getBeginColumn(), sourceBoolean.getEndLine(), sourceBoolean.getEndColumn());
+ litop.setAllPositions(sourceBoolean.getFilename(), sourceBoolean.getBeginLine(), sourceBoolean.getBeginColumn(), sourceBoolean.getEndLine(), sourceBoolean.getEndColumn());
setIdentifierParams(litop, sourceBoolean);
return litop;
}
else if (source instanceof StringIdentifier) {
StringIdentifier sourceString = (StringIdentifier) source;
LiteralOp litop = new LiteralOp(sourceString.getValue());
- litop.setAllPositions(sourceString.getBeginLine(), sourceString.getBeginColumn(), sourceString.getEndLine(), sourceString.getEndColumn());
+ litop.setAllPositions(sourceString.getFilename(), sourceString.getBeginLine(), sourceString.getBeginColumn(), sourceString.getEndLine(), sourceString.getEndColumn());
setIdentifierParams(litop, sourceString);
return litop;
}
@@ -1505,7 +1505,7 @@ public class DMLTranslator
rowUpperHops = new LiteralOp(target.getOrigDim1());
else {
rowUpperHops = new UnaryOp(target.getName(), DataType.SCALAR, ValueType.INT, Hop.OpOp1.NROW, hops.get(target.getName()));
- rowUpperHops.setAllPositions(target.getBeginLine(), target.getBeginColumn(), target.getEndLine(), target.getEndColumn());
+ rowUpperHops.setAllPositions(target.getFilename(), target.getBeginLine(), target.getBeginColumn(), target.getEndLine(), target.getEndColumn());
}
}
if (target.getColLowerBound() != null)
@@ -1543,7 +1543,7 @@ public class DMLTranslator
setIdentifierParams(leftIndexOp, target);
- leftIndexOp.setAllPositions(target.getBeginLine(), target.getBeginColumn(), target.getEndLine(), target.getEndColumn());
+ leftIndexOp.setAllPositions(target.getFilename(), target.getBeginLine(), target.getBeginColumn(), target.getEndLine(), target.getEndColumn());
leftIndexOp.setDim1(target.getOrigDim1());
leftIndexOp.setDim2(target.getOrigDim2());
@@ -1570,7 +1570,7 @@ public class DMLTranslator
rowUpperHops = new LiteralOp(source.getOrigDim1());
else {
rowUpperHops = new UnaryOp(source.getName(), DataType.SCALAR, ValueType.INT, Hop.OpOp1.NROW, hops.get(source.getName()));
- rowUpperHops.setAllPositions(source.getBeginLine(),source.getBeginColumn(), source.getEndLine(), source.getEndColumn());
+ rowUpperHops.setAllPositions(source.getFilename(), source.getBeginLine(),source.getBeginColumn(), source.getEndLine(), source.getEndColumn());
}
}
if (source.getColLowerBound() != null)
@@ -1599,7 +1599,7 @@ public class DMLTranslator
hops.get(source.getName()), rowLowerHops, rowUpperHops, colLowerHops, colUpperHops,
source.getRowLowerEqualsUpper(), source.getColLowerEqualsUpper());
- indexOp.setAllPositions(indexOp.getBeginLine(), indexOp.getBeginColumn(), indexOp.getEndLine(), indexOp.getEndColumn());
+ indexOp.setAllPositions(target.getFilename(), target.getBeginLine(), target.getBeginColumn(), target.getEndLine(), target.getEndColumn());
setIdentifierParams(indexOp, target);
return indexOp;
@@ -1657,7 +1657,7 @@ public class DMLTranslator
throw new ParseException("Unsupported parsing of binary expression: "+source.getOpCode());
}
setIdentifierParams(currBop, source.getOutput());
- currBop.setAllPositions(source.getBeginLine(), source.getBeginColumn(), source.getEndLine(), source.getEndColumn());
+ currBop.setAllPositions(source.getFilename(), source.getBeginLine(), source.getBeginColumn(), source.getEndLine(), source.getEndColumn());
return currBop;
}
@@ -1701,7 +1701,7 @@ public class DMLTranslator
op = OpOp2.NOTEQUAL;
}
currBop = new BinaryOp(target.getName(), target.getDataType(), target.getValueType(), op, left, right);
- currBop.setAllPositions(source.getBeginLine(), source.getBeginColumn(), source.getEndLine(), source.getEndColumn());
+ currBop.setAllPositions(source.getFilename(), source.getBeginLine(), source.getBeginColumn(), source.getEndLine(), source.getEndColumn());
return currBop;
}
@@ -1735,7 +1735,7 @@ public class DMLTranslator
if (source.getRight() == null) {
Hop currUop = new UnaryOp(target.getName(), target.getDataType(), target.getValueType(), Hop.OpOp1.NOT, left);
- currUop.setAllPositions(source.getBeginLine(), source.getBeginColumn(), source.getEndLine(), source.getEndColumn());
+ currUop.setAllPositions(source.getFilename(), source.getBeginLine(), source.getBeginColumn(), source.getEndLine(), source.getEndColumn());
return currUop;
}
else {
@@ -1751,7 +1751,7 @@ public class DMLTranslator
throw new RuntimeException(source.printErrorLocation() + "Unknown boolean operation " + source.getOpCode());
}
currBop = new BinaryOp(target.getName(), target.getDataType(), target.getValueType(), op, left, right);
- currBop.setAllPositions(source.getBeginLine(), source.getBeginColumn(), source.getEndLine(), source.getEndColumn());
+ currBop.setAllPositions(source.getFilename(), source.getBeginLine(), source.getBeginColumn(), source.getEndLine(), source.getEndColumn());
// setIdentifierParams(currBop,source.getOutput());
return currBop;
}
@@ -1830,9 +1830,9 @@ public class DMLTranslator
// set properties for created hops based on outputs of source expression
for ( int i=0; i < source.getOutputs().length; i++ ) {
setIdentifierParams( outputs.get(i), source.getOutputs()[i]);
- outputs.get(i).setAllPositions(source.getBeginLine(), source.getBeginColumn(), source.getEndLine(), source.getEndColumn());
+ outputs.get(i).setAllPositions(source.getFilename(), source.getBeginLine(), source.getBeginColumn(), source.getEndLine(), source.getEndColumn());
}
- currBuiltinOp.setAllPositions(source.getBeginLine(), source.getBeginColumn(), source.getEndLine(), source.getEndColumn());
+ currBuiltinOp.setAllPositions(source.getFilename(), source.getBeginLine(), source.getBeginColumn(), source.getEndLine(), source.getEndColumn());
return currBuiltinOp;
}
@@ -1952,7 +1952,7 @@ public class DMLTranslator
setIdentifierParams(currBuiltinOp, source.getOutput());
- currBuiltinOp.setAllPositions(source.getBeginLine(), source.getBeginColumn(), source.getEndLine(), source.getEndColumn());
+ currBuiltinOp.setAllPositions(source.getFilename(), source.getBeginLine(), source.getBeginColumn(), source.getEndLine(), source.getEndColumn());
return currBuiltinOp;
}
@@ -2040,7 +2040,7 @@ public class DMLTranslator
setIdentifierParams(currBuiltinOp, source.getOutput());
if( source.getOpCode()==DataExpression.DataOp.READ )
((DataOp)currBuiltinOp).setInputBlockSizes(target.getRowsInBlock(), target.getColumnsInBlock());
- currBuiltinOp.setAllPositions(source.getBeginLine(), source.getBeginColumn(), source.getEndLine(), source.getEndColumn());
+ currBuiltinOp.setAllPositions(source.getFilename(), source.getBeginLine(), source.getBeginColumn(), source.getEndLine(), source.getEndColumn());
return currBuiltinOp;
}
@@ -2102,9 +2102,9 @@ public class DMLTranslator
// set properties for created hops based on outputs of source expression
for ( int i=0; i < source.getOutputs().length; i++ ) {
setIdentifierParams( outputs.get(i), source.getOutputs()[i]);
- outputs.get(i).setAllPositions(source.getBeginLine(), source.getBeginColumn(), source.getEndLine(), source.getEndColumn());
+ outputs.get(i).setAllPositions(source.getFilename(), source.getBeginLine(), source.getBeginColumn(), source.getEndLine(), source.getEndColumn());
}
- currBuiltinOp.setAllPositions(source.getBeginLine(), source.getBeginColumn(), source.getEndLine(), source.getEndColumn());
+ currBuiltinOp.setAllPositions(source.getFilename(), source.getBeginLine(), source.getBeginColumn(), source.getEndLine(), source.getEndColumn());
return currBuiltinOp;
}
@@ -2776,7 +2776,7 @@ public class DMLTranslator
// Since the dimension of output doesnot match that of input variable for these operations
setIdentifierParams(currBuiltinOp, source.getOutput());
}
- currBuiltinOp.setAllPositions(source.getBeginLine(), source.getBeginColumn(), source.getEndLine(), source.getEndColumn());
+ currBuiltinOp.setAllPositions(source.getFilename(), source.getBeginLine(), source.getBeginColumn(), source.getEndLine(), source.getEndColumn());
return currBuiltinOp;
}
http://git-wip-us.apache.org/repos/asf/systemml/blob/648eb21d/src/main/java/org/apache/sysml/runtime/controlprogram/ExternalFunctionProgramBlock.java
----------------------------------------------------------------------
diff --git a/src/main/java/org/apache/sysml/runtime/controlprogram/ExternalFunctionProgramBlock.java b/src/main/java/org/apache/sysml/runtime/controlprogram/ExternalFunctionProgramBlock.java
index 3c72ca9..7b364d7 100644
--- a/src/main/java/org/apache/sysml/runtime/controlprogram/ExternalFunctionProgramBlock.java
+++ b/src/main/java/org/apache/sysml/runtime/controlprogram/ExternalFunctionProgramBlock.java
@@ -304,7 +304,7 @@ public class ExternalFunctionProgramBlock extends FunctionProgramBlock
else if (getOutputParams().size() > 0)
einst.setLocation(getOutputParams().get(0));
else
- einst.setLocation(this._beginLine, this._endLine, this._beginColumn, this._endColumn);
+ einst.setLocation(this.getFilename(), this._beginLine, this._endLine, this._beginColumn, this._endColumn);
_inst.add(einst);
http://git-wip-us.apache.org/repos/asf/systemml/blob/648eb21d/src/main/java/org/apache/sysml/runtime/controlprogram/ProgramBlock.java
----------------------------------------------------------------------
diff --git a/src/main/java/org/apache/sysml/runtime/controlprogram/ProgramBlock.java b/src/main/java/org/apache/sysml/runtime/controlprogram/ProgramBlock.java
index 209c566..43e93f4 100644
--- a/src/main/java/org/apache/sysml/runtime/controlprogram/ProgramBlock.java
+++ b/src/main/java/org/apache/sysml/runtime/controlprogram/ProgramBlock.java
@@ -409,21 +409,25 @@ public class ProgramBlock
// store position information for program blocks
///////////////////////////////////////////////////////////////////////////
+ public String _filename;
public int _beginLine, _beginColumn;
public int _endLine, _endColumn;
+ public void setFilename(String passed) { _filename = passed; }
public void setBeginLine(int passed) { _beginLine = passed; }
public void setBeginColumn(int passed) { _beginColumn = passed; }
public void setEndLine(int passed) { _endLine = passed; }
public void setEndColumn(int passed) { _endColumn = passed; }
- public void setAllPositions(int blp, int bcp, int elp, int ecp){
+ public void setAllPositions(String filename, int blp, int bcp, int elp, int ecp){
+ _filename = filename;
_beginLine = blp;
_beginColumn = bcp;
_endLine = elp;
_endColumn = ecp;
}
+ public String getFilename() { return _filename; }
public int getBeginLine() { return _beginLine; }
public int getBeginColumn() { return _beginColumn; }
public int getEndLine() { return _endLine; }
http://git-wip-us.apache.org/repos/asf/systemml/blob/648eb21d/src/main/java/org/apache/sysml/runtime/controlprogram/caching/CacheableData.java
----------------------------------------------------------------------
diff --git a/src/main/java/org/apache/sysml/runtime/controlprogram/caching/CacheableData.java b/src/main/java/org/apache/sysml/runtime/controlprogram/caching/CacheableData.java
index a9f604f..dbc8e07 100644
--- a/src/main/java/org/apache/sysml/runtime/controlprogram/caching/CacheableData.java
+++ b/src/main/java/org/apache/sysml/runtime/controlprogram/caching/CacheableData.java
@@ -39,6 +39,7 @@ import org.apache.sysml.parser.Expression.ValueType;
import org.apache.sysml.runtime.DMLRuntimeException;
import org.apache.sysml.runtime.controlprogram.caching.LazyWriteBuffer.RPolicy;
import org.apache.sysml.runtime.controlprogram.parfor.util.IDSequence;
+import org.apache.sysml.runtime.instructions.cp.CPInstruction;
import org.apache.sysml.runtime.instructions.cp.Data;
import org.apache.sysml.runtime.instructions.gpu.context.GPUContext;
import org.apache.sysml.runtime.instructions.gpu.context.GPUObject;
@@ -51,10 +52,12 @@ import org.apache.sysml.runtime.matrix.MatrixFormatMetaData;
import org.apache.sysml.runtime.matrix.MetaData;
import org.apache.sysml.runtime.matrix.data.FileFormatProperties;
import org.apache.sysml.runtime.matrix.data.InputInfo;
+import org.apache.sysml.runtime.matrix.data.MatrixBlock;
import org.apache.sysml.runtime.matrix.data.NumItemsByEachReducerMetaData;
import org.apache.sysml.runtime.matrix.data.OutputInfo;
import org.apache.sysml.runtime.util.LocalFileUtils;
import org.apache.sysml.runtime.util.MapReduceTool;
+import org.apache.sysml.utils.GPUStatistics;
/**
@@ -522,6 +525,10 @@ public abstract class CacheableData<T extends CacheBlock> extends Data
return _data;
}
+ public T acquireModify(T newData) throws DMLRuntimeException {
+ return acquireModify(newData, null);
+ }
+
/**
* Acquires the exclusive "write" lock for a thread that wants to throw away the
* old cache block data and link up with new cache block data. Abandons the old data
@@ -531,10 +538,11 @@ public abstract class CacheableData<T extends CacheBlock> extends Data
* Out-Status: MODIFY.
*
* @param newData new data
+ * @param opcode extended instruction opcode
* @return cacheable data
* @throws DMLRuntimeException if error occurs
*/
- public synchronized T acquireModify(T newData)
+ public synchronized T acquireModify(T newData, String opcode)
throws DMLRuntimeException
{
if( LOG.isTraceEnabled() )
@@ -562,11 +570,24 @@ public abstract class CacheableData<T extends CacheBlock> extends Data
if( DMLScript.STATISTICS ){
long t1 = System.nanoTime();
CacheStatistics.incrementAcquireMTime(t1-t0);
+ if(DMLScript.FINEGRAINED_STATISTICS && opcode != null) {
+ if(_data instanceof MatrixBlock) {
+ MatrixBlock currObject = (MatrixBlock)_data;
+ if(currObject.isInSparseFormat())
+ GPUStatistics.maintainCPMiscTimes(opcode, CPInstruction.MISC_TIMER_ACQ_MODIFY_SPARSE_MB, t1-t0);
+ else
+ GPUStatistics.maintainCPMiscTimes(opcode, CPInstruction.MISC_TIMER_ACQ_MODIFY_DENSE_MB, t1-t0);
+ }
+ }
}
return _data;
}
+ public void release() throws CacheException {
+ release(null);
+ }
+
/**
* Releases the shared ("read-only") or exclusive ("write") lock. Updates
* size information, last-access time, metadata, etc.
@@ -579,7 +600,7 @@ public abstract class CacheableData<T extends CacheBlock> extends Data
*
* @throws CacheException if CacheException occurs
*/
- public synchronized void release()
+ public synchronized void release(String opcode)
throws CacheException
{
if( LOG.isTraceEnabled() )
@@ -613,7 +634,7 @@ public abstract class CacheableData<T extends CacheBlock> extends Data
//evict blob
String filePath = getCacheFilePathAndName();
try {
- LazyWriteBuffer.writeBlock(filePath, _data);
+ LazyWriteBuffer.writeBlock(filePath, _data, opcode);
}
catch (Exception e)
{
@@ -721,6 +742,12 @@ public abstract class CacheableData<T extends CacheBlock> extends Data
exportData(fName, outputFormat, -1, formatProperties);
}
+ public synchronized void exportData (String fName, String outputFormat, int replication, FileFormatProperties formatProperties)
+ throws CacheException
+ {
+ exportData(fName, outputFormat, replication, formatProperties, null);
+ }
+
/**
* Synchronized because there might be parallel threads (parfor local) that
* access the same object (in case it was created before the loop).
@@ -736,9 +763,10 @@ public abstract class CacheableData<T extends CacheBlock> extends Data
* @param outputFormat format
* @param replication ?
* @param formatProperties file format properties
+ * @param opcode instruction opcode if available
* @throws CacheException if CacheException occurs
*/
- public synchronized void exportData (String fName, String outputFormat, int replication, FileFormatProperties formatProperties)
+ public synchronized void exportData (String fName, String outputFormat, int replication, FileFormatProperties formatProperties, String opcode)
throws CacheException
{
if( LOG.isTraceEnabled() )
@@ -816,7 +844,7 @@ public abstract class CacheableData<T extends CacheBlock> extends Data
}
finally
{
- release();
+ release(opcode);
}
}
else if( pWrite ) // pwrite with same output format
http://git-wip-us.apache.org/repos/asf/systemml/blob/648eb21d/src/main/java/org/apache/sysml/runtime/controlprogram/caching/LazyWriteBuffer.java
----------------------------------------------------------------------
diff --git a/src/main/java/org/apache/sysml/runtime/controlprogram/caching/LazyWriteBuffer.java b/src/main/java/org/apache/sysml/runtime/controlprogram/caching/LazyWriteBuffer.java
index f0eb926..c045961 100644
--- a/src/main/java/org/apache/sysml/runtime/controlprogram/caching/LazyWriteBuffer.java
+++ b/src/main/java/org/apache/sysml/runtime/controlprogram/caching/LazyWriteBuffer.java
@@ -28,7 +28,9 @@ import java.util.concurrent.Executors;
import org.apache.sysml.api.DMLScript;
import org.apache.sysml.runtime.controlprogram.parfor.stat.InfrastructureAnalyzer;
+import org.apache.sysml.runtime.instructions.cp.CPInstruction;
import org.apache.sysml.runtime.util.LocalFileUtils;
+import org.apache.sysml.utils.GPUStatistics;
public class LazyWriteBuffer
{
@@ -56,7 +58,7 @@ public class LazyWriteBuffer
_limit = (long)(CacheableData.CACHING_BUFFER_SIZE * maxMem);
}
- public static void writeBlock( String fname, CacheBlock cb )
+ public static void writeBlock( String fname, CacheBlock cb, String opcode )
throws IOException
{
//obtain basic meta data of cache block
@@ -72,6 +74,7 @@ public class LazyWriteBuffer
ByteBuffer bbuff = new ByteBuffer( lSize );
int numEvicted = 0;
+ long t1 = DMLScript.STATISTICS && DMLScript.FINEGRAINED_STATISTICS ? System.nanoTime() : 0;
//modify buffer pool
synchronized( _mQueue )
{
@@ -99,21 +102,33 @@ public class LazyWriteBuffer
_mQueue.addLast(fname, bbuff);
_size += lSize;
}
+ long t2 = DMLScript.STATISTICS && DMLScript.FINEGRAINED_STATISTICS ? System.nanoTime() : 0;
//serialize matrix (outside synchronized critical path)
- bbuff.serializeBlock(cb);
+ bbuff.serializeBlock(cb);
if( DMLScript.STATISTICS ) {
+ if(DMLScript.FINEGRAINED_STATISTICS && opcode != null) {
+ long t3 = DMLScript.STATISTICS && DMLScript.FINEGRAINED_STATISTICS ? System.nanoTime() : 0;
+ GPUStatistics.maintainCPMiscTimes(opcode, CPInstruction.MISC_TIMER_RELEASE_EVICTION, t2-t1, numEvicted);
+ GPUStatistics.maintainCPMiscTimes(opcode, CPInstruction.MISC_TIMER_RELEASE_BUFF_WRITE, t3-t2, 1);
+ }
CacheStatistics.incrementFSBuffWrites();
CacheStatistics.incrementFSWrites(numEvicted);
}
}
else
{
+ long t1 = DMLScript.STATISTICS && DMLScript.FINEGRAINED_STATISTICS ? System.nanoTime() : 0;
//write directly to local FS (bypass buffer if too large)
LocalFileUtils.writeCacheBlockToLocal(fname, cb);
- if( DMLScript.STATISTICS )
+ if( DMLScript.STATISTICS ) {
+ if(DMLScript.FINEGRAINED_STATISTICS && opcode != null) {
+ long t2 = DMLScript.STATISTICS && DMLScript.FINEGRAINED_STATISTICS ? System.nanoTime() : 0;
+ GPUStatistics.maintainCPMiscTimes(opcode, CPInstruction.MISC_TIMER_RELEASE_BUFF_WRITE, t2-t1, 1);
+ }
CacheStatistics.incrementFSWrites();
+ }
}
}
http://git-wip-us.apache.org/repos/asf/systemml/blob/648eb21d/src/main/java/org/apache/sysml/runtime/controlprogram/context/ExecutionContext.java
----------------------------------------------------------------------
diff --git a/src/main/java/org/apache/sysml/runtime/controlprogram/context/ExecutionContext.java b/src/main/java/org/apache/sysml/runtime/controlprogram/context/ExecutionContext.java
index 2b97067..8d27e3b 100644
--- a/src/main/java/org/apache/sysml/runtime/controlprogram/context/ExecutionContext.java
+++ b/src/main/java/org/apache/sysml/runtime/controlprogram/context/ExecutionContext.java
@@ -39,6 +39,7 @@ import org.apache.sysml.runtime.controlprogram.caching.FrameObject;
import org.apache.sysml.runtime.controlprogram.caching.MatrixObject;
import org.apache.sysml.runtime.controlprogram.caching.MatrixObject.UpdateType;
import org.apache.sysml.runtime.instructions.Instruction;
+import org.apache.sysml.runtime.instructions.cp.CPInstruction;
import org.apache.sysml.runtime.instructions.cp.CPOperand;
import org.apache.sysml.runtime.instructions.cp.Data;
import org.apache.sysml.runtime.instructions.cp.FunctionCallCPInstruction;
@@ -54,6 +55,7 @@ import org.apache.sysml.runtime.matrix.data.FrameBlock;
import org.apache.sysml.runtime.matrix.data.MatrixBlock;
import org.apache.sysml.runtime.matrix.data.Pair;
import org.apache.sysml.runtime.util.MapReduceTool;
+import org.apache.sysml.utils.GPUStatistics;
public class ExecutionContext {
@@ -231,13 +233,34 @@ public class ExecutionContext {
}
/**
+ * Pins a matrix variable into memory, update the finegrained statistics and returns the internal matrix block.
+ *
+ * @param varName variable name
+ * @param opcode extended opcode
+ * @return matrix block
+ * @throws DMLRuntimeException if DMLRuntimeException occurs
+ */
+ public MatrixBlock getMatrixInput(String varName, String opcode) throws DMLRuntimeException {
+ long t1 = opcode != null && DMLScript.STATISTICS && DMLScript.FINEGRAINED_STATISTICS ? System.nanoTime() : 0;
+ MatrixBlock mb = getMatrixInput(varName);
+ if(opcode != null && DMLScript.STATISTICS && DMLScript.FINEGRAINED_STATISTICS) {
+ long t2 = System.nanoTime();
+ if(mb.isInSparseFormat())
+ GPUStatistics.maintainCPMiscTimes(opcode, CPInstruction.MISC_TIMER_GET_SPARSE_MB, t2-t1);
+ else
+ GPUStatistics.maintainCPMiscTimes(opcode, CPInstruction.MISC_TIMER_GET_DENSE_MB, t2-t1);
+ }
+ return mb;
+ }
+
+ /**
* Pins a matrix variable into memory and returns the internal matrix block.
*
* @param varName variable name
* @return matrix block
* @throws DMLRuntimeException if DMLRuntimeException occurs
*/
- public MatrixBlock getMatrixInput(String varName)
+ private MatrixBlock getMatrixInput(String varName)
throws DMLRuntimeException
{
MatrixObject mo = getMatrixObject(varName);
@@ -340,16 +363,19 @@ public class ExecutionContext {
}
/**
- * Unpins a currently pinned matrix variable.
+ * Unpins a currently pinned matrix variable and update fine-grained statistics.
*
* @param varName variable name
* @throws DMLRuntimeException if DMLRuntimeException occurs
*/
- public void releaseMatrixInput(String varName)
- throws DMLRuntimeException
- {
+ public void releaseMatrixInput(String varName, String opcode) throws DMLRuntimeException {
+ long t1 = opcode != null && DMLScript.STATISTICS && DMLScript.FINEGRAINED_STATISTICS ? System.nanoTime() : 0;
MatrixObject mo = getMatrixObject(varName);
- mo.release();
+ mo.release(opcode);
+ if(opcode != null && DMLScript.STATISTICS && DMLScript.FINEGRAINED_STATISTICS) {
+ long t2 = System.nanoTime();
+ GPUStatistics.maintainCPMiscTimes(opcode, CPInstruction.MISC_TIMER_RELEASE_INPUT_MB, t2-t1);
+ }
}
public void releaseMatrixInputForGPUInstruction(String varName)
@@ -417,17 +443,18 @@ public class ExecutionContext {
}
mo.getGPUObject(getGPUContext(0)).releaseOutput();
}
+
- public void setMatrixOutput(String varName, MatrixBlock outputData)
+ public void setMatrixOutput(String varName, MatrixBlock outputData, String opcode)
throws DMLRuntimeException
{
MatrixObject mo = getMatrixObject(varName);
- mo.acquireModify(outputData);
- mo.release();
+ mo.acquireModify(outputData, opcode);
+ mo.release(opcode);
setVariable(varName, mo);
}
- public void setMatrixOutput(String varName, MatrixBlock outputData, UpdateType flag)
+ public void setMatrixOutput(String varName, MatrixBlock outputData, UpdateType flag, String opcode)
throws DMLRuntimeException
{
if( flag.isInPlace() ) {
@@ -437,7 +464,7 @@ public class ExecutionContext {
}
//default case
- setMatrixOutput(varName, outputData);
+ setMatrixOutput(varName, outputData, opcode);
}
public void setFrameOutput(String varName, FrameBlock outputData)
http://git-wip-us.apache.org/repos/asf/systemml/blob/648eb21d/src/main/java/org/apache/sysml/runtime/controlprogram/parfor/opt/OptimizerRuleBased.java
----------------------------------------------------------------------
diff --git a/src/main/java/org/apache/sysml/runtime/controlprogram/parfor/opt/OptimizerRuleBased.java b/src/main/java/org/apache/sysml/runtime/controlprogram/parfor/opt/OptimizerRuleBased.java
index bbd8fb8..118aede 100644
--- a/src/main/java/org/apache/sysml/runtime/controlprogram/parfor/opt/OptimizerRuleBased.java
+++ b/src/main/java/org/apache/sysml/runtime/controlprogram/parfor/opt/OptimizerRuleBased.java
@@ -2830,7 +2830,7 @@ public class OptimizerRuleBased extends Optimizer
//replace existing matrix object with empty matrix
MatrixObject mo = (MatrixObject)dat;
ec.cleanupMatrixObject(mo);
- ec.setMatrixOutput(rvar, new MatrixBlock((int)mo.getNumRows(), (int)mo.getNumColumns(),false));
+ ec.setMatrixOutput(rvar, new MatrixBlock((int)mo.getNumRows(), (int)mo.getNumColumns(),false), null);
//keep track of cleaned result variables
cleanedVars.add(rvar);
http://git-wip-us.apache.org/repos/asf/systemml/blob/648eb21d/src/main/java/org/apache/sysml/runtime/instructions/Instruction.java
----------------------------------------------------------------------
diff --git a/src/main/java/org/apache/sysml/runtime/instructions/Instruction.java b/src/main/java/org/apache/sysml/runtime/instructions/Instruction.java
index 0681f14..ad8cb92 100644
--- a/src/main/java/org/apache/sysml/runtime/instructions/Instruction.java
+++ b/src/main/java/org/apache/sysml/runtime/instructions/Instruction.java
@@ -21,7 +21,7 @@ package org.apache.sysml.runtime.instructions;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
-
+import org.apache.sysml.api.DMLScript;
import org.apache.sysml.lops.Lop;
import org.apache.sysml.parser.DataIdentifier;
import org.apache.sysml.runtime.DMLRuntimeException;
@@ -53,9 +53,11 @@ public abstract class Instruction
protected INSTRUCTION_TYPE type = null;
protected String instString = null;
protected String instOpcode = null;
+ private String extendedOpcode = null;
private long instID = -1;
//originating script positions
+ protected String filename = null;
protected int beginLine = -1;
protected int endLine = -1;
protected int beginCol = -1;
@@ -77,7 +79,8 @@ public abstract class Instruction
* @param beginCol beginning column position
* @param endCol ending column position
*/
- public void setLocation ( int beginLine, int endLine, int beginCol, int endCol) {
+ public void setLocation ( String filename, int beginLine, int endLine, int beginCol, int endCol) {
+ this.filename = filename;
this.beginLine = beginLine;
this.endLine = endLine;
this.beginCol = beginCol;
@@ -86,6 +89,7 @@ public abstract class Instruction
public void setLocation(Lop lop) {
if(lop != null) {
+ this.filename = lop.getFilename();
this.beginLine = lop._beginLine;
this.endLine = lop._endLine;
this.beginCol = lop._beginColumn;
@@ -95,6 +99,7 @@ public abstract class Instruction
public void setLocation(DataIdentifier id) {
if(id != null) {
+ this.filename = id.getFilename();
this.beginLine = id.getBeginLine();
this.endLine = id.getEndLine();
this.beginCol = id.getBeginColumn();
@@ -104,6 +109,7 @@ public abstract class Instruction
public void setLocation(Instruction oldInst) {
if(oldInst != null) {
+ this.filename = oldInst.filename;
this.beginLine = oldInst.beginLine;
this.endLine = oldInst.endLine;
this.beginCol = oldInst.beginCol;
@@ -152,12 +158,31 @@ public abstract class Instruction
}
public String getExtendedOpcode() {
- if( type == INSTRUCTION_TYPE.SPARK )
- return SP_INST_PREFIX + getOpcode();
- else if( type == INSTRUCTION_TYPE.GPU )
- return GPU_INST_PREFIX + getOpcode();
- else
- return getOpcode();
+ if(extendedOpcode != null)
+ return extendedOpcode;
+ if(DMLScript.FINEGRAINED_STATISTICS) {
+ String scriptInfo;
+ if(filename != null)
+ scriptInfo = " [" + filename + " " + beginLine + ":" + beginCol + "-" + endLine + ":" + endCol + "]";
+ else
+ scriptInfo = " [" + beginLine + ":" + beginCol + "-" + endLine + ":" + endCol + "]";
+ if( type == INSTRUCTION_TYPE.SPARK )
+ extendedOpcode = SP_INST_PREFIX + getOpcode() + scriptInfo;
+ else if( type == INSTRUCTION_TYPE.GPU )
+ extendedOpcode = GPU_INST_PREFIX + getOpcode() + scriptInfo;
+ else
+ extendedOpcode = getOpcode() + scriptInfo;
+ }
+ else {
+ // This ensures that there is no overhead if finegrained statistics is disabled
+ if( type == INSTRUCTION_TYPE.SPARK )
+ extendedOpcode = SP_INST_PREFIX + getOpcode();
+ else if( type == INSTRUCTION_TYPE.GPU )
+ extendedOpcode = GPU_INST_PREFIX + getOpcode();
+ else
+ extendedOpcode = getOpcode();
+ }
+ return extendedOpcode;
}
public boolean requiresLabelUpdate()
http://git-wip-us.apache.org/repos/asf/systemml/blob/648eb21d/src/main/java/org/apache/sysml/runtime/instructions/cp/AggregateBinaryCPInstruction.java
----------------------------------------------------------------------
diff --git a/src/main/java/org/apache/sysml/runtime/instructions/cp/AggregateBinaryCPInstruction.java b/src/main/java/org/apache/sysml/runtime/instructions/cp/AggregateBinaryCPInstruction.java
index 359728f..2b09843 100644
--- a/src/main/java/org/apache/sysml/runtime/instructions/cp/AggregateBinaryCPInstruction.java
+++ b/src/main/java/org/apache/sysml/runtime/instructions/cp/AggregateBinaryCPInstruction.java
@@ -70,8 +70,8 @@ public class AggregateBinaryCPInstruction extends BinaryCPInstruction
throws DMLRuntimeException
{
//get inputs
- MatrixBlock matBlock1 = ec.getMatrixInput(input1.getName());
- MatrixBlock matBlock2 = ec.getMatrixInput(input2.getName());
+ MatrixBlock matBlock1 = ec.getMatrixInput(input1.getName(), getExtendedOpcode());
+ MatrixBlock matBlock2 = ec.getMatrixInput(input2.getName(), getExtendedOpcode());
//compute matrix multiplication
AggregateBinaryOperator ab_op = (AggregateBinaryOperator) _optr;
@@ -79,8 +79,8 @@ public class AggregateBinaryCPInstruction extends BinaryCPInstruction
MatrixBlock ret = (MatrixBlock) main.aggregateBinaryOperations(matBlock1, matBlock2, new MatrixBlock(), ab_op);
//release inputs/outputs
- ec.releaseMatrixInput(input1.getName());
- ec.releaseMatrixInput(input2.getName());
- ec.setMatrixOutput(output.getName(), ret);
+ ec.releaseMatrixInput(input1.getName(), getExtendedOpcode());
+ ec.releaseMatrixInput(input2.getName(), getExtendedOpcode());
+ ec.setMatrixOutput(output.getName(), ret, getExtendedOpcode());
}
}
http://git-wip-us.apache.org/repos/asf/systemml/blob/648eb21d/src/main/java/org/apache/sysml/runtime/instructions/cp/AggregateTernaryCPInstruction.java
----------------------------------------------------------------------
diff --git a/src/main/java/org/apache/sysml/runtime/instructions/cp/AggregateTernaryCPInstruction.java b/src/main/java/org/apache/sysml/runtime/instructions/cp/AggregateTernaryCPInstruction.java
index 147436e..949462a 100644
--- a/src/main/java/org/apache/sysml/runtime/instructions/cp/AggregateTernaryCPInstruction.java
+++ b/src/main/java/org/apache/sysml/runtime/instructions/cp/AggregateTernaryCPInstruction.java
@@ -62,23 +62,23 @@ public class AggregateTernaryCPInstruction extends ComputationCPInstruction
public void processInstruction(ExecutionContext ec)
throws DMLRuntimeException
{
- MatrixBlock matBlock1 = ec.getMatrixInput(input1.getName());
- MatrixBlock matBlock2 = ec.getMatrixInput(input2.getName());
+ MatrixBlock matBlock1 = ec.getMatrixInput(input1.getName(), getExtendedOpcode());
+ MatrixBlock matBlock2 = ec.getMatrixInput(input2.getName(), getExtendedOpcode());
MatrixBlock matBlock3 = input3.isLiteral() ? null : //matrix or literal 1
- ec.getMatrixInput(input3.getName());
+ ec.getMatrixInput(input3.getName(), getExtendedOpcode());
AggregateTernaryOperator ab_op = (AggregateTernaryOperator) _optr;
MatrixBlock ret = matBlock1.aggregateTernaryOperations(
matBlock1, matBlock2, matBlock3, new MatrixBlock(), ab_op, true);
//release inputs/outputs
- ec.releaseMatrixInput(input1.getName());
- ec.releaseMatrixInput(input2.getName());
+ ec.releaseMatrixInput(input1.getName(), getExtendedOpcode());
+ ec.releaseMatrixInput(input2.getName(), getExtendedOpcode());
if( !input3.isLiteral() )
- ec.releaseMatrixInput(input3.getName());
+ ec.releaseMatrixInput(input3.getName(), getExtendedOpcode());
if( output.getDataType().isScalar() )
ec.setScalarOutput(output.getName(), new DoubleObject(ret.quickGetValue(0, 0)));
else
- ec.setMatrixOutput(output.getName(), ret);
+ ec.setMatrixOutput(output.getName(), ret, getExtendedOpcode());
}
}
http://git-wip-us.apache.org/repos/asf/systemml/blob/648eb21d/src/main/java/org/apache/sysml/runtime/instructions/cp/AggregateUnaryCPInstruction.java
----------------------------------------------------------------------
diff --git a/src/main/java/org/apache/sysml/runtime/instructions/cp/AggregateUnaryCPInstruction.java b/src/main/java/org/apache/sysml/runtime/instructions/cp/AggregateUnaryCPInstruction.java
index 8dd372a..5edce07 100644
--- a/src/main/java/org/apache/sysml/runtime/instructions/cp/AggregateUnaryCPInstruction.java
+++ b/src/main/java/org/apache/sysml/runtime/instructions/cp/AggregateUnaryCPInstruction.java
@@ -141,19 +141,19 @@ public class AggregateUnaryCPInstruction extends UnaryCPInstruction
else
{
/* Default behavior for AggregateUnary Instruction */
- MatrixBlock matBlock = ec.getMatrixInput(input1.getName());
+ MatrixBlock matBlock = ec.getMatrixInput(input1.getName(), getExtendedOpcode());
AggregateUnaryOperator au_op = (AggregateUnaryOperator) _optr;
MatrixBlock resultBlock = (MatrixBlock) matBlock.aggregateUnaryOperations(au_op, new MatrixBlock(), matBlock.getNumRows(), matBlock.getNumColumns(), new MatrixIndexes(1, 1), true);
- ec.releaseMatrixInput(input1.getName());
+ ec.releaseMatrixInput(input1.getName(), getExtendedOpcode());
if(output.getDataType() == DataType.SCALAR){
DoubleObject ret = new DoubleObject(output_name, resultBlock.getValue(0, 0));
ec.setScalarOutput(output_name, ret);
} else{
// since the computed value is a scalar, allocate a "temp" output matrix
- ec.setMatrixOutput(output_name, resultBlock);
+ ec.setMatrixOutput(output_name, resultBlock, getExtendedOpcode());
}
}
}
http://git-wip-us.apache.org/repos/asf/systemml/blob/648eb21d/src/main/java/org/apache/sysml/runtime/instructions/cp/CPInstruction.java
----------------------------------------------------------------------
diff --git a/src/main/java/org/apache/sysml/runtime/instructions/cp/CPInstruction.java b/src/main/java/org/apache/sysml/runtime/instructions/cp/CPInstruction.java
index 590dacb..c879481 100644
--- a/src/main/java/org/apache/sysml/runtime/instructions/cp/CPInstruction.java
+++ b/src/main/java/org/apache/sysml/runtime/instructions/cp/CPInstruction.java
@@ -42,6 +42,22 @@ public abstract class CPInstruction extends Instruction
protected boolean _requiresLabelUpdate = false;
+ // Generic miscellaneous timers that are applicable to all CP (and few SP) instructions
+ public final static String MISC_TIMER_GET_SPARSE_MB = "aqrs"; // time spent in bringing input sparse matrix block
+ public final static String MISC_TIMER_GET_DENSE_MB = "aqrd"; // time spent in bringing input dense matrix block
+ public final static String MISC_TIMER_ACQ_MODIFY_SPARSE_MB = "aqms"; // time spent in bringing output sparse matrix block
+ public final static String MISC_TIMER_ACQ_MODIFY_DENSE_MB = "aqmd"; // time spent in bringing output dense matrix block
+ public final static String MISC_TIMER_RELEASE_INPUT_MB = "rlsi"; // time spent in release input matrix block
+ public final static String MISC_TIMER_RELEASE_EVICTION = "rlsev";// time spent in buffer eviction of release operation
+ public final static String MISC_TIMER_RELEASE_BUFF_WRITE = "rlswr";// time spent in buffer write in release operation
+ public final static String MISC_TIMER_SPARSE_TO_DENSE = "s2d"; // time spent in sparse to dense conversion
+ public final static String MISC_TIMER_DENSE_TO_SPARSE = "d2s"; // time spent in sparse to dense conversion
+
+ // Instruction specific miscellaneous timers that were found as potential bottlenecks in one of performance analysis.
+ // SystemML committers have to be judicious about adding them by weighing the tradeoffs between reuse in future analysis and unnecessary overheads.
+ public final static String MISC_TIMER_CSR_LIX_COPY = "csrlix";// time spent in CSR-specific method to address performance issues due to repeated re-shifting on update-in-place.
+ public final static String MISC_TIMER_LIX_COPY = "lixcp";// time spent in range copy
+
public CPInstruction(String opcode, String istr) {
type = INSTRUCTION_TYPE.CONTROL_PROGRAM;
instString = istr;
http://git-wip-us.apache.org/repos/asf/systemml/blob/648eb21d/src/main/java/org/apache/sysml/runtime/instructions/cp/CentralMomentCPInstruction.java
----------------------------------------------------------------------
diff --git a/src/main/java/org/apache/sysml/runtime/instructions/cp/CentralMomentCPInstruction.java b/src/main/java/org/apache/sysml/runtime/instructions/cp/CentralMomentCPInstruction.java
index e8ca315..917e984 100644
--- a/src/main/java/org/apache/sysml/runtime/instructions/cp/CentralMomentCPInstruction.java
+++ b/src/main/java/org/apache/sysml/runtime/instructions/cp/CentralMomentCPInstruction.java
@@ -102,7 +102,7 @@ public class CentralMomentCPInstruction extends AggregateUnaryCPInstruction
* order and update the CMOperator, if needed.
*/
- MatrixBlock matBlock = ec.getMatrixInput(input1.getName());
+ MatrixBlock matBlock = ec.getMatrixInput(input1.getName(), getExtendedOpcode());
CPOperand scalarInput = (input3==null ? input2 : input3);
ScalarObject order = ec.getScalarInput(scalarInput.getName(), scalarInput.getValueType(), scalarInput.isLiteral());
@@ -117,12 +117,12 @@ public class CentralMomentCPInstruction extends AggregateUnaryCPInstruction
cmobj = matBlock.cmOperations(cm_op);
}
else {
- MatrixBlock wtBlock = ec.getMatrixInput(input2.getName());
+ MatrixBlock wtBlock = ec.getMatrixInput(input2.getName(), getExtendedOpcode());
cmobj = matBlock.cmOperations(cm_op, wtBlock);
- ec.releaseMatrixInput(input2.getName());
+ ec.releaseMatrixInput(input2.getName(), getExtendedOpcode());
}
- ec.releaseMatrixInput(input1.getName());
+ ec.releaseMatrixInput(input1.getName(), getExtendedOpcode());
double val = cmobj.getRequiredResult(_optr);
DoubleObject ret = new DoubleObject(output_name, val);
http://git-wip-us.apache.org/repos/asf/systemml/blob/648eb21d/src/main/java/org/apache/sysml/runtime/instructions/cp/CompressionCPInstruction.java
----------------------------------------------------------------------
diff --git a/src/main/java/org/apache/sysml/runtime/instructions/cp/CompressionCPInstruction.java b/src/main/java/org/apache/sysml/runtime/instructions/cp/CompressionCPInstruction.java
index 5230945..e9826e1 100644
--- a/src/main/java/org/apache/sysml/runtime/instructions/cp/CompressionCPInstruction.java
+++ b/src/main/java/org/apache/sysml/runtime/instructions/cp/CompressionCPInstruction.java
@@ -51,14 +51,14 @@ public class CompressionCPInstruction extends UnaryCPInstruction
throws DMLRuntimeException
{
//get matrix block input
- MatrixBlock in = ec.getMatrixInput(input1.getName());
+ MatrixBlock in = ec.getMatrixInput(input1.getName(), getExtendedOpcode());
//compress the matrix block
CompressedMatrixBlock cmb = new CompressedMatrixBlock(in);
cmb.compress(OptimizerUtils.getConstrainedNumThreads(-1));
//set output and release input
- ec.releaseMatrixInput(input1.getName());
- ec.setMatrixOutput(output.getName(), cmb);
+ ec.releaseMatrixInput(input1.getName(), getExtendedOpcode());
+ ec.setMatrixOutput(output.getName(), cmb, getExtendedOpcode());
}
}