You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@systemml.apache.org by "Fei Hu (JIRA)" <ji...@apache.org> on 2017/06/26 20:55:00 UTC
[jira] [Commented] (SYSTEMML-1678) Add new 1D & 2D top_k utility functions

    [ https://issues.apache.org/jira/browse/SYSTEMML-1678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16063775#comment-16063775 ] 

Fei Hu commented on SYSTEMML-1678:
----------------------------------

[PR 551|https://github.com/apache/systemml/pull/551] submitted. But there was a weird error when using nn/test/run_tests.dml to test this function. The error will happen by commenting out the lin 234 in nn/util.dml 'if(1==1){}' when running nn/test/run_tests.dml. The error is as following:
{code:java}
// Some comments here
org.apache.sysml.api.mlcontext.MLContextException: Exception when executing script
	at org.apache.sysml.api.mlcontext.MLContext.execute(MLContext.java:320)
	at org.apache.sysml.api.mlcontext.MLContext.execute(MLContext.java:293)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
	at py4j.Gateway.invoke(Gateway.java:280)
	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
	at py4j.commands.CallCommand.execute(CallCommand.java:79)
	at py4j.GatewayConnection.run(GatewayConnection.java:214)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.sysml.api.mlcontext.MLContextException: Exception occurred while executing runtime program
	at org.apache.sysml.api.mlcontext.ScriptExecutor.executeRuntimeProgram(ScriptExecutor.java:391)
	at org.apache.sysml.api.mlcontext.ScriptExecutor.execute(ScriptExecutor.java:308)
	at org.apache.sysml.api.mlcontext.MLContext.execute(MLContext.java:316)
	... 12 more
Caused by: org.apache.sysml.runtime.DMLRuntimeException: org.apache.sysml.runtime.DMLRuntimeException: ERROR: Runtime error in program block generated from statement block between lines 103 and 103 -- Error evaluating instruction: CP°extfunct°./nn/test/test.dml°top_k2d°0°0
	at org.apache.sysml.runtime.controlprogram.Program.execute(Program.java:130)
	at org.apache.sysml.api.ScriptExecutorUtils.executeRuntimeProgram(ScriptExecutorUtils.java:95)
	at org.apache.sysml.api.ScriptExecutorUtils.executeRuntimeProgram(ScriptExecutorUtils.java:54)
	at org.apache.sysml.api.mlcontext.ScriptExecutor.executeRuntimeProgram(ScriptExecutor.java:389)
	... 14 more
Caused by: org.apache.sysml.runtime.DMLRuntimeException: ERROR: Runtime error in program block generated from statement block between lines 103 and 103 -- Error evaluating instruction: CP°extfunct°./nn/test/test.dml°top_k2d°0°0
	at org.apache.sysml.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:320)
	at org.apache.sysml.runtime.controlprogram.ProgramBlock.executeInstructions(ProgramBlock.java:221)
	at org.apache.sysml.runtime.controlprogram.ProgramBlock.execute(ProgramBlock.java:167)
	at org.apache.sysml.runtime.controlprogram.Program.execute(Program.java:123)
	... 17 more
Caused by: org.apache.sysml.runtime.DMLRuntimeException: error executing function ./nn/test/test.dml::top_k2d
	at org.apache.sysml.runtime.instructions.cp.FunctionCallCPInstruction.processInstruction(FunctionCallCPInstruction.java:187)
	at org.apache.sysml.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:290)
	... 20 more
Caused by: org.apache.sysml.runtime.DMLRuntimeException: ERROR: Runtime error in function program block generated from function statement block between lines 856 and 924 -- Error evaluating function program block
	at org.apache.sysml.runtime.controlprogram.FunctionProgramBlock.execute(FunctionProgramBlock.java:122)
	at org.apache.sysml.runtime.instructions.cp.FunctionCallCPInstruction.processInstruction(FunctionCallCPInstruction.java:180)
	... 21 more
Caused by: org.apache.sysml.runtime.DMLRuntimeException: ERROR: Runtime error in program block generated from statement block between lines 919 and 919 -- Error evaluating instruction: CP°extfunct°./nn/util.dml°top_k2d°5°2°X·MATRIX·DOUBLE°2·SCALAR·INT·true°3·SCALAR·INT·true°3·SCALAR·INT·true°4·SCALAR·INT·true°values°indices
	at org.apache.sysml.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:320)
	at org.apache.sysml.runtime.controlprogram.ProgramBlock.executeInstructions(ProgramBlock.java:221)
	at org.apache.sysml.runtime.controlprogram.ProgramBlock.execute(ProgramBlock.java:167)
	at org.apache.sysml.runtime.controlprogram.FunctionProgramBlock.execute(FunctionProgramBlock.java:115)
	... 22 more
Caused by: org.apache.sysml.runtime.DMLRuntimeException: error executing function ./nn/util.dml::top_k2d
	at org.apache.sysml.runtime.instructions.cp.FunctionCallCPInstruction.processInstruction(FunctionCallCPInstruction.java:187)
	at org.apache.sysml.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:290)
	... 25 more
Caused by: org.apache.sysml.runtime.DMLRuntimeException: ERROR: Runtime error in function program block generated from function statement block between lines 331 and 363 -- Error evaluating function program block
	at org.apache.sysml.runtime.controlprogram.FunctionProgramBlock.execute(FunctionProgramBlock.java:122)
	at org.apache.sysml.runtime.instructions.cp.FunctionCallCPInstruction.processInstruction(FunctionCallCPInstruction.java:180)
	... 26 more
Caused by: org.apache.sysml.runtime.DMLRuntimeException: ERROR: Runtime error in program block generated from statement block between lines 358 and 362 -- Error evaluating instruction: CP°ctable°_mVar4017227·MATRIX·DOUBLE°_mVar4017235·MATRIX·DOUBLE°1.0·SCALAR·DOUBLE·true°_Var4017221·false°_Var4017221·false°_mVar4017237·MATRIX·DOUBLE°false
	at org.apache.sysml.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:320)
	at org.apache.sysml.runtime.controlprogram.ProgramBlock.executeInstructions(ProgramBlock.java:221)
	at org.apache.sysml.runtime.controlprogram.ProgramBlock.execute(ProgramBlock.java:167)
	at org.apache.sysml.runtime.controlprogram.FunctionProgramBlock.execute(FunctionProgramBlock.java:115)
	... 27 more
Caused by: java.lang.ArrayIndexOutOfBoundsException: 4
	at org.apache.sysml.runtime.matrix.data.MatrixBlock.quickGetValue(MatrixBlock.java:593)
	at org.apache.sysml.runtime.matrix.data.MatrixBlock.ternaryOperations(MatrixBlock.java:5381)
	at org.apache.sysml.runtime.instructions.cp.TernaryCPInstruction.processInstruction(TernaryCPInstruction.java:137)
	at org.apache.sysml.runtime.controlprogram.ProgramBlock.executeSingleInstruction(ProgramBlock.java:290)
	... 30 more
{code}

The error may be caused by the optimization in SystemML. Here is the partial plan output with "if(1==1){}":

{code:java}
PROGRAM ( size CP/SP = 124/36 )
--FUNCTIONS
----FUNCTION CALL GRAPH
------MAIN PROGRAM
--------./nn/util.dml::top_k2d
----------./nn/util.dml::top_k
------------./nn/util.dml::top_k_row
----------./nn/util.dml::transpose_NCHW_to_CNHW
----FUNCTION ./nn/util.dml::top_k_row [recompile=true]
------GENERIC (lines 292-298) [recompile=true]
--------CP ncol X.MATRIX.DOUBLE.false _Var4.SCALAR.INT
--------CP createvar _mVar5 scratch_space//_p70714_9.108.91.12//_t0/temp4 true MATRIX binaryblock 1 -1 1000 1000 0 copy
--------SPARK rand 1 ¶k¶ 1000 1000 0.0 0.0 1.0 -1 scratch_space/_p70714_9.108.91.12//_t0/ uniform 1.0 _mVar5.MATRIX.DOUBLE
--------CP createvar _mVar6 scratch_space//_p70714_9.108.91.12//_t0/temp5 true MATRIX binaryblock 1 -1 1000 1000 -1 copy
--------SPARK rangeReIndex X.MATRIX.DOUBLE r.SCALAR.INT.false r.SCALAR.INT.false 1.SCALAR.INT.true _Var4.SCALAR.INT.false _mVar6.MATRIX.DOUBLE MULTI_BLOCK
--------CP rmvar _Var4
--------CP createvar _mVar7 scratch_space//_p70714_9.108.91.12//_t0/temp6 true MATRIX binaryblock -1 1 1000 1000 -1 copy
--------SPARK r' _mVar6.MATRIX.DOUBLE _mVar7.MATRIX.DOUBLE
--------CP createvar _mVar8 scratch_space//_p70714_9.108.91.12//_t0/temp7 true MATRIX binaryblock -1 1 1000 1000 -1 copy
--------SPARK rsort _mVar7.MATRIX.DOUBLE 1.SCALAR.INT.true true.SCALAR.BOOLEAN.true true.SCALAR.BOOLEAN.true _mVar8.MATRIX.DOUBLE false
--------CP rmvar _mVar7
--------CP createvar _mVar9 scratch_space//_p70714_9.108.91.12//_t0/temp8 true MATRIX binaryblock 1 -1 1000 1000 -1 copy
--------SPARK r' _mVar8.MATRIX.DOUBLE _mVar9.MATRIX.DOUBLE
--------CP rmvar _mVar8
--------CP createvar _mVar10 scratch_space//_p70714_9.108.91.12//_t0/temp9 true MATRIX binaryblock 1 -1 1000 1000 -1 copy
--------SPARK rangeReIndex _mVar9.MATRIX.DOUBLE 1.SCALAR.INT.true 1.SCALAR.INT.true 1.SCALAR.INT.true k.SCALAR.INT.false _mVar10.MATRIX.DOUBLE MULTI_BLOCK
--------CP rmvar _mVar9
--------CP cpvar _mVar5 values
--------CP cpvar _mVar6 row
--------CP cpvar _mVar10 indices
--------CP rmvar _mVar5 _mVar6 _mVar10 X r
------GENERIC (lines 299-301) [recompile=true]
--------CP createvar _mVar11 scratch_space//_p70714_9.108.91.12//_t0/temp10 true MATRIX binaryblock 1 -1 1000 1000 -1 copy
--------SPARK chkpoint indices.MATRIX.DOUBLE _mVar11.MATRIX.DOUBLE MEMORY_AND_DISK
--------CP createvar _mVar12 scratch_space//_p70714_9.108.91.12//_t0/temp11 true MATRIX binaryblock 1 -1 1000 1000 -1 copy
--------SPARK chkpoint row.MATRIX.DOUBLE _mVar12.MATRIX.DOUBLE MEMORY_AND_DISK
--------CP rmvar indices row
--------CP cpvar _mVar11 indices
--------CP cpvar _mVar12 row
--------CP rmvar _mVar11 _mVar12
------GENERIC (lines 299-301) [recompile=true]
--------CP createvar _mVar13 scratch_space//_p70714_9.108.91.12//_t0/temp12 true MATRIX binaryblock 1 -1 1000 1000 -1 copy
--------SPARK chkpoint indices.MATRIX.DOUBLE _mVar13.MATRIX.DOUBLE MEMORY_AND_DISK
--------CP createvar _mVar14 scratch_space//_p70714_9.108.91.12//_t0/temp13 true MATRIX binaryblock 1 -1 1000 1000 -1 copy
--------SPARK chkpoint row.MATRIX.DOUBLE _mVar14.MATRIX.DOUBLE MEMORY_AND_DISK
--------CP rmvar indices row
--------CP cpvar _mVar13 indices
--------CP cpvar _mVar14 row
--------CP rmvar _mVar13 _mVar14
------GENERIC (lines 299-301) [recompile=true]
--------CP createvar _mVar15 scratch_space//_p70714_9.108.91.12//_t0/temp14 true MATRIX binaryblock 1 -1 1000 1000 -1 copy
--------SPARK chkpoint indices.MATRIX.DOUBLE _mVar15.MATRIX.DOUBLE MEMORY_AND_DISK
--------CP createvar _mVar16 scratch_space//_p70714_9.108.91.12//_t0/temp15 true MATRIX binaryblock 1 -1 1000 1000 -1 copy
--------SPARK chkpoint row.MATRIX.DOUBLE _mVar16.MATRIX.DOUBLE MEMORY_AND_DISK
--------CP rmvar indices row
--------CP cpvar _mVar15 indices
--------CP cpvar _mVar16 row
--------CP rmvar _mVar15 _mVar16
------FOR (lines 299-301) [in-place=[values]]
--------GENERIC (lines 300-300) [recompile=true]
----------CP createvar _mVar17 scratch_space//_p70714_9.108.91.12//_t0/temp16 true MATRIX binaryblock 1 1 1000 1000 -1 copy
----------SPARK rangeReIndex indices.MATRIX.DOUBLE 1.SCALAR.INT.true 1.SCALAR.INT.true i.SCALAR.INT.false i.SCALAR.INT.false _mVar17.MATRIX.DOUBLE NONE
----------CP castdts _mVar17.MATRIX.DOUBLE.false _Var18.SCALAR.DOUBLE
----------CP rmvar _mVar17
----------CP createvar _mVar19 scratch_space//_p70714_9.108.91.12//_t0/temp17 true MATRIX binaryblock 1 1 1000 1000 -1 copy
----------SPARK rangeReIndex row.MATRIX.DOUBLE 1.SCALAR.INT.true 1.SCALAR.INT.true _Var18.SCALAR.DOUBLE.false _Var18.SCALAR.DOUBLE.false _mVar19.MATRIX.DOUBLE NONE
----------CP rmvar _Var18
----------CP createvar _mVar20 scratch_space//_p70714_9.108.91.12//_t0/temp18 true MATRIX binaryblock 1 -1 1000 1000 -1 copy
----------SPARK mapLeftIndex values.MATRIX.DOUBLE _mVar19.MATRIX.DOUBLE 1.SCALAR.INT.true 1.SCALAR.INT.true i.SCALAR.INT.false i.SCALAR.INT.false _mVar20.MATRIX.DOUBLE LEFT
----------CP rmvar _mVar19 values
----------CP cpvar _mVar20 values
----------CP rmvar _mVar20
----FUNCTION ./nn/util.dml::top_k [recompile=true]
------GENERIC (lines 319-322) [recompile=true]
--------CP nrow X.MATRIX.DOUBLE.false _Var21.SCALAR.INT
--------CP createvar _mVar22 scratch_space//_p70714_9.108.91.12//_t0/temp19 true MATRIX binaryblock -1 -1 1000 1000 0 copy
--------SPARK rand ¶_Var21¶ ¶k¶ 1000 1000 0.0 0.0 1.0 -1 scratch_space/_p70714_9.108.91.12//_t0/ uniform 1.0 _mVar22.MATRIX.DOUBLE
--------CP assignvar _Var21.SCALAR.INT.false N.SCALAR.INT
--------CP rmvar _Var21
--------CP cpvar _mVar22 values
--------CP cpvar _mVar22 indices
--------CP rmvar _mVar22
------GENERIC (lines 324-328) [recompile=true]
--------CP createvar _mVar23 scratch_space//_p70714_9.108.91.12//_t0/temp20 true MATRIX binaryblock -1 -1 1000 1000 -1 copy
--------SPARK chkpoint X.MATRIX.DOUBLE _mVar23.MATRIX.DOUBLE MEMORY_AND_DISK
--------CP rmvar X
--------CP cpvar _mVar23 X
--------CP rmvar _mVar23
------GENERIC (lines 324-328) [recompile=true]
--------CP createvar _mVar24 scratch_space//_p70714_9.108.91.12//_t0/temp21 true MATRIX binaryblock -1 -1 1000 1000 -1 copy
--------SPARK chkpoint X.MATRIX.DOUBLE _mVar24.MATRIX.DOUBLE MEMORY_AND_DISK
--------CP rmvar X
--------CP cpvar _mVar24 X
--------CP rmvar _mVar24
------GENERIC (lines 324-328) [recompile=true]
--------CP createvar _mVar25 scratch_space//_p70714_9.108.91.12//_t0/temp22 true MATRIX binaryblock -1 -1 1000 1000 -1 copy
--------SPARK chkpoint X.MATRIX.DOUBLE _mVar25.MATRIX.DOUBLE MEMORY_AND_DISK
--------CP rmvar X
--------CP cpvar _mVar25 X
--------CP rmvar _mVar25
------PARFOR (lines 324-328)
--------GENERIC (lines 325-325) [recompile=true]
----------CP extfunct ./nn/util.dml top_k_row 3 2 X.MATRIX.DOUBLE r.SCALAR.INT.false k.SCALAR.INT.false value index
--------GENERIC (lines 326-327) [recompile=true]
----------CP ncol values.MATRIX.DOUBLE.false _Var26.SCALAR.INT
----------CP ncol indices.MATRIX.DOUBLE.false _Var27.SCALAR.INT
----------CP createvar _mVar28 scratch_space//_p70714_9.108.91.12//_t0/temp23 true MATRIX binaryblock -1 -1 1000 1000 -1 copy
----------SPARK leftIndex values.MATRIX.DOUBLE value.MATRIX.DOUBLE r.SCALAR.INT.false r.SCALAR.INT.false 1.SCALAR.INT.true _Var26.SCALAR.INT.false _mVar28.MATRIX.DOUBLE NONE
----------CP rmvar _Var26
----------CP createvar _mVar29 scratch_space//_p70714_9.108.91.12//_t0/temp24 true MATRIX binaryblock -1 -1 1000 1000 -1 copy
----------SPARK leftIndex indices.MATRIX.DOUBLE index.MATRIX.DOUBLE r.SCALAR.INT.false r.SCALAR.INT.false 1.SCALAR.INT.true _Var27.SCALAR.INT.false _mVar29.MATRIX.DOUBLE NONE
----------CP rmvar _Var27 indices values
----------CP cpvar _mVar28 values
----------CP cpvar _mVar29 indices
----------CP rmvar _mVar28 _mVar29 index value
----FUNCTION ./nn/util.dml::top_k2d [recompile=false]
------GENERIC (lines 349-349) [recompile=false]
--------CP assignvar 2.SCALAR.INT.true N.SCALAR.INT
------GENERIC (lines 352-352) [recompile=false]
--------CP extfunct ./nn/util.dml transpose_NCHW_to_CNHW 2 1 X.MATRIX.DOUBLE C.SCALAR.INT.false X_CNHW
--------CP rmvar X C
------GENERIC (lines 353-353) [recompile=true]
--------CP createvar _mVar1 scratch_space//_p70714_9.108.91.12//_t0/temp1 true MATRIX binaryblock -1 -1 1000 1000 -1 copy
--------SPARK r' X_CNHW.MATRIX.DOUBLE _mVar1.MATRIX.DOUBLE
--------CP cpvar _mVar1 X_NHWC
--------CP rmvar _mVar1 X_CNHW
------GENERIC (lines 356-356) [recompile=true]
--------CP extfunct ./nn/util.dml top_k 2 2 X_NHWC.MATRIX.DOUBLE k.SCALAR.INT.false values_NHWK indices_NHWK
--------CP rmvar k X_NHWC
------GENERIC (lines 358-359) [recompile=true]
--------CP createvar _mVar2 scratch_space//_p70714_9.108.91.12//_t0/temp2 true MATRIX binaryblock -1 -1 1000 1000 -1 copy
--------SPARK r' values_NHWK.MATRIX.DOUBLE _mVar2.MATRIX.DOUBLE
--------CP createvar _mVar3 scratch_space//_p70714_9.108.91.12//_t0/temp3 true MATRIX binaryblock -1 -1 1000 1000 -1 copy
--------SPARK r' indices_NHWK.MATRIX.DOUBLE _mVar3.MATRIX.DOUBLE
--------CP cpvar _mVar2 values_KNHW
--------CP cpvar _mVar3 indices_KNHW
--------CP rmvar _mVar2 _mVar3 values_NHWK indices_NHWK
------GENERIC (lines 361-361) [recompile=true]
--------CP extfunct ./nn/util.dml transpose_NCHW_to_CNHW 2 1 values_KNHW.MATRIX.DOUBLE 2.SCALAR.INT.true values
--------CP rmvar values_KNHW
------GENERIC (lines 362-362) [recompile=true]
--------CP extfunct ./nn/util.dml transpose_NCHW_to_CNHW 2 1 indices_KNHW.MATRIX.DOUBLE 2.SCALAR.INT.true indices
--------CP rmvar indices_KNHW N
----FUNCTION ./nn/util.dml::transpose_NCHW_to_CNHW [recompile=false]
------GENERIC (lines 235-271) [recompile=true]
--------CP nrow X.MATRIX.DOUBLE.false _Var30.SCALAR.INT
--------CP createvar _mVar31 scratch_space//_p70714_9.108.91.12//_t0/temp25 true MATRIX binaryblock -1 1 1000 1000 -1 copy
--------SPARK seq -1 1 1000 1000 1 ¶C¶ 1 _mVar31.MATRIX.DOUBLE
--------CP ncol X.MATRIX.DOUBLE.false _Var32.SCALAR.INT
--------CP * _Var30.SCALAR.INT.false C.SCALAR.INT.false _Var33.SCALAR.INT
--------CP - _Var30.SCALAR.INT.false 1.SCALAR.INT.true _Var34.SCALAR.INT
--------CP / _Var32.SCALAR.INT.false C.SCALAR.INT.false _Var35.SCALAR.DOUBLE
--------CP rmvar _Var32
--------CP createvar _mVar36 scratch_space//_p70714_9.108.91.12//_t0/temp26 true MATRIX binaryblock -1 1 1000 1000 -1 copy
--------SPARK seq -1 1 1000 1000 1 ¶_Var33¶ 1 _mVar36.MATRIX.DOUBLE
--------CP createvar _mVar37 scratch_space//_p70714_9.108.91.12//_t0/temp27 true MATRIX binaryblock -1 1 1000 1000 -1 copy
--------SPARK seq -1 1 1000 1000 0 ¶_Var34¶ 1 _mVar37.MATRIX.DOUBLE
--------CP rmvar _Var34
--------CP createvar _mVar38 scratch_space//_p70714_9.108.91.12//_t0/temp28 true MATRIX binaryblock -1 -1 1000 1000 -1 copy
--------SPARK rshape X.MATRIX.DOUBLE _Var33.SCALAR.INT.false _Var35.SCALAR.DOUBLE.false true.SCALAR.BOOLEAN.true _mVar38.MATRIX.DOUBLE
--------CP * _Var30.SCALAR.INT.false _Var35.SCALAR.DOUBLE.false _Var39.SCALAR.DOUBLE
--------CP rmvar _Var30 _Var35
--------CP createvar _mVar40 scratch_space//_p70714_9.108.91.12//_t0/temp29 true MATRIX binaryblock 1 -1 1000 1000 -1 copy
--------SPARK r' _mVar37.MATRIX.DOUBLE _mVar40.MATRIX.DOUBLE
--------CP rmvar _mVar37
--------CP createvar _mVar41 scratch_space//_p70714_9.108.91.12//_t0/temp30 true MATRIX binaryblock -1 1 1000 1000 -1 copy
--------SPARK +* _mVar31.MATRIX.DOUBLE C.SCALAR.INT.false _mVar40.MATRIX.DOUBLE _mVar41.MATRIX.DOUBLE
--------CP rmvar _mVar31 _mVar40
--------CP createvar _mVar42 scratch_space//_p70714_9.108.91.12//_t0/temp31 true MATRIX binaryblock -1 1 1000 1000 -1 copy
--------SPARK rshape _mVar41.MATRIX.DOUBLE _Var33.SCALAR.INT.false 1.SCALAR.INT.true true.SCALAR.BOOLEAN.true _mVar42.MATRIX.DOUBLE
--------CP rmvar _mVar41
--------CP createvar _mVar43 scratch_space//_p70714_9.108.91.12//_t0/temp32 true MATRIX binarycell -1 -1 -1 -1 -1 copy
--------SPARK ctable _mVar36.MATRIX.DOUBLE _mVar42.MATRIX.DOUBLE 1.0.SCALAR.DOUBLE.true _Var33.false _Var33.false _mVar43.MATRIX.DOUBLE false
--------CP rmvar _mVar36 _mVar42 _Var33
--------CP createvar _mVar44 scratch_space//_p70714_9.108.91.12//_t0/temp33 true MATRIX binaryblock -1 -1 1000 1000 -1 copy
--------SPARK rblk _mVar43.MATRIX.DOUBLE _mVar44.MATRIX.DOUBLE 1000 1000 true
--------CP rmvar _mVar43
--------CP createvar _mVar45 scratch_space//_p70714_9.108.91.12//_t0/temp34 true MATRIX binaryblock -1 -1 1000 1000 -1 copy
--------SPARK cpmm _mVar44.MATRIX.DOUBLE _mVar38.MATRIX.DOUBLE _mVar45.MATRIX.DOUBLE MULTI_BLOCK
--------CP rmvar _mVar44 _mVar38
--------CP createvar _mVar46 scratch_space//_p70714_9.108.91.12//_t0/temp35 true MATRIX binaryblock -1 -1 1000 1000 -1 copy
--------SPARK rshape _mVar45.MATRIX.DOUBLE C.SCALAR.INT.false _Var39.SCALAR.DOUBLE.false true.SCALAR.BOOLEAN.true _mVar46.MATRIX.DOUBLE
--------CP rmvar _mVar45 _Var39
--------CP cpvar _mVar46 out
--------CP rmvar _mVar46 X C
--MAIN PROGRAM
----GENERIC (lines 1-0) [recompile=false]
------CP createvar _mVar47 scratch_space//_p70714_9.108.91.12//_t0/temp36 true MATRIX binaryblock 2 36 1000 1000 72 copy
------SPARK rblk X.MATRIX.DOUBLE _mVar47.MATRIX.DOUBLE 1000 1000 true
------CP rmvar X
------CP cpvar _mVar47 X
------CP rmvar _mVar47
----GENERIC (lines 8-8) [recompile=false]
------CP extfunct ./nn/util.dml top_k2d 5 2 X.MATRIX.DOUBLE k.SCALAR.INT.false C.SCALAR.INT.false Hin.SCALAR.INT.false Win.SCALAR.INT.false values indices
------CP rmvar X C k Hin Win
----GENERIC (lines 10-11) [recompile=true]
------CP rmvar indices values
{code}

The following is the partial plan output without "if(1==1){}":

{code:java}
//PROGRAM ( size CP/SP = 168/52 )
--FUNCTIONS
----FUNCTION CALL GRAPH
------MAIN PROGRAM
--------./nn/util.dml::top_k2d
----------./nn/util.dml::top_k
------------./nn/util.dml::top_k_row
----FUNCTION ./nn/util.dml::top_k_row [recompile=true]
------GENERIC (lines 292-298) [recompile=true]
--------CP ncol X.MATRIX.DOUBLE.false _Var49.SCALAR.INT
--------CP createvar _mVar50 scratch_space//_p70737_9.108.91.12//_t0/temp34 true MATRIX binaryblock 1 -1 1000 1000 0 copy
--------SPARK rand 1 ¶k¶ 1000 1000 0.0 0.0 1.0 -1 scratch_space/_p70737_9.108.91.12//_t0/ uniform 1.0 _mVar50.MATRIX.DOUBLE
--------CP createvar _mVar51 scratch_space//_p70737_9.108.91.12//_t0/temp35 true MATRIX binaryblock 1 -1 1000 1000 -1 copy
--------SPARK rangeReIndex X.MATRIX.DOUBLE r.SCALAR.INT.false r.SCALAR.INT.false 1.SCALAR.INT.true _Var49.SCALAR.INT.false _mVar51.MATRIX.DOUBLE MULTI_BLOCK
--------CP rmvar _Var49
--------CP createvar _mVar52 scratch_space//_p70737_9.108.91.12//_t0/temp36 true MATRIX binaryblock -1 1 1000 1000 -1 copy
--------SPARK r' _mVar51.MATRIX.DOUBLE _mVar52.MATRIX.DOUBLE
--------CP createvar _mVar53 scratch_space//_p70737_9.108.91.12//_t0/temp37 true MATRIX binaryblock -1 1 1000 1000 -1 copy
--------SPARK rsort _mVar52.MATRIX.DOUBLE 1.SCALAR.INT.true true.SCALAR.BOOLEAN.true true.SCALAR.BOOLEAN.true _mVar53.MATRIX.DOUBLE false
--------CP rmvar _mVar52
--------CP createvar _mVar54 scratch_space//_p70737_9.108.91.12//_t0/temp38 true MATRIX binaryblock 1 -1 1000 1000 -1 copy
--------SPARK r' _mVar53.MATRIX.DOUBLE _mVar54.MATRIX.DOUBLE
--------CP rmvar _mVar53
--------CP createvar _mVar55 scratch_space//_p70737_9.108.91.12//_t0/temp39 true MATRIX binaryblock 1 -1 1000 1000 -1 copy
--------SPARK rangeReIndex _mVar54.MATRIX.DOUBLE 1.SCALAR.INT.true 1.SCALAR.INT.true 1.SCALAR.INT.true k.SCALAR.INT.false _mVar55.MATRIX.DOUBLE MULTI_BLOCK
--------CP rmvar _mVar54
--------CP cpvar _mVar50 values
--------CP cpvar _mVar51 row
--------CP cpvar _mVar55 indices
--------CP rmvar _mVar50 _mVar51 _mVar55 X r
------GENERIC (lines 299-301) [recompile=true]
--------CP createvar _mVar56 scratch_space//_p70737_9.108.91.12//_t0/temp40 true MATRIX binaryblock 1 -1 1000 1000 -1 copy
--------SPARK chkpoint indices.MATRIX.DOUBLE _mVar56.MATRIX.DOUBLE MEMORY_AND_DISK
--------CP createvar _mVar57 scratch_space//_p70737_9.108.91.12//_t0/temp41 true MATRIX binaryblock 1 -1 1000 1000 -1 copy
--------SPARK chkpoint row.MATRIX.DOUBLE _mVar57.MATRIX.DOUBLE MEMORY_AND_DISK
--------CP rmvar indices row
--------CP cpvar _mVar56 indices
--------CP cpvar _mVar57 row
--------CP rmvar _mVar56 _mVar57
------GENERIC (lines 299-301) [recompile=true]
--------CP createvar _mVar58 scratch_space//_p70737_9.108.91.12//_t0/temp42 true MATRIX binaryblock 1 -1 1000 1000 -1 copy
--------SPARK chkpoint indices.MATRIX.DOUBLE _mVar58.MATRIX.DOUBLE MEMORY_AND_DISK
--------CP createvar _mVar59 scratch_space//_p70737_9.108.91.12//_t0/temp43 true MATRIX binaryblock 1 -1 1000 1000 -1 copy
--------SPARK chkpoint row.MATRIX.DOUBLE _mVar59.MATRIX.DOUBLE MEMORY_AND_DISK
--------CP rmvar indices row
--------CP cpvar _mVar58 indices
--------CP cpvar _mVar59 row
--------CP rmvar _mVar58 _mVar59
------GENERIC (lines 299-301) [recompile=true]
--------CP createvar _mVar60 scratch_space//_p70737_9.108.91.12//_t0/temp44 true MATRIX binaryblock 1 -1 1000 1000 -1 copy
--------SPARK chkpoint indices.MATRIX.DOUBLE _mVar60.MATRIX.DOUBLE MEMORY_AND_DISK
--------CP createvar _mVar61 scratch_space//_p70737_9.108.91.12//_t0/temp45 true MATRIX binaryblock 1 -1 1000 1000 -1 copy
--------SPARK chkpoint row.MATRIX.DOUBLE _mVar61.MATRIX.DOUBLE MEMORY_AND_DISK
--------CP rmvar indices row
--------CP cpvar _mVar60 indices
--------CP cpvar _mVar61 row
--------CP rmvar _mVar60 _mVar61
------FOR (lines 299-301) [in-place=[values]]
--------GENERIC (lines 300-300) [recompile=true]
----------CP createvar _mVar62 scratch_space//_p70737_9.108.91.12//_t0/temp46 true MATRIX binaryblock 1 1 1000 1000 -1 copy
----------SPARK rangeReIndex indices.MATRIX.DOUBLE 1.SCALAR.INT.true 1.SCALAR.INT.true i.SCALAR.INT.false i.SCALAR.INT.false _mVar62.MATRIX.DOUBLE NONE
----------CP castdts _mVar62.MATRIX.DOUBLE.false _Var63.SCALAR.DOUBLE
----------CP rmvar _mVar62
----------CP createvar _mVar64 scratch_space//_p70737_9.108.91.12//_t0/temp47 true MATRIX binaryblock 1 1 1000 1000 -1 copy
----------SPARK rangeReIndex row.MATRIX.DOUBLE 1.SCALAR.INT.true 1.SCALAR.INT.true _Var63.SCALAR.DOUBLE.false _Var63.SCALAR.DOUBLE.false _mVar64.MATRIX.DOUBLE NONE
----------CP rmvar _Var63
----------CP createvar _mVar65 scratch_space//_p70737_9.108.91.12//_t0/temp48 true MATRIX binaryblock 1 -1 1000 1000 -1 copy
----------SPARK mapLeftIndex values.MATRIX.DOUBLE _mVar64.MATRIX.DOUBLE 1.SCALAR.INT.true 1.SCALAR.INT.true i.SCALAR.INT.false i.SCALAR.INT.false _mVar65.MATRIX.DOUBLE LEFT
----------CP rmvar _mVar64 values
----------CP cpvar _mVar65 values
----------CP rmvar _mVar65
----FUNCTION ./nn/util.dml::top_k [recompile=true]
------GENERIC (lines 319-322) [recompile=true]
--------CP nrow X.MATRIX.DOUBLE.false _Var66.SCALAR.INT
--------CP createvar _mVar67 scratch_space//_p70737_9.108.91.12//_t0/temp49 true MATRIX binaryblock -1 -1 1000 1000 0 copy
--------SPARK rand ¶_Var66¶ ¶k¶ 1000 1000 0.0 0.0 1.0 -1 scratch_space/_p70737_9.108.91.12//_t0/ uniform 1.0 _mVar67.MATRIX.DOUBLE
--------CP assignvar _Var66.SCALAR.INT.false N.SCALAR.INT
--------CP rmvar _Var66
--------CP cpvar _mVar67 values
--------CP cpvar _mVar67 indices
--------CP rmvar _mVar67
------GENERIC (lines 324-328) [recompile=true]
--------CP createvar _mVar68 scratch_space//_p70737_9.108.91.12//_t0/temp50 true MATRIX binaryblock -1 -1 1000 1000 -1 copy
--------SPARK chkpoint X.MATRIX.DOUBLE _mVar68.MATRIX.DOUBLE MEMORY_AND_DISK
--------CP rmvar X
--------CP cpvar _mVar68 X
--------CP rmvar _mVar68
------GENERIC (lines 324-328) [recompile=true]
--------CP createvar _mVar69 scratch_space//_p70737_9.108.91.12//_t0/temp51 true MATRIX binaryblock -1 -1 1000 1000 -1 copy
--------SPARK chkpoint X.MATRIX.DOUBLE _mVar69.MATRIX.DOUBLE MEMORY_AND_DISK
--------CP rmvar X
--------CP cpvar _mVar69 X
--------CP rmvar _mVar69
------GENERIC (lines 324-328) [recompile=true]
--------CP createvar _mVar70 scratch_space//_p70737_9.108.91.12//_t0/temp52 true MATRIX binaryblock -1 -1 1000 1000 -1 copy
--------SPARK chkpoint X.MATRIX.DOUBLE _mVar70.MATRIX.DOUBLE MEMORY_AND_DISK
--------CP rmvar X
--------CP cpvar _mVar70 X
--------CP rmvar _mVar70
------PARFOR (lines 324-328)
--------GENERIC (lines 325-325) [recompile=true]
----------CP extfunct ./nn/util.dml top_k_row 3 2 X.MATRIX.DOUBLE r.SCALAR.INT.false k.SCALAR.INT.false value index
--------GENERIC (lines 326-327) [recompile=true]
----------CP ncol values.MATRIX.DOUBLE.false _Var71.SCALAR.INT
----------CP ncol indices.MATRIX.DOUBLE.false _Var72.SCALAR.INT
----------CP createvar _mVar73 scratch_space//_p70737_9.108.91.12//_t0/temp53 true MATRIX binaryblock -1 -1 1000 1000 -1 copy
----------SPARK leftIndex values.MATRIX.DOUBLE value.MATRIX.DOUBLE r.SCALAR.INT.false r.SCALAR.INT.false 1.SCALAR.INT.true _Var71.SCALAR.INT.false _mVar73.MATRIX.DOUBLE NONE
----------CP rmvar _Var71
----------CP createvar _mVar74 scratch_space//_p70737_9.108.91.12//_t0/temp54 true MATRIX binaryblock -1 -1 1000 1000 -1 copy
----------SPARK leftIndex indices.MATRIX.DOUBLE index.MATRIX.DOUBLE r.SCALAR.INT.false r.SCALAR.INT.false 1.SCALAR.INT.true _Var72.SCALAR.INT.false _mVar74.MATRIX.DOUBLE NONE
----------CP rmvar _Var72 indices values
----------CP cpvar _mVar73 values
----------CP cpvar _mVar74 indices
----------CP rmvar _mVar73 _mVar74 index value
----FUNCTION ./nn/util.dml::top_k2d [recompile=false]
------GENERIC (lines 349-353) [recompile=true]
--------CP * 2.SCALAR.INT.true C.SCALAR.INT.false _Var0.SCALAR.INT
--------CP createvar _mVar1 scratch_space//_p70737_9.108.91.12//_t0/temp1 true MATRIX binaryblock -1 1 1000 1000 -1 copy
--------SPARK seq -1 1 1000 1000 1 ¶C¶ 1 _mVar1.MATRIX.DOUBLE
--------CP - 2.SCALAR.INT.true 1.SCALAR.INT.true _Var2.SCALAR.INT
--------CP / 36.SCALAR.INT.true C.SCALAR.INT.false _Var3.SCALAR.DOUBLE
--------CP createvar _mVar4 scratch_space//_p70737_9.108.91.12//_t0/temp2 true MATRIX binaryblock -1 1 1000 1000 -1 copy
--------SPARK seq -1 1 1000 1000 1 ¶_Var0¶ 1 _mVar4.MATRIX.DOUBLE
--------CP createvar _mVar5 scratch_space//_p70737_9.108.91.12//_t0/temp3 true MATRIX binaryblock 2 1 1000 1000 -1 copy
--------CP seq 2 1 1000 1000 0 ¶_Var2¶ 1 _mVar5.MATRIX.DOUBLE
--------CP rmvar _Var2
--------CP createvar _mVar6 scratch_space//_p70737_9.108.91.12//_t0/temp4 true MATRIX binaryblock -1 -1 1000 1000 72 copy
--------SPARK rshape X.MATRIX.DOUBLE _Var0.SCALAR.INT.false _Var3.SCALAR.DOUBLE.false true.SCALAR.BOOLEAN.true _mVar6.MATRIX.DOUBLE
--------CP * 2.SCALAR.INT.true _Var3.SCALAR.DOUBLE.false _Var7.SCALAR.DOUBLE
--------CP rmvar _Var3
--------CP createvar _mVar8 scratch_space//_p70737_9.108.91.12//_t0/temp5 true MATRIX binaryblock 1 2 1000 1000 -1 copy
--------CP r' _mVar5.MATRIX.DOUBLE _mVar8.MATRIX.DOUBLE 8
--------CP rmvar _mVar5
--------CP createvar _mVar9 scratch_space//_p70737_9.108.91.12//_t0/temp6 true MATRIX binaryblock -1 1 1000 1000 -1 copy
--------SPARK +* _mVar1.MATRIX.DOUBLE C.SCALAR.INT.false _mVar8.MATRIX.DOUBLE _mVar9.MATRIX.DOUBLE
--------CP rmvar _mVar1 _mVar8
--------CP createvar _mVar10 scratch_space//_p70737_9.108.91.12//_t0/temp7 true MATRIX binaryblock -1 1 1000 1000 -1 copy
--------SPARK rshape _mVar9.MATRIX.DOUBLE _Var0.SCALAR.INT.false 1.SCALAR.INT.true true.SCALAR.BOOLEAN.true _mVar10.MATRIX.DOUBLE
--------CP rmvar _mVar9
--------CP createvar _mVar11 scratch_space//_p70737_9.108.91.12//_t0/temp8 true MATRIX binarycell -1 -1 -1 -1 -1 copy
--------SPARK ctable _mVar4.MATRIX.DOUBLE _mVar10.MATRIX.DOUBLE 1.0.SCALAR.DOUBLE.true _Var0.false _Var0.false _mVar11.MATRIX.DOUBLE false
--------CP rmvar _mVar4 _mVar10 _Var0
--------CP createvar _mVar12 scratch_space//_p70737_9.108.91.12//_t0/temp9 true MATRIX binaryblock -1 -1 1000 1000 -1 copy
--------SPARK rblk _mVar11.MATRIX.DOUBLE _mVar12.MATRIX.DOUBLE 1000 1000 true
--------CP rmvar _mVar11
--------CP createvar _mVar13 scratch_space//_p70737_9.108.91.12//_t0/temp10 true MATRIX binaryblock -1 -1 1000 1000 -1 copy
--------SPARK cpmm _mVar12.MATRIX.DOUBLE _mVar6.MATRIX.DOUBLE _mVar13.MATRIX.DOUBLE MULTI_BLOCK
--------CP rmvar _mVar12 _mVar6
--------CP createvar _mVar14 scratch_space//_p70737_9.108.91.12//_t0/temp11 true MATRIX binaryblock -1 -1 1000 1000 -1 copy
--------SPARK rshape _mVar13.MATRIX.DOUBLE C.SCALAR.INT.false _Var7.SCALAR.DOUBLE.false true.SCALAR.BOOLEAN.true _mVar14.MATRIX.DOUBLE
--------CP rmvar _mVar13 _Var7
--------CP createvar _mVar15 scratch_space//_p70737_9.108.91.12//_t0/temp12 true MATRIX binaryblock -1 -1 1000 1000 -1 copy
--------SPARK r' _mVar14.MATRIX.DOUBLE _mVar15.MATRIX.DOUBLE
--------CP rmvar _mVar14
--------CP assignvar 2.SCALAR.INT.true N.SCALAR.INT
--------CP cpvar _mVar15 X_NHWC
--------CP rmvar _mVar15 X C
------GENERIC (lines 356-356) [recompile=true]
--------CP extfunct ./nn/util.dml top_k 2 2 X_NHWC.MATRIX.DOUBLE k.SCALAR.INT.false values_NHWK indices_NHWK
--------CP rmvar k X_NHWC
------GENERIC (lines 358-362) [recompile=true]
--------CP createvar _mVar16 scratch_space//_p70737_9.108.91.12//_t0/temp13 true MATRIX binaryblock -1 -1 1000 1000 -1 copy
--------SPARK r' values_NHWK.MATRIX.DOUBLE _mVar16.MATRIX.DOUBLE
--------CP createvar _mVar17 scratch_space//_p70737_9.108.91.12//_t0/temp14 true MATRIX binaryblock 2 1 1000 1000 -1 copy
--------CP seq 2 1 1000 1000 1 2 1 _mVar17.MATRIX.DOUBLE
--------CP createvar _mVar18 scratch_space//_p70737_9.108.91.12//_t0/temp15 true MATRIX binaryblock -1 -1 1000 1000 -1 copy
--------SPARK r' indices_NHWK.MATRIX.DOUBLE _mVar18.MATRIX.DOUBLE
--------CP nrow _mVar16.MATRIX.DOUBLE.false _Var19.SCALAR.INT
--------CP ncol _mVar16.MATRIX.DOUBLE.false _Var20.SCALAR.INT
--------CP nrow _mVar18.MATRIX.DOUBLE.false _Var21.SCALAR.INT
--------CP ncol _mVar18.MATRIX.DOUBLE.false _Var22.SCALAR.INT
--------CP * _Var19.SCALAR.INT.false 2.SCALAR.INT.true _Var23.SCALAR.INT
--------CP - _Var19.SCALAR.INT.false 1.SCALAR.INT.true _Var24.SCALAR.INT
--------CP / _Var20.SCALAR.INT.false 2.SCALAR.INT.true _Var25.SCALAR.DOUBLE
--------CP rmvar _Var20
--------CP * _Var21.SCALAR.INT.false 2.SCALAR.INT.true _Var26.SCALAR.INT
--------CP - _Var21.SCALAR.INT.false 1.SCALAR.INT.true _Var27.SCALAR.INT
--------CP / _Var22.SCALAR.INT.false 2.SCALAR.INT.true _Var28.SCALAR.DOUBLE
--------CP rmvar _Var22
--------CP createvar _mVar29 scratch_space//_p70737_9.108.91.12//_t0/temp16 true MATRIX binaryblock -1 1 1000 1000 -1 copy
--------SPARK seq -1 1 1000 1000 1 ¶_Var23¶ 1 _mVar29.MATRIX.DOUBLE
--------CP createvar _mVar30 scratch_space//_p70737_9.108.91.12//_t0/temp17 true MATRIX binaryblock -1 1 1000 1000 -1 copy
--------SPARK seq -1 1 1000 1000 0 ¶_Var24¶ 1 _mVar30.MATRIX.DOUBLE
--------CP rmvar _Var24
--------CP createvar _mVar31 scratch_space//_p70737_9.108.91.12//_t0/temp18 true MATRIX binaryblock -1 -1 1000 1000 -1 copy
--------SPARK rshape _mVar16.MATRIX.DOUBLE _Var23.SCALAR.INT.false _Var25.SCALAR.DOUBLE.false true.SCALAR.BOOLEAN.true _mVar31.MATRIX.DOUBLE
--------CP rmvar _mVar16
--------CP * _Var19.SCALAR.INT.false _Var25.SCALAR.DOUBLE.false _Var32.SCALAR.DOUBLE
--------CP rmvar _Var19 _Var25
--------CP createvar _mVar33 scratch_space//_p70737_9.108.91.12//_t0/temp19 true MATRIX binaryblock -1 1 1000 1000 -1 copy
--------SPARK seq -1 1 1000 1000 1 ¶_Var26¶ 1 _mVar33.MATRIX.DOUBLE
--------CP createvar _mVar34 scratch_space//_p70737_9.108.91.12//_t0/temp20 true MATRIX binaryblock -1 1 1000 1000 -1 copy
--------SPARK seq -1 1 1000 1000 0 ¶_Var27¶ 1 _mVar34.MATRIX.DOUBLE
--------CP rmvar _Var27
--------CP createvar _mVar35 scratch_space//_p70737_9.108.91.12//_t0/temp21 true MATRIX binaryblock -1 -1 1000 1000 -1 copy
--------SPARK rshape _mVar18.MATRIX.DOUBLE _Var26.SCALAR.INT.false _Var28.SCALAR.DOUBLE.false true.SCALAR.BOOLEAN.true _mVar35.MATRIX.DOUBLE
--------CP rmvar _mVar18
--------CP * _Var21.SCALAR.INT.false _Var28.SCALAR.DOUBLE.false _Var36.SCALAR.DOUBLE
--------CP rmvar _Var21 _Var28
--------CP createvar _mVar37 scratch_space//_p70737_9.108.91.12//_t0/temp22 true MATRIX binaryblock 1 -1 1000 1000 -1 copy
--------SPARK r' _mVar30.MATRIX.DOUBLE _mVar37.MATRIX.DOUBLE
--------CP rmvar _mVar30
--------CP createvar _mVar38 scratch_space//_p70737_9.108.91.12//_t0/temp23 true MATRIX binaryblock 1 -1 1000 1000 -1 copy
--------SPARK r' _mVar34.MATRIX.DOUBLE _mVar38.MATRIX.DOUBLE
--------CP rmvar _mVar34
--------CP createvar _mVar39 scratch_space//_p70737_9.108.91.12//_t0/temp24 true MATRIX binaryblock 2 1 1000 1000 -1 copy
--------SPARK +* _mVar17.MATRIX.DOUBLE 2.SCALAR.INT.true _mVar37.MATRIX.DOUBLE _mVar39.MATRIX.DOUBLE
--------CP rmvar _mVar37
--------CP createvar _mVar40 scratch_space//_p70737_9.108.91.12//_t0/temp25 true MATRIX binaryblock 2 1 1000 1000 -1 copy
--------SPARK +* _mVar17.MATRIX.DOUBLE 2.SCALAR.INT.true _mVar38.MATRIX.DOUBLE _mVar40.MATRIX.DOUBLE
--------CP rmvar _mVar17 _mVar38
--------CP createvar _mVar41 scratch_space//_p70737_9.108.91.12//_t0/temp26 true MATRIX binarycell -1 -1 -1 -1 -1 copy
--------SPARK ctable _mVar29.MATRIX.DOUBLE _mVar39.MATRIX.DOUBLE 1.0.SCALAR.DOUBLE.true _Var23.false _Var23.false _mVar41.MATRIX.DOUBLE false
--------CP rmvar _mVar29 _mVar39 _Var23
--------CP createvar _mVar42 scratch_space//_p70737_9.108.91.12//_t0/temp27 true MATRIX binarycell -1 -1 -1 -1 -1 copy
--------SPARK ctable _mVar33.MATRIX.DOUBLE _mVar40.MATRIX.DOUBLE 1.0.SCALAR.DOUBLE.true _Var26.false _Var26.false _mVar42.MATRIX.DOUBLE false
--------CP rmvar _mVar33 _mVar40 _Var26
--------CP createvar _mVar43 scratch_space//_p70737_9.108.91.12//_t0/temp28 true MATRIX binaryblock -1 -1 1000 1000 -1 copy
--------SPARK rblk _mVar41.MATRIX.DOUBLE _mVar43.MATRIX.DOUBLE 1000 1000 true
--------CP rmvar _mVar41
--------CP createvar _mVar44 scratch_space//_p70737_9.108.91.12//_t0/temp29 true MATRIX binaryblock -1 -1 1000 1000 -1 copy
--------SPARK rblk _mVar42.MATRIX.DOUBLE _mVar44.MATRIX.DOUBLE 1000 1000 true
--------CP rmvar _mVar42
--------CP createvar _mVar45 scratch_space//_p70737_9.108.91.12//_t0/temp30 true MATRIX binaryblock -1 -1 1000 1000 -1 copy
--------SPARK cpmm _mVar43.MATRIX.DOUBLE _mVar31.MATRIX.DOUBLE _mVar45.MATRIX.DOUBLE MULTI_BLOCK
--------CP rmvar _mVar43 _mVar31
--------CP createvar _mVar46 scratch_space//_p70737_9.108.91.12//_t0/temp31 true MATRIX binaryblock -1 -1 1000 1000 -1 copy
--------SPARK cpmm _mVar44.MATRIX.DOUBLE _mVar35.MATRIX.DOUBLE _mVar46.MATRIX.DOUBLE MULTI_BLOCK
--------CP rmvar _mVar44 _mVar35
--------CP createvar _mVar47 scratch_space//_p70737_9.108.91.12//_t0/temp32 true MATRIX binaryblock 2 -1 1000 1000 -1 copy
--------SPARK rshape _mVar45.MATRIX.DOUBLE 2.SCALAR.INT.true _Var32.SCALAR.DOUBLE.false true.SCALAR.BOOLEAN.true _mVar47.MATRIX.DOUBLE
--------CP rmvar _mVar45 _Var32
--------CP createvar _mVar48 scratch_space//_p70737_9.108.91.12//_t0/temp33 true MATRIX binaryblock 2 -1 1000 1000 -1 copy
--------SPARK rshape _mVar46.MATRIX.DOUBLE 2.SCALAR.INT.true _Var36.SCALAR.DOUBLE.false true.SCALAR.BOOLEAN.true _mVar48.MATRIX.DOUBLE
--------CP rmvar _mVar46 _Var36
--------CP cpvar _mVar47 values
--------CP cpvar _mVar48 indices
--------CP rmvar _mVar47 _mVar48 N values_NHWK indices_NHWK
--MAIN PROGRAM
----GENERIC (lines 1-0) [recompile=false]
------CP createvar _mVar75 scratch_space//_p70737_9.108.91.12//_t0/temp55 true MATRIX binaryblock 2 36 1000 1000 72 copy
------SPARK rblk X.MATRIX.DOUBLE _mVar75.MATRIX.DOUBLE 1000 1000 true
------CP rmvar X
------CP cpvar _mVar75 X
------CP rmvar _mVar75
----GENERIC (lines 8-8) [recompile=false]
------CP extfunct ./nn/util.dml top_k2d 5 2 X.MATRIX.DOUBLE k.SCALAR.INT.false C.SCALAR.INT.false Hin.SCALAR.INT.false Win.SCALAR.INT.false values indices
------CP rmvar X C k Hin Win
----GENERIC (lines 10-11) [recompile=true]
------CP rmvar indices values
{code}







> Add new 1D & 2D top_k utility functions
> ---------------------------------------
>
>                 Key: SYSTEMML-1678
>                 URL: https://issues.apache.org/jira/browse/SYSTEMML-1678
>             Project: SystemML
>          Issue Type: Sub-task
>            Reporter: Mike Dusenberry
>            Assignee: Fei Hu
>
> We should add new {{top_k}} and {{top_k2d}} utility functions (in {{nn/util.dml}}) that accept a matrix {{X}} and return matrices {{values}} and {{indices}} with the top {{k}} values (i.e. probabilities) and associated indices (i.e. classes) along a certain dimension.  This will be modeled after the [{{top_k}} function in TensorFlow | https://www.tensorflow.org/api_docs/python/tf/nn/top_k]  For the 1D case, {{top_k}} will operate on the columns dimension.  A typical use case is that in which {{X}} is the output of a {{softmax}} layer (so each row contains a set of normalized class probabilities), and {{values}} and {{indices}} will contain rows with the top {{k}} probabilities and class indices as described above.  For the 2D case, {{top_k}} will operate on the channels dimension.  A typical use case here is that in which {{X}} is the output of a {{softmax2d}} layer (so each channel contains a set of normalized class probabilities), and {{values}} and {{indices}} will contain the top {{k}} probabilities and indices along the channel axis.  This scenario would be common in an image segmentation problem, in which every pixel of the output image will have a set of class probabilities along the channel axis.
> Having these {{top-k}} functions will allow us to extract either predict a single class for each item, or the top {{k}} classes, and therefore may be more useful that a {{predict_class}} function.
> Although we will use {{values}} and {{indices}} as the names of the returned matrices within the functions, in practice, one is likely to name the results {{probs}} and {{classes}} in the calling environment.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)