You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@systemml.apache.org by "Janardhan (JIRA)" <ji...@apache.org> on 2019/04/13 08:22:00 UTC
[jira] [Assigned] (SYSTEMML-1158) MLContext values not found when
data is read in DML script
[ https://issues.apache.org/jira/browse/SYSTEMML-1158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Janardhan reassigned SYSTEMML-1158:
-----------------------------------
Assignee: Janardhan
> MLContext values not found when data is read in DML script
> ----------------------------------------------------------
>
> Key: SYSTEMML-1158
> URL: https://issues.apache.org/jira/browse/SYSTEMML-1158
> Project: SystemML
> Issue Type: Bug
> Components: APIs
> Affects Versions: SystemML 0.11
> Environment: IBM DSX
> Reporter: Felix Schüler
> Assignee: Janardhan
> Priority: Major
>
> When executing a DML script through MLContext that reads data from a file, resulting variables can not be found in the MLContext because they are removed from the context. This does not happen when the data is initialized randomly inside the script.
> This example works fine with MLContext.getTuple
> {code}
> data = rand(100, 10)
> c = ncol(data)
> r = nrow(data)
> stats = rand(rows=1, cols=c)
> for (i in 0:(r - 1)) {
> stats = (stats + data[i + 1,])
> }
> stats = (stats / r)
> {code}
> When I replace the random matrix initialization with {code} read("~/abalone_transformed", format="csv"){code} I get the following exception:
> {code}
> Name: org.apache.sysml.api.mlcontext.MLContextException
> Message: Variable 'stats' not found
> StackTrace: at org.apache.sysml.api.mlcontext.MLResults.getData(MLResults.java:103)
> at org.apache.sysml.api.mlcontext.MLResults.outputValue(MLResults.java:1996)
> at org.apache.sysml.api.mlcontext.MLResults.getTuple(MLResults.java:663)
> {code}
> The instructions for the working case:
> {code}
> PROGRAM ( size CP/SP = 28/0 )
> --MAIN PROGRAM
> ----GENERIC (lines 1-4) [recompile=false]
> ------CP createvar _mVar167234 scratch_space//_p664_10.143.133.52//_t0/temp83636 true MATRIX binaryblock 3 3 1000 1000 9 copy
> ------CP rand 3 3 1000 1000 0.0 1.0 1.0 -1 uniform 1.0 48 _mVar167234.MATRIX.DOUBLE
> ------CP createvar _mVar167235 scratch_space//_p664_10.143.133.52//_t0/temp83637 true MATRIX binaryblock 1 3 1000 1000 3 copy
> ------CP rand 1 3 1000 1000 0.0 1.0 1.0 -1 uniform 1.0 48 _mVar167235.MATRIX.DOUBLE
> ------CP assignvar 3.SCALAR.INT.true c.SCALAR.INT
> ------CP assignvar 3.SCALAR.INT.true r.SCALAR.INT
> ------CP cpvar _mVar167234 data
> ------CP cpvar _mVar167235 stats
> ------CP rmvar _mVar167234
> ------CP rmvar _mVar167235
> ----GENERIC (lines 5-7) [recompile=false]
> ----FOR (lines 5-7)
> ------GENERIC (lines 6-6) [recompile=false]
> --------CP + i.SCALAR.INT.false 1.SCALAR.INT.true _Var167236.SCALAR.INT
> --------CP createvar _mVar167237 scratch_space//_p664_10.143.133.52//_t0/temp83638 true MATRIX binaryblock 1 3 1000 1000 -1 copy
> --------CP rangeReIndex data.MATRIX.DOUBLE _Var167236.SCALAR.INT.false _Var167236.SCALAR.INT.false 1.SCALAR.INT.true 3.SCALAR.INT.true _mVar167237.MATRIX.DOUBLE
> --------CP rmvar _Var167236
> --------CP createvar _mVar167238 scratch_space//_p664_10.143.133.52//_t0/temp83639 true MATRIX binaryblock 1 3 1000 1000 -1 copy
> --------CP + stats.MATRIX.DOUBLE _mVar167237.MATRIX.DOUBLE _mVar167238.MATRIX.DOUBLE
> --------CP rmvar _mVar167237
> --------CP rmvar stats
> --------CP cpvar _mVar167238 stats
> --------CP rmvar _mVar167238
> ----GENERIC (lines 8-11) [recompile=false]
> ------CP createvar _mVar167239 scratch_space//_p664_10.143.133.52//_t0/temp83640 true MATRIX binaryblock 1 3 1000 1000 -1 copy
> ------CP / stats.MATRIX.DOUBLE 3.SCALAR.INT.true _mVar167239.MATRIX.DOUBLE
> ------CP rmvar stats
> ------CP cpvar _mVar167239 stats
> ------CP rmvar _mVar167239
> ------CP rmvar r
> ------CP rmvar c
> ------CP rmvar stats
> {code}
> Failing case (note that this is in a notebook reading a file from object storage)
> {code}
> PROGRAM ( size CP/SP = 38/7 )
> --MAIN PROGRAM
> ----GENERIC (lines 1-4) [recompile=true]
> ------CP createvar pREADdata ~/abalone_transformed false MATRIX csv -1 -1 -1 -1 -1 copy false , true 0.0
> ------CP createvar _mVar150511 scratch_space//_p664_10.143.133.52//_t0/temp75273 true MATRIX binaryblock -1 -1 1000 1000 -1 copy
> ------SPARK csvrblk pREADdata.MATRIX.DOUBLE _mVar150511.MATRIX.DOUBLE 1000 1000 false , true 0.0
> ------CP createvar _mVar150512 scratch_space//_p664_10.143.133.52//_t0/temp75274 true MATRIX binaryblock -1 -1 1000 1000 -1 copy
> ------SPARK chkpoint _mVar150511.MATRIX.DOUBLE _mVar150512.MATRIX.DOUBLE MEMORY_AND_DISK
> ------CP rmvar _mVar150511
> ------CP cpvar _mVar150512 data
> ------CP rmvar _mVar150512
> ----GENERIC (lines 1-4) [recompile=true]
> ------CP ncol data.MATRIX.DOUBLE.false _Var150513.SCALAR.INT
> ------CP nrow data.MATRIX.DOUBLE.false _Var150514.SCALAR.INT
> ------CP createvar _mVar150515 scratch_space//_p664_10.143.133.52//_t0/temp75275 true MATRIX binaryblock 1 -1 1000 1000 -1 copy
> ------SPARK rand 1 ¶_Var150513¶ 1000 1000 0.0 1.0 1.0 -1 scratch_space/_p664_10.143.133.52//_t0/ uniform 1.0 _mVar150515.MATRIX.DOUBLE
> ------CP assignvar _Var150513.SCALAR.INT.false c.SCALAR.INT
> ------CP assignvar _Var150514.SCALAR.INT.false r.SCALAR.INT
> ------CP rmvar _Var150513
> ------CP rmvar _Var150514
> ------CP cpvar _mVar150515 stats
> ------CP rmvar _mVar150515
> ----GENERIC (lines 5-7) [recompile=true]
> ------CP createvar _mVar150516 scratch_space//_p664_10.143.133.52//_t0/temp75276 true MATRIX binaryblock -1 -1 1000 1000 -1 copy
> ------SPARK chkpoint data.MATRIX.DOUBLE _mVar150516.MATRIX.DOUBLE MEMORY_AND_DISK
> ------CP rmvar data
> ------CP cpvar _mVar150516 data
> ------CP rmvar _mVar150516
> ----FOR (lines 5-7)
> ------CP - r.SCALAR.INT.false 1.SCALAR.INT.true _Var150517.SCALAR.INT
> ------CP rmvar _Var150517
> ------GENERIC (lines 6-6) [recompile=true]
> --------CP + i.SCALAR.INT.false 1.SCALAR.INT.true _Var150518.SCALAR.INT
> --------CP ncol data.MATRIX.DOUBLE.false _Var150519.SCALAR.INT
> --------CP createvar _mVar150520 scratch_space//_p664_10.143.133.52//_t0/temp75277 true MATRIX binaryblock 1 -1 1000 1000 -1 copy
> --------SPARK rangeReIndex data.MATRIX.DOUBLE _Var150518.SCALAR.INT.false _Var150518.SCALAR.INT.false 1.SCALAR.INT.true _Var150519.SCALAR.INT.false _mVar150520.MATRIX.DOUBLE MULTI_BLOCK
> --------CP rmvar _Var150518
> --------CP rmvar _Var150519
> --------CP createvar _mVar150521 scratch_space//_p664_10.143.133.52//_t0/temp75278 true MATRIX binaryblock 1 -1 1000 1000 -1 copy
> --------SPARK + stats.MATRIX.DOUBLE _mVar150520.MATRIX.DOUBLE _mVar150521.MATRIX.DOUBLE
> --------CP rmvar _mVar150520
> --------CP rmvar stats
> --------CP cpvar _mVar150521 stats
> --------CP rmvar _mVar150521
> ----GENERIC (lines 8-11) [recompile=true]
> ------CP createvar _mVar150522 scratch_space//_p664_10.143.133.52//_t0/temp75279 true MATRIX binaryblock 1 -1 1000 1000 -1 copy
> ------SPARK / stats.MATRIX.DOUBLE r.SCALAR.INT.false _mVar150522.MATRIX.DOUBLE
> ------CP rmvar stats
> ------CP cpvar _mVar150522 stats
> ------CP rmvar _mVar150522
> ------CP rmvar r
> ------CP rmvar c
> ------CP rmvar stats
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)