[jira] [Commented] (SYSTEMML-1277) DataFrames With `mllib.Vector` Columns Are No Longer Converted to Matrices.

["Mike Dusenberry (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SYSTEMML-1277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15870515#comment-15870515 ] 

Mike Dusenberry commented on SYSTEMML-1277:
-------------------------------------------

cc [~deron]

> DataFrames With `mllib.Vector` Columns Are No Longer Converted to Matrices.
> ---------------------------------------------------------------------------
>
>                 Key: SYSTEMML-1277
>                 URL: https://issues.apache.org/jira/browse/SYSTEMML-1277
>             Project: SystemML
>          Issue Type: Bug
>    Affects Versions: SystemML 0.13
>            Reporter: Mike Dusenberry
>            Priority: Blocker
>
> Recently, we made the switch from the old {{mllib.Vector}} to the new {{ml.Vector}} type.  Unfortunately, this leaves us with the issue of no longer recognizing DataFrames with {{mllib.Vector}} columns during conversion, and thus, we (1) do not correctly convert to SystemML {{Matrix}} objects, (2) instead fall back on conversion to {{Frame}} objects, and then (3) fail completely when the ensuing DML script is expecting to operated on matrices.
> Given a Spark {{DataFrame}} {{X_df}} of type {{DataFrame\[__INDEX: int, sample: vector\]}}, where {{vector}} is of type {{mllib.Vector}}, the following script will now fail (did not previously):
> {code}
> script = """
> # Scale images to [-1,1]
> X = X / 255
> X = X * 2 - 1
> """
> outputs = ("X")
> script = dml(script).input(X=X_df).output(*outputs)
> X = ml.execute(script).get(*outputs)
> X
> {code}
> {code}
> Caused by: org.apache.sysml.api.mlcontext.MLContextException: Exception occurred while validating script
> 	at org.apache.sysml.api.mlcontext.ScriptExecutor.validateScript(ScriptExecutor.java:487)
> 	at org.apache.sysml.api.mlcontext.ScriptExecutor.execute(ScriptExecutor.java:280)
> 	at org.apache.sysml.api.mlcontext.MLContext.execute(MLContext.java:293)
> 	... 12 more
> Caused by: org.apache.sysml.parser.LanguageException: Invalid Parameters : ERROR: null -- line 4, column 4 -- Invalid Datatypes for operation FRAME SCALAR
> 	at org.apache.sysml.parser.Expression.raiseValidateError(Expression.java:549)
> 	at org.apache.sysml.parser.Expression.computeDataType(Expression.java:415)
> 	at org.apache.sysml.parser.Expression.computeDataType(Expression.java:386)
> 	at org.apache.sysml.parser.BinaryExpression.validateExpression(BinaryExpression.java:130)
> 	at org.apache.sysml.parser.StatementBlock.validate(StatementBlock.java:567)
> 	at org.apache.sysml.parser.DMLTranslator.validateParseTree(DMLTranslator.java:140)
> 	at org.apache.sysml.api.mlcontext.ScriptExecutor.validateScript(ScriptExecutor.java:485)
> 	... 14 more
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)