[jira] [Commented] (SYSTEMML-880) Push-down loop structures in Python DSL

["Niketan Pansare (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SYSTEMML-880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15589478#comment-15589478 ] 

Niketan Pansare commented on SYSTEMML-880:
------------------------------------------

Design decisions of PR https://github.com/apache/incubator-systemml/pull/270:

1. In the initial version, I decided not to parse and walk the Python AST. The key assumption is that the code to be pushed-down has to map to PyDML. This will help us to identify the gaps in PyDML as well (for example: dot(H, H.transpose()) instead of H.dot(H.transpose)). Also, cases such as sml.matrix is not handled.

2. The inputs and outputs need to be specified. There is a tradeoff here between simplicity of the API+Implementation and optimization scope.
 - Alternative to the current implementation is that inputs+outputs are not specified and the function to be pushed-down has access to all the variables in the global scope. In this case, we walk down the AST and identify all the variables on RHS and treat them as outputs and all other variables not created as input (hand-waving lil bit here).
- Another alternative is to not require providing variable name in string format: 'H', but this was difficult to implement due to pass-by-value and point 1.

Also, I could not find a way to enable the decorator (@parallelize) on statement block instead of function.

import systemml as sml
from systemml import random
sml.setSparkContext(sc)

m, n = 100, 20
k = 40
V = sml.random.uniform(size=(m, n))
W = sml.random.uniform(size=(m, k))
H = sml.random.uniform(size=(k, n))

@sml.parallelize
def push_down_loop(inputs, outputs):
    max_iters = 200
    for i in range(0, max_iters):
        H = H * dot(W.transpose(), V)/dot(W.transpose(), dot(W, H))
        W = W * dot(V, H.transpose())/dot(W, dot(H, H.transpose()))

# The returned values as Matrix class
ret = push_down_loop(inputs={'H': H, 'W':W, 'V': V}, outputs=['H', 'W'])
H = ret[0].toNumPy()
W = ret[1].toNumPy()

> Push-down loop structures in Python DSL
> ---------------------------------------
>
>                 Key: SYSTEMML-880
>                 URL: https://issues.apache.org/jira/browse/SYSTEMML-880
>             Project: SystemML
>          Issue Type: Task
>            Reporter: Niketan Pansare
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)