[jira] [Closed] (SYSTEMML-2020) Perftest bivariate statistics w/ codegen causes unnecessary sparse-dense conversions

["Matthias Boehm (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SYSTEMML-2020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Matthias Boehm closed SYSTEMML-2020.
------------------------------------
       Resolution: Fixed
         Assignee: Matthias Boehm
    Fix Version/s: SystemML 1.0

> Perftest bivariate statistics w/ codegen causes unnecessary sparse-dense conversions
> ------------------------------------------------------------------------------------
>
>                 Key: SYSTEMML-2020
>                 URL: https://issues.apache.org/jira/browse/SYSTEMML-2020
>             Project: SystemML
>          Issue Type: Bug
>            Reporter: Matthias Boehm
>            Assignee: Matthias Boehm
>             Fix For: SystemML 1.0
>
>
> {code}
> [8,1] scale-categorical
> [3,1] scale-categorical
> [10,1] scale-categorical
> [6,1] scale-categorical
> [4,1] scale-categorical
> [1,1] scale-categorical
> [2,1] scale-categorical
> [7,1] scale-categorical
> [5,1] scale-categorical
> [9,1] scale-categorical
> [13,1] scale-scale
> [13,2] scale-categorical
> [14,1] scale-scale
> [11,1] scale-scale
> [14,2] scale-categorical
> [12,1] scale-scale
> [12,2] scale-categorical
> [11,2] scale-categorical
> [15,1] scale-scale
> [16,1] scale-scale
> [15,2] scale-categorical
> [16,2] scale-categorical
> [11,3] scale-categorical
> [12,3] scale-categorical
> [15,3] scale-categorical
> [12,4] scale-categorical
> [14,3] scale-categorical
> [16,3] scale-categorical
> [13,3] scale-categorical
> [15,4] scale-categorical
> [12,5] scale-categorical
> [16,4] scale-categorical
> [13,4] scale-categorical
> [15,5] scale-categorical
> [11,4] scale-categorical
> [12,6] scale-categorical
> [13,5] scale-categorical
> [16,5] scale-categorical
> [14,4] scale-categorical
> [15,6] scale-categorical
> [11,5] scale-categorical
> [12,7] scale-categorical
> [13,6] scale-categorical
> [14,5] scale-categorical
> [16,6] scale-categorical
> [12,8] scale-categorical
> [13,7] scale-categorical
> [15,7] scale-categorical
> [14,6] scale-categorical
> [11,6] scale-categorical
> [16,7] scale-categorical
> [12,9] scale-categorical
> [13,8] scale-categorical
> [14,7] scale-categorical
> [11,7] scale-categorical
> [16,8] scale-categorical
> [14,8] scale-categorical
> [13,9] scale-categorical
> [12,10] scale-categorical
> [11,8] scale-categorical
> [15,8] scale-categorical
> [16,9] scale-categorical
> [14,9] scale-categorical
> [13,10] scale-categorical
> [11,9] scale-categorical
> [15,9] scale-categorical
> [16,10] scale-categorical
> [14,10] scale-categorical
> [11,10] scale-categorical
> [15,10] scale-categorical
> [19,1] scale-scale
> [17,1] scale-scale
> [19,2] scale-categorical
> [17,2] scale-categorical
> [18,1] scale-scale
> [18,2] scale-categorical
> [19,3] scale-categorical
> [20,1] scale-scale
> [17,3] scale-categorical
> [20,2] scale-categorical
> [18,3] scale-categorical
> [19,4] scale-categorical
> [20,3] scale-categorical
> [17,4] scale-categorical
> [19,5] scale-categorical
> [18,4] scale-categorical
> [17,5] scale-categorical
> [19,6] scale-categorical
> [20,4] scale-categorical
> [18,5] scale-categorical
> [17,6] scale-categorical
> [19,7] scale-categorical
> [20,5] scale-categorical
> [18,6] scale-categorical
> [17,7] scale-categorical
> [19,8] scale-categorical
> [20,6] scale-categorical
> [18,7] scale-categorical
> [17,8] scale-categorical
> [19,9] scale-categorical
> [17,9] scale-categorical
> [18,8] scale-categorical
> [20,7] scale-categorical
> [19,10] scale-categorical
> [18,9] scale-categorical
> [20,8] scale-categorical
> [17,10] scale-categorical
> [20,9] scale-categorical
> [18,10] scale-categorical
> [20,10] scale-categorical
> [8,2] categorical-categorical
> [3,2] categorical-categorical
> [4,2] categorical-categorical
> [10,2] categorical-categorical
> [9,2] categorical-categorical
> [6,2] categorical-categorical
> [2,2] categorical-categorical
> [1,2] categorical-categorical
> [7,2] categorical-categorical
> [5,2] categorical-categorical
> 17/11/17 16:08:05 WARN codegen.SpoofOperator: codegen.TMP43: Converted 566x640, nnz=9829 sideways input matrix from sparse to dense.
> 17/11/17 16:08:05 WARN codegen.SpoofOperator: codegen.TMP22: Converted 727x640, nnz=9886 sideways input matrix from sparse to dense.
> 17/11/17 16:08:05 WARN codegen.SpoofOperator: codegen.TMP57: Converted 1100x640, nnz=9934 sideways input matrix from sparse to dense.
> 17/11/17 16:08:05 WARN codegen.SpoofOperator: codegen.TMP50: Converted 1100x640, nnz=9934 sideways input matrix from sparse to dense.
> 17/11/17 16:08:05 WARN codegen.SpoofOperator: codegen.TMP79: Converted 1100x640, nnz=9922 sideways input matrix from sparse to dense.
> 17/11/17 16:08:05 WARN codegen.SpoofOperator: codegen.TMP64: Converted 1100x640, nnz=9941 sideways input matrix from sparse to dense.
> 17/11/17 16:08:05 WARN codegen.SpoofOperator: codegen.TMP36: Converted 1051x640, nnz=9924 sideways input matrix from sparse to dense.
> 17/11/17 16:08:05 WARN codegen.SpoofOperator: codegen.TMP29: Converted 870x640, nnz=9916 sideways input matrix from sparse to dense.
> 17/11/17 16:08:05 WARN codegen.SpoofOperator: codegen.TMP71: Converted 1100x640, nnz=9923 sideways input matrix from sparse to dense.
> 17/11/17 16:08:05 WARN codegen.SpoofOperator: codegen.TMP85: Converted 1100x640, nnz=9938 sideways input matrix from sparse to dense.
> [1,2] ordinal-ordinal
> [9,3] categorical-categorical
> [2,2] ordinal-ordinal
> [8,3] categorical-categorical
> [10,3] categorical-categorical
> [5,2] ordinal-ordinal
> [6,3] categorical-categorical
> 17/11/17 16:08:05 WARN codegen.SpoofOperator: codegen.TMP85: Converted 1100x1100, nnz=9961 sideways input matrix from sparse to dense.
> [7,3] categorical-categorical
> [4,2] ordinal-ordinal
> 17/11/17 16:08:05 WARN codegen.SpoofOperator: codegen.TMP57: Converted 727x1100, nnz=9935 sideways input matrix from sparse to dense.
> 17/11/17 16:08:05 WARN codegen.SpoofOperator: codegen.TMP57: Converted 566x1100, nnz=9918 sideways input matrix from sparse to dense.
> 17/11/17 16:08:05 WARN codegen.SpoofOperator: codegen.TMP57: Converted 1100x1100, nnz=9964 sideways input matrix from sparse to dense.
> 17/11/17 16:08:05 WARN codegen.SpoofOperator: codegen.TMP57: Converted 1100x1100, nnz=9958 sideways input matrix from sparse to dense.
> [3,2] ordinal-ordinal
> [9,4] categorical-categorical
> [8,4] categorical-categorical
> 17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP57: Converted 566x429, nnz=9790 sideways input matrix from sparse to dense.
> 17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP57: Converted 727x429, nnz=9849 sideways input matrix from sparse to dense.
> [6,4] categorical-categorical
> [10,4] categorical-categorical
> 17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP57: Converted 1100x429, nnz=9900 sideways input matrix from sparse to dense.
> 17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP57: Converted 1100x429, nnz=9904 sideways input matrix from sparse to dense.
> [7,4] categorical-categorical
> 17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP85: Converted 1100x429, nnz=9915 sideways input matrix from sparse to dense.
> [9,5] categorical-categorical
> [8,5] categorical-categorical
> 17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP57: Converted 727x635, nnz=9889 sideways input matrix from sparse to dense.
> 17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP85: Converted 566x635, nnz=9848 sideways input matrix from sparse to dense.
> [10,5] categorical-categorical
> [6,5] categorical-categorical
> 17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP57: Converted 1100x635, nnz=9922 sideways input matrix from sparse to dense.
> [7,5] categorical-categorical
> [1,3] categorical-categorical
> [2,3] categorical-categorical
> 17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP57: Converted 1100x635, nnz=9930 sideways input matrix from sparse to dense.
> [5,3] categorical-categorical
> 17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP85: Converted 1100x1100, nnz=9959 sideways input matrix from sparse to dense.
> [9,6] categorical-categorical
> [8,6] categorical-categorical
> 17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP57: Converted 1100x1100, nnz=9963 sideways input matrix from sparse to dense.
> 17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP57: Converted 1100x1100, nnz=9969 sideways input matrix from sparse to dense.
> 17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP57: Converted 1100x635, nnz=9926 sideways input matrix from sparse to dense.
> 17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP85: Converted 566x1100, nnz=9930 sideways input matrix from sparse to dense.
> 17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP57: Converted 727x1100, nnz=9940 sideways input matrix from sparse to dense.
> [10,6] categorical-categorical
> 17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP57: Converted 1100x1100, nnz=9958 sideways input matrix from sparse to dense.
> [8,7] categorical-categorical
> [7,6] categorical-categorical
> 17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP85: Converted 566x1100, nnz=9915 sideways input matrix from sparse to dense.
> [6,6] categorical-categorical
> 17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP57: Converted 1100x1100, nnz=9966 sideways input matrix from sparse to dense.
> 17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP57: Converted 1100x1100, nnz=9960 sideways input matrix from sparse to dense.
> [9,7] categorical-categorical
> 17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP57: Converted 727x1100, nnz=9928 sideways input matrix from sparse to dense.
> [3,3] categorical-categorical
> 17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP85: Converted 870x1100, nnz=9957 sideways input matrix from sparse to dense.
> [8,8] categorical-categorical
> [4,3] categorical-categorical
> [9,8] categorical-categorical
> 17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP57: Converted 566x1100, nnz=9915 sideways input matrix from sparse to dense.
> 17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP85: Converted 1051x1100, nnz=9961 sideways input matrix from sparse to dense.
> 17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP57: Converted 727x1100, nnz=9933 sideways input matrix from sparse to dense.
> [10,7] categorical-categorical
> 17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP57: Converted 1100x1100, nnz=1100 sideways input matrix from sparse to dense.
> [6,7] categorical-categorical
> [3,3] ordinal-ordinal
> [7,7] categorical-categorical
> 17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP85: Converted 1100x1100, nnz=9949 sideways input matrix from sparse to dense.
> [1,3] ordinal-ordinal
> [8,9] categorical-categorical
> 17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP85: Converted 566x1100, nnz=9922 sideways input matrix from sparse to dense.
> [2,3] ordinal-ordinal
> [5,3] ordinal-ordinal
> [9,9] categorical-categorical
> 17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP57: Converted 1100x1100, nnz=9964 sideways input matrix from sparse to dense.
> 17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP85: Converted 727x1100, nnz=9940 sideways input matrix from sparse to dense.
> [8,10] categorical-categorical
> 17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP85: Converted 566x444, nnz=9795 sideways input matrix from sparse to dense.
> [3,4] categorical-categorical
> [6,8] categorical-categorical
> [10,8] categorical-categorical
> 17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP85: Converted 870x429, nnz=9883 sideways input matrix from sparse to dense.
> 17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP85: Converted 1100x1100, nnz=9959 sideways input matrix from sparse to dense.
> 17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP85: Converted 1100x1100, nnz=9954 sideways input matrix from sparse to dense.
> [9,10] categorical-categorical
> [4,3] ordinal-ordinal
> 17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP85: Converted 727x444, nnz=9830 sideways input matrix from sparse to dense.
> [3,4] ordinal-ordinal
> [7,8] categorical-categorical
> 17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP57: Converted 1100x1100, nnz=9960 sideways input matrix from sparse to dense.
> [6,9] categorical-categorical
> [10,9] categorical-categorical
> 17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP85: Converted 1100x1100, nnz=9957 sideways input matrix from sparse to dense.
> 17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP85: Converted 1100x1100, nnz=9963 sideways input matrix from sparse to dense.
> [7,9] categorical-categorical
> 17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP85: Converted 1100x1100, nnz=9964 sideways input matrix from sparse to dense.
> [10,10] categorical-categorical
> [6,10] categorical-categorical
> 17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP57: Converted 1100x444, nnz=9900 sideways input matrix from sparse to dense.
> 17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP57: Converted 1100x444, nnz=9902 sideways input matrix from sparse to dense.
> [7,10] categorical-categorical
> 17/11/17 16:08:06 WARN codegen.SpoofOperator: codegen.TMP85: Converted 1100x444, nnz=9885 sideways input matrix from sparse to dense.
> [3,5] categorical-categorical
> [1,4] categorical-categorical
> 17/11/17 16:08:07 WARN codegen.SpoofOperator: codegen.TMP85: Converted 870x635, nnz=9922 sideways input matrix from sparse to dense.
> [5,4] categorical-categorical
> [4,4] categorical-categorical
> [2,4] categorical-categorical
> 17/11/17 16:08:07 WARN codegen.SpoofOperator: codegen.TMP57: Converted 1100x429, nnz=9905 sideways input matrix from sparse to dense.
> 17/11/17 16:08:07 WARN codegen.SpoofOperator: codegen.TMP57: Converted 1100x429, nnz=9888 sideways input matrix from sparse to dense.
> 17/11/17 16:08:07 WARN codegen.SpoofOperator: codegen.TMP57: Converted 1051x429, nnz=9890 sideways input matrix from sparse to dense.
> 17/11/17 16:08:07 WARN codegen.SpoofOperator: codegen.TMP57: Converted 1100x429, nnz=9881 sideways input matrix from sparse to dense.
> [5,4] ordinal-ordinal
> [1,4] ordinal-ordinal
> [2,4] ordinal-ordinal
> [4,4] ordinal-ordinal
> [3,5] ordinal-ordinal
> [5,5] categorical-categorical
> [1,5] categorical-categorical
> 17/11/17 16:08:07 WARN codegen.SpoofOperator: codegen.TMP57: Converted 1100x635, nnz=9920 sideways input matrix from sparse to dense.
> 17/11/17 16:08:07 WARN codegen.SpoofOperator: codegen.TMP85: Converted 1100x635, nnz=9933 sideways input matrix from sparse to dense.
> [2,5] categorical-categorical
> [3,6] categorical-categorical
> 17/11/17 16:08:07 WARN codegen.SpoofOperator: codegen.TMP57: Converted 1100x635, nnz=9932 sideways input matrix from sparse to dense.
> [4,5] categorical-categorical
> 17/11/17 16:08:07 WARN codegen.SpoofOperator: codegen.TMP57: Converted 870x1100, nnz=9953 sideways input matrix from sparse to dense.
> 17/11/17 16:08:07 WARN codegen.SpoofOperator: codegen.TMP57: Converted 1051x635, nnz=9939 sideways input matrix from sparse to dense.
> [5,5] ordinal-ordinal
> [1,5] ordinal-ordinal
> [2,5] ordinal-ordinal
> [4,5] ordinal-ordinal
> [3,7] categorical-categorical
> 17/11/17 16:08:07 WARN codegen.SpoofOperator: codegen.TMP57: Converted 870x1100, nnz=9950 sideways input matrix from sparse to dense.
> [4,6] categorical-categorical
> [5,6] categorical-categorical
> 17/11/17 16:08:07 WARN codegen.SpoofOperator: codegen.TMP85: Converted 1100x1100, nnz=9943 sideways input matrix from sparse to dense.
> [2,6] categorical-categorical
> 17/11/17 16:08:07 WARN codegen.SpoofOperator: codegen.TMP85: Converted 1100x1100, nnz=9960 sideways input matrix from sparse to dense.
> [1,6] categorical-categorical
> 17/11/17 16:08:07 WARN codegen.SpoofOperator: codegen.TMP85: Converted 1051x1100, nnz=9955 sideways input matrix from sparse to dense.
> 17/11/17 16:08:07 WARN codegen.SpoofOperator: codegen.TMP85: Converted 1100x1100, nnz=9962 sideways input matrix from sparse to dense.
> [3,8] categorical-categorical
> 17/11/17 16:08:07 WARN codegen.SpoofOperator: codegen.TMP57: Converted 870x1100, nnz=9959 sideways input matrix from sparse to dense.
> [1,7] categorical-categorical
> 17/11/17 16:08:07 WARN codegen.SpoofOperator: codegen.TMP57: Converted 1100x1100, nnz=9964 sideways input matrix from sparse to dense.
> [4,7] categorical-categorical
> [3,9] categorical-categorical
> [5,7] categorical-categorical
> 17/11/17 16:08:07 WARN codegen.SpoofOperator: codegen.TMP85: Converted 1051x1100, nnz=9955 sideways input matrix from sparse to dense.
> 17/11/17 16:08:07 WARN codegen.SpoofOperator: codegen.TMP57: Converted 1100x1100, nnz=9949 sideways input matrix from sparse to dense.
> [2,7] categorical-categorical
> 17/11/17 16:08:07 WARN codegen.SpoofOperator: codegen.TMP85: Converted 870x1100, nnz=9940 sideways input matrix from sparse to dense.
> 17/11/17 16:08:07 WARN codegen.SpoofOperator: codegen.TMP57: Converted 1100x1100, nnz=9962 sideways input matrix from sparse to dense.
> [1,8] categorical-categorical
> [3,10] categorical-categorical
> 17/11/17 16:08:07 WARN codegen.SpoofOperator: codegen.TMP85: Converted 1100x1100, nnz=9952 sideways input matrix from sparse to dense.
> 17/11/17 16:08:07 WARN codegen.SpoofOperator: codegen.TMP57: Converted 870x444, nnz=9867 sideways input matrix from sparse to dense.
> [4,8] categorical-categorical
> 17/11/17 16:08:07 WARN codegen.SpoofOperator: codegen.TMP57: Converted 1051x1100, nnz=9952 sideways input matrix from sparse to dense.
> [2,8] categorical-categorical
> [5,8] categorical-categorical
> 17/11/17 16:08:07 WARN codegen.SpoofOperator: codegen.TMP57: Converted 1100x1100, nnz=9952 sideways input matrix from sparse to dense.
> 17/11/17 16:08:07 WARN codegen.SpoofOperator: codegen.TMP85: Converted 1100x1100, nnz=9958 sideways input matrix from sparse to dense.
> [1,9] categorical-categorical
> 17/11/17 16:08:07 WARN codegen.SpoofOperator: codegen.TMP85: Converted 1100x1100, nnz=9959 sideways input matrix from sparse to dense.
> [4,9] categorical-categorical
> [5,9] categorical-categorical
> 17/11/17 16:08:07 WARN codegen.SpoofOperator: codegen.TMP57: Converted 1051x1100, nnz=9962 sideways input matrix from sparse to dense.
> 17/11/17 16:08:07 WARN codegen.SpoofOperator: codegen.TMP85: Converted 1100x1100, nnz=9961 sideways input matrix from sparse to dense.
> [2,9] categorical-categorical
> 17/11/17 16:08:07 WARN codegen.SpoofOperator: codegen.TMP85: Converted 1100x1100, nnz=9961 sideways input matrix from sparse to dense.
> [1,10] categorical-categorical
> 17/11/17 16:08:07 WARN codegen.SpoofOperator: codegen.TMP57: Converted 1100x444, nnz=9906 sideways input matrix from sparse to dense.
> [5,10] categorical-categorical
> [4,10] categorical-categorical
> 17/11/17 16:08:07 WARN codegen.SpoofOperator: codegen.TMP85: Converted 1100x444, nnz=9895 sideways input matrix from sparse to dense.
> 17/11/17 16:08:07 WARN codegen.SpoofOperator: codegen.TMP85: Converted 1051x444, nnz=9895 sideways input matrix from sparse to dense.
> [2,10] categorical-categorical
> 17/11/17 16:08:07 WARN codegen.SpoofOperator: codegen.TMP57: Converted 1100x444, nnz=9887 sideways input matrix from sparse to dense.
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)