[jira] [Resolved] (SYSTEMML-678) MLContext parallelization

["Sebastian Baunsgaard (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SYSTEMML-678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sebastian Baunsgaard resolved SYSTEMML-678.
-------------------------------------------
    Fix Version/s: SystemML 0.10
       Resolution: Done

> MLContext parallelization
> -------------------------
>
>                 Key: SYSTEMML-678
>                 URL: https://issues.apache.org/jira/browse/SYSTEMML-678
>             Project: SystemDS
>          Issue Type: Question
>          Components: Algorithms, Parser, Runtime
>    Affects Versions: SystemML 0.10
>            Reporter: Johannes Wilke
>            Priority: Major
>             Fix For: SystemML 0.10
>
>
> I try to execute script in the MLContext. It is executing, but it dont parallel. For smaller scripts, it works fine. But this script doesnt and it is not clear why. I think it is because of the 4 loop levels, but I am not sure. 
> Is there a documentation what is parallizable and what isnt?
> If I change the main while-loop, i wish to parallize, to a parfor loop it works.
> Here is the script:
> X = read($Xin)
> P = read($Pin)
> #errorMatrix = matrix(0.0,rows=1,cols=1)
> j = 1
> sum = 0
> while (j <=nrow(X) & sum >= 0){ # this should be parallelized 
> #parfor(j in 1: nrow(X),check=0){
> 	first = TRUE
> 	windows = matrix(0,rows=1,cols=1)
> 	offsetPreWindowDefinitions = 0
> 	sumWindowLength = 0
> 	mastercount = 0
> 	totalwindowLength = 0
> 	s = 0
> 	for(i in 1: nrow(P)){
> 		if((as.scalar(P[i,1])*as.scalar(P[i,2]))>totalwindowLength){
> 			totalwindowLength = (as.scalar(P[i,1])*as.scalar(P[i,2]))
> 		}
> 		s = s+1
> 	}
> 	lastWindow = matrix(0,rows=sum(P[,1]),cols=1)
> 	
> 	for(i in 1:nrow(P)){# for every Window-Definition
> 		
> 		for(k in 1: as.integer(as.scalar(P[i,1]))){# for every pnum
> 			column = matrix(0,rows=as.integer(as.scalar(P[1,4])),cols=1)
> 			for(l in 1: nrow(column)+1){
> 				offsetPreWindowDefinitions = totalwindowLength - (as.scalar(P[i,1])*as.scalar(P[i,2]))
> 				tsindex = ((k-1) * as.scalar(P[i,2])) + l-1 + offsetPreWindowDefinitions
> 				if(l==nrow(column)+1){
> 					lastWindow[sumWindowLength+k,1] = X[j,tsindex+1]
> 				} else {
> 					
> 					column[l,1] = X[j,tsindex+1]
> 				}
> 				mastercount = mastercount +1
> 				#print(mastercount)
> 			}
> 			if(first){
> 				first = FALSE;
> 				windows = column
> 			} else {
> 				windows = cbind(windows,column)
> 			}
> 		}
> 		
> 		sumWindowLength = sumWindowLength + as.scalar(P[i,1])
> 	}
> 	
> 	
> 	result = matrix(14.3,rows=as.integer(as.scalar(P[1,4])),cols=1)
> 	for(i in totalwindowLength:as.integer(as.scalar(P[1,4]))+totalwindowLength-1){
> 		result[i-totalwindowLength+1,1] = X[j,i+1]	
> 		s = s+1
> 	}
> 	params = solve(windows,result)
> 	print(j)
> 	predict = matrix(0,rows=1, cols=1)
> 	for(i in 1:nrow(lastWindow)){
> 		predict[1,1] = predict[1,1] + (params[i,1] * lastWindow[i,1])
> 		s = s+1
> 	}
> 	
> 	predictscalar = as.scalar(predict[1,1])
> 	targetscalar = as.scalar(X[j,ncol(X)])
> 	sum = sum + ((targetscalar - predictscalar) * (targetscalar - predictscalar))
> 	
> 	
> 	
> 	j = j+1
> 	#write(lastWindow, "/media/johannes/Data/Seafile/UNI/Beleg/sysml_output/lWOut.csv", format="csv", header=TRUE, sep=",", sparse=TRUE);
> 	#write(windows, "/media/johannes/Data/Seafile/UNI/Beleg/sysml_output/windowsOut.csv", format="csv", header=TRUE, sep=",", sparse=TRUE);
> 	#write(result, "/media/johannes/Data/Seafile/UNI/Beleg/sysml_output/resultOut.csv", format="csv", header=TRUE, sep=",", sparse=TRUE);
> }
> print(sum/nrow(X))
> I hope that you can help me!



--
This message was sent by Atlassian Jira
(v8.3.4#803005)