[jira] [Commented] (SYSTEMML-1471) Support PreparedScript for MLContext

["Deron Eriksson (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SYSTEMML-1471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15959961#comment-15959961 ] 

Deron Eriksson commented on SYSTEMML-1471:
------------------------------------------

I think you might be able to do something like this with the existing API. You could create a ScoringScriptExecutor class that extends ScriptExecutor. On this class, create a prepare(Script) method that contains:
{code}
	setup(script);
	parseScript();
	liveVariableAnalysis();
	validateScript();
	constructHops();
	rewriteHops();
	rewritePersistentReadsAndWrites();
	constructLops();
	generateRuntimeProgram();
	showExplanation();
	globalDataFlowOptimization();
	countCompiledMRJobsAndSparkInstructions();
	initializeCachingAndScratchSpace();
	cleanupRuntimeProgram();
{code}

Then override ScriptExecutor's execute(Script) method and have it contain:
{code}
	script.clearAll();
	if(statistics) {
		Statistics.startRunTimer();
	}

	createAndInitializeExecutionContext();
	executeRuntimeProgram();
	cleanupAfterExecution();

	// add symbol table to MLResults
	MLResults mlResults = new MLResults(script);
	script.setResults(mlResults);

	if (statistics) {
		Statistics.stopRunTimer();
		System.out.println(Statistics.display(statisticsMaxHeavyHitters));
	}

	return mlResults;
{code}

In the calling code, have something like:
{code}
ScoringScriptExecutor sse = new ScoringScriptExecutor();
sse.prepare(script); // create the dml program
while (....) {
	...
	MLResults results = ml.execute(script, sse); // execute the dml program
}
{code}



> Support PreparedScript for MLContext
> ------------------------------------
>
>                 Key: SYSTEMML-1471
>                 URL: https://issues.apache.org/jira/browse/SYSTEMML-1471
>             Project: SystemML
>          Issue Type: Improvement
>            Reporter: Niketan Pansare
>
> The intent of this JIRA is three-fold:
> 1. Allow MLContext to be used in prediction scenario.
> 2. Consolidate the code of JMLC and MLContext.
> 3. Explore what extensions are needed in SystemML to support Spark streaming.
> For prediction scenario, it is important to reduce the parsing/validation overhead as much as possible and reusing the JMLC infrastructure might be a good step in that direction. It is also important that MLContext continues to support dynamic recompilation and other optimization as the input size could be small (similar to JMLC), but could also be large (if window size is large, making MLContext ideal for this scenario). 
> {code}
> val streamingContext = new StreamingContext(sc, SLIDE_INTERVAL)
> val windowDStream  = .....window(WINDOW_LENGTH, SLIDE_INTERVAL)
> val preparedScript = ....prepareScript(....)
> windowDStream.foreachRDD(currentWindow => {
> if (currentWindow.count() > 0) {
>   ml.execute(preparedScript.in("X", currentWindow.toDF()))
>   ...
> }
> })
> {code}
> [~deron] [~mboehm7] [~reinwald] [~freiss] [~mwdusenb@us.ibm.com] [~nakul02] Is this something that interest anyone of you ?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)