Re: Gxuides about running SystemML by spark cluster

[Matthias Boehm <mb...@us.ibm.com>]

thanks again for catching
https://issues.apache.org/jira/browse/SYSTEMML-609, yes the change is in
SystemML head now, so please rebuild SystemML or use one of our nightly
builds (https://sparktc.ibmcloud.com/repo/latest/). Thanks.

For running SystemML on Spark, you have multiple options (
http://apache.github.io/incubator-systemml/#running-systemml). Either use
MLContext or spark-submit. Since our documentation does not show many
examples for spark-submit yet, here is a typical command line invocation:

../spark/bin/spark-submit \
      --class org.apache.sysml.api.DMLScript \
      --master yarn-client \
      --num-executors 10 \
      --driver-memory 20g \
      --executor-memory 60g \
      --executor-cores 24 \
      --queue default \
      --conf spark.driver.maxResultSize=0 \
      ./SystemML.jar \
      -f test.dml -stats -exec hybrid_spark -nvargs ...

Everything else is similar to the hadoop invocation. We also provide you a
script that simplifies this configuration:
https://github.com/apache/incubator-systemml/blob/master/scripts/sparkDML.sh
. Keep in mind that if you want to run in yarn-cluster, you should put the
DML script and potentially SystemML-config into HDFS too.

Regards,
Matthias




From:	Wenjie Zhuang <ka...@vt.edu>
To:	dev@systemml.incubator.apache.org
Cc:	Matthias Boehm/Almaden/IBM@IBMUS
Date:	04/02/2016 07:50 PM
Subject:	Re: Gxuides about running SystemML by spark cluster



Hi,

I try to run StepLinearRegDS.dml by spark yarn mode today. And I get the
following result. Is it correct?

Thanks.

BEGIN STEPWISE LINEAR REGRESSION SCRIPT
Reading X and Y...
Best AIC without any features: 4123.134539784949
Best AIC 4068.2916533784332 achieved with feature: 22
Running linear regression with selected features...
Computing the statistics...
Writing the output matrix...



On Sat, Apr 2, 2016 at 8:37 AM, Wenjie Zhuang <ka...@vt.edu> wrote:
  Hi,

  I am now trying to run experiments about SystemML on spark cluster. Could
  you please share some guides about how to run StepLinearRegDS.dml by
  spark cluster?  The official guide I find is most about hadoop.

  Thanks & Have a good weekend!