You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@systemml.apache.org by Sourav Mazumder <so...@gmail.com> on 2015/12/08 17:25:52 UTC
DML script parsing error in Spark
Hi,
I'm facing issue while parsing any DML script.
What I'm trying something like -
val ml = new MLContext(sc)ml.reset()
ml.registerInput("X", Xfc, 3569, 4)
ml.registerInput("Y", yDc, 1, 4)
ml.registerOutput("beta_out")
val cmdLineParams = Map("X" -> " ", "Y" -> " ")
val outputs =
ml.execute("/home/system-ml-0.9.0-SNAPSHOT/algorithms/GLM2.dml",
cmdLineParams).
I'm getting following error -
com.ibm.bi.dml.parser.ParseException: ERROR: Cannot translate the parse
tree into DMLProgram:null at
com.ibm.bi.dml.parser.antlr4.DMLParserWrapper.doParse(DMLParserWrapper.java:250)
at
com.ibm.bi.dml.parser.antlr4.DMLParserWrapper.parse(DMLParserWrapper.java:143)
at
com.ibm.bi.dml.api.MLContext.executeUsingSimplifiedCompilationChain(MLContext.java:1285)
at
com.ibm.bi.dml.api.MLContext.compileAndExecuteScript(MLContext.java:1204)
at
com.ibm.bi.dml.api.MLContext.compileAndExecuteScript(MLContext.java:1150)
at com.ibm.bi.dml.api.MLContext.execute(MLContext.java:632) at
com.ibm.bi.dml.api.MLContext.execute(MLContext.java:667) at
com.ibm.bi.dml.api.MLContext.execute(MLContext.java:680) at
$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:33) at
$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:38) at
$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:40) at
$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:42) at
$iwC$$iwC$$iwC$$iwC.<init>(<console>:44) at
$iwC$$iwC$$iwC.<init>(<console>:46) at $iwC$$iwC.<init>(<console>:48) at
$iwC.<init>(<console>:50) at <init>(<console>:52) at .<init>(<console>:56)
at .<clinit>(<console>) at .<init>(<console>:7) at .<clinit>(<console>) at
$print(<console>) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497) at
org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)
at
org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1340)
at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871) at
org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819)
Regards,
Sourav
Re: DML script parsing error in Spark
Posted by Sourav Mazumder <so...@gmail.com>.
Got a solution to this problem. I was not passing the right value of the
Response matrix. With the proper value of response matrix this problem does
not occur.
Regards,
Sourav
On Tue, Dec 8, 2015 at 11:12 AM, Sourav Mazumder <
sourav.mazumder00@gmail.com> wrote:
> Hi Shirish,
>
> Passing 'B' as " " (as below) through cmdLineArg did not help. I get the
> same error.
>
> val cmdLineParams = Map("X" -> " ", "Y" -> " ", "B" -> " ")
> val outputs =
> ml.execute("/home/system-ml-0.9.0-SNAPSHOT/algorithms/GLM2.dml",
> cmdLineParams)
>
> Any idea what else I need to pass ? I'm trying to run the DMLs only on
> Spark (no Hadoop) and trying to come out with a use case.
>
> Regards,
> Sourav
>
> On Tue, Dec 8, 2015 at 9:27 AM, Shirish Tatikonda <
> shirish.tatikonda@gmail.com> wrote:
>
>> Sourav,
>>
>> In cmdLineParams, you also need to pass B (output location) -- it does
>> not
>> have a default value.
>>
>> In general, a DML script may have two types of "inputs":
>>
>> 1. $ parameters in the script (e.g., $X, $Y, $dfam, etc. in
>> GLM.dml) which gets populated from command line arguments. The default
>> arguments are specified via *ifdef() *in the script.
>> 2. data that you pass-in via *read().*
>>
>> cmdLineParams in your example refers to (1).
>>
>> For read(), the exact method in which the data is passed depends on the
>> way
>> you invoke the script. In a typical scenario, the data comes from HDFS
>> files. Additionally, in the context of MLContext, it may also come from
>> RDDs/DataFrames via *registerInput()*. In the context of JMLC, the data
>> can
>> come in as an in-memory data structure (e.g., double[][] array).
>>
>> Hope that clarifies the difference.
>>
>> Shirish
>>
>>
>> On Tue, Dec 8, 2015 at 8:25 AM, Sourav Mazumder <
>> sourav.mazumder00@gmail.com
>> > wrote:
>>
>> > Hi,
>> >
>> > I'm facing issue while parsing any DML script.
>> >
>> > What I'm trying something like -
>> >
>> > val ml = new MLContext(sc)ml.reset()
>> > ml.registerInput("X", Xfc, 3569, 4)
>> > ml.registerInput("Y", yDc, 1, 4)
>> > ml.registerOutput("beta_out")
>> >
>> > val cmdLineParams = Map("X" -> " ", "Y" -> " ")
>> >
>> > val outputs =
>> > ml.execute("/home/system-ml-0.9.0-SNAPSHOT/algorithms/GLM2.dml",
>> > cmdLineParams).
>> >
>> > I'm getting following error -
>> >
>> > com.ibm.bi.dml.parser.ParseException: ERROR: Cannot translate the parse
>> > tree into DMLProgram:null at
>> >
>> >
>> com.ibm.bi.dml.parser.antlr4.DMLParserWrapper.doParse(DMLParserWrapper.java:250)
>> > at
>> >
>> >
>> com.ibm.bi.dml.parser.antlr4.DMLParserWrapper.parse(DMLParserWrapper.java:143)
>> > at
>> >
>> >
>> com.ibm.bi.dml.api.MLContext.executeUsingSimplifiedCompilationChain(MLContext.java:1285)
>> > at
>> >
>> com.ibm.bi.dml.api.MLContext.compileAndExecuteScript(MLContext.java:1204)
>> > at
>> >
>> com.ibm.bi.dml.api.MLContext.compileAndExecuteScript(MLContext.java:1150)
>> > at com.ibm.bi.dml.api.MLContext.execute(MLContext.java:632) at
>> > com.ibm.bi.dml.api.MLContext.execute(MLContext.java:667) at
>> > com.ibm.bi.dml.api.MLContext.execute(MLContext.java:680) at
>> > $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:33) at
>> > $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:38) at
>> > $iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:40) at
>> > $iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:42) at
>> > $iwC$$iwC$$iwC$$iwC.<init>(<console>:44) at
>> > $iwC$$iwC$$iwC.<init>(<console>:46) at $iwC$$iwC.<init>(<console>:48) at
>> > $iwC.<init>(<console>:50) at <init>(<console>:52) at
>> .<init>(<console>:56)
>> > at .<clinit>(<console>) at .<init>(<console>:7) at .<clinit>(<console>)
>> at
>> > $print(<console>) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>> > Method) at
>> >
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> > at
>> >
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> > at java.lang.reflect.Method.invoke(Method.java:497) at
>> >
>> org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)
>> > at
>> >
>> org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1340)
>> > at
>> org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840)
>> > at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871) at
>> > org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819)
>> >
>> >
>> > Regards,
>> > Sourav
>> >
>>
>
>
Re: DML script parsing error in Spark
Posted by Sourav Mazumder <so...@gmail.com>.
Hi Niketan,
Just sent a mail to this forum stating that this problem is solved.
I was passing wrong dimension while creating Y using ml.registerInput().
That was what creating problem.
Once I pass the right dimension it works.
Thanks for your help.
Regards,
Sourav
On Tue, Dec 8, 2015 at 12:32 PM, Niketan Pansare <np...@us.ibm.com> wrote:
> Hi Sourav,
>
> I tried a similar example on our internal cluster and everything seems to
> work.
>
> The parser might have provided some error/warning messages before it
> throws exception. Can you please provide the entire trace ?
> For example:
> scala> val outputs =
> ml.execute("/home/system-ml-0.9.0-SNAPSHOT/algorithms/GLM2.dml",
> cmdLineParams)
> 15/12/08 12:00:56 WARN hops.OptimizerUtils: Auto-disable multi-threaded
> text read for 'text' and 'csv' due to thread contention on JRE < 1.8
> (java.version=1.7.0_91).
> 15/12/08 12:00:56 WARN parser.StatementBlock: WARNING:
> ../algorithms/GLM.dml -- line 361, column 14 -- Initialization of w depends
> on if-else execution
> 15/12/08 12:00:56 WARN parser.StatementBlock: WARNING:
> ../algorithms/GLM.dml -- line 361, column 14 -- Initialization of w depends
> on while execution
> 15/12/08 12:00:56 WARN parser.StatementBlock: WARNING:
> ../algorithms/GLM.dml -- line 161, column 0 -- Variable termination_code
> defined with different value type in if and else clause.
>
> Thanks,
>
> Niketan Pansare
> IBM Almaden Research Center
> E-mail: npansar At us.ibm.com
> http://researcher.watson.ibm.com/researcher/view.php?person=us-npansar
>
> [image: Inactive hide details for Sourav Mazumder ---12/08/2015 11:17:43
> AM---Hi Shirish, Passing 'B' as " " (as below) through cmdLine]Sourav
> Mazumder ---12/08/2015 11:17:43 AM---Hi Shirish, Passing 'B' as " " (as
> below) through cmdLineArg did not help. I get the
>
> From: Sourav Mazumder <so...@gmail.com>
> To: dev@systemml.incubator.apache.org
> Date: 12/08/2015 11:17 AM
> Subject: Re: DML script parsing error in Spark
> ------------------------------
>
>
>
> Hi Shirish,
>
> Passing 'B' as " " (as below) through cmdLineArg did not help. I get the
> same error.
>
> val cmdLineParams = Map("X" -> " ", "Y" -> " ", "B" -> " ")
> val outputs =
> ml.execute("/home/system-ml-0.9.0-SNAPSHOT/algorithms/GLM2.dml",
> cmdLineParams)
>
> Any idea what else I need to pass ? I'm trying to run the DMLs only on
> Spark (no Hadoop) and trying to come out with a use case.
>
> Regards,
> Sourav
>
> On Tue, Dec 8, 2015 at 9:27 AM, Shirish Tatikonda <
> shirish.tatikonda@gmail.com> wrote:
>
> > Sourav,
> >
> > In cmdLineParams, you also need to pass B (output location) -- it does
> not
> > have a default value.
> >
> > In general, a DML script may have two types of "inputs":
> >
> > 1. $ parameters in the script (e.g., $X, $Y, $dfam, etc. in
> > GLM.dml) which gets populated from command line arguments. The default
> > arguments are specified via *ifdef() *in the script.
> > 2. data that you pass-in via *read().*
> >
> > cmdLineParams in your example refers to (1).
> >
> > For read(), the exact method in which the data is passed depends on the
> way
> > you invoke the script. In a typical scenario, the data comes from HDFS
> > files. Additionally, in the context of MLContext, it may also come from
> > RDDs/DataFrames via *registerInput()*. In the context of JMLC, the data
> can
> > come in as an in-memory data structure (e.g., double[][] array).
> >
> > Hope that clarifies the difference.
> >
> > Shirish
> >
> >
> > On Tue, Dec 8, 2015 at 8:25 AM, Sourav Mazumder <
> > sourav.mazumder00@gmail.com
> > > wrote:
> >
> > > Hi,
> > >
> > > I'm facing issue while parsing any DML script.
> > >
> > > What I'm trying something like -
> > >
> > > val ml = new MLContext(sc)ml.reset()
> > > ml.registerInput("X", Xfc, 3569, 4)
> > > ml.registerInput("Y", yDc, 1, 4)
> > > ml.registerOutput("beta_out")
> > >
> > > val cmdLineParams = Map("X" -> " ", "Y" -> " ")
> > >
> > > val outputs =
> > > ml.execute("/home/system-ml-0.9.0-SNAPSHOT/algorithms/GLM2.dml",
> > > cmdLineParams).
> > >
> > > I'm getting following error -
> > >
> > > com.ibm.bi.dml.parser.ParseException: ERROR: Cannot translate the parse
> > > tree into DMLProgram:null at
> > >
> > >
> >
> com.ibm.bi.dml.parser.antlr4.DMLParserWrapper.doParse(DMLParserWrapper.java:250)
> > > at
> > >
> > >
> >
> com.ibm.bi.dml.parser.antlr4.DMLParserWrapper.parse(DMLParserWrapper.java:143)
> > > at
> > >
> > >
> >
> com.ibm.bi.dml.api.MLContext.executeUsingSimplifiedCompilationChain(MLContext.java:1285)
> > > at
> > >
> com.ibm.bi.dml.api.MLContext.compileAndExecuteScript(MLContext.java:1204)
> > > at
> > >
> com.ibm.bi.dml.api.MLContext.compileAndExecuteScript(MLContext.java:1150)
> > > at com.ibm.bi.dml.api.MLContext.execute(MLContext.java:632) at
> > > com.ibm.bi.dml.api.MLContext.execute(MLContext.java:667) at
> > > com.ibm.bi.dml.api.MLContext.execute(MLContext.java:680) at
> > > $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:33) at
> > > $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:38) at
> > > $iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:40) at
> > > $iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:42) at
> > > $iwC$$iwC$$iwC$$iwC.<init>(<console>:44) at
> > > $iwC$$iwC$$iwC.<init>(<console>:46) at $iwC$$iwC.<init>(<console>:48)
> at
> > > $iwC.<init>(<console>:50) at <init>(<console>:52) at
> > .<init>(<console>:56)
> > > at .<clinit>(<console>) at .<init>(<console>:7) at .<clinit>(<console>)
> > at
> > > $print(<console>) at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> > > Method) at
> > >
> > >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> > > at
> > >
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > > at java.lang.reflect.Method.invoke(Method.java:497) at
> > >
> >
> org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)
> > > at
> > >
> >
> org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1340)
> > > at
> org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840)
> > > at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871) at
> > > org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819)
> > >
> > >
> > > Regards,
> > > Sourav
> > >
> >
>
>
>
Re: DML script parsing error in Spark
Posted by Niketan Pansare <np...@us.ibm.com>.
Hi Sourav,
I tried a similar example on our internal cluster and everything seems to
work.
The parser might have provided some error/warning messages before it throws
exception. Can you please provide the entire trace ?
For example:
scala> val outputs = ml.execute
("/home/system-ml-0.9.0-SNAPSHOT/algorithms/GLM2.dml", cmdLineParams)
15/12/08 12:00:56 WARN hops.OptimizerUtils: Auto-disable multi-threaded
text read for 'text' and 'csv' due to thread contention on JRE < 1.8
(java.version=1.7.0_91).
15/12/08 12:00:56 WARN parser.StatementBlock:
WARNING: ../algorithms/GLM.dml -- line 361, column 14 -- Initialization of
w depends on if-else execution
15/12/08 12:00:56 WARN parser.StatementBlock:
WARNING: ../algorithms/GLM.dml -- line 361, column 14 -- Initialization of
w depends on while execution
15/12/08 12:00:56 WARN parser.StatementBlock:
WARNING: ../algorithms/GLM.dml -- line 161, column 0 -- Variable
termination_code defined with different value type in if and else clause.
Thanks,
Niketan Pansare
IBM Almaden Research Center
E-mail: npansar At us.ibm.com
http://researcher.watson.ibm.com/researcher/view.php?person=us-npansar
From: Sourav Mazumder <so...@gmail.com>
To: dev@systemml.incubator.apache.org
Date: 12/08/2015 11:17 AM
Subject: Re: DML script parsing error in Spark
Hi Shirish,
Passing 'B' as " " (as below) through cmdLineArg did not help. I get the
same error.
val cmdLineParams = Map("X" -> " ", "Y" -> " ", "B" -> " ")
val outputs =
ml.execute("/home/system-ml-0.9.0-SNAPSHOT/algorithms/GLM2.dml",
cmdLineParams)
Any idea what else I need to pass ? I'm trying to run the DMLs only on
Spark (no Hadoop) and trying to come out with a use case.
Regards,
Sourav
On Tue, Dec 8, 2015 at 9:27 AM, Shirish Tatikonda <
shirish.tatikonda@gmail.com> wrote:
> Sourav,
>
> In cmdLineParams, you also need to pass B (output location) -- it does
not
> have a default value.
>
> In general, a DML script may have two types of "inputs":
>
> 1. $ parameters in the script (e.g., $X, $Y, $dfam, etc. in
> GLM.dml) which gets populated from command line arguments. The default
> arguments are specified via *ifdef() *in the script.
> 2. data that you pass-in via *read().*
>
> cmdLineParams in your example refers to (1).
>
> For read(), the exact method in which the data is passed depends on the
way
> you invoke the script. In a typical scenario, the data comes from HDFS
> files. Additionally, in the context of MLContext, it may also come from
> RDDs/DataFrames via *registerInput()*. In the context of JMLC, the data
can
> come in as an in-memory data structure (e.g., double[][] array).
>
> Hope that clarifies the difference.
>
> Shirish
>
>
> On Tue, Dec 8, 2015 at 8:25 AM, Sourav Mazumder <
> sourav.mazumder00@gmail.com
> > wrote:
>
> > Hi,
> >
> > I'm facing issue while parsing any DML script.
> >
> > What I'm trying something like -
> >
> > val ml = new MLContext(sc)ml.reset()
> > ml.registerInput("X", Xfc, 3569, 4)
> > ml.registerInput("Y", yDc, 1, 4)
> > ml.registerOutput("beta_out")
> >
> > val cmdLineParams = Map("X" -> " ", "Y" -> " ")
> >
> > val outputs =
> > ml.execute("/home/system-ml-0.9.0-SNAPSHOT/algorithms/GLM2.dml",
> > cmdLineParams).
> >
> > I'm getting following error -
> >
> > com.ibm.bi.dml.parser.ParseException: ERROR: Cannot translate the parse
> > tree into DMLProgram:null at
> >
> >
> com.ibm.bi.dml.parser.antlr4.DMLParserWrapper.doParse
(DMLParserWrapper.java:250)
> > at
> >
> >
> com.ibm.bi.dml.parser.antlr4.DMLParserWrapper.parse
(DMLParserWrapper.java:143)
> > at
> >
> >
> com.ibm.bi.dml.api.MLContext.executeUsingSimplifiedCompilationChain
(MLContext.java:1285)
> > at
> > com.ibm.bi.dml.api.MLContext.compileAndExecuteScript
(MLContext.java:1204)
> > at
> > com.ibm.bi.dml.api.MLContext.compileAndExecuteScript
(MLContext.java:1150)
> > at com.ibm.bi.dml.api.MLContext.execute(MLContext.java:632) at
> > com.ibm.bi.dml.api.MLContext.execute(MLContext.java:667) at
> > com.ibm.bi.dml.api.MLContext.execute(MLContext.java:680) at
> > $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:33) at
> > $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:38) at
> > $iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:40) at
> > $iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:42) at
> > $iwC$$iwC$$iwC$$iwC.<init>(<console>:44) at
> > $iwC$$iwC$$iwC.<init>(<console>:46) at $iwC$$iwC.<init>(<console>:48)
at
> > $iwC.<init>(<console>:50) at <init>(<console>:52) at
> .<init>(<console>:56)
> > at .<clinit>(<console>) at .<init>(<console>:7) at .<clinit>(<console>)
> at
> > $print(<console>) at sun.reflect.NativeMethodAccessorImpl.invoke0
(Native
> > Method) at
> >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke
(NativeMethodAccessorImpl.java:62)
> > at
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke
(DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:497) at
> >
> org.apache.spark.repl.SparkIMain$ReadEvalPrint.call
(SparkIMain.scala:1065)
> > at
> >
> org.apache.spark.repl.SparkIMain$Request.loadAndRun
(SparkIMain.scala:1340)
> > at org.apache.spark.repl.SparkIMain.loadAndRunReq$1
(SparkIMain.scala:840)
> > at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871) at
> > org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819)
> >
> >
> > Regards,
> > Sourav
> >
>
Re: DML script parsing error in Spark
Posted by Sourav Mazumder <so...@gmail.com>.
Hi Shirish,
Passing 'B' as " " (as below) through cmdLineArg did not help. I get the
same error.
val cmdLineParams = Map("X" -> " ", "Y" -> " ", "B" -> " ")
val outputs =
ml.execute("/home/system-ml-0.9.0-SNAPSHOT/algorithms/GLM2.dml",
cmdLineParams)
Any idea what else I need to pass ? I'm trying to run the DMLs only on
Spark (no Hadoop) and trying to come out with a use case.
Regards,
Sourav
On Tue, Dec 8, 2015 at 9:27 AM, Shirish Tatikonda <
shirish.tatikonda@gmail.com> wrote:
> Sourav,
>
> In cmdLineParams, you also need to pass B (output location) -- it does not
> have a default value.
>
> In general, a DML script may have two types of "inputs":
>
> 1. $ parameters in the script (e.g., $X, $Y, $dfam, etc. in
> GLM.dml) which gets populated from command line arguments. The default
> arguments are specified via *ifdef() *in the script.
> 2. data that you pass-in via *read().*
>
> cmdLineParams in your example refers to (1).
>
> For read(), the exact method in which the data is passed depends on the way
> you invoke the script. In a typical scenario, the data comes from HDFS
> files. Additionally, in the context of MLContext, it may also come from
> RDDs/DataFrames via *registerInput()*. In the context of JMLC, the data can
> come in as an in-memory data structure (e.g., double[][] array).
>
> Hope that clarifies the difference.
>
> Shirish
>
>
> On Tue, Dec 8, 2015 at 8:25 AM, Sourav Mazumder <
> sourav.mazumder00@gmail.com
> > wrote:
>
> > Hi,
> >
> > I'm facing issue while parsing any DML script.
> >
> > What I'm trying something like -
> >
> > val ml = new MLContext(sc)ml.reset()
> > ml.registerInput("X", Xfc, 3569, 4)
> > ml.registerInput("Y", yDc, 1, 4)
> > ml.registerOutput("beta_out")
> >
> > val cmdLineParams = Map("X" -> " ", "Y" -> " ")
> >
> > val outputs =
> > ml.execute("/home/system-ml-0.9.0-SNAPSHOT/algorithms/GLM2.dml",
> > cmdLineParams).
> >
> > I'm getting following error -
> >
> > com.ibm.bi.dml.parser.ParseException: ERROR: Cannot translate the parse
> > tree into DMLProgram:null at
> >
> >
> com.ibm.bi.dml.parser.antlr4.DMLParserWrapper.doParse(DMLParserWrapper.java:250)
> > at
> >
> >
> com.ibm.bi.dml.parser.antlr4.DMLParserWrapper.parse(DMLParserWrapper.java:143)
> > at
> >
> >
> com.ibm.bi.dml.api.MLContext.executeUsingSimplifiedCompilationChain(MLContext.java:1285)
> > at
> > com.ibm.bi.dml.api.MLContext.compileAndExecuteScript(MLContext.java:1204)
> > at
> > com.ibm.bi.dml.api.MLContext.compileAndExecuteScript(MLContext.java:1150)
> > at com.ibm.bi.dml.api.MLContext.execute(MLContext.java:632) at
> > com.ibm.bi.dml.api.MLContext.execute(MLContext.java:667) at
> > com.ibm.bi.dml.api.MLContext.execute(MLContext.java:680) at
> > $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:33) at
> > $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:38) at
> > $iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:40) at
> > $iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:42) at
> > $iwC$$iwC$$iwC$$iwC.<init>(<console>:44) at
> > $iwC$$iwC$$iwC.<init>(<console>:46) at $iwC$$iwC.<init>(<console>:48) at
> > $iwC.<init>(<console>:50) at <init>(<console>:52) at
> .<init>(<console>:56)
> > at .<clinit>(<console>) at .<init>(<console>:7) at .<clinit>(<console>)
> at
> > $print(<console>) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> > Method) at
> >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> > at
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:497) at
> >
> org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)
> > at
> >
> org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1340)
> > at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840)
> > at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871) at
> > org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819)
> >
> >
> > Regards,
> > Sourav
> >
>
Re: DML script parsing error in Spark
Posted by Shirish Tatikonda <sh...@gmail.com>.
Sourav,
In cmdLineParams, you also need to pass B (output location) -- it does not
have a default value.
In general, a DML script may have two types of "inputs":
1. $ parameters in the script (e.g., $X, $Y, $dfam, etc. in
GLM.dml) which gets populated from command line arguments. The default
arguments are specified via *ifdef() *in the script.
2. data that you pass-in via *read().*
cmdLineParams in your example refers to (1).
For read(), the exact method in which the data is passed depends on the way
you invoke the script. In a typical scenario, the data comes from HDFS
files. Additionally, in the context of MLContext, it may also come from
RDDs/DataFrames via *registerInput()*. In the context of JMLC, the data can
come in as an in-memory data structure (e.g., double[][] array).
Hope that clarifies the difference.
Shirish
On Tue, Dec 8, 2015 at 8:25 AM, Sourav Mazumder <sourav.mazumder00@gmail.com
> wrote:
> Hi,
>
> I'm facing issue while parsing any DML script.
>
> What I'm trying something like -
>
> val ml = new MLContext(sc)ml.reset()
> ml.registerInput("X", Xfc, 3569, 4)
> ml.registerInput("Y", yDc, 1, 4)
> ml.registerOutput("beta_out")
>
> val cmdLineParams = Map("X" -> " ", "Y" -> " ")
>
> val outputs =
> ml.execute("/home/system-ml-0.9.0-SNAPSHOT/algorithms/GLM2.dml",
> cmdLineParams).
>
> I'm getting following error -
>
> com.ibm.bi.dml.parser.ParseException: ERROR: Cannot translate the parse
> tree into DMLProgram:null at
>
> com.ibm.bi.dml.parser.antlr4.DMLParserWrapper.doParse(DMLParserWrapper.java:250)
> at
>
> com.ibm.bi.dml.parser.antlr4.DMLParserWrapper.parse(DMLParserWrapper.java:143)
> at
>
> com.ibm.bi.dml.api.MLContext.executeUsingSimplifiedCompilationChain(MLContext.java:1285)
> at
> com.ibm.bi.dml.api.MLContext.compileAndExecuteScript(MLContext.java:1204)
> at
> com.ibm.bi.dml.api.MLContext.compileAndExecuteScript(MLContext.java:1150)
> at com.ibm.bi.dml.api.MLContext.execute(MLContext.java:632) at
> com.ibm.bi.dml.api.MLContext.execute(MLContext.java:667) at
> com.ibm.bi.dml.api.MLContext.execute(MLContext.java:680) at
> $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:33) at
> $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:38) at
> $iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:40) at
> $iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:42) at
> $iwC$$iwC$$iwC$$iwC.<init>(<console>:44) at
> $iwC$$iwC$$iwC.<init>(<console>:46) at $iwC$$iwC.<init>(<console>:48) at
> $iwC.<init>(<console>:50) at <init>(<console>:52) at .<init>(<console>:56)
> at .<clinit>(<console>) at .<init>(<console>:7) at .<clinit>(<console>) at
> $print(<console>) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method) at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497) at
> org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)
> at
> org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1340)
> at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840)
> at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871) at
> org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819)
>
>
> Regards,
> Sourav
>