peoplehum/template-Labelling-Topics-with-wikipedia - pio train error (scala 2.11.6, pio 0.10.0-incubating, spark 2.1.0, postgreSQL 9.5) - Exception in thread "main" java.lang.NoSuchMethodError: org.apache.spark.sql.SQLContext.createDataFrame(Lorg/apache/spark/rdd/RDD;Lscala/reflect/api/TypeTags$TypeTag;)Lorg/apache/spark/sql/DataFrame;

[Barthe van Doorn <ba...@gmail.com>]

I am getting error when I train template-Labelling-Topics-with-wikipedia
engine.

I already changed scalaVersion in build.sbt to 2.11.6 and build was
successful.h

pio train -- --master spark://xx.xx.xx.xx:7077
exits with error: Exception in thread "main" java.lang.NoSuchMethodError:
org.apache.spark.sql.SQLContext.createDataFrame(Lorg/apache/spark/rdd/RDD;Lscala/reflect/api/TypeTags$TypeTag;)Lorg/apache/spark/sql/DataFrame;

Here is output:
[INFO] [Engine] Extracting datasource params...
[INFO] [WorkflowUtils$] No 'name' is found. Default empty String will be
used.
[INFO] [Engine] Datasource params: (,DataSourceParam(test_wiki,0))
[INFO] [Engine] Extracting preparator params...
[INFO] [Engine] Preparator params: (,Empty)
[INFO] [Engine] Extracting serving params...
[INFO] [Engine] Serving params: (,Empty)
[INFO] [log] Logging initialized @3791ms
[INFO] [Server] jetty-9.2.z-SNAPSHOT
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@3520963
<https://github.com/o.s.j.s.ServletContextHandler/template-Labelling-Topics-with-wikipedia/commit/3520963d>
{/jobs,null,AVAILABLE}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@388d14e
<https://github.com/o.s.j.s.ServletContextHandler/template-Labelling-Topics-with-wikipedia/commit/388d14e>
{/jobs/json,null,AVAILABLE}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@1cd4356
<https://github.com/o.s.j.s.ServletContextHandler/template-Labelling-Topics-with-wikipedia/commit/1cd43562>
{/jobs/job,null,AVAILABLE}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@5993929
<https://github.com/o.s.j.s.ServletContextHandler/template-Labelling-Topics-with-wikipedia/commit/59939293>
{/jobs/job/json,null,AVAILABLE}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@68b366e
<https://github.com/o.s.j.s.ServletContextHandler/template-Labelling-Topics-with-wikipedia/commit/68b366e2>
{/stages,null,AVAILABLE}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@2d74c81
<https://github.com/o.s.j.s.ServletContextHandler/template-Labelling-Topics-with-wikipedia/commit/2d74c81b>
{/stages/json,null,AVAILABLE}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@10b687f
<https://github.com/o.s.j.s.ServletContextHandler/template-Labelling-Topics-with-wikipedia/commit/10b687f2>
{/stages/stage,null,AVAILABLE}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@55f4887
<https://github.com/o.s.j.s.ServletContextHandler/template-Labelling-Topics-with-wikipedia/commit/55f4887d>
{/stages/stage/json,null,AVAILABLE}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@2683705
<https://github.com/o.s.j.s.ServletContextHandler/template-Labelling-Topics-with-wikipedia/commit/26837057>
{/stages/pool,null,AVAILABLE}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@2dbc408
<https://github.com/o.s.j.s.ServletContextHandler/template-Labelling-Topics-with-wikipedia/commit/2dbc408c>
{/stages/pool/json,null,AVAILABLE}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@991cbde
<https://github.com/o.s.j.s.ServletContextHandler/template-Labelling-Topics-with-wikipedia/commit/991cbde>
{/storage,null,AVAILABLE}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@78d71df
<https://github.com/o.s.j.s.ServletContextHandler/template-Labelling-Topics-with-wikipedia/commit/78d71df1>
{/storage/json,null,AVAILABLE}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@456bcb7
<https://github.com/o.s.j.s.ServletContextHandler/template-Labelling-Topics-with-wikipedia/commit/456bcb74>
{/storage/rdd,null,AVAILABLE}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@a9c5b75
<https://github.com/o.s.j.s.ServletContextHandler/template-Labelling-Topics-with-wikipedia/commit/a9c5b75>
{/storage/rdd/json,null,AVAILABLE}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@dd4aec3
<https://github.com/o.s.j.s.ServletContextHandler/template-Labelling-Topics-with-wikipedia/commit/dd4aec3>
{/environment,null,AVAILABLE}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@4dac121
<https://github.com/o.s.j.s.ServletContextHandler/template-Labelling-Topics-with-wikipedia/commit/4dac121d>
{/environment/json,null,AVAILABLE}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@568750b
<https://github.com/o.s.j.s.ServletContextHandler/template-Labelling-Topics-with-wikipedia/commit/568750b7>
{/executors,null,AVAILABLE}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@3e28fee
<https://github.com/o.s.j.s.ServletContextHandler/template-Labelling-Topics-with-wikipedia/commit/3e28fee1>
{/executors/json,null,AVAILABLE}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@16a2ed5
<https://github.com/o.s.j.s.ServletContextHandler/template-Labelling-Topics-with-wikipedia/commit/16a2ed51>
{/executors/threadDump,null,AVAILABLE}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@3b33296
<https://github.com/o.s.j.s.ServletContextHandler/template-Labelling-Topics-with-wikipedia/commit/3b332962>
{/executors/threadDump/json,null,AVAILABLE}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@57ddd45
<https://github.com/o.s.j.s.ServletContextHandler/template-Labelling-Topics-with-wikipedia/commit/57ddd45b>
{/static,null,AVAILABLE}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@2e7bf7b
<https://github.com/o.s.j.s.ServletContextHandler/template-Labelling-Topics-with-wikipedia/commit/2e7bf7b7>
{/,null,AVAILABLE}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@2fb25f4
<https://github.com/o.s.j.s.ServletContextHandler/template-Labelling-Topics-with-wikipedia/commit/2fb25f4c>
{/api,null,AVAILABLE}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@de81be1
<https://github.com/o.s.j.s.ServletContextHandler/template-Labelling-Topics-with-wikipedia/commit/de81be1>
{/jobs/job/kill,null,AVAILABLE}
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@342e690
<https://github.com/o.s.j.s.ServletContextHandler/template-Labelling-Topics-with-wikipedia/commit/342e690b>
{/stages/stage/kill,null,AVAILABLE}
[INFO] [ServerConnector] Started ServerConnector@7df28f1
<https://github.com/ServerConnector/template-Labelling-Topics-with-wikipedia/commit/7df28f1>
{HTTP/1.1}{192.168.5.140:4040}
[INFO] [Server] Started @3956ms
[INFO] [ContextHandler] Started o.s.j.s.ServletContextHandler@411cd15
<https://github.com/o.s.j.s.ServletContextHandler/template-Labelling-Topics-with-wikipedia/commit/411cd156>
{/metrics/json,null,AVAILABLE}
[INFO] [Engine$] EngineWorkflow.train
[INFO] [Engine$] DataSource: com.coviam.wikiClassifier.engine.DataSource@
6d6ac39
<https://github.com/com.coviam.wikiClassifier.engine.DataSource/template-Labelling-Topics-with-wikipedia/commit/6d6ac396>
[INFO] [Engine$] Preparator:
com.coviam.wikiClassifier.engine.DataPreparator@519c6fc
<https://github.com/com.coviam.wikiClassifier.engine.DataPreparator/template-Labelling-Topics-with-wikipedia/commit/519c6fcc>
[INFO] [Engine$] AlgorithmList: List(
com.coviam.wikiClassifier.engine.Algorithm@22da2fe
<https://github.com/com.coviam.wikiClassifier.engine.Algorithm/template-Labelling-Topics-with-wikipedia/commit/22da2fe6>
)
[INFO] [Engine$] Data sanity check is on.
[INFO] [Engine$] com.coviam.wikiClassifier.engine.TrainingData supports
data sanity check. Performing check.
(total observation,2)

[INFO] [ServerConnector] Stopped ServerConnector@7df28f1
<https://github.com/ServerConnector/template-Labelling-Topics-with-wikipedia/commit/7df28f1>
{HTTP/1.1}{192.168.5.140:4040}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@342e690
<https://github.com/o.s.j.s.ServletContextHandler/template-Labelling-Topics-with-wikipedia/commit/342e690b>
{/stages/stage/kill,null,UNAVAILABLE}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@de81be1
<https://github.com/o.s.j.s.ServletContextHandler/template-Labelling-Topics-with-wikipedia/commit/de81be1>
{/jobs/job/kill,null,UNAVAILABLE}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@2fb25f4
<https://github.com/o.s.j.s.ServletContextHandler/template-Labelling-Topics-with-wikipedia/commit/2fb25f4c>
{/api,null,UNAVAILABLE}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@2e7bf7b
<https://github.com/o.s.j.s.ServletContextHandler/template-Labelling-Topics-with-wikipedia/commit/2e7bf7b7>
{/,null,UNAVAILABLE}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@57ddd45
<https://github.com/o.s.j.s.ServletContextHandler/template-Labelling-Topics-with-wikipedia/commit/57ddd45b>
{/static,null,UNAVAILABLE}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@3b33296
<https://github.com/o.s.j.s.ServletContextHandler/template-Labelling-Topics-with-wikipedia/commit/3b332962>
{/executors/threadDump/json,null,UNAVAILABLE}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@16a2ed5
<https://github.com/o.s.j.s.ServletContextHandler/template-Labelling-Topics-with-wikipedia/commit/16a2ed51>
{/executors/threadDump,null,UNAVAILABLE}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@3e28fee
<https://github.com/o.s.j.s.ServletContextHandler/template-Labelling-Topics-with-wikipedia/commit/3e28fee1>
{/executors/json,null,UNAVAILABLE}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@568750b
<https://github.com/o.s.j.s.ServletContextHandler/template-Labelling-Topics-with-wikipedia/commit/568750b7>
{/executors,null,UNAVAILABLE}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@4dac121
<https://github.com/o.s.j.s.ServletContextHandler/template-Labelling-Topics-with-wikipedia/commit/4dac121d>
{/environment/json,null,UNAVAILABLE}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@dd4aec3
<https://github.com/o.s.j.s.ServletContextHandler/template-Labelling-Topics-with-wikipedia/commit/dd4aec3>
{/environment,null,UNAVAILABLE}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@a9c5b75
<https://github.com/o.s.j.s.ServletContextHandler/template-Labelling-Topics-with-wikipedia/commit/a9c5b75>
{/storage/rdd/json,null,UNAVAILABLE}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@456bcb7
<https://github.com/o.s.j.s.ServletContextHandler/template-Labelling-Topics-with-wikipedia/commit/456bcb74>
{/storage/rdd,null,UNAVAILABLE}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@78d71df
<https://github.com/o.s.j.s.ServletContextHandler/template-Labelling-Topics-with-wikipedia/commit/78d71df1>
{/storage/json,null,UNAVAILABLE}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@991cbde
<https://github.com/o.s.j.s.ServletContextHandler/template-Labelling-Topics-with-wikipedia/commit/991cbde>
{/storage,null,UNAVAILABLE}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@2dbc408
<https://github.com/o.s.j.s.ServletContextHandler/template-Labelling-Topics-with-wikipedia/commit/2dbc408c>
{/stages/pool/json,null,UNAVAILABLE}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@2683705
<https://github.com/o.s.j.s.ServletContextHandler/template-Labelling-Topics-with-wikipedia/commit/26837057>
{/stages/pool,null,UNAVAILABLE}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@55f4887
<https://github.com/o.s.j.s.ServletContextHandler/template-Labelling-Topics-with-wikipedia/commit/55f4887d>
{/stages/stage/json,null,UNAVAILABLE}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@10b687f
<https://github.com/o.s.j.s.ServletContextHandler/template-Labelling-Topics-with-wikipedia/commit/10b687f2>
{/stages/stage,null,UNAVAILABLE}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@2d74c81
<https://github.com/o.s.j.s.ServletContextHandler/template-Labelling-Topics-with-wikipedia/commit/2d74c81b>
{/stages/json,null,UNAVAILABLE}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@68b366e
<https://github.com/o.s.j.s.ServletContextHandler/template-Labelling-Topics-with-wikipedia/commit/68b366e2>
{/stages,null,UNAVAILABLE}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@5993929
<https://github.com/o.s.j.s.ServletContextHandler/template-Labelling-Topics-with-wikipedia/commit/59939293>
{/jobs/job/json,null,UNAVAILABLE}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@1cd4356
<https://github.com/o.s.j.s.ServletContextHandler/template-Labelling-Topics-with-wikipedia/commit/1cd43562>
{/jobs/job,null,UNAVAILABLE}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@388d14e
<https://github.com/o.s.j.s.ServletContextHandler/template-Labelling-Topics-with-wikipedia/commit/388d14e>
{/jobs/json,null,UNAVAILABLE}
[INFO] [ContextHandler] Stopped o.s.j.s.ServletContextHandler@3520963
<https://github.com/o.s.j.s.ServletContextHandler/template-Labelling-Topics-with-wikipedia/commit/3520963d>
{/jobs,null,UNAVAILABLE}
Exception in thread "main" java.lang.NoSuchMethodError:
org.apache.spark.sql.SQLContext.createDataFrame(Lorg/apache/spark/rdd/RDD;Lscala/reflect/api/TypeTags$TypeTag;)Lorg/apache/spark/sql/DataFrame;
at
com.coviam.wikiClassifier.engine.DataPreparator.prepare(DataPreparator.scala:19)
at
com.coviam.wikiClassifier.engine.DataPreparator.prepare(DataPreparator.scala:13)
at
org.apache.predictionio.controller.PPreparator.prepareBase(PPreparator.scala:37)
at org.apache.predictionio.controller.Engine$.train(Engine.scala:671)
at org.apache.predictionio.controller.Engine.train(Engine.scala:177)
at
org.apache.predictionio.workflow.CoreWorkflow$.runTrain(CoreWorkflow.scala:67)
at
org.apache.predictionio.workflow.CreateWorkflow$.main(CreateWorkflow.scala:250)
at
org.apache.predictionio.workflow.CreateWorkflow.main(CreateWorkflow.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:738)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)