You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@griffin.apache.org by phani priyank <ph...@gmail.com> on 2020/04/04 11:18:00 UTC
Griffin Error
Dear Team,
I need your help. I'm trying this job given in the documentation
https://griffin.apache.org/docs/quickstart.html . I have the following
versions installed in my system.
Hadoop:3.2.0
scala: 2.13.1
Spark: 3.0.0
Java: 1.8.0
Hive: 3.1.2
I'm submitting the spark job to run data quality using griffin. I'm getting
below exception can you pls help me in this regard.
2020-04-04 08:37:35,456 INFO cluster.YarnClientSchedulerBackend:
SchedulerBackend is ready for scheduling beginning after waiting
maxRegisteredResourcesWaitingTime: 30000000000(ns)
Exception in thread "main" java.lang.NoSuchMethodError:
scala.Predef$.refArrayOps([Ljava/lang/Object;)Lscala/
collection/mutable/ArrayOps;
at org.apache.griffin.measure.datasource.connector.batch.
HiveBatchDataConnector.<init>(HiveBatchDataConnector.scala:47)
at org.apache.griffin.measure.datasource.connector.
DataConnectorFactory$$anonfun$getDataConnector$1.apply(
DataConnectorFactory.scala:63)
at org.apache.griffin.measure.datasource.connector.
DataConnectorFactory$$anonfun$getDataConnector$1.apply(
DataConnectorFactory.scala:62)
at scala.util.Try$.apply(Try.scala:213)
at org.apache.griffin.measure.datasource.connector.
DataConnectorFactory$.getDataConnector(DataConnectorFactory.scala:61)
at org.apache.griffin.measure.datasource.DataSourceFactory$$
anonfun$1.apply(DataSourceFactory.scala:58)
at org.apache.griffin.measure.datasource.DataSourceFactory$$
anonfun$1.apply(DataSourceFactory.scala:57)
at scala.collection.TraversableLike.$anonfun$
flatMap$1(TraversableLike.scala:245)
at scala.collection.immutable.List.foreach(List.scala:392)
at scala.collection.TraversableLike.flatMap(
TraversableLike.scala:245)
at scala.collection.TraversableLike.flatMap$(
TraversableLike.scala:242)
at scala.collection.immutable.List.flatMap(List.scala:355)
at org.apache.griffin.measure.datasource.DataSourceFactory$.
org$apache$griffin$measure$datasource$DataSourceFactory$$getDataSource(
DataSourceFactory.scala:57)
at org.apache.griffin.measure.datasource.DataSourceFactory$$
anonfun$getDataSources$1.apply(DataSourceFactory.scala:40)
at org.apache.griffin.measure.datasource.DataSourceFactory$$
anonfun$getDataSources$1.apply(DataSourceFactory.scala:38)
at scala.collection.TraversableLike.$anonfun$
flatMap$1(TraversableLike.scala:245)
at scala.collection.immutable.List.foreach(List.scala:392)
at scala.collection.TraversableLike.flatMap(
TraversableLike.scala:245)
at scala.collection.TraversableLike.flatMap$(
TraversableLike.scala:242)
at scala.collection.immutable.List.flatMap(List.scala:355)
at org.apache.griffin.measure.datasource.DataSourceFactory$.
getDataSources(DataSourceFactory.scala:38)
at org.apache.griffin.measure.launch.batch.BatchDQApp$$
anonfun$run$1.apply$mcZ$sp(BatchDQApp.scala:75)
at org.apache.griffin.measure.launch.batch.BatchDQApp$$
anonfun$run$1.apply(BatchDQApp.scala:67)
at org.apache.griffin.measure.launch.batch.BatchDQApp$$
anonfun$run$1.apply(BatchDQApp.scala:67)
at scala.util.Try$.apply(Try.scala:213)
at org.apache.griffin.measure.launch.batch.BatchDQApp.run(
BatchDQApp.scala:67)
at org.apache.griffin.measure.Application$.main(Application.
scala:88)
at org.apache.griffin.measure.Application.main(Application.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(
NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(
DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(
SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org
<http://org.apache.spark.deploy.sparksubmit.org/>$apache$spark$
deploy$SparkSubmit$$runMain(SparkSubmit.scala:928)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(
SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
at org.apache.spark.deploy.SparkSubmit.doSubmit(
SparkSubmit.scala:90)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(
SparkSubmit.scala:1007)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1016)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Re: Griffin Error
Posted by Aleksandr Borgatin <ab...@griddynamics.com>.
Hi!
The error message “java.lang.NoSuchMethodError” indicates that you are using an incompatible version of Scala.
Griffin is developed on version 2.11.8. try to use it
https://github.com/apache/griffin/blob/master/measure/pom.xml#L37 <https://github.com/apache/griffin/blob/master/measure/pom.xml#L37>
> On 6 Apr 2020, at 13:38, Chitral Verma <ch...@gmail.com> wrote:
>
> Hi Are you able to submit the regular jobs?
>
> Griffin is based on Spark 2.2x relying on Hadoop 2.x and expects the class
> to be provided.
>
> This maybe the cause of the problem.
>
> Regards
>
> On Mon, 6 Apr, 2020, 15:58 phani priyank, <ph...@gmail.com> wrote:
>
>> Dear Team,
>>
>> I need your help. I'm trying this job given in the documentation
>> https://griffin.apache.org/docs/quickstart.html . I have the following
>> versions installed in my system.
>>
>> Hadoop:3.2.0
>> scala: 2.13.1
>> Spark: 3.0.0
>> Java: 1.8.0
>> Hive: 3.1.2
>>
>> I'm submitting the spark job to run data quality using griffin. I'm getting
>> below exception can you pls help me in this regard.
>>
>> 2020-04-04 08:37:35,456 INFO cluster.YarnClientSchedulerBackend:
>> SchedulerBackend is ready for scheduling beginning after waiting
>> maxRegisteredResourcesWaitingTime: 30000000000(ns)
>> Exception in thread "main" java.lang.NoSuchMethodError:
>> scala.Predef$.refArrayOps([Ljava/lang/Object;)Lscala/
>> collection/mutable/ArrayOps;
>> at org.apache.griffin.measure.datasource.connector.batch.
>> HiveBatchDataConnector.<init>(HiveBatchDataConnector.scala:47)
>> at org.apache.griffin.measure.datasource.connector.
>> DataConnectorFactory$$anonfun$getDataConnector$1.apply(
>> DataConnectorFactory.scala:63)
>> at org.apache.griffin.measure.datasource.connector.
>> DataConnectorFactory$$anonfun$getDataConnector$1.apply(
>> DataConnectorFactory.scala:62)
>> at scala.util.Try$.apply(Try.scala:213)
>> at org.apache.griffin.measure.datasource.connector.
>> DataConnectorFactory$.getDataConnector(DataConnectorFactory.scala:61)
>> at org.apache.griffin.measure.datasource.DataSourceFactory$$
>> anonfun$1.apply(DataSourceFactory.scala:58)
>> at org.apache.griffin.measure.datasource.DataSourceFactory$$
>> anonfun$1.apply(DataSourceFactory.scala:57)
>> at scala.collection.TraversableLike.$anonfun$
>> flatMap$1(TraversableLike.scala:245)
>> at scala.collection.immutable.List.foreach(List.scala:392)
>> at scala.collection.TraversableLike.flatMap(
>> TraversableLike.scala:245)
>> at scala.collection.TraversableLike.flatMap$(
>> TraversableLike.scala:242)
>> at scala.collection.immutable.List.flatMap(List.scala:355)
>> at org.apache.griffin.measure.datasource.DataSourceFactory$.
>> org$apache$griffin$measure$datasource$DataSourceFactory$$getDataSource(
>> DataSourceFactory.scala:57)
>> at org.apache.griffin.measure.datasource.DataSourceFactory$$
>> anonfun$getDataSources$1.apply(DataSourceFactory.scala:40)
>> at org.apache.griffin.measure.datasource.DataSourceFactory$$
>> anonfun$getDataSources$1.apply(DataSourceFactory.scala:38)
>> at scala.collection.TraversableLike.$anonfun$
>> flatMap$1(TraversableLike.scala:245)
>> at scala.collection.immutable.List.foreach(List.scala:392)
>> at scala.collection.TraversableLike.flatMap(
>> TraversableLike.scala:245)
>> at scala.collection.TraversableLike.flatMap$(
>> TraversableLike.scala:242)
>> at scala.collection.immutable.List.flatMap(List.scala:355)
>> at org.apache.griffin.measure.datasource.DataSourceFactory$.
>> getDataSources(DataSourceFactory.scala:38)
>> at org.apache.griffin.measure.launch.batch.BatchDQApp$$
>> anonfun$run$1.apply$mcZ$sp(BatchDQApp.scala:75)
>> at org.apache.griffin.measure.launch.batch.BatchDQApp$$
>> anonfun$run$1.apply(BatchDQApp.scala:67)
>> at org.apache.griffin.measure.launch.batch.BatchDQApp$$
>> anonfun$run$1.apply(BatchDQApp.scala:67)
>> at scala.util.Try$.apply(Try.scala:213)
>> at org.apache.griffin.measure.launch.batch.BatchDQApp.run(
>> BatchDQApp.scala:67)
>> at org.apache.griffin.measure.Application$.main(Application.
>> scala:88)
>> at org.apache.griffin.measure.Application.main(Application.scala)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at sun.reflect.NativeMethodAccessorImpl.invoke(
>> NativeMethodAccessorImpl.java:62)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(
>> DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:498)
>> at org.apache.spark.deploy.JavaMainApplication.start(
>> SparkApplication.scala:52)
>> at org.apache.spark.deploy.SparkSubmit.org
>> <http://org.apache.spark.deploy.sparksubmit.org/>$apache$spark$
>> deploy$SparkSubmit$$runMain(SparkSubmit.scala:928)
>> at org.apache.spark.deploy.SparkSubmit.doRunMain$1(
>> SparkSubmit.scala:180)
>> at
>> org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
>> at org.apache.spark.deploy.SparkSubmit.doSubmit(
>> SparkSubmit.scala:90)
>> at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(
>> SparkSubmit.scala:1007)
>> at
>> org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1016)
>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>>
Re: Griffin Error
Posted by Chitral Verma <ch...@gmail.com>.
Hi Are you able to submit the regular jobs?
Griffin is based on Spark 2.2x relying on Hadoop 2.x and expects the class
to be provided.
This maybe the cause of the problem.
Regards
On Mon, 6 Apr, 2020, 15:58 phani priyank, <ph...@gmail.com> wrote:
> Dear Team,
>
> I need your help. I'm trying this job given in the documentation
> https://griffin.apache.org/docs/quickstart.html . I have the following
> versions installed in my system.
>
> Hadoop:3.2.0
> scala: 2.13.1
> Spark: 3.0.0
> Java: 1.8.0
> Hive: 3.1.2
>
> I'm submitting the spark job to run data quality using griffin. I'm getting
> below exception can you pls help me in this regard.
>
> 2020-04-04 08:37:35,456 INFO cluster.YarnClientSchedulerBackend:
> SchedulerBackend is ready for scheduling beginning after waiting
> maxRegisteredResourcesWaitingTime: 30000000000(ns)
> Exception in thread "main" java.lang.NoSuchMethodError:
> scala.Predef$.refArrayOps([Ljava/lang/Object;)Lscala/
> collection/mutable/ArrayOps;
> at org.apache.griffin.measure.datasource.connector.batch.
> HiveBatchDataConnector.<init>(HiveBatchDataConnector.scala:47)
> at org.apache.griffin.measure.datasource.connector.
> DataConnectorFactory$$anonfun$getDataConnector$1.apply(
> DataConnectorFactory.scala:63)
> at org.apache.griffin.measure.datasource.connector.
> DataConnectorFactory$$anonfun$getDataConnector$1.apply(
> DataConnectorFactory.scala:62)
> at scala.util.Try$.apply(Try.scala:213)
> at org.apache.griffin.measure.datasource.connector.
> DataConnectorFactory$.getDataConnector(DataConnectorFactory.scala:61)
> at org.apache.griffin.measure.datasource.DataSourceFactory$$
> anonfun$1.apply(DataSourceFactory.scala:58)
> at org.apache.griffin.measure.datasource.DataSourceFactory$$
> anonfun$1.apply(DataSourceFactory.scala:57)
> at scala.collection.TraversableLike.$anonfun$
> flatMap$1(TraversableLike.scala:245)
> at scala.collection.immutable.List.foreach(List.scala:392)
> at scala.collection.TraversableLike.flatMap(
> TraversableLike.scala:245)
> at scala.collection.TraversableLike.flatMap$(
> TraversableLike.scala:242)
> at scala.collection.immutable.List.flatMap(List.scala:355)
> at org.apache.griffin.measure.datasource.DataSourceFactory$.
> org$apache$griffin$measure$datasource$DataSourceFactory$$getDataSource(
> DataSourceFactory.scala:57)
> at org.apache.griffin.measure.datasource.DataSourceFactory$$
> anonfun$getDataSources$1.apply(DataSourceFactory.scala:40)
> at org.apache.griffin.measure.datasource.DataSourceFactory$$
> anonfun$getDataSources$1.apply(DataSourceFactory.scala:38)
> at scala.collection.TraversableLike.$anonfun$
> flatMap$1(TraversableLike.scala:245)
> at scala.collection.immutable.List.foreach(List.scala:392)
> at scala.collection.TraversableLike.flatMap(
> TraversableLike.scala:245)
> at scala.collection.TraversableLike.flatMap$(
> TraversableLike.scala:242)
> at scala.collection.immutable.List.flatMap(List.scala:355)
> at org.apache.griffin.measure.datasource.DataSourceFactory$.
> getDataSources(DataSourceFactory.scala:38)
> at org.apache.griffin.measure.launch.batch.BatchDQApp$$
> anonfun$run$1.apply$mcZ$sp(BatchDQApp.scala:75)
> at org.apache.griffin.measure.launch.batch.BatchDQApp$$
> anonfun$run$1.apply(BatchDQApp.scala:67)
> at org.apache.griffin.measure.launch.batch.BatchDQApp$$
> anonfun$run$1.apply(BatchDQApp.scala:67)
> at scala.util.Try$.apply(Try.scala:213)
> at org.apache.griffin.measure.launch.batch.BatchDQApp.run(
> BatchDQApp.scala:67)
> at org.apache.griffin.measure.Application$.main(Application.
> scala:88)
> at org.apache.griffin.measure.Application.main(Application.scala)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(
> NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.spark.deploy.JavaMainApplication.start(
> SparkApplication.scala:52)
> at org.apache.spark.deploy.SparkSubmit.org
> <http://org.apache.spark.deploy.sparksubmit.org/>$apache$spark$
> deploy$SparkSubmit$$runMain(SparkSubmit.scala:928)
> at org.apache.spark.deploy.SparkSubmit.doRunMain$1(
> SparkSubmit.scala:180)
> at
> org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
> at org.apache.spark.deploy.SparkSubmit.doSubmit(
> SparkSubmit.scala:90)
> at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(
> SparkSubmit.scala:1007)
> at
> org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1016)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>