You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@kylin.apache.org by washyou112 <wa...@163.com> on 2021/08/25 07:38:25 UTC
Kylin 4.0 error in the first step of building cube
Kylin 4.0 error in the first step of building cubekylin.log：2021-08-24 10:45:00,232 ERROR [Thread-1] application.JobMonitor : Job failed the 1 times.
java.lang.NoSuchMethodError: com.fasterxml.jackson.databind.JsonMappingException.<init>(Ljava/io/Closeable;Ljava/lang/String;)V
at com.fasterxml.jackson.module.scala.JacksonModule$class.setupModule(JacksonModule.scala:61)
at com.fasterxml.jackson.module.scala.DefaultScalaModule.setupModule(DefaultScalaModule.scala:17)
at com.fasterxml.jackson.databind.ObjectMapper.registerModule(ObjectMapper.java:718)
at org.apache.spark.util.JsonProtocol$.<init>(JsonProtocol.scala:60)
at org.apache.spark.util.JsonProtocol$.<clinit>(JsonProtocol.scala)
at org.apache.spark.scheduler.EventLoggingListener$.initEventLog(EventLoggingListener.scala:353)
at org.apache.spark.scheduler.EventLoggingListener.start(EventLoggingListener.scala:135)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:533)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2549)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:944)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:935)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:935)
at org.apache.kylin.engine.spark.application.SparkApplication.execute(SparkApplication.java:283)
at org.apache.kylin.engine.spark.application.SparkApplication.execute(SparkApplication.java:89)
at org.apache.spark.application.JobWorker$$anon$2.run(JobWorker.scala:55)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2021-08-24 10:45:00,246 ERROR [Thread-1] application.JobWorkSpace : Job failed eventually. Reason: Error occurred when generate retry configuration.
java.util.NoSuchElementException: spark.executor.memory
at org.apache.spark.SparkConf$$anonfun$get$1.apply(SparkConf.scala:245)
at org.apache.spark.SparkConf$$anonfun$get$1.apply(SparkConf.scala:245)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.SparkConf.get(SparkConf.scala:245)
at org.apache.spark.autoheal.ExceptionTerminator$.incMemory(ExceptionTerminator.scala:70)
at org.apache.spark.autoheal.ExceptionTerminator$.resolveException(ExceptionTerminator.scala:45)
at org.apache.spark.application.JobMonitor.handleResourceLack(JobMonitor.scala:53)
at org.apache.spark.application.JobMonitor$$anon$1.onReceive(JobMonitor.scala:33)
at org.apache.spark.scheduler.KylinJobEventLoop$$anonfun$onReceive$1.apply(KylinJobEventLoop.scala:42)
at org.apache.spark.scheduler.KylinJobEventLoop$$anonfun$onReceive$1.apply(KylinJobEventLoop.scala:42)
at scala.collection.Iterator$class.foreach(Iterator.scala:893)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at org.apache.spark.scheduler.KylinJobEventLoop.onReceive(KylinJobEventLoop.scala:42)
at org.apache.spark.scheduler.KylinJobEventLoop.onReceive(KylinJobEventLoop.scala:29)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
Is this an jar package conflict problem? How to solve it?
| |
washyou112
|
|
washyou112@163.com
|
签名由网易邮箱大师定制
Re: Kylin 4.0 error in the first step of building cube
Posted by washyou112 <wa...@163.com>.
I would like to ask according to my configuration, the choice of the version of kylin. Do you have any suggestions?
| |
washyou112
|
|
washyou112@163.com
|
签名由网易邮箱大师定制
On 8/25/2021 16:52，Xiaoxiang Yu<xx...@apache.org> wrote：
On each hadoop platform/env we tested, we do NOT use the spark provided by env(HDP, CDH or AWS EMR), but download specific version of Apache Spark. For CDH 6.2, we use Apache Spark 3.1.1 (and I think Spark 2.4.6 should works as well).
--
Best wishes to you !
From ：Xiaoxiang Yu
At 2021-08-25 16:34:15, "washyou112" <wa...@163.com> wrote:
Maybe spark 2.4.0 no match
This is my env information
| |
washyou112
|
|
washyou112@163.com
|
签名由网易邮箱大师定制
On 8/25/2021 15:57，Xiaoxiang Yu<xx...@apache.org> wrote：
Hi,
This looks caused by class load conflict with jar provided by env. Here is a list of Hadoop Platform we verified and test : https://cwiki.apache.org/confluence/display/KYLIN/Support+Hadoop+Version+Matrix+of+Kylin+4.0.0 .
Please let us know your env information(version), or you can choose to use our recommended hadoop platform.
--
Best wishes to you !
From ：Xiaoxiang Yu
在 2021-08-25 15:38:25，"washyou112" <wa...@163.com> 写道：
Kylin 4.0 error in the first step of building cube
kylin.log：
2021-08-24 10:45:00,232 ERROR [Thread-1] application.JobMonitor : Job failed the 1 times.
java.lang.NoSuchMethodError: com.fasterxml.jackson.databind.JsonMappingException.<init>(Ljava/io/Closeable;Ljava/lang/String;)V
at com.fasterxml.jackson.module.scala.JacksonModule$class.setupModule(JacksonModule.scala:61)
at com.fasterxml.jackson.module.scala.DefaultScalaModule.setupModule(DefaultScalaModule.scala:17)
at com.fasterxml.jackson.databind.ObjectMapper.registerModule(ObjectMapper.java:718)
at org.apache.spark.util.JsonProtocol$.<init>(JsonProtocol.scala:60)
at org.apache.spark.util.JsonProtocol$.<clinit>(JsonProtocol.scala)
at org.apache.spark.scheduler.EventLoggingListener$.initEventLog(EventLoggingListener.scala:353)
at org.apache.spark.scheduler.EventLoggingListener.start(EventLoggingListener.scala:135)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:533)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2549)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:944)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:935)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:935)
at org.apache.kylin.engine.spark.application.SparkApplication.execute(SparkApplication.java:283)
at org.apache.kylin.engine.spark.application.SparkApplication.execute(SparkApplication.java:89)
at org.apache.spark.application.JobWorker$$anon$2.run(JobWorker.scala:55)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2021-08-24 10:45:00,246 ERROR [Thread-1] application.JobWorkSpace : Job failed eventually. Reason: Error occurred when generate retry configuration.
java.util.NoSuchElementException: spark.executor.memory
at org.apache.spark.SparkConf$$anonfun$get$1.apply(SparkConf.scala:245)
at org.apache.spark.SparkConf$$anonfun$get$1.apply(SparkConf.scala:245)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.SparkConf.get(SparkConf.scala:245)
at org.apache.spark.autoheal.ExceptionTerminator$.incMemory(ExceptionTerminator.scala:70)
at org.apache.spark.autoheal.ExceptionTerminator$.resolveException(ExceptionTerminator.scala:45)
at org.apache.spark.application.JobMonitor.handleResourceLack(JobMonitor.scala:53)
at org.apache.spark.application.JobMonitor$$anon$1.onReceive(JobMonitor.scala:33)
at org.apache.spark.scheduler.KylinJobEventLoop$$anonfun$onReceive$1.apply(KylinJobEventLoop.scala:42)
at org.apache.spark.scheduler.KylinJobEventLoop$$anonfun$onReceive$1.apply(KylinJobEventLoop.scala:42)
at scala.collection.Iterator$class.foreach(Iterator.scala:893)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at org.apache.spark.scheduler.KylinJobEventLoop.onReceive(KylinJobEventLoop.scala:42)
at org.apache.spark.scheduler.KylinJobEventLoop.onReceive(KylinJobEventLoop.scala:29)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
Is this an jar package conflict problem? How to solve it?
| |
washyou112
|
|
washyou112@163.com
|
签名由网易邮箱大师定制
Re: Kylin 4.0 error in the first step of building cube
Posted by washyou112 <wa...@163.com>.
That should be the problem ,spark 2.4.0 may be too low for kylin 4.0, i should replace a later version of spark.
Thank you very much for your help.
| |
washyou112
|
|
washyou112@163.com
|
签名由网易邮箱大师定制
On 8/25/2021 16:52，Xiaoxiang Yu<xx...@apache.org> wrote：
On each hadoop platform/env we tested, we do NOT use the spark provided by env(HDP, CDH or AWS EMR), but download specific version of Apache Spark. For CDH 6.2, we use Apache Spark 3.1.1 (and I think Spark 2.4.6 should works as well).
--
Best wishes to you !
From ：Xiaoxiang Yu
At 2021-08-25 16:34:15, "washyou112" <wa...@163.com> wrote:
Maybe spark 2.4.0 no match
This is my env information
| |
washyou112
|
|
washyou112@163.com
|
签名由网易邮箱大师定制
On 8/25/2021 15:57，Xiaoxiang Yu<xx...@apache.org> wrote：
Hi,
This looks caused by class load conflict with jar provided by env. Here is a list of Hadoop Platform we verified and test : https://cwiki.apache.org/confluence/display/KYLIN/Support+Hadoop+Version+Matrix+of+Kylin+4.0.0 .
Please let us know your env information(version), or you can choose to use our recommended hadoop platform.
--
Best wishes to you !
From ：Xiaoxiang Yu
在 2021-08-25 15:38:25，"washyou112" <wa...@163.com> 写道：
Kylin 4.0 error in the first step of building cube
kylin.log：
2021-08-24 10:45:00,232 ERROR [Thread-1] application.JobMonitor : Job failed the 1 times.
java.lang.NoSuchMethodError: com.fasterxml.jackson.databind.JsonMappingException.<init>(Ljava/io/Closeable;Ljava/lang/String;)V
at com.fasterxml.jackson.module.scala.JacksonModule$class.setupModule(JacksonModule.scala:61)
at com.fasterxml.jackson.module.scala.DefaultScalaModule.setupModule(DefaultScalaModule.scala:17)
at com.fasterxml.jackson.databind.ObjectMapper.registerModule(ObjectMapper.java:718)
at org.apache.spark.util.JsonProtocol$.<init>(JsonProtocol.scala:60)
at org.apache.spark.util.JsonProtocol$.<clinit>(JsonProtocol.scala)
at org.apache.spark.scheduler.EventLoggingListener$.initEventLog(EventLoggingListener.scala:353)
at org.apache.spark.scheduler.EventLoggingListener.start(EventLoggingListener.scala:135)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:533)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2549)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:944)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:935)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:935)
at org.apache.kylin.engine.spark.application.SparkApplication.execute(SparkApplication.java:283)
at org.apache.kylin.engine.spark.application.SparkApplication.execute(SparkApplication.java:89)
at org.apache.spark.application.JobWorker$$anon$2.run(JobWorker.scala:55)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2021-08-24 10:45:00,246 ERROR [Thread-1] application.JobWorkSpace : Job failed eventually. Reason: Error occurred when generate retry configuration.
java.util.NoSuchElementException: spark.executor.memory
at org.apache.spark.SparkConf$$anonfun$get$1.apply(SparkConf.scala:245)
at org.apache.spark.SparkConf$$anonfun$get$1.apply(SparkConf.scala:245)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.SparkConf.get(SparkConf.scala:245)
at org.apache.spark.autoheal.ExceptionTerminator$.incMemory(ExceptionTerminator.scala:70)
at org.apache.spark.autoheal.ExceptionTerminator$.resolveException(ExceptionTerminator.scala:45)
at org.apache.spark.application.JobMonitor.handleResourceLack(JobMonitor.scala:53)
at org.apache.spark.application.JobMonitor$$anon$1.onReceive(JobMonitor.scala:33)
at org.apache.spark.scheduler.KylinJobEventLoop$$anonfun$onReceive$1.apply(KylinJobEventLoop.scala:42)
at org.apache.spark.scheduler.KylinJobEventLoop$$anonfun$onReceive$1.apply(KylinJobEventLoop.scala:42)
at scala.collection.Iterator$class.foreach(Iterator.scala:893)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at org.apache.spark.scheduler.KylinJobEventLoop.onReceive(KylinJobEventLoop.scala:42)
at org.apache.spark.scheduler.KylinJobEventLoop.onReceive(KylinJobEventLoop.scala:29)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
Is this an jar package conflict problem? How to solve it?
| |
washyou112
|
|
washyou112@163.com
|
签名由网易邮箱大师定制
Re: Kylin 4.0 error in the first step of building cube
Posted by Xiaoxiang Yu <xx...@apache.org>.
On each hadoop platform/env we tested, we do NOT use the spark provided by env(HDP, CDH or AWS EMR), but download specific version of Apache Spark. For CDH 6.2, we use Apache Spark 3.1.1 (and I think Spark 2.4.6 should works as well).
--
Best wishes to you !
From ：Xiaoxiang Yu
At 2021-08-25 16:34:15, "washyou112" <wa...@163.com> wrote:
Maybe spark 2.4.0 no match
This is my env information
| |
washyou112
|
|
washyou112@163.com
|
签名由网易邮箱大师定制
On 8/25/2021 15:57，Xiaoxiang Yu<xx...@apache.org> wrote：
Hi,
This looks caused by class load conflict with jar provided by env. Here is a list of Hadoop Platform we verified and test : https://cwiki.apache.org/confluence/display/KYLIN/Support+Hadoop+Version+Matrix+of+Kylin+4.0.0 .
Please let us know your env information(version), or you can choose to use our recommended hadoop platform.
--
Best wishes to you !
From ：Xiaoxiang Yu
在 2021-08-25 15:38:25，"washyou112" <wa...@163.com> 写道：
Kylin 4.0 error in the first step of building cube
kylin.log：
2021-08-24 10:45:00,232 ERROR [Thread-1] application.JobMonitor : Job failed the 1 times.
java.lang.NoSuchMethodError: com.fasterxml.jackson.databind.JsonMappingException.<init>(Ljava/io/Closeable;Ljava/lang/String;)V
at com.fasterxml.jackson.module.scala.JacksonModule$class.setupModule(JacksonModule.scala:61)
at com.fasterxml.jackson.module.scala.DefaultScalaModule.setupModule(DefaultScalaModule.scala:17)
at com.fasterxml.jackson.databind.ObjectMapper.registerModule(ObjectMapper.java:718)
at org.apache.spark.util.JsonProtocol$.<init>(JsonProtocol.scala:60)
at org.apache.spark.util.JsonProtocol$.<clinit>(JsonProtocol.scala)
at org.apache.spark.scheduler.EventLoggingListener$.initEventLog(EventLoggingListener.scala:353)
at org.apache.spark.scheduler.EventLoggingListener.start(EventLoggingListener.scala:135)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:533)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2549)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:944)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:935)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:935)
at org.apache.kylin.engine.spark.application.SparkApplication.execute(SparkApplication.java:283)
at org.apache.kylin.engine.spark.application.SparkApplication.execute(SparkApplication.java:89)
at org.apache.spark.application.JobWorker$$anon$2.run(JobWorker.scala:55)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2021-08-24 10:45:00,246 ERROR [Thread-1] application.JobWorkSpace : Job failed eventually. Reason: Error occurred when generate retry configuration.
java.util.NoSuchElementException: spark.executor.memory
at org.apache.spark.SparkConf$$anonfun$get$1.apply(SparkConf.scala:245)
at org.apache.spark.SparkConf$$anonfun$get$1.apply(SparkConf.scala:245)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.SparkConf.get(SparkConf.scala:245)
at org.apache.spark.autoheal.ExceptionTerminator$.incMemory(ExceptionTerminator.scala:70)
at org.apache.spark.autoheal.ExceptionTerminator$.resolveException(ExceptionTerminator.scala:45)
at org.apache.spark.application.JobMonitor.handleResourceLack(JobMonitor.scala:53)
at org.apache.spark.application.JobMonitor$$anon$1.onReceive(JobMonitor.scala:33)
at org.apache.spark.scheduler.KylinJobEventLoop$$anonfun$onReceive$1.apply(KylinJobEventLoop.scala:42)
at org.apache.spark.scheduler.KylinJobEventLoop$$anonfun$onReceive$1.apply(KylinJobEventLoop.scala:42)
at scala.collection.Iterator$class.foreach(Iterator.scala:893)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at org.apache.spark.scheduler.KylinJobEventLoop.onReceive(KylinJobEventLoop.scala:42)
at org.apache.spark.scheduler.KylinJobEventLoop.onReceive(KylinJobEventLoop.scala:29)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
Is this an jar package conflict problem? How to solve it?
| |
washyou112
|
|
washyou112@163.com
|
签名由网易邮箱大师定制
Re: Kylin 4.0 error in the first step of building cube
Posted by washyou112 <wa...@163.com>.
Maybe spark 2.4.0 no match
This is my env information
| |
washyou112
|
|
washyou112@163.com
|
签名由网易邮箱大师定制
On 8/25/2021 15:57，Xiaoxiang Yu<xx...@apache.org> wrote：
Hi,
This looks caused by class load conflict with jar provided by env. Here is a list of Hadoop Platform we verified and test : https://cwiki.apache.org/confluence/display/KYLIN/Support+Hadoop+Version+Matrix+of+Kylin+4.0.0 .
Please let us know your env information(version), or you can choose to use our recommended hadoop platform.
--
Best wishes to you !
From ：Xiaoxiang Yu
在 2021-08-25 15:38:25，"washyou112" <wa...@163.com> 写道：
Kylin 4.0 error in the first step of building cube
kylin.log：
2021-08-24 10:45:00,232 ERROR [Thread-1] application.JobMonitor : Job failed the 1 times.
java.lang.NoSuchMethodError: com.fasterxml.jackson.databind.JsonMappingException.<init>(Ljava/io/Closeable;Ljava/lang/String;)V
at com.fasterxml.jackson.module.scala.JacksonModule$class.setupModule(JacksonModule.scala:61)
at com.fasterxml.jackson.module.scala.DefaultScalaModule.setupModule(DefaultScalaModule.scala:17)
at com.fasterxml.jackson.databind.ObjectMapper.registerModule(ObjectMapper.java:718)
at org.apache.spark.util.JsonProtocol$.<init>(JsonProtocol.scala:60)
at org.apache.spark.util.JsonProtocol$.<clinit>(JsonProtocol.scala)
at org.apache.spark.scheduler.EventLoggingListener$.initEventLog(EventLoggingListener.scala:353)
at org.apache.spark.scheduler.EventLoggingListener.start(EventLoggingListener.scala:135)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:533)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2549)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:944)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:935)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:935)
at org.apache.kylin.engine.spark.application.SparkApplication.execute(SparkApplication.java:283)
at org.apache.kylin.engine.spark.application.SparkApplication.execute(SparkApplication.java:89)
at org.apache.spark.application.JobWorker$$anon$2.run(JobWorker.scala:55)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2021-08-24 10:45:00,246 ERROR [Thread-1] application.JobWorkSpace : Job failed eventually. Reason: Error occurred when generate retry configuration.
java.util.NoSuchElementException: spark.executor.memory
at org.apache.spark.SparkConf$$anonfun$get$1.apply(SparkConf.scala:245)
at org.apache.spark.SparkConf$$anonfun$get$1.apply(SparkConf.scala:245)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.SparkConf.get(SparkConf.scala:245)
at org.apache.spark.autoheal.ExceptionTerminator$.incMemory(ExceptionTerminator.scala:70)
at org.apache.spark.autoheal.ExceptionTerminator$.resolveException(ExceptionTerminator.scala:45)
at org.apache.spark.application.JobMonitor.handleResourceLack(JobMonitor.scala:53)
at org.apache.spark.application.JobMonitor$$anon$1.onReceive(JobMonitor.scala:33)
at org.apache.spark.scheduler.KylinJobEventLoop$$anonfun$onReceive$1.apply(KylinJobEventLoop.scala:42)
at org.apache.spark.scheduler.KylinJobEventLoop$$anonfun$onReceive$1.apply(KylinJobEventLoop.scala:42)
at scala.collection.Iterator$class.foreach(Iterator.scala:893)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at org.apache.spark.scheduler.KylinJobEventLoop.onReceive(KylinJobEventLoop.scala:42)
at org.apache.spark.scheduler.KylinJobEventLoop.onReceive(KylinJobEventLoop.scala:29)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
Is this an jar package conflict problem? How to solve it?
| |
washyou112
|
|
washyou112@163.com
|
签名由网易邮箱大师定制
Re:Kylin 4.0 error in the first step of building cube
Posted by Xiaoxiang Yu <xx...@apache.org>.
Hi,
This looks caused by class load conflict with jar provided by env. Here is a list of Hadoop Platform we verified and test : https://cwiki.apache.org/confluence/display/KYLIN/Support+Hadoop+Version+Matrix+of+Kylin+4.0.0 .
Please let us know your env information(version), or you can choose to use our recommended hadoop platform.
--
Best wishes to you !
From ：Xiaoxiang Yu
在 2021-08-25 15:38:25，"washyou112" <wa...@163.com> 写道：
Kylin 4.0 error in the first step of building cube
kylin.log：
2021-08-24 10:45:00,232 ERROR [Thread-1] application.JobMonitor : Job failed the 1 times.
java.lang.NoSuchMethodError: com.fasterxml.jackson.databind.JsonMappingException.<init>(Ljava/io/Closeable;Ljava/lang/String;)V
at com.fasterxml.jackson.module.scala.JacksonModule$class.setupModule(JacksonModule.scala:61)
at com.fasterxml.jackson.module.scala.DefaultScalaModule.setupModule(DefaultScalaModule.scala:17)
at com.fasterxml.jackson.databind.ObjectMapper.registerModule(ObjectMapper.java:718)
at org.apache.spark.util.JsonProtocol$.<init>(JsonProtocol.scala:60)
at org.apache.spark.util.JsonProtocol$.<clinit>(JsonProtocol.scala)
at org.apache.spark.scheduler.EventLoggingListener$.initEventLog(EventLoggingListener.scala:353)
at org.apache.spark.scheduler.EventLoggingListener.start(EventLoggingListener.scala:135)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:533)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2549)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:944)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:935)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:935)
at org.apache.kylin.engine.spark.application.SparkApplication.execute(SparkApplication.java:283)
at org.apache.kylin.engine.spark.application.SparkApplication.execute(SparkApplication.java:89)
at org.apache.spark.application.JobWorker$$anon$2.run(JobWorker.scala:55)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2021-08-24 10:45:00,246 ERROR [Thread-1] application.JobWorkSpace : Job failed eventually. Reason: Error occurred when generate retry configuration.
java.util.NoSuchElementException: spark.executor.memory
at org.apache.spark.SparkConf$$anonfun$get$1.apply(SparkConf.scala:245)
at org.apache.spark.SparkConf$$anonfun$get$1.apply(SparkConf.scala:245)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.SparkConf.get(SparkConf.scala:245)
at org.apache.spark.autoheal.ExceptionTerminator$.incMemory(ExceptionTerminator.scala:70)
at org.apache.spark.autoheal.ExceptionTerminator$.resolveException(ExceptionTerminator.scala:45)
at org.apache.spark.application.JobMonitor.handleResourceLack(JobMonitor.scala:53)
at org.apache.spark.application.JobMonitor$$anon$1.onReceive(JobMonitor.scala:33)
at org.apache.spark.scheduler.KylinJobEventLoop$$anonfun$onReceive$1.apply(KylinJobEventLoop.scala:42)
at org.apache.spark.scheduler.KylinJobEventLoop$$anonfun$onReceive$1.apply(KylinJobEventLoop.scala:42)
at scala.collection.Iterator$class.foreach(Iterator.scala:893)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at org.apache.spark.scheduler.KylinJobEventLoop.onReceive(KylinJobEventLoop.scala:42)
at org.apache.spark.scheduler.KylinJobEventLoop.onReceive(KylinJobEventLoop.scala:29)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
Is this an jar package conflict problem? How to solve it?
| |
washyou112
|
|
washyou112@163.com
|
签名由网易邮箱大师定制