You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@kylin.apache.org by 听说海的那边有宝藏 <fe...@qq.com> on 2019/05/31 07:18:57 UTC
spark build cube ,has a problem
您好： when I use spark(hdp2.6.5) to build cube，a poblem has arisen:
what should I do?
19/05/31 14:27:49 WARN scheduler.TaskSetManager: Lost task 0.0 in stage 1.0 (TID 2, data1.test, executor 10): java.lang.ExceptionInInitializerError
at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.getNewWriter(HFileOutputFormat2.java:247)
at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:194)
at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:152)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply$mcV$sp(PairRDDFunctions.scala:1125)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1123)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1123)
at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1353)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1131)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1102)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:325)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: Could not create interface org.apache.hadoop.hbase.regionserver.MetricsRegionServerSourceFactory Is the hadoop compatibility jar on the classpath?
at org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:73)
at org.apache.hadoop.hbase.io.MetricsIO.<init>(MetricsIO.java:31)
at org.apache.hadoop.hbase.io.hfile.HFile.<clinit>(HFile.java:192)
... 15 more
Caused by: java.util.NoSuchElementException
at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:365)
at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
at org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:59)
... 17 more
回复：spark build cube ,has a problem
Posted by 听说海的那边有宝藏 <fe...@qq.com>.
I'm so sorry,it take effect.but Another issue arises
19/06/11 10:06:24 INFO spark.MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 172.10.3.161:56895 19/06/11 10:06:24 INFO spark.MapOutputTrackerMaster: Size of output statuses for shuffle 0 is 158 bytes 19/06/11 10:06:30 WARN scheduler.TaskSetManager: Lost task 0.0 in stage 1.0 (TID 2, data1.test, executor 5): java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/util/Counter at org.apache.hadoop.metrics2.lib.MutableHistogram.<init>(MutableHistogram.java:42) at org.apache.hadoop.metrics2.lib.MutableRangeHistogram.<init>(MutableRangeHistogram.java:41) at org.apache.hadoop.metrics2.lib.MutableTimeHistogram.<init>(MutableTimeHistogram.java:42) at org.apache.hadoop.metrics2.lib.MutableTimeHistogram.<init>(MutableTimeHistogram.java:38) at org.apache.hadoop.metrics2.lib.DynamicMetricsRegistry.newTimeHistogram(DynamicMetricsRegistry.java:262) at org.apache.hadoop.hbase.io.MetricsIOSourceImpl.<init>(MetricsIOSourceImpl.java:49) at org.apache.hadoop.hbase.io.MetricsIOSourceImpl.<init>(MetricsIOSourceImpl.java:36) at org.apache.hadoop.hbase.regionserver.MetricsRegionServerSourceFactoryImpl.createIO(MetricsRegionServerSourceFactoryImpl.java:89) at org.apache.hadoop.hbase.io.MetricsIO.<init>(MetricsIO.java:32) at org.apache.hadoop.hbase.io.hfile.HFile.<clinit>(HFile.java:192) at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.getNewWriter(HFileOutputFormat2.java:247) at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:194) at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:152) at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply$mcV$sp(PairRDDFunctions.scala:1125) at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1123) at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1123) at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1353) at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1131) at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1102) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) at org.apache.spark.scheduler.Task.run(Task.scala:99) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:325) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.util.Counter at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 25 more
------------------ 原始邮件 ------------------
发件人: "听说海的那边有宝藏"<fe...@qq.com>;
发送时间: 2019年6月11日(星期二) 上午10:03
收件人: "user"<us...@kylin.apache.org>;
主题: 回复：spark build cube ,has a problem
nichunen:
First of all, thank you for your reply.
I tried in this way,but it is no effective:
Hadoop cluster(Five machine)
cp /usr/hdp/2.6.5.0-292/hive2/lib/{hbase-hadoop2-compat-1.1.2.2.6.5.0-292-tests.jar,hbase-hadoop2-compat-1.1.2.2.6.5.0-292.jar} /usr/hdp/2.6.5.0-292/spark2/jars/
cp /usr/hdp/2.6.5.0-292/hbase/lib/{hbase-hadoop-compat-1.1.2.2.6.5.0-292.jar,hbase-hadoop-compat.jar} /usr/hdp/2.6.5.0-292/spark2/jars/
kylin machine
cp /usr/hdp/2.6.5.0-292/hive2/lib/{hbase-hadoop2-compat-1.1.2.2.6.5.0-292-tests.jar,hbase-hadoop2-compat-1.1.2.2.6.5.0-292.jar} /usr/local/apache-kylin-2.5.2-bin-hbase1x/spark/jars/
cp /usr/hdp/2.6.5.0-292/hbase/lib/{hbase-hadoop-compat-1.1.2.2.6.5.0-292.jar,hbase-hadoop-compat.jar} /usr/local/apache-kylin-2.5.2-bin-hbase1x/spark/jars/
Is there any other way?
------------------ 原始邮件 ------------------
发件人: "nichunen"<ni...@apache.org>;
发送时间: 2019年6月7日(星期五) 晚上11:01
收件人: "user@kylin.apache.org"<us...@kylin.apache.org>;
主题: Re:spark build cube ,has a problem
Hi,
I think it’s a known issue with some versions of hadoop. By copying hbase-hadoop2-compat-*.jar and hbase-hadoop-compat-*.jar in your env to path $SPARK_HOME/jars/ may fix this.
Best regards,
Ni Chunen / George
On 05/31/2019 15:18，听说海的那边有宝藏<fe...@qq.com> wrote：
您好： when I use spark(hdp2.6.5) to build cube，a poblem has arisen:
what should I do?
19/05/31 14:27:49 WARN scheduler.TaskSetManager: Lost task 0.0 in stage 1.0 (TID 2, data1.test, executor 10): java.lang.ExceptionInInitializerError
at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.getNewWriter(HFileOutputFormat2.java:247)
at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:194)
at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:152)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply$mcV$sp(PairRDDFunctions.scala:1125)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1123)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1123)
at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1353)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1131)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1102)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:325)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: Could not create interface org.apache.hadoop.hbase.regionserver.MetricsRegionServerSourceFactory Is the hadoop compatibility jar on the classpath?
at org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:73)
at org.apache.hadoop.hbase.io.MetricsIO.<init>(MetricsIO.java:31)
at org.apache.hadoop.hbase.io.hfile.HFile.<clinit>(HFile.java:192)
... 15 more
Caused by: java.util.NoSuchElementException
at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:365)
at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
at org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:59)
... 17 more
Re:spark build cube ,has a problem
Posted by nichunen <ni...@apache.org>.
Hi,
I think it’s a known issue with some versions of hadoop. By copying hbase-hadoop2-compat-*.jar and hbase-hadoop-compat-*.jar in your env to path $SPARK_HOME/jars/ may fix this.
Best regards,
Ni Chunen / George
On 05/31/2019 15:18，听说海的那边有宝藏<fe...@qq.com> wrote：
您好：
when I use spark(hdp2.6.5) to build cube，a poblem has arisen:
what should I do?
19/05/31 14:27:49 WARN scheduler.TaskSetManager: Lost task 0.0 in stage 1.0 (TID 2, data1.test, executor 10): java.lang.ExceptionInInitializerError
at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.getNewWriter(HFileOutputFormat2.java:247)
at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:194)
at org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:152)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply$mcV$sp(PairRDDFunctions.scala:1125)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1123)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1123)
at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1353)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1131)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1102)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:325)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: Could not create interface org.apache.hadoop.hbase.regionserver.MetricsRegionServerSourceFactory Is the hadoop compatibility jar on the classpath?
at org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:73)
at org.apache.hadoop.hbase.io.MetricsIO.<init>(MetricsIO.java:31)
at org.apache.hadoop.hbase.io.hfile.HFile.<clinit>(HFile.java:192)
... 15 more
Caused by: java.util.NoSuchElementException
at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:365)
at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
at org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:59)
... 17 more