You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kylin.apache.org by Jason Hale <ja...@koddi.com> on 2016/07/27 22:20:49 UTC
HCatalogFormat error
I have set up a Kylin instance on the master node of my Hadoop cluster. I
was trying on a separate client node, but had some permission issues, so to
simplify the test case, I've just installed it on master. Now I am getting
the below error.
To correct this, I've tried the solution to distribute the jars in
https://issues.apache.org/jira/browse/KYLIN-1082 using '
kylin.job.mr.lib.dir'.
I'm not sure how to append to 'kylin.hive.dependency' as I cannot find
information on that (perhaps I'm not looking in the right place). But the
lib dir setting did not help and it still is unable to find that class.
On #2 Step Name: Extract Fact Table Distinct Columns
Kylin executes with the following parameters:
-conf /opt/kylin/bin/../conf/kylin_job_conf.xml -cubename Testing -output
/kylin/kylin_metadata/kylin-40827168-d18f-4b17-a613-3febe773ce2c/Testing/fact_distinct_columns
-segmentname 19700101000000_20160731000000 -statisticsenabled true
-statisticsoutput
/kylin/kylin_metadata/kylin-40827168-d18f-4b17-a613-3febe773ce2c/Testing/statistics
-statisticssamplingpercent 100 -jobname
Kylin_Fact_Distinct_Columns_Testing_Step
Error Msg:
2016-07-27 21:54:03,387 ERROR [pool-6-thread-2]
execution.AbstractExecutable:116 : error running Executable
java.lang.NoClassDefFoundError:
org/apache/hive/hcatalog/mapreduce/HCatInputFormat
at
org.apache.kylin.source.hive.HiveMRInput$HiveTableInputFormat.configureJob(HiveMRInput.java:81)
at
org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.setupMapper(FactDistinctColumnsJob.java:111)
at
org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.run(FactDistinctColumnsJob.java:91)
at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:91)
at
org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:121)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:114)
at
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:50)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:114)
at
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:124)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException:
org.apache.hive.hcatalog.mapreduce.HCatInputFormat
at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1720)
at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1571)
... 12 more
2016-07-27 21:54:03,399 INFO [pool-6-thread-2]
manager.ExecutableManager:274 : job
id:40827168-d18f-4b17-a613-3febe773ce2c-01 from RUNNING to ERROR
2016-07-27 21:54:03,399 ERROR [pool-6-thread-2]
execution.AbstractExecutable:116 : error running Executable
org.apache.kylin.job.exception.ExecuteException:
java.lang.NoClassDefFoundError:
org/apache/hive/hcatalog/mapreduce/HCatInputFormat
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:124)
at
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:50)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:114)
at
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:124)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NoClassDefFoundError:
org/apache/hive/hcatalog/mapreduce/HCatInputFormat
at
org.apache.kylin.source.hive.HiveMRInput$HiveTableInputFormat.configureJob(HiveMRInput.java:81)
at
org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.setupMapper(FactDistinctColumnsJob.java:111)
at
org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.run(FactDistinctColumnsJob.java:91)
at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:91)
at
org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:121)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:114)
... 6 more
Caused by: java.lang.ClassNotFoundException:
org.apache.hive.hcatalog.mapreduce.HCatInputFormat
at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1720)
at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1571)
... 12 more
2016-07-27 21:54:03,410 INFO [pool-6-thread-2]
manager.ExecutableManager:274 : job id:40827168-d18f-4b17-a613-3febe773ce2c
from RUNNING to ERROR
2016-07-27 21:54:03,410 ERROR [pool-6-thread-2]
threadpool.DefaultScheduler:128 : ExecuteException
job:40827168-d18f-4b17-a613-3febe773ce2c
org.apache.kylin.job.exception.ExecuteException:
org.apache.kylin.job.exception.ExecuteException:
java.lang.NoClassDefFoundError:
org/apache/hive/hcatalog/mapreduce/HCatInputFormat
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:124)
at
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:124)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.kylin.job.exception.ExecuteException:
java.lang.NoClassDefFoundError:
org/apache/hive/hcatalog/mapreduce/HCatInputFormat
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:124)
at
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:50)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:114)
... 4 more
Caused by: java.lang.NoClassDefFoundError:
org/apache/hive/hcatalog/mapreduce/HCatInputFormat
at
org.apache.kylin.source.hive.HiveMRInput$HiveTableInputFormat.configureJob(HiveMRInput.java:81)
at
org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.setupMapper(FactDistinctColumnsJob.java:111)
at
org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.run(FactDistinctColumnsJob.java:91)
at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:91)
at
org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:121)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:114)
... 6 more
Caused by: java.lang.ClassNotFoundException:
org.apache.hive.hcatalog.mapreduce.HCatInputFormat
at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1720)
at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1571)
... 12 more
Re: HCatalogFormat error
Posted by Li Yang <li...@apache.org>.
> My understanding was that the 'kylin.job.mr.lib.dir' setting would
distribute the jars through the hadoop tmpjars property for Kylin to use.
This is correct. However not the most ideal path.
'kylin.sh' should detect hive/hcat installation and distribute the jars
automatically. User should not have to set 'kylin.job.mr.lib.dir' as of
v1.5.3.
Thanks for the env info. Have verified Kylin on EMR once and it worked at
that time. Will double check again.
Yang
On Wed, Aug 3, 2016 at 10:51 PM, Jason Hale <ja...@koddi.com> wrote:
> Thanks for the response Li Yang. This was an EMR cluster which I don't have
> running now. I switched to setting up a HDP sandbox to get it up and
> running for testing purposes. If I get a chance to spin up the EMR cluster
> again, I will look into this further.
>
> To answer your question, though, it was the latest version of Kylin, 1.5.3,
> and I believe hadoop 2.4 on EMR, so this could very well have been the
> issue.
> My understanding was that the 'kylin.job.mr.lib.dir' setting would
> distribute the jars through the hadoop tmpjars property for Kylin to use.
> Is this not correct, or not available on this version?
>
> On Tue, Aug 2, 2016 at 11:52 PM, Li Yang <li...@apache.org> wrote:
>
> > What's your Kylin version?
> >
> > If it is 1.5.x, your problem is detecting the right hive jar on the Kylin
> > node.
> >
> > Checkout bin/find-hive-dependency.sh. See if it returns right hive path.
> >
> > On Thu, Jul 28, 2016 at 6:20 AM, Jason Hale <ja...@koddi.com> wrote:
> >
> > > I have set up a Kylin instance on the master node of my Hadoop
> cluster. I
> > > was trying on a separate client node, but had some permission issues,
> so
> > to
> > > simplify the test case, I've just installed it on master. Now I am
> > getting
> > > the below error.
> > >
> > > To correct this, I've tried the solution to distribute the jars in
> > > https://issues.apache.org/jira/browse/KYLIN-1082 using '
> > > kylin.job.mr.lib.dir'.
> > > I'm not sure how to append to 'kylin.hive.dependency' as I cannot find
> > > information on that (perhaps I'm not looking in the right place). But
> the
> > > lib dir setting did not help and it still is unable to find that class.
> > >
> > >
> > > On #2 Step Name: Extract Fact Table Distinct Columns
> > >
> > > Kylin executes with the following parameters:
> > >
> > > -conf /opt/kylin/bin/../conf/kylin_job_conf.xml -cubename Testing
> -output
> > >
> > >
> > /kylin/kylin_metadata/kylin-40827168-d18f-4b17-a613-
> 3febe773ce2c/Testing/fact_distinct_columns
> > > -segmentname 19700101000000_20160731000000 -statisticsenabled true
> > > -statisticsoutput
> > >
> > >
> > /kylin/kylin_metadata/kylin-40827168-d18f-4b17-a613-
> 3febe773ce2c/Testing/statistics
> > > -statisticssamplingpercent 100 -jobname
> > > Kylin_Fact_Distinct_Columns_Testing_Step
> > >
> > > Error Msg:
> > >
> > > 2016-07-27 21:54:03,387 ERROR [pool-6-thread-2]
> > > execution.AbstractExecutable:116 : error running Executable
> > > java.lang.NoClassDefFoundError:
> > > org/apache/hive/hcatalog/mapreduce/HCatInputFormat
> > > at
> > >
> > >
> > org.apache.kylin.source.hive.HiveMRInput$HiveTableInputFormat.
> configureJob(HiveMRInput.java:81)
> > > at
> > >
> > >
> > org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.setupMapper(
> FactDistinctColumnsJob.java:111)
> > > at
> > >
> > >
> > org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.
> run(FactDistinctColumnsJob.java:91)
> > > at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:91)
> > > at
> > >
> > >
> > org.apache.kylin.engine.mr.common.MapReduceExecutable.
> doWork(MapReduceExecutable.java:121)
> > > at
> > >
> > >
> > org.apache.kylin.job.execution.AbstractExecutable.
> execute(AbstractExecutable.java:114)
> > > at
> > >
> > >
> > org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(
> DefaultChainedExecutable.java:50)
> > > at
> > >
> > >
> > org.apache.kylin.job.execution.AbstractExecutable.
> execute(AbstractExecutable.java:114)
> > > at
> > >
> > >
> > org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(
> DefaultScheduler.java:124)
> > > at
> > >
> > >
> > java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1145)
> > > at
> > >
> > >
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:615)
> > > at java.lang.Thread.run(Thread.java:745)
> > > Caused by: java.lang.ClassNotFoundException:
> > > org.apache.hive.hcatalog.mapreduce.HCatInputFormat
> > > at
> > >
> > >
> > org.apache.catalina.loader.WebappClassLoader.loadClass(
> WebappClassLoader.java:1720)
> > > at
> > >
> > >
> > org.apache.catalina.loader.WebappClassLoader.loadClass(
> WebappClassLoader.java:1571)
> > > ... 12 more
> > > 2016-07-27 21:54:03,399 INFO [pool-6-thread-2]
> > > manager.ExecutableManager:274 : job
> > > id:40827168-d18f-4b17-a613-3febe773ce2c-01 from RUNNING to ERROR
> > > 2016-07-27 21:54:03,399 ERROR [pool-6-thread-2]
> > > execution.AbstractExecutable:116 : error running Executable
> > > org.apache.kylin.job.exception.ExecuteException:
> > > java.lang.NoClassDefFoundError:
> > > org/apache/hive/hcatalog/mapreduce/HCatInputFormat
> > > at
> > >
> > >
> > org.apache.kylin.job.execution.AbstractExecutable.
> execute(AbstractExecutable.java:124)
> > > at
> > >
> > >
> > org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(
> DefaultChainedExecutable.java:50)
> > > at
> > >
> > >
> > org.apache.kylin.job.execution.AbstractExecutable.
> execute(AbstractExecutable.java:114)
> > > at
> > >
> > >
> > org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(
> DefaultScheduler.java:124)
> > > at
> > >
> > >
> > java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1145)
> > > at
> > >
> > >
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:615)
> > > at java.lang.Thread.run(Thread.java:745)
> > > Caused by: java.lang.NoClassDefFoundError:
> > > org/apache/hive/hcatalog/mapreduce/HCatInputFormat
> > > at
> > >
> > >
> > org.apache.kylin.source.hive.HiveMRInput$HiveTableInputFormat.
> configureJob(HiveMRInput.java:81)
> > > at
> > >
> > >
> > org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.setupMapper(
> FactDistinctColumnsJob.java:111)
> > > at
> > >
> > >
> > org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.
> run(FactDistinctColumnsJob.java:91)
> > > at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:91)
> > > at
> > >
> > >
> > org.apache.kylin.engine.mr.common.MapReduceExecutable.
> doWork(MapReduceExecutable.java:121)
> > > at
> > >
> > >
> > org.apache.kylin.job.execution.AbstractExecutable.
> execute(AbstractExecutable.java:114)
> > > ... 6 more
> > > Caused by: java.lang.ClassNotFoundException:
> > > org.apache.hive.hcatalog.mapreduce.HCatInputFormat
> > > at
> > >
> > >
> > org.apache.catalina.loader.WebappClassLoader.loadClass(
> WebappClassLoader.java:1720)
> > > at
> > >
> > >
> > org.apache.catalina.loader.WebappClassLoader.loadClass(
> WebappClassLoader.java:1571)
> > > ... 12 more
> > > 2016-07-27 21:54:03,410 INFO [pool-6-thread-2]
> > > manager.ExecutableManager:274 : job
> > id:40827168-d18f-4b17-a613-3febe773ce2c
> > > from RUNNING to ERROR
> > > 2016-07-27 21:54:03,410 ERROR [pool-6-thread-2]
> > > threadpool.DefaultScheduler:128 : ExecuteException
> > > job:40827168-d18f-4b17-a613-3febe773ce2c
> > > org.apache.kylin.job.exception.ExecuteException:
> > > org.apache.kylin.job.exception.ExecuteException:
> > > java.lang.NoClassDefFoundError:
> > > org/apache/hive/hcatalog/mapreduce/HCatInputFormat
> > > at
> > >
> > >
> > org.apache.kylin.job.execution.AbstractExecutable.
> execute(AbstractExecutable.java:124)
> > > at
> > >
> > >
> > org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(
> DefaultScheduler.java:124)
> > > at
> > >
> > >
> > java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1145)
> > > at
> > >
> > >
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:615)
> > > at java.lang.Thread.run(Thread.java:745)
> > > Caused by: org.apache.kylin.job.exception.ExecuteException:
> > > java.lang.NoClassDefFoundError:
> > > org/apache/hive/hcatalog/mapreduce/HCatInputFormat
> > > at
> > >
> > >
> > org.apache.kylin.job.execution.AbstractExecutable.
> execute(AbstractExecutable.java:124)
> > > at
> > >
> > >
> > org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(
> DefaultChainedExecutable.java:50)
> > > at
> > >
> > >
> > org.apache.kylin.job.execution.AbstractExecutable.
> execute(AbstractExecutable.java:114)
> > > ... 4 more
> > > Caused by: java.lang.NoClassDefFoundError:
> > > org/apache/hive/hcatalog/mapreduce/HCatInputFormat
> > > at
> > >
> > >
> > org.apache.kylin.source.hive.HiveMRInput$HiveTableInputFormat.
> configureJob(HiveMRInput.java:81)
> > > at
> > >
> > >
> > org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.setupMapper(
> FactDistinctColumnsJob.java:111)
> > > at
> > >
> > >
> > org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.
> run(FactDistinctColumnsJob.java:91)
> > > at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:91)
> > > at
> > >
> > >
> > org.apache.kylin.engine.mr.common.MapReduceExecutable.
> doWork(MapReduceExecutable.java:121)
> > > at
> > >
> > >
> > org.apache.kylin.job.execution.AbstractExecutable.
> execute(AbstractExecutable.java:114)
> > > ... 6 more
> > > Caused by: java.lang.ClassNotFoundException:
> > > org.apache.hive.hcatalog.mapreduce.HCatInputFormat
> > > at
> > >
> > >
> > org.apache.catalina.loader.WebappClassLoader.loadClass(
> WebappClassLoader.java:1720)
> > > at
> > >
> > >
> > org.apache.catalina.loader.WebappClassLoader.loadClass(
> WebappClassLoader.java:1571)
> > > ... 12 more
> > >
> >
>
Re: HCatalogFormat error
Posted by Jason Hale <ja...@koddi.com>.
Thanks for the response Li Yang. This was an EMR cluster which I don't have
running now. I switched to setting up a HDP sandbox to get it up and
running for testing purposes. If I get a chance to spin up the EMR cluster
again, I will look into this further.
To answer your question, though, it was the latest version of Kylin, 1.5.3,
and I believe hadoop 2.4 on EMR, so this could very well have been the
issue.
My understanding was that the 'kylin.job.mr.lib.dir' setting would
distribute the jars through the hadoop tmpjars property for Kylin to use.
Is this not correct, or not available on this version?
On Tue, Aug 2, 2016 at 11:52 PM, Li Yang <li...@apache.org> wrote:
> What's your Kylin version?
>
> If it is 1.5.x, your problem is detecting the right hive jar on the Kylin
> node.
>
> Checkout bin/find-hive-dependency.sh. See if it returns right hive path.
>
> On Thu, Jul 28, 2016 at 6:20 AM, Jason Hale <ja...@koddi.com> wrote:
>
> > I have set up a Kylin instance on the master node of my Hadoop cluster. I
> > was trying on a separate client node, but had some permission issues, so
> to
> > simplify the test case, I've just installed it on master. Now I am
> getting
> > the below error.
> >
> > To correct this, I've tried the solution to distribute the jars in
> > https://issues.apache.org/jira/browse/KYLIN-1082 using '
> > kylin.job.mr.lib.dir'.
> > I'm not sure how to append to 'kylin.hive.dependency' as I cannot find
> > information on that (perhaps I'm not looking in the right place). But the
> > lib dir setting did not help and it still is unable to find that class.
> >
> >
> > On #2 Step Name: Extract Fact Table Distinct Columns
> >
> > Kylin executes with the following parameters:
> >
> > -conf /opt/kylin/bin/../conf/kylin_job_conf.xml -cubename Testing -output
> >
> >
> /kylin/kylin_metadata/kylin-40827168-d18f-4b17-a613-3febe773ce2c/Testing/fact_distinct_columns
> > -segmentname 19700101000000_20160731000000 -statisticsenabled true
> > -statisticsoutput
> >
> >
> /kylin/kylin_metadata/kylin-40827168-d18f-4b17-a613-3febe773ce2c/Testing/statistics
> > -statisticssamplingpercent 100 -jobname
> > Kylin_Fact_Distinct_Columns_Testing_Step
> >
> > Error Msg:
> >
> > 2016-07-27 21:54:03,387 ERROR [pool-6-thread-2]
> > execution.AbstractExecutable:116 : error running Executable
> > java.lang.NoClassDefFoundError:
> > org/apache/hive/hcatalog/mapreduce/HCatInputFormat
> > at
> >
> >
> org.apache.kylin.source.hive.HiveMRInput$HiveTableInputFormat.configureJob(HiveMRInput.java:81)
> > at
> >
> >
> org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.setupMapper(FactDistinctColumnsJob.java:111)
> > at
> >
> >
> org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.run(FactDistinctColumnsJob.java:91)
> > at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:91)
> > at
> >
> >
> org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:121)
> > at
> >
> >
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:114)
> > at
> >
> >
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:50)
> > at
> >
> >
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:114)
> > at
> >
> >
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:124)
> > at
> >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > at
> >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > at java.lang.Thread.run(Thread.java:745)
> > Caused by: java.lang.ClassNotFoundException:
> > org.apache.hive.hcatalog.mapreduce.HCatInputFormat
> > at
> >
> >
> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1720)
> > at
> >
> >
> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1571)
> > ... 12 more
> > 2016-07-27 21:54:03,399 INFO [pool-6-thread-2]
> > manager.ExecutableManager:274 : job
> > id:40827168-d18f-4b17-a613-3febe773ce2c-01 from RUNNING to ERROR
> > 2016-07-27 21:54:03,399 ERROR [pool-6-thread-2]
> > execution.AbstractExecutable:116 : error running Executable
> > org.apache.kylin.job.exception.ExecuteException:
> > java.lang.NoClassDefFoundError:
> > org/apache/hive/hcatalog/mapreduce/HCatInputFormat
> > at
> >
> >
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:124)
> > at
> >
> >
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:50)
> > at
> >
> >
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:114)
> > at
> >
> >
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:124)
> > at
> >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > at
> >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > at java.lang.Thread.run(Thread.java:745)
> > Caused by: java.lang.NoClassDefFoundError:
> > org/apache/hive/hcatalog/mapreduce/HCatInputFormat
> > at
> >
> >
> org.apache.kylin.source.hive.HiveMRInput$HiveTableInputFormat.configureJob(HiveMRInput.java:81)
> > at
> >
> >
> org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.setupMapper(FactDistinctColumnsJob.java:111)
> > at
> >
> >
> org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.run(FactDistinctColumnsJob.java:91)
> > at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:91)
> > at
> >
> >
> org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:121)
> > at
> >
> >
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:114)
> > ... 6 more
> > Caused by: java.lang.ClassNotFoundException:
> > org.apache.hive.hcatalog.mapreduce.HCatInputFormat
> > at
> >
> >
> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1720)
> > at
> >
> >
> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1571)
> > ... 12 more
> > 2016-07-27 21:54:03,410 INFO [pool-6-thread-2]
> > manager.ExecutableManager:274 : job
> id:40827168-d18f-4b17-a613-3febe773ce2c
> > from RUNNING to ERROR
> > 2016-07-27 21:54:03,410 ERROR [pool-6-thread-2]
> > threadpool.DefaultScheduler:128 : ExecuteException
> > job:40827168-d18f-4b17-a613-3febe773ce2c
> > org.apache.kylin.job.exception.ExecuteException:
> > org.apache.kylin.job.exception.ExecuteException:
> > java.lang.NoClassDefFoundError:
> > org/apache/hive/hcatalog/mapreduce/HCatInputFormat
> > at
> >
> >
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:124)
> > at
> >
> >
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:124)
> > at
> >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > at
> >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > at java.lang.Thread.run(Thread.java:745)
> > Caused by: org.apache.kylin.job.exception.ExecuteException:
> > java.lang.NoClassDefFoundError:
> > org/apache/hive/hcatalog/mapreduce/HCatInputFormat
> > at
> >
> >
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:124)
> > at
> >
> >
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:50)
> > at
> >
> >
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:114)
> > ... 4 more
> > Caused by: java.lang.NoClassDefFoundError:
> > org/apache/hive/hcatalog/mapreduce/HCatInputFormat
> > at
> >
> >
> org.apache.kylin.source.hive.HiveMRInput$HiveTableInputFormat.configureJob(HiveMRInput.java:81)
> > at
> >
> >
> org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.setupMapper(FactDistinctColumnsJob.java:111)
> > at
> >
> >
> org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.run(FactDistinctColumnsJob.java:91)
> > at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:91)
> > at
> >
> >
> org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:121)
> > at
> >
> >
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:114)
> > ... 6 more
> > Caused by: java.lang.ClassNotFoundException:
> > org.apache.hive.hcatalog.mapreduce.HCatInputFormat
> > at
> >
> >
> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1720)
> > at
> >
> >
> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1571)
> > ... 12 more
> >
>
Re: HCatalogFormat error
Posted by Li Yang <li...@apache.org>.
What's your Kylin version?
If it is 1.5.x, your problem is detecting the right hive jar on the Kylin
node.
Checkout bin/find-hive-dependency.sh. See if it returns right hive path.
On Thu, Jul 28, 2016 at 6:20 AM, Jason Hale <ja...@koddi.com> wrote:
> I have set up a Kylin instance on the master node of my Hadoop cluster. I
> was trying on a separate client node, but had some permission issues, so to
> simplify the test case, I've just installed it on master. Now I am getting
> the below error.
>
> To correct this, I've tried the solution to distribute the jars in
> https://issues.apache.org/jira/browse/KYLIN-1082 using '
> kylin.job.mr.lib.dir'.
> I'm not sure how to append to 'kylin.hive.dependency' as I cannot find
> information on that (perhaps I'm not looking in the right place). But the
> lib dir setting did not help and it still is unable to find that class.
>
>
> On #2 Step Name: Extract Fact Table Distinct Columns
>
> Kylin executes with the following parameters:
>
> -conf /opt/kylin/bin/../conf/kylin_job_conf.xml -cubename Testing -output
>
> /kylin/kylin_metadata/kylin-40827168-d18f-4b17-a613-3febe773ce2c/Testing/fact_distinct_columns
> -segmentname 19700101000000_20160731000000 -statisticsenabled true
> -statisticsoutput
>
> /kylin/kylin_metadata/kylin-40827168-d18f-4b17-a613-3febe773ce2c/Testing/statistics
> -statisticssamplingpercent 100 -jobname
> Kylin_Fact_Distinct_Columns_Testing_Step
>
> Error Msg:
>
> 2016-07-27 21:54:03,387 ERROR [pool-6-thread-2]
> execution.AbstractExecutable:116 : error running Executable
> java.lang.NoClassDefFoundError:
> org/apache/hive/hcatalog/mapreduce/HCatInputFormat
> at
>
> org.apache.kylin.source.hive.HiveMRInput$HiveTableInputFormat.configureJob(HiveMRInput.java:81)
> at
>
> org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.setupMapper(FactDistinctColumnsJob.java:111)
> at
>
> org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.run(FactDistinctColumnsJob.java:91)
> at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:91)
> at
>
> org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:121)
> at
>
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:114)
> at
>
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:50)
> at
>
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:114)
> at
>
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:124)
> at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hive.hcatalog.mapreduce.HCatInputFormat
> at
>
> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1720)
> at
>
> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1571)
> ... 12 more
> 2016-07-27 21:54:03,399 INFO [pool-6-thread-2]
> manager.ExecutableManager:274 : job
> id:40827168-d18f-4b17-a613-3febe773ce2c-01 from RUNNING to ERROR
> 2016-07-27 21:54:03,399 ERROR [pool-6-thread-2]
> execution.AbstractExecutable:116 : error running Executable
> org.apache.kylin.job.exception.ExecuteException:
> java.lang.NoClassDefFoundError:
> org/apache/hive/hcatalog/mapreduce/HCatInputFormat
> at
>
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:124)
> at
>
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:50)
> at
>
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:114)
> at
>
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:124)
> at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NoClassDefFoundError:
> org/apache/hive/hcatalog/mapreduce/HCatInputFormat
> at
>
> org.apache.kylin.source.hive.HiveMRInput$HiveTableInputFormat.configureJob(HiveMRInput.java:81)
> at
>
> org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.setupMapper(FactDistinctColumnsJob.java:111)
> at
>
> org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.run(FactDistinctColumnsJob.java:91)
> at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:91)
> at
>
> org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:121)
> at
>
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:114)
> ... 6 more
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hive.hcatalog.mapreduce.HCatInputFormat
> at
>
> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1720)
> at
>
> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1571)
> ... 12 more
> 2016-07-27 21:54:03,410 INFO [pool-6-thread-2]
> manager.ExecutableManager:274 : job id:40827168-d18f-4b17-a613-3febe773ce2c
> from RUNNING to ERROR
> 2016-07-27 21:54:03,410 ERROR [pool-6-thread-2]
> threadpool.DefaultScheduler:128 : ExecuteException
> job:40827168-d18f-4b17-a613-3febe773ce2c
> org.apache.kylin.job.exception.ExecuteException:
> org.apache.kylin.job.exception.ExecuteException:
> java.lang.NoClassDefFoundError:
> org/apache/hive/hcatalog/mapreduce/HCatInputFormat
> at
>
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:124)
> at
>
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:124)
> at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.kylin.job.exception.ExecuteException:
> java.lang.NoClassDefFoundError:
> org/apache/hive/hcatalog/mapreduce/HCatInputFormat
> at
>
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:124)
> at
>
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:50)
> at
>
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:114)
> ... 4 more
> Caused by: java.lang.NoClassDefFoundError:
> org/apache/hive/hcatalog/mapreduce/HCatInputFormat
> at
>
> org.apache.kylin.source.hive.HiveMRInput$HiveTableInputFormat.configureJob(HiveMRInput.java:81)
> at
>
> org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.setupMapper(FactDistinctColumnsJob.java:111)
> at
>
> org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.run(FactDistinctColumnsJob.java:91)
> at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:91)
> at
>
> org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:121)
> at
>
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:114)
> ... 6 more
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hive.hcatalog.mapreduce.HCatInputFormat
> at
>
> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1720)
> at
>
> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1571)
> ... 12 more
>