You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kylin.apache.org by 张磊 <12...@qq.com> on 2016/10/17 09:03:55 UTC
some question about kylin
I install kylin(apache-kylin-1.5.4.1-hbase1.x-bin) in EMR5.0.0， while i operate on web, there is some error, i do not install hcatalog in emr. How can i resolve this question? Thanks!
java.lang.NoClassDefFoundError: org/apache/hive/hcatalog/mapreduce/HCatInputFormat at org.apache.kylin.source.hive.HiveMRInput$HiveTableInputFormat.configureJob(HiveMRInput.java:89) at org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.setupMapper(FactDistinctColumnsJob.java:123) at org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.run(FactDistinctColumnsJob.java:103) at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:88) at org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:120) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:113) at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:57) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:113) at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:136) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)
上一封 下一封
« 返回
上一封 下一封
Re: some question about kylin
Posted by "Billy(Yiming) Liu" <li...@gmail.com>.
Have you built the cube successfully? The Insight Tab lists query-able
tables only.
2016-10-18 11:14 GMT+08:00 张磊 <12...@qq.com>:
> hi
> in model menu, there are some tables; but in insight menu,there is no
> tables, the file of kylin.log do not contain errors, why?
>
> ------------------ 原始邮件 ------------------
> *发件人:* "ShaoFeng Shi";<sh...@apache.org>;
> *发送时间:* 2016年10月17日(星期一) 晚上10:00
> *收件人:* "dev"<de...@kylin.apache.org>;
> *主题:* Re: some question about kylin
>
> Hi,
>
> Usually this is caused that HBase's script didn't inheritate the
> "HBASE_CLASSPATH" environment varaible (Kylin depends on it to attache Hive
> jars to classpath).
>
> Go to the HBase installation path, and then grep "HBASE_CLASSPATH" in the
> `hbase` shell file, you should find something like:
>
> export
> HBASE_CLASSPATH=$HADOOP_CONF:$HADOOP_HOME/*:$HADOOP_HOME/
> lib/*:$ZOOKEEPER_HOME/*:$ZOOKEEPER_HOME/lib/*:$HBASE_CLASSPATH
>
> Above is a good sample, as it will keep the original value of
> $HBASE_CLASSPATH; But in some distribution like EMR 5.0, it just set the
> HBASE_CLASSPATH value instead of appending, then it will cause the hive jar
> not on Kylin's classpath. You need manually correct that.
>
> I will add this to FAQ page.
>
>
> 2016-10-17 17:03 GMT+08:00 张磊 <12...@qq.com>:
>
> > I install kylin(apache-kylin-1.5.4.1-hbase1.x-bin) in EMR5.0.0， while i
> > operate on web, there is some error, i do not install hcatalog in emr.
> How
> > can i resolve this question? Thanks!
> >
> > java.lang.NoClassDefFoundError: org/apache/hive/hcatalog/
> mapreduce/HCatInputFormat
> > at org.apache.kylin.source.hive.HiveMRInput$HiveTableInputFormat.
> > configureJob(HiveMRInput.java:89) at org.apache.kylin.engine.mr.
> > steps.FactDistinctColumnsJob.setupMapper(FactDistinctColumnsJob.java:
> 123)
> > at org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.
> > run(FactDistinctColumnsJob.java:103) at
> org.apache.kylin.engine.mr
> > .MRUtil.runMRJob(MRUtil.java:88) at org.apache.kylin.engine.mr.
> > common.MapReduceExecutable.doWork(MapReduceExecutable.java:120) at
> > org.apache.kylin.job.execution.AbstractExecutable.
> > execute(AbstractExecutable.java:113) at org.apache.kylin.job.
> > execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:
> 57)
> > at org.apache.kylin.job.execution.AbstractExecutable.
> > execute(AbstractExecutable.java:113) at org.apache.kylin.job.impl.
> > threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:136)
> > at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> >
> >
> >
> >
> > 上一封 下一封
> > « 返回
> >
> >
> >
> >
> >
> >
> >
> >
> > 上一封 下一封
>
>
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>
>
--
With Warm regards
Yiming Liu (刘一鸣)
回复： some question about kylin
Posted by 张磊 <12...@qq.com>.
hi
in model menu, there are some tables; but in insight menu,there is no tables, the file of kylin.log do not contain errors, why?
------------------ 原始邮件 ------------------
发件人: "ShaoFeng Shi";<sh...@apache.org>;
发送时间: 2016年10月17日(星期一) 晚上10:00
收件人: "dev"<de...@kylin.apache.org>;
主题: Re: some question about kylin
Hi,
Usually this is caused that HBase's script didn't inheritate the
"HBASE_CLASSPATH" environment varaible (Kylin depends on it to attache Hive
jars to classpath).
Go to the HBase installation path, and then grep "HBASE_CLASSPATH" in the
`hbase` shell file, you should find something like:
export
HBASE_CLASSPATH=$HADOOP_CONF:$HADOOP_HOME/*:$HADOOP_HOME/lib/*:$ZOOKEEPER_HOME/*:$ZOOKEEPER_HOME/lib/*:$HBASE_CLASSPATH
Above is a good sample, as it will keep the original value of
$HBASE_CLASSPATH; But in some distribution like EMR 5.0, it just set the
HBASE_CLASSPATH value instead of appending, then it will cause the hive jar
not on Kylin's classpath. You need manually correct that.
I will add this to FAQ page.
2016-10-17 17:03 GMT+08:00 张磊 <12...@qq.com>:
> I install kylin(apache-kylin-1.5.4.1-hbase1.x-bin) in EMR5.0.0， while i
> operate on web, there is some error, i do not install hcatalog in emr. How
> can i resolve this question? Thanks!
>
> java.lang.NoClassDefFoundError: org/apache/hive/hcatalog/mapreduce/HCatInputFormat
> at org.apache.kylin.source.hive.HiveMRInput$HiveTableInputFormat.
> configureJob(HiveMRInput.java:89) at org.apache.kylin.engine.mr.
> steps.FactDistinctColumnsJob.setupMapper(FactDistinctColumnsJob.java:123)
> at org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.
> run(FactDistinctColumnsJob.java:103) at org.apache.kylin.engine.mr
> .MRUtil.runMRJob(MRUtil.java:88) at org.apache.kylin.engine.mr.
> common.MapReduceExecutable.doWork(MapReduceExecutable.java:120) at
> org.apache.kylin.job.execution.AbstractExecutable.
> execute(AbstractExecutable.java:113) at org.apache.kylin.job.
> execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:57)
> at org.apache.kylin.job.execution.AbstractExecutable.
> execute(AbstractExecutable.java:113) at org.apache.kylin.job.impl.
> threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:136)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
>
>
>
> 上一封 下一封
> « 返回
>
>
>
>
>
>
>
>
> 上一封 下一封
--
Best regards,
Shaofeng Shi 史少锋
回复： some question about kylin
Posted by 张磊 <12...@qq.com>.
hbase command --> /usr/bin/hbase
i read /usr/bin/hbase this file,the content is
export HADOOP_CONF=${HADOOP_CONF:-/etc/hadoop/conf}
export ZOOKEEPER_HOME=${ZOOKEEPER_HOME:-/usr/lib/zookeeper}
export HBASE_CLASSPATH=$HADOOP_CONF:$HADOOP_HOME/*:$HADOOP_HOME/lib/*:$ZOOKEEPER_HOME/*:$ZOOKEEPER_HOME/lib/*:$HBASE_CLASSPATH
the HBASE_CLASSPATH is appending not only set
my question is
1、when i run sample.sh, hive have kylin_cal_dt、kylin_category_groupings、kylin_sales three tables
but in Kylin Web Interface , in learn_kylin project, model data source can load those three tables,but can not load other tables(i create table in hive by own)
2、i create a project in Kylin Web Interface, but can not load one table(contain kylin_cal_dt and other sample table)
How can i solve thoes two problems, and kylin.log contain no error logs?
------------------ 原始邮件 ------------------
发件人: "ShaoFeng Shi";<sh...@apache.org>;
发送时间: 2016年10月18日(星期二) 下午4:37
收件人: "dev"<de...@kylin.apache.org>;
主题: Re: some question about kylin
kylin.sh doesn't have issue; What need to check is the "hbase" command in
hbase installation folder.
2016-10-18 16:04 GMT+08:00 张磊 <12...@qq.com>:
> hi
> I read the kylin.sh script, i find these sentences
>
>
> function retrieveDependency() export HBASE_CLASSPATH=${HBASE_
> CLASSPATH}:${hive_dependency}
>
>
> Do you mean i should set ${HBASE_CLASSPATH} value?
>
>
> And this value should be from the command hbase classpath?
>
>
>
>
>
>
> ------------------ 原始邮件 ------------------
> 发件人: "ShaoFeng Shi";<sh...@apache.org>;
> 发送时间: 2016年10月17日(星期一) 晚上10:00
> 收件人: "dev"<de...@kylin.apache.org>;
>
> 主题: Re: some question about kylin
>
>
>
> Hi,
>
> Usually this is caused that HBase's script didn't inheritate the
> "HBASE_CLASSPATH" environment varaible (Kylin depends on it to attache Hive
> jars to classpath).
>
> Go to the HBase installation path, and then grep "HBASE_CLASSPATH" in the
> `hbase` shell file, you should find something like:
>
> export
> HBASE_CLASSPATH=$HADOOP_CONF:$HADOOP_HOME/*:$HADOOP_HOME/
> lib/*:$ZOOKEEPER_HOME/*:$ZOOKEEPER_HOME/lib/*:$HBASE_CLASSPATH
>
> Above is a good sample, as it will keep the original value of
> $HBASE_CLASSPATH; But in some distribution like EMR 5.0, it just set the
> HBASE_CLASSPATH value instead of appending, then it will cause the hive jar
> not on Kylin's classpath. You need manually correct that.
>
> I will add this to FAQ page.
>
>
> 2016-10-17 17:03 GMT+08:00 张磊 <12...@qq.com>:
>
> > I install kylin(apache-kylin-1.5.4.1-hbase1.x-bin) in EMR5.0.0， while i
> > operate on web, there is some error, i do not install hcatalog in emr.
> How
> > can i resolve this question? Thanks!
> >
> > java.lang.NoClassDefFoundError: org/apache/hive/hcatalog/
> mapreduce/HCatInputFormat
> > at org.apache.kylin.source.hive.HiveMRInput$HiveTableInputFormat.
> > configureJob(HiveMRInput.java:89) at org.apache.kylin.engine.mr.
> > steps.FactDistinctColumnsJob.setupMapper(FactDistinctColumnsJob.java:
> 123)
> > at org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.
> > run(FactDistinctColumnsJob.java:103) at
> org.apache.kylin.engine.mr
> > .MRUtil.runMRJob(MRUtil.java:88) at org.apache.kylin.engine.mr.
> > common.MapReduceExecutable.doWork(MapReduceExecutable.java:120) at
> > org.apache.kylin.job.execution.AbstractExecutable.
> > execute(AbstractExecutable.java:113) at org.apache.kylin.job.
> > execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:
> 57)
> > at org.apache.kylin.job.execution.AbstractExecutable.
> > execute(AbstractExecutable.java:113) at org.apache.kylin.job.impl.
> > threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:136)
> > at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> >
> >
> >
> >
> > 上一封 下一封
> > « 返回
> >
> >
> >
> >
> >
> >
> >
> >
> > 上一封 下一封
>
>
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>
--
Best regards,
Shaofeng Shi 史少锋
回复： java.lang.NoClassDefFoundError: org/apache/hive/hcatalog/mapreduce/HCatInputFormat
Posted by 张磊 <12...@qq.com>.
I got it,thanks
------------------ 原始邮件 ------------------
发件人: "ShaoFeng Shi";<sh...@apache.org>;
发送时间: 2016年10月28日(星期五) 上午10:33
收件人: "dev"<de...@kylin.apache.org>;
主题: Re: java.lang.NoClassDefFoundError: org/apache/hive/hcatalog/mapreduce/HCatInputFormat
Today I start a Kylin 1.5.4.1-hbase1.x package on a EMR cluster (separately
request a client node), got this NoClassDefFoundError, finally solved by
changing the /usr/lib/hbase/conf/hbase-env.sh; I recorded it in the FAQ
page, just for your reference:
https://kylin.apache.org/docs15/gettingstarted/faq.html
2016-10-20 9:19 GMT+08:00 张磊 <12...@qq.com>:
> find-hive-dependency.sh is ok
> I say find-hbase-dependency.sh this script,
> when i mkdir -p /usr/lib/oozie/lib, the variable HBASE_ENV_INIT is set true
> i do not edit /usr/lib/hbase/conf/hbase-env.sh and Change the first line
> from
> export HBASE_CLASSPATH=/etc/hadoop/conf
> to
> export HBASE_CLASSPATH=$HBASE_CLASSPATH:/etc/hadoop/conf
>
>
>
>
> ------------------ 原始邮件 ------------------
> 发件人: "ShaoFeng Shi";<sh...@apache.org>;
> 发送时间: 2016年10月19日(星期三) 下午5:38
> 收件人: "dev"<de...@kylin.apache.org>;
>
> 主题: Re: java.lang.NoClassDefFoundError: org/apache/hive/hcatalog/
> mapreduce/HCatInputFormat
>
>
>
> In 1.5.4 kylin change to load hcat from /usr/lib/hive-hcatalog, you can try
> to modify the file to
> https://github.com/apache/kylin/blob/kylin-1.5.4.1/build/bin/find-hive-
> dependency.sh#L89
>
> 2016-10-19 15:48 GMT+08:00 张磊 <12...@qq.com>:
>
> > it is ok,thanks
> > if my emr cluster has a directory /usr/lib/oozie/lib，HBASE_ENV_
> INIT＝true,
> > i do not change /usr/lib/hbase/conf/hbase-env.sh, i do not konw why you
> > judge by /usr/lib/oozie/lib directory
> >
> >
> > if [ -n is_aws ] && [ -d "/usr/lib/oozie/lib" ]; then
> > export HBASE_ENV_INIT="true"
> > fi
> >
> >
> >
> >
> >
> > ------------------ 原始邮件 ------------------
> > 发件人: "liuyiming.vip";<li...@gmail.com>;
> > 发送时间: 2016年10月19日(星期三) 下午2:47
> > 收件人: "dev"<de...@kylin.apache.org>;
> >
> > 主题: Re: java.lang.NoClassDefFoundError: org/apache/hive/hcatalog/
> > mapreduce/HCatInputFormat
> >
> >
> >
> > Edit /usr/lib/hbase/conf/hbase-env.sh
> > Change the first line from
> > export HBASE_CLASSPATH=/etc/hadoop/conf
> >
> > to
> > export HBASE_CLASSPATH=$HBASE_CLASSPATH:/etc/hadoop/conf
> >
> > 2016-10-19 14:24 GMT+08:00 张磊 <12...@qq.com>:
> >
> > > Hi when i build the sample kylin_sales_cube, in the kylin.log,i find
> > > java.lang.NoClassDefFoundError. but in the log i find some messages, i
> > do
> > > not konw why?
> > >
> > >
> > > /usr/lib/hive-hcatalog/share/hcatalog/hive-hcatalog-core-2.
> > 1.0-amzn-0.jar
> > > is loaded, why class is not foung?
> > >
> > >
> > > 016-10-18 23:17:59,902 INFO [pool-8-thread-3]
> > > common.AbstractHadoopJob:203 : Hive Dependencies After Filtered:
> > > /usr/lib/hive/lib/hive-metastore-2.1.0-amzn-0.jar,/
> > > usr/lib/hive/lib/hive-exec-2.1.0-amzn-0.jar,/usr/lib/hive-
> > > hcatalog/share/hcatalog/hive-hcatalog-core-2.1.0-amzn-0.jar
> > >
> > >
> > > 2016-10-18 23:17:59,907 INFO [pool-8-thread-3]
> > > common.AbstractHadoopJob:371 : Job 'tmpjars' updated --
> > > file:/usr/lib/hive/lib/hive-metastore-2.1.0-amzn-0.jar,
> > > file:/usr/lib/hive/lib/hive-exec-2.1.0-amzn-0.jar,file:/
> > > usr/lib/hive-hcatalog/share/hcatalog/hive-hcatalog-core-2.
> 1.0-amzn-0.jar
> > >
> > >
> > > 2016-10-18 23:17:59,934 ERROR [pool-8-thread-3]
> > > threadpool.DefaultScheduler:140 : ExecuteException
> > > job:845380a7-63b7-4315-8d71-6bb290018790
> > > org.apache.kylin.job.exception.ExecuteException: org.apache.kylin.job.
> > exception.ExecuteException:
> > > java.lang.NoClassDefFoundError: org/apache/hive/hcatalog/
> > > mapreduce/HCatInputFormat
> > > at org.apache.kylin.job.execution.AbstractExecutable.
> > > execute(AbstractExecutable.java:123)
> > > at org.apache.kylin.job.impl.threadpool.DefaultScheduler$
> > > JobRunner.run(DefaultScheduler.java:136)
> > > at java.util.concurrent.ThreadPoolExecutor.runWorker(
> > > ThreadPoolExecutor.java:1142)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > > ThreadPoolExecutor.java:617)
> > > at java.lang.Thread.run(Thread.java:745)
> > > Caused by: org.apache.kylin.job.exception.ExecuteException: java.lang.
> > NoClassDefFoundError:
> > > org/apache/hive/hcatalog/mapreduce/HCatInputFormat
> > > at org.apache.kylin.job.execution.AbstractExecutable.
> > > execute(AbstractExecutable.java:123)
> > > at org.apache.kylin.job.execution.DefaultChainedExecutable.
> > doWork(
> > > DefaultChainedExecutable.java:57)
> > > at org.apache.kylin.job.execution.AbstractExecutable.
> > > execute(AbstractExecutable.java:113)
> > > ... 4 more
> > > Caused by: java.lang.NoClassDefFoundError: org/apache/hive/hcatalog/
> > > mapreduce/HCatInputFormat
> > > at org.apache.kylin.source.hive.HiveMRInput$
> > HiveTableInputFormat.
> > > configureJob(HiveMRInput.java:89)
> > > at org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.
> > > setupMapper(FactDistinctColumnsJob.java:123)
> > > at org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.
> > > run(FactDistinctColumnsJob.java:103)
> > > at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:88)
> > > at org.apache.kylin.engine.mr.common.MapReduceExecutable.
> > > doWork(MapReduceExecutable.java:120)
> > > at org.apache.kylin.job.execution.AbstractExecutable.
> > > execute(AbstractExecutable.java:113)
> > > ... 6 more
> > > Caused by: java.lang.ClassNotFoundException: org.apache.hive.hcatalog.
> > > mapreduce.HCatInputFormat
> > > at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(
> > > WebappClassLoaderBase.java:1858)
> > > at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(
> > > WebappClassLoaderBase.java:1701)
> > > ... 12 more
> >
> >
> >
> >
> > --
> > With Warm regards
> >
> > Yiming Liu (刘一鸣)
> >
>
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>
--
Best regards,
Shaofeng Shi 史少锋
Re: java.lang.NoClassDefFoundError: org/apache/hive/hcatalog/mapreduce/HCatInputFormat
Posted by ShaoFeng Shi <sh...@apache.org>.
Today I start a Kylin 1.5.4.1-hbase1.x package on a EMR cluster (separately
request a client node), got this NoClassDefFoundError, finally solved by
changing the /usr/lib/hbase/conf/hbase-env.sh; I recorded it in the FAQ
page, just for your reference:
https://kylin.apache.org/docs15/gettingstarted/faq.html
2016-10-20 9:19 GMT+08:00 张磊 <12...@qq.com>:
> find-hive-dependency.sh is ok
> I say find-hbase-dependency.sh this script,
> when i mkdir -p /usr/lib/oozie/lib, the variable HBASE_ENV_INIT is set true
> i do not edit /usr/lib/hbase/conf/hbase-env.sh and Change the first line
> from
> export HBASE_CLASSPATH=/etc/hadoop/conf
> to
> export HBASE_CLASSPATH=$HBASE_CLASSPATH:/etc/hadoop/conf
>
>
>
>
> ------------------ 原始邮件 ------------------
> 发件人: "ShaoFeng Shi";<sh...@apache.org>;
> 发送时间: 2016年10月19日(星期三) 下午5:38
> 收件人: "dev"<de...@kylin.apache.org>;
>
> 主题: Re: java.lang.NoClassDefFoundError: org/apache/hive/hcatalog/
> mapreduce/HCatInputFormat
>
>
>
> In 1.5.4 kylin change to load hcat from /usr/lib/hive-hcatalog, you can try
> to modify the file to
> https://github.com/apache/kylin/blob/kylin-1.5.4.1/build/bin/find-hive-
> dependency.sh#L89
>
> 2016-10-19 15:48 GMT+08:00 张磊 <12...@qq.com>:
>
> > it is ok,thanks
> > if my emr cluster has a directory /usr/lib/oozie/lib，HBASE_ENV_
> INIT＝true,
> > i do not change /usr/lib/hbase/conf/hbase-env.sh, i do not konw why you
> > judge by /usr/lib/oozie/lib directory
> >
> >
> > if [ -n is_aws ] && [ -d "/usr/lib/oozie/lib" ]; then
> > export HBASE_ENV_INIT="true"
> > fi
> >
> >
> >
> >
> >
> > ------------------ 原始邮件 ------------------
> > 发件人: "liuyiming.vip";<li...@gmail.com>;
> > 发送时间: 2016年10月19日(星期三) 下午2:47
> > 收件人: "dev"<de...@kylin.apache.org>;
> >
> > 主题: Re: java.lang.NoClassDefFoundError: org/apache/hive/hcatalog/
> > mapreduce/HCatInputFormat
> >
> >
> >
> > Edit /usr/lib/hbase/conf/hbase-env.sh
> > Change the first line from
> > export HBASE_CLASSPATH=/etc/hadoop/conf
> >
> > to
> > export HBASE_CLASSPATH=$HBASE_CLASSPATH:/etc/hadoop/conf
> >
> > 2016-10-19 14:24 GMT+08:00 张磊 <12...@qq.com>:
> >
> > > Hi when i build the sample kylin_sales_cube, in the kylin.log,i find
> > > java.lang.NoClassDefFoundError. but in the log i find some messages, i
> > do
> > > not konw why?
> > >
> > >
> > > /usr/lib/hive-hcatalog/share/hcatalog/hive-hcatalog-core-2.
> > 1.0-amzn-0.jar
> > > is loaded, why class is not foung?
> > >
> > >
> > > 016-10-18 23:17:59,902 INFO [pool-8-thread-3]
> > > common.AbstractHadoopJob:203 : Hive Dependencies After Filtered:
> > > /usr/lib/hive/lib/hive-metastore-2.1.0-amzn-0.jar,/
> > > usr/lib/hive/lib/hive-exec-2.1.0-amzn-0.jar,/usr/lib/hive-
> > > hcatalog/share/hcatalog/hive-hcatalog-core-2.1.0-amzn-0.jar
> > >
> > >
> > > 2016-10-18 23:17:59,907 INFO [pool-8-thread-3]
> > > common.AbstractHadoopJob:371 : Job 'tmpjars' updated --
> > > file:/usr/lib/hive/lib/hive-metastore-2.1.0-amzn-0.jar,
> > > file:/usr/lib/hive/lib/hive-exec-2.1.0-amzn-0.jar,file:/
> > > usr/lib/hive-hcatalog/share/hcatalog/hive-hcatalog-core-2.
> 1.0-amzn-0.jar
> > >
> > >
> > > 2016-10-18 23:17:59,934 ERROR [pool-8-thread-3]
> > > threadpool.DefaultScheduler:140 : ExecuteException
> > > job:845380a7-63b7-4315-8d71-6bb290018790
> > > org.apache.kylin.job.exception.ExecuteException: org.apache.kylin.job.
> > exception.ExecuteException:
> > > java.lang.NoClassDefFoundError: org/apache/hive/hcatalog/
> > > mapreduce/HCatInputFormat
> > > at org.apache.kylin.job.execution.AbstractExecutable.
> > > execute(AbstractExecutable.java:123)
> > > at org.apache.kylin.job.impl.threadpool.DefaultScheduler$
> > > JobRunner.run(DefaultScheduler.java:136)
> > > at java.util.concurrent.ThreadPoolExecutor.runWorker(
> > > ThreadPoolExecutor.java:1142)
> > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > > ThreadPoolExecutor.java:617)
> > > at java.lang.Thread.run(Thread.java:745)
> > > Caused by: org.apache.kylin.job.exception.ExecuteException: java.lang.
> > NoClassDefFoundError:
> > > org/apache/hive/hcatalog/mapreduce/HCatInputFormat
> > > at org.apache.kylin.job.execution.AbstractExecutable.
> > > execute(AbstractExecutable.java:123)
> > > at org.apache.kylin.job.execution.DefaultChainedExecutable.
> > doWork(
> > > DefaultChainedExecutable.java:57)
> > > at org.apache.kylin.job.execution.AbstractExecutable.
> > > execute(AbstractExecutable.java:113)
> > > ... 4 more
> > > Caused by: java.lang.NoClassDefFoundError: org/apache/hive/hcatalog/
> > > mapreduce/HCatInputFormat
> > > at org.apache.kylin.source.hive.HiveMRInput$
> > HiveTableInputFormat.
> > > configureJob(HiveMRInput.java:89)
> > > at org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.
> > > setupMapper(FactDistinctColumnsJob.java:123)
> > > at org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.
> > > run(FactDistinctColumnsJob.java:103)
> > > at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:88)
> > > at org.apache.kylin.engine.mr.common.MapReduceExecutable.
> > > doWork(MapReduceExecutable.java:120)
> > > at org.apache.kylin.job.execution.AbstractExecutable.
> > > execute(AbstractExecutable.java:113)
> > > ... 6 more
> > > Caused by: java.lang.ClassNotFoundException: org.apache.hive.hcatalog.
> > > mapreduce.HCatInputFormat
> > > at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(
> > > WebappClassLoaderBase.java:1858)
> > > at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(
> > > WebappClassLoaderBase.java:1701)
> > > ... 12 more
> >
> >
> >
> >
> > --
> > With Warm regards
> >
> > Yiming Liu (刘一鸣)
> >
>
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>
--
Best regards,
Shaofeng Shi 史少锋
回复： java.lang.NoClassDefFoundError: org/apache/hive/hcatalog/mapreduce/HCatInputFormat
Posted by 张磊 <12...@qq.com>.
find-hive-dependency.sh is ok
I say find-hbase-dependency.sh this script,
when i mkdir -p /usr/lib/oozie/lib, the variable HBASE_ENV_INIT is set true
i do not edit /usr/lib/hbase/conf/hbase-env.sh and Change the first line from
export HBASE_CLASSPATH=/etc/hadoop/conf
to
export HBASE_CLASSPATH=$HBASE_CLASSPATH:/etc/hadoop/conf
------------------ 原始邮件 ------------------
发件人: "ShaoFeng Shi";<sh...@apache.org>;
发送时间: 2016年10月19日(星期三) 下午5:38
收件人: "dev"<de...@kylin.apache.org>;
主题: Re: java.lang.NoClassDefFoundError: org/apache/hive/hcatalog/mapreduce/HCatInputFormat
In 1.5.4 kylin change to load hcat from /usr/lib/hive-hcatalog, you can try
to modify the file to
https://github.com/apache/kylin/blob/kylin-1.5.4.1/build/bin/find-hive-dependency.sh#L89
2016-10-19 15:48 GMT+08:00 张磊 <12...@qq.com>:
> it is ok,thanks
> if my emr cluster has a directory /usr/lib/oozie/lib，HBASE_ENV_INIT＝true,
> i do not change /usr/lib/hbase/conf/hbase-env.sh, i do not konw why you
> judge by /usr/lib/oozie/lib directory
>
>
> if [ -n is_aws ] && [ -d "/usr/lib/oozie/lib" ]; then
> export HBASE_ENV_INIT="true"
> fi
>
>
>
>
>
> ------------------ 原始邮件 ------------------
> 发件人: "liuyiming.vip";<li...@gmail.com>;
> 发送时间: 2016年10月19日(星期三) 下午2:47
> 收件人: "dev"<de...@kylin.apache.org>;
>
> 主题: Re: java.lang.NoClassDefFoundError: org/apache/hive/hcatalog/
> mapreduce/HCatInputFormat
>
>
>
> Edit /usr/lib/hbase/conf/hbase-env.sh
> Change the first line from
> export HBASE_CLASSPATH=/etc/hadoop/conf
>
> to
> export HBASE_CLASSPATH=$HBASE_CLASSPATH:/etc/hadoop/conf
>
> 2016-10-19 14:24 GMT+08:00 张磊 <12...@qq.com>:
>
> > Hi when i build the sample kylin_sales_cube, in the kylin.log,i find
> > java.lang.NoClassDefFoundError. but in the log i find some messages, i
> do
> > not konw why?
> >
> >
> > /usr/lib/hive-hcatalog/share/hcatalog/hive-hcatalog-core-2.
> 1.0-amzn-0.jar
> > is loaded, why class is not foung?
> >
> >
> > 016-10-18 23:17:59,902 INFO [pool-8-thread-3]
> > common.AbstractHadoopJob:203 : Hive Dependencies After Filtered:
> > /usr/lib/hive/lib/hive-metastore-2.1.0-amzn-0.jar,/
> > usr/lib/hive/lib/hive-exec-2.1.0-amzn-0.jar,/usr/lib/hive-
> > hcatalog/share/hcatalog/hive-hcatalog-core-2.1.0-amzn-0.jar
> >
> >
> > 2016-10-18 23:17:59,907 INFO [pool-8-thread-3]
> > common.AbstractHadoopJob:371 : Job 'tmpjars' updated --
> > file:/usr/lib/hive/lib/hive-metastore-2.1.0-amzn-0.jar,
> > file:/usr/lib/hive/lib/hive-exec-2.1.0-amzn-0.jar,file:/
> > usr/lib/hive-hcatalog/share/hcatalog/hive-hcatalog-core-2.1.0-amzn-0.jar
> >
> >
> > 2016-10-18 23:17:59,934 ERROR [pool-8-thread-3]
> > threadpool.DefaultScheduler:140 : ExecuteException
> > job:845380a7-63b7-4315-8d71-6bb290018790
> > org.apache.kylin.job.exception.ExecuteException: org.apache.kylin.job.
> exception.ExecuteException:
> > java.lang.NoClassDefFoundError: org/apache/hive/hcatalog/
> > mapreduce/HCatInputFormat
> > at org.apache.kylin.job.execution.AbstractExecutable.
> > execute(AbstractExecutable.java:123)
> > at org.apache.kylin.job.impl.threadpool.DefaultScheduler$
> > JobRunner.run(DefaultScheduler.java:136)
> > at java.util.concurrent.ThreadPoolExecutor.runWorker(
> > ThreadPoolExecutor.java:1142)
> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> > Caused by: org.apache.kylin.job.exception.ExecuteException: java.lang.
> NoClassDefFoundError:
> > org/apache/hive/hcatalog/mapreduce/HCatInputFormat
> > at org.apache.kylin.job.execution.AbstractExecutable.
> > execute(AbstractExecutable.java:123)
> > at org.apache.kylin.job.execution.DefaultChainedExecutable.
> doWork(
> > DefaultChainedExecutable.java:57)
> > at org.apache.kylin.job.execution.AbstractExecutable.
> > execute(AbstractExecutable.java:113)
> > ... 4 more
> > Caused by: java.lang.NoClassDefFoundError: org/apache/hive/hcatalog/
> > mapreduce/HCatInputFormat
> > at org.apache.kylin.source.hive.HiveMRInput$
> HiveTableInputFormat.
> > configureJob(HiveMRInput.java:89)
> > at org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.
> > setupMapper(FactDistinctColumnsJob.java:123)
> > at org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.
> > run(FactDistinctColumnsJob.java:103)
> > at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:88)
> > at org.apache.kylin.engine.mr.common.MapReduceExecutable.
> > doWork(MapReduceExecutable.java:120)
> > at org.apache.kylin.job.execution.AbstractExecutable.
> > execute(AbstractExecutable.java:113)
> > ... 6 more
> > Caused by: java.lang.ClassNotFoundException: org.apache.hive.hcatalog.
> > mapreduce.HCatInputFormat
> > at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(
> > WebappClassLoaderBase.java:1858)
> > at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(
> > WebappClassLoaderBase.java:1701)
> > ... 12 more
>
>
>
>
> --
> With Warm regards
>
> Yiming Liu (刘一鸣)
>
--
Best regards,
Shaofeng Shi 史少锋
Re: java.lang.NoClassDefFoundError: org/apache/hive/hcatalog/mapreduce/HCatInputFormat
Posted by ShaoFeng Shi <sh...@apache.org>.
In 1.5.4 kylin change to load hcat from /usr/lib/hive-hcatalog, you can try
to modify the file to
https://github.com/apache/kylin/blob/kylin-1.5.4.1/build/bin/find-hive-dependency.sh#L89
2016-10-19 15:48 GMT+08:00 张磊 <12...@qq.com>:
> it is ok,thanks
> if my emr cluster has a directory /usr/lib/oozie/lib，HBASE_ENV_INIT＝true,
> i do not change /usr/lib/hbase/conf/hbase-env.sh, i do not konw why you
> judge by /usr/lib/oozie/lib directory
>
>
> if [ -n is_aws ] && [ -d "/usr/lib/oozie/lib" ]; then
> export HBASE_ENV_INIT="true"
> fi
>
>
>
>
>
> ------------------ 原始邮件 ------------------
> 发件人: "liuyiming.vip";<li...@gmail.com>;
> 发送时间: 2016年10月19日(星期三) 下午2:47
> 收件人: "dev"<de...@kylin.apache.org>;
>
> 主题: Re: java.lang.NoClassDefFoundError: org/apache/hive/hcatalog/
> mapreduce/HCatInputFormat
>
>
>
> Edit /usr/lib/hbase/conf/hbase-env.sh
> Change the first line from
> export HBASE_CLASSPATH=/etc/hadoop/conf
>
> to
> export HBASE_CLASSPATH=$HBASE_CLASSPATH:/etc/hadoop/conf
>
> 2016-10-19 14:24 GMT+08:00 张磊 <12...@qq.com>:
>
> > Hi when i build the sample kylin_sales_cube, in the kylin.log,i find
> > java.lang.NoClassDefFoundError. but in the log i find some messages, i
> do
> > not konw why?
> >
> >
> > /usr/lib/hive-hcatalog/share/hcatalog/hive-hcatalog-core-2.
> 1.0-amzn-0.jar
> > is loaded, why class is not foung?
> >
> >
> > 016-10-18 23:17:59,902 INFO [pool-8-thread-3]
> > common.AbstractHadoopJob:203 : Hive Dependencies After Filtered:
> > /usr/lib/hive/lib/hive-metastore-2.1.0-amzn-0.jar,/
> > usr/lib/hive/lib/hive-exec-2.1.0-amzn-0.jar,/usr/lib/hive-
> > hcatalog/share/hcatalog/hive-hcatalog-core-2.1.0-amzn-0.jar
> >
> >
> > 2016-10-18 23:17:59,907 INFO [pool-8-thread-3]
> > common.AbstractHadoopJob:371 : Job 'tmpjars' updated --
> > file:/usr/lib/hive/lib/hive-metastore-2.1.0-amzn-0.jar,
> > file:/usr/lib/hive/lib/hive-exec-2.1.0-amzn-0.jar,file:/
> > usr/lib/hive-hcatalog/share/hcatalog/hive-hcatalog-core-2.1.0-amzn-0.jar
> >
> >
> > 2016-10-18 23:17:59,934 ERROR [pool-8-thread-3]
> > threadpool.DefaultScheduler:140 : ExecuteException
> > job:845380a7-63b7-4315-8d71-6bb290018790
> > org.apache.kylin.job.exception.ExecuteException: org.apache.kylin.job.
> exception.ExecuteException:
> > java.lang.NoClassDefFoundError: org/apache/hive/hcatalog/
> > mapreduce/HCatInputFormat
> > at org.apache.kylin.job.execution.AbstractExecutable.
> > execute(AbstractExecutable.java:123)
> > at org.apache.kylin.job.impl.threadpool.DefaultScheduler$
> > JobRunner.run(DefaultScheduler.java:136)
> > at java.util.concurrent.ThreadPoolExecutor.runWorker(
> > ThreadPoolExecutor.java:1142)
> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> > Caused by: org.apache.kylin.job.exception.ExecuteException: java.lang.
> NoClassDefFoundError:
> > org/apache/hive/hcatalog/mapreduce/HCatInputFormat
> > at org.apache.kylin.job.execution.AbstractExecutable.
> > execute(AbstractExecutable.java:123)
> > at org.apache.kylin.job.execution.DefaultChainedExecutable.
> doWork(
> > DefaultChainedExecutable.java:57)
> > at org.apache.kylin.job.execution.AbstractExecutable.
> > execute(AbstractExecutable.java:113)
> > ... 4 more
> > Caused by: java.lang.NoClassDefFoundError: org/apache/hive/hcatalog/
> > mapreduce/HCatInputFormat
> > at org.apache.kylin.source.hive.HiveMRInput$
> HiveTableInputFormat.
> > configureJob(HiveMRInput.java:89)
> > at org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.
> > setupMapper(FactDistinctColumnsJob.java:123)
> > at org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.
> > run(FactDistinctColumnsJob.java:103)
> > at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:88)
> > at org.apache.kylin.engine.mr.common.MapReduceExecutable.
> > doWork(MapReduceExecutable.java:120)
> > at org.apache.kylin.job.execution.AbstractExecutable.
> > execute(AbstractExecutable.java:113)
> > ... 6 more
> > Caused by: java.lang.ClassNotFoundException: org.apache.hive.hcatalog.
> > mapreduce.HCatInputFormat
> > at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(
> > WebappClassLoaderBase.java:1858)
> > at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(
> > WebappClassLoaderBase.java:1701)
> > ... 12 more
>
>
>
>
> --
> With Warm regards
>
> Yiming Liu (刘一鸣)
>
--
Best regards,
Shaofeng Shi 史少锋
回复： java.lang.NoClassDefFoundError: org/apache/hive/hcatalog/mapreduce/HCatInputFormat
Posted by 张磊 <12...@qq.com>.
it is ok,thanks
if my emr cluster has a directory /usr/lib/oozie/lib，HBASE_ENV_INIT＝true, i do not change /usr/lib/hbase/conf/hbase-env.sh, i do not konw why you judge by /usr/lib/oozie/lib directory
if [ -n is_aws ] && [ -d "/usr/lib/oozie/lib" ]; then
export HBASE_ENV_INIT="true"
fi
------------------ 原始邮件 ------------------
发件人: "liuyiming.vip";<li...@gmail.com>;
发送时间: 2016年10月19日(星期三) 下午2:47
收件人: "dev"<de...@kylin.apache.org>;
主题: Re: java.lang.NoClassDefFoundError: org/apache/hive/hcatalog/mapreduce/HCatInputFormat
Edit /usr/lib/hbase/conf/hbase-env.sh
Change the first line from
export HBASE_CLASSPATH=/etc/hadoop/conf
to
export HBASE_CLASSPATH=$HBASE_CLASSPATH:/etc/hadoop/conf
2016-10-19 14:24 GMT+08:00 张磊 <12...@qq.com>:
> Hi when i build the sample kylin_sales_cube, in the kylin.log,i find
> java.lang.NoClassDefFoundError. but in the log i find some messages, i do
> not konw why?
>
>
> /usr/lib/hive-hcatalog/share/hcatalog/hive-hcatalog-core-2.1.0-amzn-0.jar
> is loaded, why class is not foung?
>
>
> 016-10-18 23:17:59,902 INFO [pool-8-thread-3]
> common.AbstractHadoopJob:203 : Hive Dependencies After Filtered:
> /usr/lib/hive/lib/hive-metastore-2.1.0-amzn-0.jar,/
> usr/lib/hive/lib/hive-exec-2.1.0-amzn-0.jar,/usr/lib/hive-
> hcatalog/share/hcatalog/hive-hcatalog-core-2.1.0-amzn-0.jar
>
>
> 2016-10-18 23:17:59,907 INFO [pool-8-thread-3]
> common.AbstractHadoopJob:371 : Job 'tmpjars' updated --
> file:/usr/lib/hive/lib/hive-metastore-2.1.0-amzn-0.jar,
> file:/usr/lib/hive/lib/hive-exec-2.1.0-amzn-0.jar,file:/
> usr/lib/hive-hcatalog/share/hcatalog/hive-hcatalog-core-2.1.0-amzn-0.jar
>
>
> 2016-10-18 23:17:59,934 ERROR [pool-8-thread-3]
> threadpool.DefaultScheduler:140 : ExecuteException
> job:845380a7-63b7-4315-8d71-6bb290018790
> org.apache.kylin.job.exception.ExecuteException: org.apache.kylin.job.exception.ExecuteException:
> java.lang.NoClassDefFoundError: org/apache/hive/hcatalog/
> mapreduce/HCatInputFormat
> at org.apache.kylin.job.execution.AbstractExecutable.
> execute(AbstractExecutable.java:123)
> at org.apache.kylin.job.impl.threadpool.DefaultScheduler$
> JobRunner.run(DefaultScheduler.java:136)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.kylin.job.exception.ExecuteException: java.lang.NoClassDefFoundError:
> org/apache/hive/hcatalog/mapreduce/HCatInputFormat
> at org.apache.kylin.job.execution.AbstractExecutable.
> execute(AbstractExecutable.java:123)
> at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(
> DefaultChainedExecutable.java:57)
> at org.apache.kylin.job.execution.AbstractExecutable.
> execute(AbstractExecutable.java:113)
> ... 4 more
> Caused by: java.lang.NoClassDefFoundError: org/apache/hive/hcatalog/
> mapreduce/HCatInputFormat
> at org.apache.kylin.source.hive.HiveMRInput$HiveTableInputFormat.
> configureJob(HiveMRInput.java:89)
> at org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.
> setupMapper(FactDistinctColumnsJob.java:123)
> at org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.
> run(FactDistinctColumnsJob.java:103)
> at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:88)
> at org.apache.kylin.engine.mr.common.MapReduceExecutable.
> doWork(MapReduceExecutable.java:120)
> at org.apache.kylin.job.execution.AbstractExecutable.
> execute(AbstractExecutable.java:113)
> ... 6 more
> Caused by: java.lang.ClassNotFoundException: org.apache.hive.hcatalog.
> mapreduce.HCatInputFormat
> at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(
> WebappClassLoaderBase.java:1858)
> at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(
> WebappClassLoaderBase.java:1701)
> ... 12 more
--
With Warm regards
Yiming Liu (刘一鸣)
Re: java.lang.NoClassDefFoundError: org/apache/hive/hcatalog/mapreduce/HCatInputFormat
Posted by "Billy(Yiming) Liu" <li...@gmail.com>.
Edit /usr/lib/hbase/conf/hbase-env.sh
Change the first line from
export HBASE_CLASSPATH=/etc/hadoop/conf
to
export HBASE_CLASSPATH=$HBASE_CLASSPATH:/etc/hadoop/conf
2016-10-19 14:24 GMT+08:00 张磊 <12...@qq.com>:
> Hi when i build the sample kylin_sales_cube, in the kylin.log,i find
> java.lang.NoClassDefFoundError. but in the log i find some messages, i do
> not konw why?
>
>
> /usr/lib/hive-hcatalog/share/hcatalog/hive-hcatalog-core-2.1.0-amzn-0.jar
> is loaded, why class is not foung?
>
>
> 016-10-18 23:17:59,902 INFO [pool-8-thread-3]
> common.AbstractHadoopJob:203 : Hive Dependencies After Filtered:
> /usr/lib/hive/lib/hive-metastore-2.1.0-amzn-0.jar,/
> usr/lib/hive/lib/hive-exec-2.1.0-amzn-0.jar,/usr/lib/hive-
> hcatalog/share/hcatalog/hive-hcatalog-core-2.1.0-amzn-0.jar
>
>
> 2016-10-18 23:17:59,907 INFO [pool-8-thread-3]
> common.AbstractHadoopJob:371 : Job 'tmpjars' updated --
> file:/usr/lib/hive/lib/hive-metastore-2.1.0-amzn-0.jar,
> file:/usr/lib/hive/lib/hive-exec-2.1.0-amzn-0.jar,file:/
> usr/lib/hive-hcatalog/share/hcatalog/hive-hcatalog-core-2.1.0-amzn-0.jar
>
>
> 2016-10-18 23:17:59,934 ERROR [pool-8-thread-3]
> threadpool.DefaultScheduler:140 : ExecuteException
> job:845380a7-63b7-4315-8d71-6bb290018790
> org.apache.kylin.job.exception.ExecuteException: org.apache.kylin.job.exception.ExecuteException:
> java.lang.NoClassDefFoundError: org/apache/hive/hcatalog/
> mapreduce/HCatInputFormat
> at org.apache.kylin.job.execution.AbstractExecutable.
> execute(AbstractExecutable.java:123)
> at org.apache.kylin.job.impl.threadpool.DefaultScheduler$
> JobRunner.run(DefaultScheduler.java:136)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.kylin.job.exception.ExecuteException: java.lang.NoClassDefFoundError:
> org/apache/hive/hcatalog/mapreduce/HCatInputFormat
> at org.apache.kylin.job.execution.AbstractExecutable.
> execute(AbstractExecutable.java:123)
> at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(
> DefaultChainedExecutable.java:57)
> at org.apache.kylin.job.execution.AbstractExecutable.
> execute(AbstractExecutable.java:113)
> ... 4 more
> Caused by: java.lang.NoClassDefFoundError: org/apache/hive/hcatalog/
> mapreduce/HCatInputFormat
> at org.apache.kylin.source.hive.HiveMRInput$HiveTableInputFormat.
> configureJob(HiveMRInput.java:89)
> at org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.
> setupMapper(FactDistinctColumnsJob.java:123)
> at org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.
> run(FactDistinctColumnsJob.java:103)
> at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:88)
> at org.apache.kylin.engine.mr.common.MapReduceExecutable.
> doWork(MapReduceExecutable.java:120)
> at org.apache.kylin.job.execution.AbstractExecutable.
> execute(AbstractExecutable.java:113)
> ... 6 more
> Caused by: java.lang.ClassNotFoundException: org.apache.hive.hcatalog.
> mapreduce.HCatInputFormat
> at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(
> WebappClassLoaderBase.java:1858)
> at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(
> WebappClassLoaderBase.java:1701)
> ... 12 more
--
With Warm regards
Yiming Liu (刘一鸣)
java.lang.NoClassDefFoundError: org/apache/hive/hcatalog/mapreduce/HCatInputFormat
Posted by 张磊 <12...@qq.com>.
Hi when i build the sample kylin_sales_cube, in the kylin.log,i find java.lang.NoClassDefFoundError. but in the log i find some messages, i do not konw why?
/usr/lib/hive-hcatalog/share/hcatalog/hive-hcatalog-core-2.1.0-amzn-0.jar is loaded, why class is not foung?
016-10-18 23:17:59,902 INFO [pool-8-thread-3] common.AbstractHadoopJob:203 : Hive Dependencies After Filtered: /usr/lib/hive/lib/hive-metastore-2.1.0-amzn-0.jar,/usr/lib/hive/lib/hive-exec-2.1.0-amzn-0.jar,/usr/lib/hive-hcatalog/share/hcatalog/hive-hcatalog-core-2.1.0-amzn-0.jar
2016-10-18 23:17:59,907 INFO [pool-8-thread-3] common.AbstractHadoopJob:371 : Job 'tmpjars' updated -- file:/usr/lib/hive/lib/hive-metastore-2.1.0-amzn-0.jar,file:/usr/lib/hive/lib/hive-exec-2.1.0-amzn-0.jar,file:/usr/lib/hive-hcatalog/share/hcatalog/hive-hcatalog-core-2.1.0-amzn-0.jar
2016-10-18 23:17:59,934 ERROR [pool-8-thread-3] threadpool.DefaultScheduler:140 : ExecuteException job:845380a7-63b7-4315-8d71-6bb290018790
org.apache.kylin.job.exception.ExecuteException: org.apache.kylin.job.exception.ExecuteException: java.lang.NoClassDefFoundError: org/apache/hive/hcatalog/mapreduce/HCatInputFormat
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:123)
at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:136)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.kylin.job.exception.ExecuteException: java.lang.NoClassDefFoundError: org/apache/hive/hcatalog/mapreduce/HCatInputFormat
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:123)
at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:57)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:113)
... 4 more
Caused by: java.lang.NoClassDefFoundError: org/apache/hive/hcatalog/mapreduce/HCatInputFormat
at org.apache.kylin.source.hive.HiveMRInput$HiveTableInputFormat.configureJob(HiveMRInput.java:89)
at org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.setupMapper(FactDistinctColumnsJob.java:123)
at org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.run(FactDistinctColumnsJob.java:103)
at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:88)
at org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:120)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:113)
... 6 more
Caused by: java.lang.ClassNotFoundException: org.apache.hive.hcatalog.mapreduce.HCatInputFormat
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1858)
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1701)
... 12 more
Re: some question about kylin
Posted by ShaoFeng Shi <sh...@apache.org>.
kylin.sh doesn't have issue; What need to check is the "hbase" command in
hbase installation folder.
2016-10-18 16:04 GMT+08:00 张磊 <12...@qq.com>:
> hi
> I read the kylin.sh script, i find these sentences
>
>
> function retrieveDependency() export HBASE_CLASSPATH=${HBASE_
> CLASSPATH}:${hive_dependency}
>
>
> Do you mean i should set ${HBASE_CLASSPATH} value?
>
>
> And this value should be from the command hbase classpath?
>
>
>
>
>
>
> ------------------ 原始邮件 ------------------
> 发件人: "ShaoFeng Shi";<sh...@apache.org>;
> 发送时间: 2016年10月17日(星期一) 晚上10:00
> 收件人: "dev"<de...@kylin.apache.org>;
>
> 主题: Re: some question about kylin
>
>
>
> Hi,
>
> Usually this is caused that HBase's script didn't inheritate the
> "HBASE_CLASSPATH" environment varaible (Kylin depends on it to attache Hive
> jars to classpath).
>
> Go to the HBase installation path, and then grep "HBASE_CLASSPATH" in the
> `hbase` shell file, you should find something like:
>
> export
> HBASE_CLASSPATH=$HADOOP_CONF:$HADOOP_HOME/*:$HADOOP_HOME/
> lib/*:$ZOOKEEPER_HOME/*:$ZOOKEEPER_HOME/lib/*:$HBASE_CLASSPATH
>
> Above is a good sample, as it will keep the original value of
> $HBASE_CLASSPATH; But in some distribution like EMR 5.0, it just set the
> HBASE_CLASSPATH value instead of appending, then it will cause the hive jar
> not on Kylin's classpath. You need manually correct that.
>
> I will add this to FAQ page.
>
>
> 2016-10-17 17:03 GMT+08:00 张磊 <12...@qq.com>:
>
> > I install kylin(apache-kylin-1.5.4.1-hbase1.x-bin) in EMR5.0.0， while i
> > operate on web, there is some error, i do not install hcatalog in emr.
> How
> > can i resolve this question? Thanks!
> >
> > java.lang.NoClassDefFoundError: org/apache/hive/hcatalog/
> mapreduce/HCatInputFormat
> > at org.apache.kylin.source.hive.HiveMRInput$HiveTableInputFormat.
> > configureJob(HiveMRInput.java:89) at org.apache.kylin.engine.mr.
> > steps.FactDistinctColumnsJob.setupMapper(FactDistinctColumnsJob.java:
> 123)
> > at org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.
> > run(FactDistinctColumnsJob.java:103) at
> org.apache.kylin.engine.mr
> > .MRUtil.runMRJob(MRUtil.java:88) at org.apache.kylin.engine.mr.
> > common.MapReduceExecutable.doWork(MapReduceExecutable.java:120) at
> > org.apache.kylin.job.execution.AbstractExecutable.
> > execute(AbstractExecutable.java:113) at org.apache.kylin.job.
> > execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:
> 57)
> > at org.apache.kylin.job.execution.AbstractExecutable.
> > execute(AbstractExecutable.java:113) at org.apache.kylin.job.impl.
> > threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:136)
> > at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> >
> >
> >
> >
> > 上一封 下一封
> > « 返回
> >
> >
> >
> >
> >
> >
> >
> >
> > 上一封 下一封
>
>
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>
--
Best regards,
Shaofeng Shi 史少锋
回复： some question about kylin
Posted by 张磊 <12...@qq.com>.
hi
I read the kylin.sh script, i find these sentences
function retrieveDependency() export HBASE_CLASSPATH=${HBASE_CLASSPATH}:${hive_dependency}
Do you mean i should set ${HBASE_CLASSPATH} value?
And this value should be from the command hbase classpath?
------------------ 原始邮件 ------------------
发件人: "ShaoFeng Shi";<sh...@apache.org>;
发送时间: 2016年10月17日(星期一) 晚上10:00
收件人: "dev"<de...@kylin.apache.org>;
主题: Re: some question about kylin
Hi,
Usually this is caused that HBase's script didn't inheritate the
"HBASE_CLASSPATH" environment varaible (Kylin depends on it to attache Hive
jars to classpath).
Go to the HBase installation path, and then grep "HBASE_CLASSPATH" in the
`hbase` shell file, you should find something like:
export
HBASE_CLASSPATH=$HADOOP_CONF:$HADOOP_HOME/*:$HADOOP_HOME/lib/*:$ZOOKEEPER_HOME/*:$ZOOKEEPER_HOME/lib/*:$HBASE_CLASSPATH
Above is a good sample, as it will keep the original value of
$HBASE_CLASSPATH; But in some distribution like EMR 5.0, it just set the
HBASE_CLASSPATH value instead of appending, then it will cause the hive jar
not on Kylin's classpath. You need manually correct that.
I will add this to FAQ page.
2016-10-17 17:03 GMT+08:00 张磊 <12...@qq.com>:
> I install kylin(apache-kylin-1.5.4.1-hbase1.x-bin) in EMR5.0.0， while i
> operate on web, there is some error, i do not install hcatalog in emr. How
> can i resolve this question? Thanks!
>
> java.lang.NoClassDefFoundError: org/apache/hive/hcatalog/mapreduce/HCatInputFormat
> at org.apache.kylin.source.hive.HiveMRInput$HiveTableInputFormat.
> configureJob(HiveMRInput.java:89) at org.apache.kylin.engine.mr.
> steps.FactDistinctColumnsJob.setupMapper(FactDistinctColumnsJob.java:123)
> at org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.
> run(FactDistinctColumnsJob.java:103) at org.apache.kylin.engine.mr
> .MRUtil.runMRJob(MRUtil.java:88) at org.apache.kylin.engine.mr.
> common.MapReduceExecutable.doWork(MapReduceExecutable.java:120) at
> org.apache.kylin.job.execution.AbstractExecutable.
> execute(AbstractExecutable.java:113) at org.apache.kylin.job.
> execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:57)
> at org.apache.kylin.job.execution.AbstractExecutable.
> execute(AbstractExecutable.java:113) at org.apache.kylin.job.impl.
> threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:136)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
>
>
>
> 上一封 下一封
> « 返回
>
>
>
>
>
>
>
>
> 上一封 下一封
--
Best regards,
Shaofeng Shi 史少锋
Re: some question about kylin
Posted by ShaoFeng Shi <sh...@apache.org>.
Hi,
Usually this is caused that HBase's script didn't inheritate the
"HBASE_CLASSPATH" environment varaible (Kylin depends on it to attache Hive
jars to classpath).
Go to the HBase installation path, and then grep "HBASE_CLASSPATH" in the
`hbase` shell file, you should find something like:
export
HBASE_CLASSPATH=$HADOOP_CONF:$HADOOP_HOME/*:$HADOOP_HOME/lib/*:$ZOOKEEPER_HOME/*:$ZOOKEEPER_HOME/lib/*:$HBASE_CLASSPATH
Above is a good sample, as it will keep the original value of
$HBASE_CLASSPATH; But in some distribution like EMR 5.0, it just set the
HBASE_CLASSPATH value instead of appending, then it will cause the hive jar
not on Kylin's classpath. You need manually correct that.
I will add this to FAQ page.
2016-10-17 17:03 GMT+08:00 张磊 <12...@qq.com>:
> I install kylin(apache-kylin-1.5.4.1-hbase1.x-bin) in EMR5.0.0， while i
> operate on web, there is some error, i do not install hcatalog in emr. How
> can i resolve this question? Thanks!
>
> java.lang.NoClassDefFoundError: org/apache/hive/hcatalog/mapreduce/HCatInputFormat
> at org.apache.kylin.source.hive.HiveMRInput$HiveTableInputFormat.
> configureJob(HiveMRInput.java:89) at org.apache.kylin.engine.mr.
> steps.FactDistinctColumnsJob.setupMapper(FactDistinctColumnsJob.java:123)
> at org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.
> run(FactDistinctColumnsJob.java:103) at org.apache.kylin.engine.mr
> .MRUtil.runMRJob(MRUtil.java:88) at org.apache.kylin.engine.mr.
> common.MapReduceExecutable.doWork(MapReduceExecutable.java:120) at
> org.apache.kylin.job.execution.AbstractExecutable.
> execute(AbstractExecutable.java:113) at org.apache.kylin.job.
> execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:57)
> at org.apache.kylin.job.execution.AbstractExecutable.
> execute(AbstractExecutable.java:113) at org.apache.kylin.job.impl.
> threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:136)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
>
>
>
> 上一封 下一封
> « 返回
>
>
>
>
>
>
>
>
> 上一封 下一封
--
Best regards,
Shaofeng Shi 史少锋