You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kylin.apache.org by mvishnubhatta <mv...@argusinformation.com> on 2020/04/09 01:22:59 UTC
Kylin Start fails with NoClassDefFoundError on AWS EMR Edge Node
Hi,
I am trying to set up a Kylin server on an AWS EMR edge node. When doing
that the web URL throws the error that "The origin server did not find a
current representation for the target resource or is not willing to disclose
that one exists".
The tomcat localhost log file shows a NoClassDefFoundError for the class
org/apache/hadoop/hive/metastore/api/NoSuchObjectException
But I see this class in the hive library
/usr/lib/hive/lib/hive-metastore-2.3.5-amzn-0.jar and the path is exported
as HIVE_LIB and as HIVE_HOME (used HIVE_LIB based on suggestion here
https://issues.apache.org/jira/browse/KYLIN-2511).
This appears to be a case where some CLASSPATH somewhere is not correctly
including this path.
So, I tried modifying the kylin.properties file to add a property called
kylin.job.mr.lib.dir pointing to the /usr/lib/hive/lib/ but that didnt help
either. (This too was based on another suggested workaround here).
Can someone point out what I am missing?
Here are the full details:
Hadoop version is Hadoop 2.8.5-amzn-4
Kylin version is 3.0.1
Hive 2.3.5-amzn-0
I have an AWS EMR Cluster and a separate EC2 machine that is set up as an
edge node. From this edge node I am able to run hive queries to connect to
the EMR Master node. Hbase also works on this node.
On this edge node, I are trying to start a Kylin server. I followed the
instructions given here:
https://kylin.apache.org/docs/install/kylin_aws_emr.html
<https://kylin.apache.org/docs/install/kylin_aws_emr.html>
I exported the following variables:
export KYLIN_HOME=/usr/local/kylin/apache-kylin-3.0.1-bin-hbase1x
#export HADOOP_HOME=/usr/lib/hadoop/etc/hadoop
export HADOOP_HOME=/usr/lib/hadoop
export HBASE_HOME=/usr/lib/hbase/
export HIVE_HOME=/usr/lib/hive/lib/
export HADOOP_CONF_DIR=/etc/hadoop/conf
export HIVE_LIB=/usr/lib/hive/lib
export HIVE_CONF=/etc/hive/conf
export HCAT_HOME=/usr/lib/hive-hcatalog
The command I run is: $KYLIN_HOME/bin/kylin.sh start
After starting it, I get this message:
/A new Kylin instance is started by hadoop. To stop it, run 'kylin.sh stop'
Check the log at
/usr/local/kylin/apache-kylin-3.0.1-bin-hbase1x/logs/kylin.log
Web UI is at http://ip-xxxxxxxxxxxxxxxxxxxxx.compute.internal:7070/kylin/
When I navigate to the URL, I get the following error.
"The origin server did not find a current representation for the target
resource or is not willing to disclose that one exists."
When I look into the log file which is at
$KYLIN_HOME/tomcat/logs/localhost.2020-04-08.log I see the following error:
SEVERE: Exception sending context initialized event to listener instance of
class org.springframework.web.context.ContextLoaderListener
org.springframework.beans.factory.BeanCreationException: Error creating bean
with name
'org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerMapping':
Invocation of init method failed; nested exception is
java.lang.NoClassDefFoundError:
org/apache/hadoop/hive/metastore/api/NoSuchObjectException
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1628)
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:555)
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
at
org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at
org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
at
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
at
org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:761)
at
org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:867)
at
org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:543)
at
org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:443)
at
org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:325)
at
org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:107)
at
org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:5157)
at
org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5680)
at
org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
at
org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:1018)
at
org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:994)
at
org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652)
at
org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:1127)
at
org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:2021)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NoClassDefFoundError:
org/apache/hadoop/hive/metastore/api/NoSuchObjectException
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.getDeclaredMethods(Class.java:1975)
at
org.springframework.util.ReflectionUtils.getDeclaredMethods(ReflectionUtils.java:613)
at
org.springframework.util.ReflectionUtils.doWithMethods(ReflectionUtils.java:524)
at
org.springframework.core.MethodIntrospector.selectMethods(MethodIntrospector.java:68)
at
org.springframework.web.servlet.handler.AbstractHandlerMethodMapping.detectHandlerMethods(AbstractHandlerMethodMapping.java:230)
at
org.springframework.web.servlet.handler.AbstractHandlerMethodMapping.initHandlerMethods(AbstractHandlerMethodMapping.java:214)
at
org.springframework.web.servlet.handler.AbstractHandlerMethodMapping.afterPropertiesSet(AbstractHandlerMethodMapping.java:184)
at
org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerMapping.afterPropertiesSet(RequestMappingHandlerMapping.java:127)
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1687)
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1624)
... 25 more
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.hive.metastore.api.NoSuchObjectException
at
org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1928)
--
Sent from: http://apache-kylin.74782.x6.nabble.com/
Re: Re:Kylin Start fails with NoClassDefFoundError on AWS EMR Edge
Node
Posted by mvishnubhatta <mv...@argusinformation.com>.
I was finally able to get this to work on both the Master and the Edge nodes.
The change needed was already a known issue documented in the 1st FAQ here:
https://kylin.apache.org/docs15/gettingstarted/faq.html
The issue was that the the /usr/lib/hbase/conf/hbase-env.sh was exporting
CLASSPATH as below.
export HBASE_CLASSPATH=/etc/hadoop/conf
which needs to be changed to
export HBASE_CLASSPATH=/etc/hadoop/conf:$HBASE_CLASSPATH
Thanks for your time and help on this.
--
Sent from: http://apache-kylin.74782.x6.nabble.com/
Re: Re:Kylin Start fails with NoClassDefFoundError on AWS EMR Edge
Node
Posted by mvishnubhatta <mv...@argusinformation.com>.
Hi,
I had not tried setting this up on the Master node earlier, but when I tried
it on the Master node now, I still get the same error in tomcat/localhost
log file.
/
SEVERE: Exception sending context initialized event to listener instance of
class org.springframework.web.context.ContextLoaderListener
org.springframework.beans.factory.BeanCreationException: Error creating bean
with name
'org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerMapping':
Invocation of init method failed; nested exception is
java.lang.NoClassDefFoundError:
org/apache/hadoop/hive/metastore/api/NoSuchObjectException/
Madhav.
--
Sent from: http://apache-kylin.74782.x6.nabble.com/
Re:Kylin Start fails with NoClassDefFoundError on AWS EMR Edge Node
Posted by Xiaoxiang Yu <xx...@apache.org>.
Hi friend,
I have a suggestion and I am not sure if it is the best practice, and I don't know if it works. But maybe you can have a try if you are willing to do it.
I guess maybe you have installed Kylin successfully on Master node of EMR cluster, right? After kylin instance was started successfully(if you are using the kylin 3.0 or above), you will see some files with cached-*-dependency.sh under $KYLIN_HOME/bin, in all these files, you can find the location of jars that is needed by Kylin. You may try copy the jars from master node to edge node in the same folder, then restart Kylin in edge node.
On the another hand, you don't need to modify the property "kylin.job.mr.lib.dir" when starting a Kylin instance.
Following :
cached-hadoop-conf-dir.sh
cached-hbase-dependency.sh
cached-hive-dependency.sh
cached-kafka-dependency.sh
cached-spark-dependency.sh
[root@cdh-master all]# cat bin/cached-hive-dependency.sh
export hive_dependency=/opt/cloudera/parcels/CDH-5.7.6-1.cdh5.7.6.p0.6/bin/../lib/hive/conf:/opt/cloudera/parcels/CDH-5.7.6-1.cdh5.7.6.p0.6/bin/../lib/hive/lib/hive-jdbc-standalone.jar:/opt/cloudera/parcels/CDH-5.7.6-1.cdh5.7.6.p0.6/bin/../lib/hive/lib/junit-4.11.jar:/opt/cloudera/parcels/CDH-5.7.6-1.cdh5.7.6.p0.6/bin/../lib/hive/lib/commons-collections-3.2.2.jar:/opt/cloudera/parcels/CDH-5.7.6-1.cdh5.7.6.p0.6/bin/../lib/hive/lib/pentaho-aggdesigner-algorithm-5.1.5-jhyde.jar:/opt/cloudera/parcels/CDH-5.7.6-1.cdh5.7.6.p0.6/bin/../lib/hive/lib/hive-service.jar:/opt/cloudera/parcels/CDH-5.7.6-1.cdh5.7.6.p0.6/bin/../lib/hive/lib/metrics-jvm-3.0.2.jar:/opt/cloudera/parcels/CDH-5.7.6-1.cdh5.7.6.p0.6/bin/../lib/hive/lib/ant-launcher-1.9.1.jar:/opt/cloudera/parcels/CDH-5.7.6-1.cdh5.7.6.p0.6/bin/../lib/hive/lib/log4j-1.2.16.jar:/opt/cloudera/parcels/CDH-5.7.6-1.cdh5.7.6.p0.6/bin/../lib/hive/lib/datanucleus-api-jdo-3.2.6.jar:/opt/cloudera/parcels/CDH-5.7.6-1.cdh5.7.6.p0.6/bin/../lib/hive/lib/hive-cli-1.1.0-cdh5.7.6.jar:/opt/cloudera/parcels/CDH-5.7.6-1.cdh5.7.6.p0.6/bin/../lib/hive/lib/hive-ant.jar:/opt/cloudera/parcels/CDH-5.7.6-1.cdh5.7.6.p0.6/bin/../lib/hive/lib
.....
....
--
Best wishes to you !
From ：Xiaoxiang Yu
At 2020-04-09 09:22:59, "mvishnubhatta" <mv...@argusinformation.com> wrote:
>Hi,
>
>I am trying to set up a Kylin server on an AWS EMR edge node. When doing
>that the web URL throws the error that "The origin server did not find a
>current representation for the target resource or is not willing to disclose
>that one exists".
>
>The tomcat localhost log file shows a NoClassDefFoundError for the class
>org/apache/hadoop/hive/metastore/api/NoSuchObjectException
>
>But I see this class in the hive library
>/usr/lib/hive/lib/hive-metastore-2.3.5-amzn-0.jar and the path is exported
>as HIVE_LIB and as HIVE_HOME (used HIVE_LIB based on suggestion here
>https://issues.apache.org/jira/browse/KYLIN-2511).
>
>This appears to be a case where some CLASSPATH somewhere is not correctly
>including this path.
>
>So, I tried modifying the kylin.properties file to add a property called
>kylin.job.mr.lib.dir pointing to the /usr/lib/hive/lib/ but that didnt help
>either. (This too was based on another suggested workaround here).
>
>Can someone point out what I am missing?
>
>Here are the full details:
>
>Hadoop version is Hadoop 2.8.5-amzn-4
>Kylin version is 3.0.1
>Hive 2.3.5-amzn-0
>
>I have an AWS EMR Cluster and a separate EC2 machine that is set up as an
>edge node. From this edge node I am able to run hive queries to connect to
>the EMR Master node. Hbase also works on this node.
>
>On this edge node, I are trying to start a Kylin server. I followed the
>instructions given here:
>https://kylin.apache.org/docs/install/kylin_aws_emr.html
><https://kylin.apache.org/docs/install/kylin_aws_emr.html>
>
>I exported the following variables:
>export KYLIN_HOME=/usr/local/kylin/apache-kylin-3.0.1-bin-hbase1x
>#export HADOOP_HOME=/usr/lib/hadoop/etc/hadoop
>export HADOOP_HOME=/usr/lib/hadoop
>export HBASE_HOME=/usr/lib/hbase/
>export HIVE_HOME=/usr/lib/hive/lib/
>export HADOOP_CONF_DIR=/etc/hadoop/conf
>export HIVE_LIB=/usr/lib/hive/lib
>export HIVE_CONF=/etc/hive/conf
>export HCAT_HOME=/usr/lib/hive-hcatalog
>
>
>The command I run is: $KYLIN_HOME/bin/kylin.sh start
>
>After starting it, I get this message:
>
>/A new Kylin instance is started by hadoop. To stop it, run 'kylin.sh stop'
>Check the log at
>/usr/local/kylin/apache-kylin-3.0.1-bin-hbase1x/logs/kylin.log
>Web UI is at http://ip-xxxxxxxxxxxxxxxxxxxxx.compute.internal:7070/kylin/
>
>When I navigate to the URL, I get the following error.
>
>"The origin server did not find a current representation for the target
>resource or is not willing to disclose that one exists."
>
>When I look into the log file which is at
>$KYLIN_HOME/tomcat/logs/localhost.2020-04-08.log I see the following error:
>
>SEVERE: Exception sending context initialized event to listener instance of
>class org.springframework.web.context.ContextLoaderListener
>org.springframework.beans.factory.BeanCreationException: Error creating bean
>with name
>'org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerMapping':
>Invocation of init method failed; nested exception is
>java.lang.NoClassDefFoundError:
>org/apache/hadoop/hive/metastore/api/NoSuchObjectException
> at
>org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1628)
> at
>org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:555)
> at
>org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
> at
>org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
> at
>org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
> at
>org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
> at
>org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
> at
>org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:761)
> at
>org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:867)
> at
>org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:543)
> at
>org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:443)
> at
>org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:325)
> at
>org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:107)
> at
>org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:5157)
> at
>org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5680)
> at
>org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
> at
>org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:1018)
> at
>org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:994)
> at
>org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652)
> at
>org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:1127)
> at
>org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:2021)
> at
>java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
>Caused by: java.lang.NoClassDefFoundError:
>org/apache/hadoop/hive/metastore/api/NoSuchObjectException
> at java.lang.Class.getDeclaredMethods0(Native Method)
> at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
> at java.lang.Class.getDeclaredMethods(Class.java:1975)
> at
>org.springframework.util.ReflectionUtils.getDeclaredMethods(ReflectionUtils.java:613)
> at
>org.springframework.util.ReflectionUtils.doWithMethods(ReflectionUtils.java:524)
> at
>org.springframework.core.MethodIntrospector.selectMethods(MethodIntrospector.java:68)
> at
>org.springframework.web.servlet.handler.AbstractHandlerMethodMapping.detectHandlerMethods(AbstractHandlerMethodMapping.java:230)
> at
>org.springframework.web.servlet.handler.AbstractHandlerMethodMapping.initHandlerMethods(AbstractHandlerMethodMapping.java:214)
> at
>org.springframework.web.servlet.handler.AbstractHandlerMethodMapping.afterPropertiesSet(AbstractHandlerMethodMapping.java:184)
> at
>org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerMapping.afterPropertiesSet(RequestMappingHandlerMapping.java:127)
> at
>org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1687)
> at
>org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1624)
> ... 25 more
>Caused by: java.lang.ClassNotFoundException:
>org.apache.hadoop.hive.metastore.api.NoSuchObjectException
> at
>org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1928)
>
>
>
>
>--
>Sent from: http://apache-kylin.74782.x6.nabble.com/
Re: Kylin Start fails with NoClassDefFoundError on AWS EMR Edge
Node
Posted by mvishnubhatta <mv...@argusinformation.com>.
Thanks for the response. I tried /usr/lib/hive and it did not work. So I was
just trying out with /usr/lib/hive/lib. I checked again by changing it back
to /usr/lib/hive but it doesn't work.
I tried running a hive query and it does work (with /usr/lib/hive as
HVIE_HOME). Here is the test, for your information.
Logging initialized using configuration in
file:/etc/hive/conf.dist/hive-log4j2.properties Async: true
hive> INSERT INTO test_emr_from_airflow select '20200407','x','y',
CURRENT_TIMESTAMP;
Query ID = hadoop_20200409131925_720f87d1-067e-47be-80f6-5b7e296b0c79
Total jobs = 1
Launching Job 1 out of 1
Status: Running (Executing on YARN cluster with App id
application_1585860045700_0150)
----------------------------------------------------------------------------------------------
VERTICES MODE STATUS TOTAL COMPLETED RUNNING PENDING
FAILED KILLED
----------------------------------------------------------------------------------------------
Map 1 .......... container SUCCEEDED 1 1 0 0
0 0
----------------------------------------------------------------------------------------------
VERTICES: 01/01 [==========================>>] 100% ELAPSED TIME: 7.38 s
----------------------------------------------------------------------------------------------
Loading data to table default.test_emr_from_airflow
OK
Time taken: 20.152 seconds
hive>
--
Sent from: http://apache-kylin.74782.x6.nabble.com/
Re: Kylin Start fails with NoClassDefFoundError on AWS EMR Edge Node
Posted by Yaqian Zhang <Ya...@126.com>.
Hi：
I see that HIVE_HOME=/usr/lib/hive/lib/ in your environment variables, why not /usr/lib/hive/?
> 在 2020年4月9日，09:22，mvishnubhatta <mv...@argusinformation.com> 写道：
>
> /usr/lib/hive/l
Re: Kylin Start fails with NoClassDefFoundError on AWS EMR Edge Node
Posted by Yaqian Zhang <Ya...@126.com>.
Hi:
Can you execute the select command in hive to check if hive is working properly？
> 在 2020年4月9日，09:22，mvishnubhatta <mv...@argusinformation.com> 写道：
>
> org/apache/hadoop/hive/metastore/api/NoSuchObjectException