You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@kylin.apache.org by ShaoFeng Shi <sh...@apache.org> on 2017/12/22 09:49:11 UTC
Re: Kylin on EMR Installation Error
Please check https://issues.apache.org/jira/browse/KYLIN-3129
2017-09-06 22:31 GMT+08:00 ShaoFeng Shi <sh...@apache.org>:
> Thanks, I will try EMR 5.8 later;
>
> At this moment, you can try emr 5.5; one thing to pay attention is, you
> need copy a configuration from hbase-site.xml to KYLIN_HOME/conf/kylin_job_conf.xml
> (replace the zk quorum name with yours):
>
> <property>
> <name>hbase.zookeeper.quorum</name>
> <value>ip-nn-nn-nn-nn.ap-northeast-2.compute.internal</value>
> </property>
>
>
> 2017-09-06 21:07 GMT+08:00 Mukund Agilisium <mu...@gmail.com>:
>
>> Hi
>>
>> I am using EMR 5.8.0 , the latest one.
>>
>> Thanks & Regards
>> Mukund
>>
>> On Wed, Sep 6, 2017 at 8:44 AM, ShaoFeng Shi <sh...@apache.org>
>> wrote:
>>
>>> Hi Mukund,
>>>
>>> What's the EMR version?
>>>
>>> Check if there is other version's jar in EMR node, e.g,:
>>>
>>> sudo find / -name joda-time*.jar
>>>
>>>
>>> 2017-09-06 1:52 GMT+08:00 Mukund Agilisium <mu...@gmail.com>:
>>>
>>>> Hi All
>>>>
>>>> I am trying to install Kylin apache-kylin-2.1.0-bin-hbase1x on EMR and
>>>> getting the following errors.
>>>>
>>>> [root@ip-172-31-35-25 lib]# pwd
>>>> /etc/apache-kylin-2.1.0-bin-hbase1x/tomcat/webapps/kylin/WEB-INF/lib
>>>> [root@ip-172-31-35-25 lib]# ls -al | grep joda
>>>> -rw-r--r-- 1 root root 573912 May 6 2016 joda-time-2.2.jar
>>>>
>>>> Looks like the jar is available inside the webinf/lib as well.
>>>>
>>>> Any help is appreciated!!
>>>>
>>>> *kylin.log*
>>>> 2017-09-05 12:06:45,097 ERROR [localhost-startStop-1]
>>>> context.ContextLoader:351 : Context initialization failed
>>>> org.springframework.beans.factory.BeanCreationException: Error
>>>> creating bean with name 'org.springframework.web.servl
>>>> et.mvc.method.annotation.RequestMappingHandlerAdapter': Instantiation
>>>> of bean failed; nested exception is org.springframework.beans.BeanInstantiationException:
>>>> Failed to instantiate [org.springframework.web.servl
>>>> et.mvc.method.annotation.RequestMappingHandlerAdapter]: Constructor
>>>> threw exception; nested exception is java.lang.NoSuchMethodError:
>>>> org.joda.time.format.DateTimeFormatter.withZoneUTC()Lorg/jod
>>>> a/time/format/DateTimeFormatter;
>>>> at org.springframework.beans.factory.support.AbstractAutowireCa
>>>> pableBeanFactory.instantiateBean(AbstractAutowireCapableBean
>>>> Factory.java:1109)
>>>> at org.springframework.beans.factory.support.AbstractAutowireCa
>>>> pableBeanFactory.createBeanInstance(AbstractAutowireCapableB
>>>> eanFactory.java:1054)
>>>> at org.springframework.beans.factory.support.AbstractAutowireCa
>>>> pableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFac
>>>> tory.java:510)
>>>> at org.springframework.beans.factory.support.AbstractAutowireCa
>>>> pableBeanFactory.createBean(AbstractAutowireCapableBeanFacto
>>>> ry.java:482)
>>>> at org.springframework.beans.factory.support.AbstractBeanFactor
>>>> y$1.getObject(AbstractBeanFactory.java:306)
>>>> at org.springframework.beans.factory.support.DefaultSingletonBe
>>>> anRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
>>>> at org.springframework.beans.factory.support.AbstractBeanFactor
>>>> y.doGetBean(AbstractBeanFactory.java:302)
>>>> at org.springframework.beans.factory.support.AbstractBeanFactor
>>>> y.getBean(AbstractBeanFactory.java:197)
>>>> at org.springframework.beans.factory.support.DefaultListableBea
>>>> nFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:778)
>>>> at org.springframework.context.support.AbstractApplicationConte
>>>> xt.finishBeanFactoryInitialization(AbstractApplicationContext.java:839)
>>>> at org.springframework.context.support.AbstractApplicationConte
>>>> xt.refresh(AbstractApplicationContext.java:538)
>>>> at org.springframework.web.context.ContextLoader.configureAndRe
>>>> freshWebApplicationContext(ContextLoader.java:444)
>>>> at org.springframework.web.context.ContextLoader.initWebApplica
>>>> tionContext(ContextLoader.java:326)
>>>> ......
>>>>
>>>>
>>>> Caused by: org.springframework.beans.BeanInstantiationException:
>>>> Failed to instantiate [org.springframework.web.servl
>>>> et.mvc.method.annotation.RequestMappingHandlerAdapter]: Constructor
>>>> threw exception; nested exception is java.lang.NoSuchMethodError:
>>>> org.joda.time.format.DateTimeFormatter.withZoneUTC()Lorg/jod
>>>> a/time/format/DateTimeFormatter;
>>>> at org.springframework.beans.BeanUtils.instantiateClass(BeanUti
>>>> ls.java:163)
>>>> at org.springframework.beans.factory.support.SimpleInstantiatio
>>>> nStrategy.instantiate(SimpleInstantiationStrategy.java:89)
>>>> at org.springframework.beans.factory.support.AbstractAutowireCa
>>>> pableBeanFactory.instantiateBean(AbstractAutowireCapableBean
>>>> Factory.java:1102)
>>>> ... 26 more
>>>>
>>>>
>>>> SEVERE: Exception sending context initialized event to listener
>>>> instance of class org.springframework.web.context.ContextLoaderListener
>>>> org.springframework.beans.factory.BeanCreationException: Error
>>>> creating bean with name 'org.springframework.web.servl
>>>> et.mvc.method.annotation.RequestMappingHandlerAdapter': Instantiation
>>>> of bean failed; nested exception is org.springframework.beans.BeanInstantiationException:
>>>> Failed to instantiate [org.springframework.web.servl
>>>> et.mvc.method.annotation.RequestMappingHandlerAdapter]: Constructor
>>>> threw exception; nested exception is java.lang.NoSuchMethodError:
>>>> org.joda.time.format.DateTimeFormatter.withZoneUTC()Lorg/jod
>>>> a/time/format/DateTimeFormatter;
>>>> at org.springframework.beans.factory.support.AbstractAutowireCa
>>>> pableBeanFactory.instantiateBean(AbstractAutowireCapableBean
>>>> Factory.java:1109)
>>>> at org.springframework.beans.factory.support.AbstractAutowireCa
>>>> pableBeanFactory.createBeanInstance(AbstractAutowireCapableB
>>>> eanFactory.java:1054)
>>>> at org.springframework.beans.factory.support.AbstractAutowireCa
>>>> pableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFac
>>>> tory.java:510)
>>>> at org.springframework.beans.factory.support.AbstractAutowireCa
>>>> pableBeanFactory.createBean(AbstractAutowireCapableBeanFacto
>>>> ry.java:482)
>>>> at org.springframework.beans.factory.support.AbstractBeanFactor
>>>> y$1.getObject(AbstractBeanFactory.java:306)
>>>> at org.springframework.beans.factory.support.DefaultSingletonBe
>>>> anRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
>>>> at org.springframework.beans.factory.support.AbstractBeanFactor
>>>> y.doGetBean(AbstractBeanFactory.java:302)
>>>> at org.springframework.beans.factory.support.AbstractBeanFactor
>>>> y.getBean(AbstractBeanFactory.java:197)
>>>> at org.springframework.beans.factory.support.DefaultListableBea
>>>> nFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:778)
>>>> at org.springframework.context.support.AbstractApplicationConte
>>>> xt.finishBeanFactoryInitialization(AbstractApplicationContext.java:839)
>>>>
>>>> *kylin.out*
>>>>
>>>> There is not keystore files in the folder
>>>>
>>>> [root@ip-172-31-35-25 conf]# pwd
>>>> /etc/apache-kylin-2.1.0-bin-hbase1x/tomcat/conf
>>>> [root@ip-172-31-35-25 conf]# ls *.keystore
>>>> ls: cannot access *.keystore: No such file or directory
>>>>
>>>>
>>>> SEVERE: Failed to load keystore type JKS with path conf/.keystore due
>>>> to /etc/apache-kylin-2.1.0-bin-hbase1x/tomcat/conf/.keystore (No such
>>>> file or directory)
>>>> java.io.FileNotFoundException: /etc/apache-kylin-2.1.0-bin-hbase1x/tomcat/conf/.keystore
>>>> (No such file or directory)
>>>> at java.io.FileInputStream.open0(Native Method)
>>>> at java.io.FileInputStream.open(FileInputStream.java:195)
>>>> at java.io.FileInputStream.<init>(FileInputStream.java:138)
>>>> at java.io.FileInputStream.<init>(FileInputStream.java:93)
>>>>
>>>>
>>>> Sep 05, 2017 12:06:38 PM org.apache.catalina.core.StandardService
>>>> initInternal
>>>> SEVERE: Failed to initialize connector [Connector[HTTP/1.1-7443]]
>>>> org.apache.catalina.LifecycleException: Failed to initialize component
>>>> [Connector[HTTP/1.1-7443]]
>>>> at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.ja
>>>> va:106)
>>>> at org.apache.catalina.core.StandardService.initInternal(Standa
>>>> rdService.java:560)
>>>> at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.ja
>>>> va:102)
>>>> at org.apache.catalina.core.StandardServer.initInternal(Standar
>>>> dServer.java:838)
>>>> at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.ja
>>>> va:102)
>>>>
>>>
>>>
>>>
>>> --
>>> Best regards,
>>>
>>> Shaofeng Shi 史少锋
>>>
>>>
>>
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>
>
--
Best regards,
Shaofeng Shi 史少锋
Re: Kylin on EMR Installation Error
Posted by ShaoFeng Shi <sh...@apache.org>.
That's good; Thanks for the updating. Enjoy Kylin!
2017-12-25 15:39 GMT+08:00 朱真龙 <18...@qq.com>:
> Hi, ShaoFeng,
>
> I have add the patch to my kylin2.1 cluster which visit two
> versions of hadoop(hdp2.3.2 and hdp2.6.0 ) , now the bug is resolved .
> thank you for your help .
>
> ------------------ 原始邮件 ------------------
> *发件人:* "ShaoFeng Shi";<sh...@apache.org>;
> *发送时间:* 2017年12月25日(星期一) 中午1:59
> *收件人:* "user"<us...@kylin.apache.org>;
> *主题:* Re: Kylin on EMR Installation Error
>
> Hi zhenlong,
>
> I'm afraid that patch is not all the changes; suggest you download the
> formal 2.2 release.
>
> 2017-12-25 11:53 GMT+08:00 朱真龙 <18...@qq.com>:
>
>> Hi, ShaoFeng,
>>
>> thank you , I'll add the patch to kylin2.1 and retry . thank you
>> again.
>>
>>
>> ------------------ 原始邮件 ------------------
>> *发件人:* "ShaoFeng Shi";<sh...@apache.org>;
>> *发送时间:* 2017年12月25日(星期一) 中午11:09
>> *收件人:* "user"<us...@kylin.apache.org>;
>> *主题:* Re: Kylin on EMR Installation Error
>>
>> Hi zhenlong,
>>
>> Thanks for the detailed analysis. You're correct, there was some
>> inconsistency in Kylin 2.1, it should have been fixed in 2.2, could you
>> please upgrade Kylin and then verify?
>>
>> The related JIRA is:
>> https://issues.apache.org/jira/browse/KYLIN-2865
>> https://issues.apache.org/jira/browse/KYLIN-2766
>>
>> 2017-12-25 10:56 GMT+08:00 朱真龙 <18...@qq.com>:
>>
>>> thank you for your response.after check https://issues.apache.o
>>> rg/jira/browse/KYLIN-3129 ,I find it's not the same problem as mine
>>> and I have found the way to resolved my problem. I used kylin between two
>>> hadoop cluster ,like this:
>>>
>>>
>>>
>>> its works well when model is not too big. but some model with many row
>>> keys does not . when I configured fifteen columns for rowkey and setted
>>> dict encode like this :
>>> when running to "Build Dimension Dictionary" step, I got error as
>>> follow( it means after writted snapshot to hdfs, when submit these hdfs
>>> file to hbase ,the files and hbaseTable are in different hdfs cluster , one
>>> in cluster 1 and one in cluster 2):
>>>
>>>
>>> through looked into source code the error referred, I found that ,when
>>> writting "Dimension Dictionary" to hbase , kylin will judge if your
>>> keyvalue is larger than the configured Max_KeyValue size
>>> (hbase.client.keyvalue.maxsize is setted in hbase-site.xml),if you not
>>> setted ,it will get 10485760(10MB) as default . if your keyValue size is
>>> smaller than hbase.client.keyvalue.maxsize ,then kylin
>>> will set your key and value into hbase's put object then submit. if your
>>> keyvalue is bigger than hbase.client.keyvalue.maxsize ,it will write
>>> snapshot into hdfs first then submit it via hbase table. but I think it's
>>> maybe a bug on kylin2.1 when use kylin between two hadoop cluster and
>>> build Dimension with keyvalue bigger than hbase.client.keyvalue.maxsize.
>>>
>>>
>>> when keyvlaue size is bigger than hbase.client.keyvalue.maxsize , writte
>>> snapshot into hdfs( it got the hdfs path from hive table instead of hbase,
>>> and they are in different hdfs cluster) . so if you are using one hadoop
>>> cluster(hive and hbase are on one hadoop cluster ),it will works well . if
>>> you are using two hadoop cluster and your keyvalue size is smaller than
>>> hbase.client.keyvalue.maxsize ,it will also works well (use hbase's put
>>> object). but if you are using two hadoop cluster and your keyvalue is
>>> bigger than hbase.client.keyvalue.maxsize , you will get then same error
>>> like mine(beause your hive and hbase are in different hdfs cluster) .
>>>
>>> I checked my hbase.client.keyvalue.maxsize in hbase-site.xml,
>>> and found ambari set 1MB as default ,and kylin set 10MB as default .so I
>>> changed this value to 10MB , and now my model works well. If you find a
>>> better way to resolve this problem ,please tell me , thank you.
>>>
>>>
>>> apache-kylin-2.1.0-src\apache-kylin-2.1.0\storage-hbase\src\
>>> main\java\org\apache\kylin\storage\hbase\HBaseResourceStore.java
>>>
>>>
>>>
>>>
>>> ------------------ 原始邮件 ------------------
>>> *发件人:* "ShaoFeng Shi";<sh...@apache.org>;
>>> *发送时间:* 2017年12月22日(星期五) 下午5:49
>>> *收件人:* "user"<us...@kylin.apache.org>;
>>> *主题:* Re: Kylin on EMR Installation Error
>>>
>>> Please check https://issues.apache.org/jira/browse/KYLIN-3129
>>>
>>> 2017-09-06 22:31 GMT+08:00 ShaoFeng Shi <sh...@apache.org>:
>>>
>>>> Thanks, I will try EMR 5.8 later;
>>>>
>>>> At this moment, you can try emr 5.5; one thing to pay attention is, you
>>>> need copy a configuration from hbase-site.xml to
>>>> KYLIN_HOME/conf/kylin_job_conf.xml (replace the zk quorum name with
>>>> yours):
>>>>
>>>> <property>
>>>> <name>hbase.zookeeper.quorum</name>
>>>> <value>ip-nn-nn-nn-nn.ap-northeast-2.compute.internal</value>
>>>> </property>
>>>>
>>>>
>>>> 2017-09-06 21:07 GMT+08:00 Mukund Agilisium <mukund.agilisium@gmail.com
>>>> >:
>>>>
>>>>> Hi
>>>>>
>>>>> I am using EMR 5.8.0 , the latest one.
>>>>>
>>>>> Thanks & Regards
>>>>> Mukund
>>>>>
>>>>> On Wed, Sep 6, 2017 at 8:44 AM, ShaoFeng Shi <sh...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> Hi Mukund,
>>>>>>
>>>>>> What's the EMR version?
>>>>>>
>>>>>> Check if there is other version's jar in EMR node, e.g,:
>>>>>>
>>>>>> sudo find / -name joda-time*.jar
>>>>>>
>>>>>>
>>>>>> 2017-09-06 1:52 GMT+08:00 Mukund Agilisium <
>>>>>> mukund.agilisium@gmail.com>:
>>>>>>
>>>>>>> Hi All
>>>>>>>
>>>>>>> I am trying to install Kylin apache-kylin-2.1.0-bin-hbase1x on EMR
>>>>>>> and getting the following errors.
>>>>>>>
>>>>>>> [root@ip-172-31-35-25 lib]# pwd
>>>>>>> /etc/apache-kylin-2.1.0-bin-hbase1x/tomcat/webapps/kylin/WEB-INF/lib
>>>>>>> [root@ip-172-31-35-25 lib]# ls -al | grep joda
>>>>>>> -rw-r--r-- 1 root root 573912 May 6 2016 joda-time-2.2.jar
>>>>>>>
>>>>>>> Looks like the jar is available inside the webinf/lib as well.
>>>>>>>
>>>>>>> Any help is appreciated!!
>>>>>>>
>>>>>>> *kylin.log*
>>>>>>> 2017-09-05 12:06:45,097 ERROR [localhost-startStop-1]
>>>>>>> context.ContextLoader:351 : Context initialization failed
>>>>>>> org.springframework.beans.factory.BeanCreationException: Error
>>>>>>> creating bean with name 'org.springframework.web.servl
>>>>>>> et.mvc.method.annotation.RequestMappingHandlerAdapter':
>>>>>>> Instantiation of bean failed; nested exception is
>>>>>>> org.springframework.beans.BeanInstantiationException: Failed to
>>>>>>> instantiate [org.springframework.web.servl
>>>>>>> et.mvc.method.annotation.RequestMappingHandlerAdapter]: Constructor
>>>>>>> threw exception; nested exception is java.lang.NoSuchMethodError:
>>>>>>> org.joda.time.format.DateTimeFormatter.withZoneUTC()Lorg/jod
>>>>>>> a/time/format/DateTimeFormatter;
>>>>>>> at org.springframework.beans.fact
>>>>>>> ory.support.AbstractAutowireCapableBeanFactory.instantiateBe
>>>>>>> an(AbstractAutowireCapableBeanFactory.java:1109)
>>>>>>> at org.springframework.beans.fact
>>>>>>> ory.support.AbstractAutowireCapableBeanFactory.createBeanIns
>>>>>>> tance(AbstractAutowireCapableBeanFactory.java:1054)
>>>>>>> at org.springframework.beans.fact
>>>>>>> ory.support.AbstractAutowireCapableBeanFactory.doCreateBean(
>>>>>>> AbstractAutowireCapableBeanFactory.java:510)
>>>>>>> at org.springframework.beans.fact
>>>>>>> ory.support.AbstractAutowireCapableBeanFactory.createBean(Ab
>>>>>>> stractAutowireCapableBeanFactory.java:482)
>>>>>>> at org.springframework.beans.fact
>>>>>>> ory.support.AbstractBeanFactory$1.getObject(AbstractBeanFact
>>>>>>> ory.java:306)
>>>>>>> at org.springframework.beans.fact
>>>>>>> ory.support.DefaultSingletonBeanRegistry.getSingleton(Defaul
>>>>>>> tSingletonBeanRegistry.java:230)
>>>>>>> at org.springframework.beans.fact
>>>>>>> ory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactor
>>>>>>> y.java:302)
>>>>>>> at org.springframework.beans.fact
>>>>>>> ory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.
>>>>>>> java:197)
>>>>>>> at org.springframework.beans.fact
>>>>>>> ory.support.DefaultListableBeanFactory.preInstantiateSinglet
>>>>>>> ons(DefaultListableBeanFactory.java:778)
>>>>>>> at org.springframework.context.su
>>>>>>> pport.AbstractApplicationContext.finishBeanFactoryInitializa
>>>>>>> tion(AbstractApplicationContext.java:839)
>>>>>>> at org.springframework.context.su
>>>>>>> pport.AbstractApplicationContext.refresh(AbstractApplication
>>>>>>> Context.java:538)
>>>>>>> at org.springframework.web.contex
>>>>>>> t.ContextLoader.configureAndRefreshWebApplicationContext(Con
>>>>>>> textLoader.java:444)
>>>>>>> at org.springframework.web.contex
>>>>>>> t.ContextLoader.initWebApplicationContext(ContextLoader.java:326)
>>>>>>> ......
>>>>>>>
>>>>>>>
>>>>>>> Caused by: org.springframework.beans.BeanInstantiationException:
>>>>>>> Failed to instantiate [org.springframework.web.servl
>>>>>>> et.mvc.method.annotation.RequestMappingHandlerAdapter]: Constructor
>>>>>>> threw exception; nested exception is java.lang.NoSuchMethodError:
>>>>>>> org.joda.time.format.DateTimeFormatter.withZoneUTC()Lorg/jod
>>>>>>> a/time/format/DateTimeFormatter;
>>>>>>> at org.springframework.beans.Bean
>>>>>>> Utils.instantiateClass(BeanUtils.java:163)
>>>>>>> at org.springframework.beans.fact
>>>>>>> ory.support.SimpleInstantiationStrategy.instantiate(SimpleIn
>>>>>>> stantiationStrategy.java:89)
>>>>>>> at org.springframework.beans.fact
>>>>>>> ory.support.AbstractAutowireCapableBeanFactory.instantiateBe
>>>>>>> an(AbstractAutowireCapableBeanFactory.java:1102)
>>>>>>> ... 26 more
>>>>>>>
>>>>>>>
>>>>>>> SEVERE: Exception sending context initialized event to listener
>>>>>>> instance of class org.springframework.web.contex
>>>>>>> t.ContextLoaderListener
>>>>>>> org.springframework.beans.factory.BeanCreationException: Error
>>>>>>> creating bean with name 'org.springframework.web.servl
>>>>>>> et.mvc.method.annotation.RequestMappingHandlerAdapter':
>>>>>>> Instantiation of bean failed; nested exception is
>>>>>>> org.springframework.beans.BeanInstantiationException: Failed to
>>>>>>> instantiate [org.springframework.web.servl
>>>>>>> et.mvc.method.annotation.RequestMappingHandlerAdapter]: Constructor
>>>>>>> threw exception; nested exception is java.lang.NoSuchMethodError:
>>>>>>> org.joda.time.format.DateTimeFormatter.withZoneUTC()Lorg/jod
>>>>>>> a/time/format/DateTimeFormatter;
>>>>>>> at org.springframework.beans.fact
>>>>>>> ory.support.AbstractAutowireCapableBeanFactory.instantiateBe
>>>>>>> an(AbstractAutowireCapableBeanFactory.java:1109)
>>>>>>> at org.springframework.beans.fact
>>>>>>> ory.support.AbstractAutowireCapableBeanFactory.createBeanIns
>>>>>>> tance(AbstractAutowireCapableBeanFactory.java:1054)
>>>>>>> at org.springframework.beans.fact
>>>>>>> ory.support.AbstractAutowireCapableBeanFactory.doCreateBean(
>>>>>>> AbstractAutowireCapableBeanFactory.java:510)
>>>>>>> at org.springframework.beans.fact
>>>>>>> ory.support.AbstractAutowireCapableBeanFactory.createBean(Ab
>>>>>>> stractAutowireCapableBeanFactory.java:482)
>>>>>>> at org.springframework.beans.fact
>>>>>>> ory.support.AbstractBeanFactory$1.getObject(AbstractBeanFact
>>>>>>> ory.java:306)
>>>>>>> at org.springframework.beans.fact
>>>>>>> ory.support.DefaultSingletonBeanRegistry.getSingleton(Defaul
>>>>>>> tSingletonBeanRegistry.java:230)
>>>>>>> at org.springframework.beans.fact
>>>>>>> ory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactor
>>>>>>> y.java:302)
>>>>>>> at org.springframework.beans.fact
>>>>>>> ory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.
>>>>>>> java:197)
>>>>>>> at org.springframework.beans.fact
>>>>>>> ory.support.DefaultListableBeanFactory.preInstantiateSinglet
>>>>>>> ons(DefaultListableBeanFactory.java:778)
>>>>>>> at org.springframework.context.su
>>>>>>> pport.AbstractApplicationContext.finishBeanFactoryInitializa
>>>>>>> tion(AbstractApplicationContext.java:839)
>>>>>>>
>>>>>>> *kylin.out*
>>>>>>>
>>>>>>> There is not keystore files in the folder
>>>>>>>
>>>>>>> [root@ip-172-31-35-25 conf]# pwd
>>>>>>> /etc/apache-kylin-2.1.0-bin-hbase1x/tomcat/conf
>>>>>>> [root@ip-172-31-35-25 conf]# ls *.keystore
>>>>>>> ls: cannot access *.keystore: No such file or directory
>>>>>>>
>>>>>>>
>>>>>>> SEVERE: Failed to load keystore type JKS with path conf/.keystore
>>>>>>> due to /etc/apache-kylin-2.1.0-bin-hbase1x/tomcat/conf/.keystore
>>>>>>> (No such file or directory)
>>>>>>> java.io.FileNotFoundException: /etc/apache-kylin-2.1.0-bin-hbase1x/tomcat/conf/.keystore
>>>>>>> (No such file or directory)
>>>>>>> at java.io.FileInputStream.open0(Native Method)
>>>>>>> at java.io.FileInputStream.open(FileInputStream.java:195)
>>>>>>> at java.io.FileInputStream.<init>(FileInputStream.java:138)
>>>>>>> at java.io.FileInputStream.<init>(FileInputStream.java:93)
>>>>>>>
>>>>>>>
>>>>>>> Sep 05, 2017 12:06:38 PM org.apache.catalina.core.StandardService
>>>>>>> initInternal
>>>>>>> SEVERE: Failed to initialize connector [Connector[HTTP/1.1-7443]]
>>>>>>> org.apache.catalina.LifecycleException: Failed to initialize
>>>>>>> component [Connector[HTTP/1.1-7443]]
>>>>>>> at org.apache.catalina.util.Lifec
>>>>>>> ycleBase.init(LifecycleBase.java:106)
>>>>>>> at org.apache.catalina.core.Stand
>>>>>>> ardService.initInternal(StandardService.java:560)
>>>>>>> at org.apache.catalina.util.Lifec
>>>>>>> ycleBase.init(LifecycleBase.java:102)
>>>>>>> at org.apache.catalina.core.Stand
>>>>>>> ardServer.initInternal(StandardServer.java:838)
>>>>>>> at org.apache.catalina.util.Lifec
>>>>>>> ycleBase.init(LifecycleBase.java:102)
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Best regards,
>>>>>>
>>>>>> Shaofeng Shi 史少锋
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Best regards,
>>>>
>>>> Shaofeng Shi 史少锋
>>>>
>>>>
>>>
>>>
>>> --
>>> Best regards,
>>>
>>> Shaofeng Shi 史少锋
>>>
>>>
>>
>>
>> --
>> Best regards,
>>
>> Shaofeng Shi 史少锋
>>
>>
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>
>
--
Best regards,
Shaofeng Shi 史少锋
回复： Kylin on EMR Installation Error
Posted by 朱真龙 <18...@qq.com>.
Hi, ShaoFeng,
I have add the patch to my kylin2.1 cluster which visit two versions of hadoop(hdp2.3.2 and hdp2.6.0 ) , now the bug is resolved . thank you for your help .
------------------ 原始邮件 ------------------
发件人: "ShaoFeng Shi";<sh...@apache.org>;
发送时间: 2017年12月25日(星期一) 中午1:59
收件人: "user"<us...@kylin.apache.org>;
主题: Re: Kylin on EMR Installation Error
Hi zhenlong,
I'm afraid that patch is not all the changes; suggest you download the formal 2.2 release.
2017-12-25 11:53 GMT+08:00 朱真龙 <18...@qq.com>:
Hi, ShaoFeng,
thank you , I'll add the patch to kylin2.1 and retry . thank you again.
------------------ 原始邮件 ------------------
发件人: "ShaoFeng Shi";<sh...@apache.org>;
发送时间: 2017年12月25日(星期一) 中午11:09
收件人: "user"<us...@kylin.apache.org>;
主题: Re: Kylin on EMR Installation Error
Hi zhenlong,
Thanks for the detailed analysis. You're correct, there was some inconsistency in Kylin 2.1, it should have been fixed in 2.2, could you please upgrade Kylin and then verify?
The related JIRA is:
https://issues.apache.org/jira/browse/KYLIN-2865
https://issues.apache.org/jira/browse/KYLIN-2766
2017-12-25 10:56 GMT+08:00 朱真龙 <18...@qq.com>:
thank you for your response.after check https://issues.apache.org/jira/browse/KYLIN-3129 ,I find it's not the same problem as mine and I have found the way to resolved my problem. I used kylin between two hadoop cluster ,like this:
its works well when model is not too big. but some model with many row keys does not . when I configured fifteen columns for rowkey and setted dict encode like this :
when running to "Build Dimension Dictionary" step, I got error as follow( it means after writted snapshot to hdfs, when submit these hdfs file to hbase ,the files and hbaseTable are in different hdfs cluster , one in cluster 1 and one in cluster 2):
through looked into source code the error referred, I found that ,when writting "Dimension Dictionary" to hbase , kylin will judge if your keyvalue is larger than the configured Max_KeyValue size (hbase.client.keyvalue.maxsize is setted in hbase-site.xml),if you not setted ,it will get 10485760(10MB) as default . if your keyValue size is smaller than hbase.client.keyvalue.maxsize ,then kylin
will set your key and value into hbase's put object then submit. if your keyvalue is bigger than hbase.client.keyvalue.maxsize ,it will write snapshot into hdfs first then submit it via hbase table. but I think it's maybe a bug on kylin2.1 when use kylin between two hadoop cluster and build Dimension with keyvalue bigger than hbase.client.keyvalue.maxsize.
when keyvlaue size is bigger than hbase.client.keyvalue.maxsize , writte snapshot into hdfs( it got the hdfs path from hive table instead of hbase, and they are in different hdfs cluster) . so if you are using one hadoop cluster(hive and hbase are on one hadoop cluster ),it will works well . if you are using two hadoop cluster and your keyvalue size is smaller than hbase.client.keyvalue.maxsize ,it will also works well (use hbase's put object). but if you are using two hadoop cluster and your keyvalue is bigger than hbase.client.keyvalue.maxsize , you will get then same error like mine(beause your hive and hbase are in different hdfs cluster) .
I checked my hbase.client.keyvalue.maxsize in hbase-site.xml, and found ambari set 1MB as default ,and kylin set 10MB as default .so I changed this value to 10MB , and now my model works well. If you find a better way to resolve this problem ,please tell me , thank you.
apache-kylin-2.1.0-src\apache-kylin-2.1.0\storage-hbase\src\main\java\org\apache\kylin\storage\hbase\HBaseResourceStore.java
------------------ 原始邮件 ------------------
发件人: "ShaoFeng Shi";<sh...@apache.org>;
发送时间: 2017年12月22日(星期五) 下午5:49
收件人: "user"<us...@kylin.apache.org>;
主题: Re: Kylin on EMR Installation Error
Please check https://issues.apache.org/jira/browse/KYLIN-3129
2017-09-06 22:31 GMT+08:00 ShaoFeng Shi <sh...@apache.org>:
Thanks, I will try EMR 5.8 later;
At this moment, you can try emr 5.5; one thing to pay attention is, you need copy a configuration from hbase-site.xml to KYLIN_HOME/conf/kylin_job_conf.xml (replace the zk quorum name with yours):
<property>
<name>hbase.zookeeper.quorum</name>
<value>ip-nn-nn-nn-nn.ap-northeast-2.compute.internal</value>
</property>
2017-09-06 21:07 GMT+08:00 Mukund Agilisium <mu...@gmail.com>:
Hi
I am using EMR 5.8.0 , the latest one.
Thanks & Regards
Mukund
On Wed, Sep 6, 2017 at 8:44 AM, ShaoFeng Shi <sh...@apache.org> wrote:
Hi Mukund,
What's the EMR version?
Check if there is other version's jar in EMR node, e.g,:
sudo find / -name joda-time*.jar
2017-09-06 1:52 GMT+08:00 Mukund Agilisium <mu...@gmail.com>:
Hi All
I am trying to install Kylin apache-kylin-2.1.0-bin-hbase1x on EMR and getting the following errors.
[root@ip-172-31-35-25 lib]# pwd
/etc/apache-kylin-2.1.0-bin-hbase1x/tomcat/webapps/kylin/WEB-INF/lib
[root@ip-172-31-35-25 lib]# ls -al | grep joda
-rw-r--r-- 1 root root 573912 May 6 2016 joda-time-2.2.jar
Looks like the jar is available inside the webinf/lib as well.
Any help is appreciated!!
kylin.log
2017-09-05 12:06:45,097 ERROR [localhost-startStop-1] context.ContextLoader:351 : Context initialization failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter': Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter]: Constructor threw exception; nested exception is java.lang.NoSuchMethodError: org.joda.time.format.DateTimeFormatter.withZoneUTC()Lorg/joda/time/format/DateTimeFormatter;
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateBean(AbstractAutowireCapableBeanFactory.java:1109)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1054)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:510)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:778)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:839)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:538)
at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:444)
at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:326)
......
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter]: Constructor threw exception; nested exception is java.lang.NoSuchMethodError: org.joda.time.format.DateTimeFormatter.withZoneUTC()Lorg/joda/time/format/DateTimeFormatter;
at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:163)
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:89)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateBean(AbstractAutowireCapableBeanFactory.java:1102)
... 26 more
SEVERE: Exception sending context initialized event to listener instance of class org.springframework.web.context.ContextLoaderListener
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter': Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter]: Constructor threw exception; nested exception is java.lang.NoSuchMethodError: org.joda.time.format.DateTimeFormatter.withZoneUTC()Lorg/joda/time/format/DateTimeFormatter;
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateBean(AbstractAutowireCapableBeanFactory.java:1109)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1054)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:510)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:778)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:839)
kylin.out
There is not keystore files in the folder
[root@ip-172-31-35-25 conf]# pwd
/etc/apache-kylin-2.1.0-bin-hbase1x/tomcat/conf
[root@ip-172-31-35-25 conf]# ls *.keystore
ls: cannot access *.keystore: No such file or directory
SEVERE: Failed to load keystore type JKS with path conf/.keystore due to /etc/apache-kylin-2.1.0-bin-hbase1x/tomcat/conf/.keystore (No such file or directory)
java.io.FileNotFoundException: /etc/apache-kylin-2.1.0-bin-hbase1x/tomcat/conf/.keystore (No such file or directory)
at java.io.FileInputStream.open0(Native Method)
at java.io.FileInputStream.open(FileInputStream.java:195)
at java.io.FileInputStream.<init>(FileInputStream.java:138)
at java.io.FileInputStream.<init>(FileInputStream.java:93)
Sep 05, 2017 12:06:38 PM org.apache.catalina.core.StandardService initInternal
SEVERE: Failed to initialize connector [Connector[HTTP/1.1-7443]]
org.apache.catalina.LifecycleException: Failed to initialize component [Connector[HTTP/1.1-7443]]
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:106)
at org.apache.catalina.core.StandardService.initInternal(StandardService.java:560)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:102)
at org.apache.catalina.core.StandardServer.initInternal(StandardServer.java:838)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:102)
--
Best regards,
Shaofeng Shi 史少锋
--
Best regards,
Shaofeng Shi 史少锋
--
Best regards,
Shaofeng Shi 史少锋
--
Best regards,
Shaofeng Shi 史少锋
--
Best regards,
Shaofeng Shi 史少锋
Re: Kylin on EMR Installation Error
Posted by ShaoFeng Shi <sh...@apache.org>.
Hi zhenlong,
I'm afraid that patch is not all the changes; suggest you download the
formal 2.2 release.
2017-12-25 11:53 GMT+08:00 朱真龙 <18...@qq.com>:
> Hi, ShaoFeng,
>
> thank you , I'll add the patch to kylin2.1 and retry . thank you
> again.
>
>
> ------------------ 原始邮件 ------------------
> *发件人:* "ShaoFeng Shi";<sh...@apache.org>;
> *发送时间:* 2017年12月25日(星期一) 中午11:09
> *收件人:* "user"<us...@kylin.apache.org>;
> *主题:* Re: Kylin on EMR Installation Error
>
> Hi zhenlong,
>
> Thanks for the detailed analysis. You're correct, there was some
> inconsistency in Kylin 2.1, it should have been fixed in 2.2, could you
> please upgrade Kylin and then verify?
>
> The related JIRA is:
> https://issues.apache.org/jira/browse/KYLIN-2865
> https://issues.apache.org/jira/browse/KYLIN-2766
>
> 2017-12-25 10:56 GMT+08:00 朱真龙 <18...@qq.com>:
>
>> thank you for your response.after check https://issues.apache.o
>> rg/jira/browse/KYLIN-3129 ,I find it's not the same problem as mine and
>> I have found the way to resolved my problem. I used kylin between two
>> hadoop cluster ,like this:
>>
>>
>>
>> its works well when model is not too big. but some model with many row
>> keys does not . when I configured fifteen columns for rowkey and setted
>> dict encode like this :
>> when running to "Build Dimension Dictionary" step, I got error as
>> follow( it means after writted snapshot to hdfs, when submit these hdfs
>> file to hbase ,the files and hbaseTable are in different hdfs cluster , one
>> in cluster 1 and one in cluster 2):
>>
>>
>> through looked into source code the error referred, I found that ,when
>> writting "Dimension Dictionary" to hbase , kylin will judge if your
>> keyvalue is larger than the configured Max_KeyValue size
>> (hbase.client.keyvalue.maxsize is setted in hbase-site.xml),if you not
>> setted ,it will get 10485760(10MB) as default . if your keyValue size is
>> smaller than hbase.client.keyvalue.maxsize ,then kylin
>> will set your key and value into hbase's put object then submit. if your
>> keyvalue is bigger than hbase.client.keyvalue.maxsize ,it will write
>> snapshot into hdfs first then submit it via hbase table. but I think it's
>> maybe a bug on kylin2.1 when use kylin between two hadoop cluster and
>> build Dimension with keyvalue bigger than hbase.client.keyvalue.maxsize.
>>
>> when keyvlaue size is bigger than hbase.client.keyvalue.maxsize , writte
>> snapshot into hdfs( it got the hdfs path from hive table instead of hbase,
>> and they are in different hdfs cluster) . so if you are using one hadoop
>> cluster(hive and hbase are on one hadoop cluster ),it will works well . if
>> you are using two hadoop cluster and your keyvalue size is smaller than
>> hbase.client.keyvalue.maxsize ,it will also works well (use hbase's put
>> object). but if you are using two hadoop cluster and your keyvalue is
>> bigger than hbase.client.keyvalue.maxsize , you will get then same error
>> like mine(beause your hive and hbase are in different hdfs cluster) .
>>
>> I checked my hbase.client.keyvalue.maxsize in hbase-site.xml, and
>> found ambari set 1MB as default ,and kylin set 10MB as default .so I
>> changed this value to 10MB , and now my model works well. If you find a
>> better way to resolve this problem ,please tell me , thank you.
>>
>>
>> apache-kylin-2.1.0-src\apache-kylin-2.1.0\storage-hbase\src\
>> main\java\org\apache\kylin\storage\hbase\HBaseResourceStore.java
>>
>>
>>
>>
>> ------------------ 原始邮件 ------------------
>> *发件人:* "ShaoFeng Shi";<sh...@apache.org>;
>> *发送时间:* 2017年12月22日(星期五) 下午5:49
>> *收件人:* "user"<us...@kylin.apache.org>;
>> *主题:* Re: Kylin on EMR Installation Error
>>
>> Please check https://issues.apache.org/jira/browse/KYLIN-3129
>>
>> 2017-09-06 22:31 GMT+08:00 ShaoFeng Shi <sh...@apache.org>:
>>
>>> Thanks, I will try EMR 5.8 later;
>>>
>>> At this moment, you can try emr 5.5; one thing to pay attention is, you
>>> need copy a configuration from hbase-site.xml to
>>> KYLIN_HOME/conf/kylin_job_conf.xml (replace the zk quorum name with
>>> yours):
>>>
>>> <property>
>>> <name>hbase.zookeeper.quorum</name>
>>> <value>ip-nn-nn-nn-nn.ap-northeast-2.compute.internal</value>
>>> </property>
>>>
>>>
>>> 2017-09-06 21:07 GMT+08:00 Mukund Agilisium <mu...@gmail.com>
>>> :
>>>
>>>> Hi
>>>>
>>>> I am using EMR 5.8.0 , the latest one.
>>>>
>>>> Thanks & Regards
>>>> Mukund
>>>>
>>>> On Wed, Sep 6, 2017 at 8:44 AM, ShaoFeng Shi <sh...@apache.org>
>>>> wrote:
>>>>
>>>>> Hi Mukund,
>>>>>
>>>>> What's the EMR version?
>>>>>
>>>>> Check if there is other version's jar in EMR node, e.g,:
>>>>>
>>>>> sudo find / -name joda-time*.jar
>>>>>
>>>>>
>>>>> 2017-09-06 1:52 GMT+08:00 Mukund Agilisium <mukund.agilisium@gmail.com
>>>>> >:
>>>>>
>>>>>> Hi All
>>>>>>
>>>>>> I am trying to install Kylin apache-kylin-2.1.0-bin-hbase1x on EMR
>>>>>> and getting the following errors.
>>>>>>
>>>>>> [root@ip-172-31-35-25 lib]# pwd
>>>>>> /etc/apache-kylin-2.1.0-bin-hbase1x/tomcat/webapps/kylin/WEB-INF/lib
>>>>>> [root@ip-172-31-35-25 lib]# ls -al | grep joda
>>>>>> -rw-r--r-- 1 root root 573912 May 6 2016 joda-time-2.2.jar
>>>>>>
>>>>>> Looks like the jar is available inside the webinf/lib as well.
>>>>>>
>>>>>> Any help is appreciated!!
>>>>>>
>>>>>> *kylin.log*
>>>>>> 2017-09-05 12:06:45,097 ERROR [localhost-startStop-1]
>>>>>> context.ContextLoader:351 : Context initialization failed
>>>>>> org.springframework.beans.factory.BeanCreationException: Error
>>>>>> creating bean with name 'org.springframework.web.servl
>>>>>> et.mvc.method.annotation.RequestMappingHandlerAdapter':
>>>>>> Instantiation of bean failed; nested exception is
>>>>>> org.springframework.beans.BeanInstantiationException: Failed to
>>>>>> instantiate [org.springframework.web.servl
>>>>>> et.mvc.method.annotation.RequestMappingHandlerAdapter]: Constructor
>>>>>> threw exception; nested exception is java.lang.NoSuchMethodError:
>>>>>> org.joda.time.format.DateTimeFormatter.withZoneUTC()Lorg/jod
>>>>>> a/time/format/DateTimeFormatter;
>>>>>> at org.springframework.beans.fact
>>>>>> ory.support.AbstractAutowireCapableBeanFactory.instantiateBe
>>>>>> an(AbstractAutowireCapableBeanFactory.java:1109)
>>>>>> at org.springframework.beans.fact
>>>>>> ory.support.AbstractAutowireCapableBeanFactory.createBeanIns
>>>>>> tance(AbstractAutowireCapableBeanFactory.java:1054)
>>>>>> at org.springframework.beans.fact
>>>>>> ory.support.AbstractAutowireCapableBeanFactory.doCreateBean(
>>>>>> AbstractAutowireCapableBeanFactory.java:510)
>>>>>> at org.springframework.beans.fact
>>>>>> ory.support.AbstractAutowireCapableBeanFactory.createBean(Ab
>>>>>> stractAutowireCapableBeanFactory.java:482)
>>>>>> at org.springframework.beans.fact
>>>>>> ory.support.AbstractBeanFactory$1.getObject(AbstractBeanFact
>>>>>> ory.java:306)
>>>>>> at org.springframework.beans.fact
>>>>>> ory.support.DefaultSingletonBeanRegistry.getSingleton(Defaul
>>>>>> tSingletonBeanRegistry.java:230)
>>>>>> at org.springframework.beans.fact
>>>>>> ory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactor
>>>>>> y.java:302)
>>>>>> at org.springframework.beans.fact
>>>>>> ory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
>>>>>> at org.springframework.beans.fact
>>>>>> ory.support.DefaultListableBeanFactory.preInstantiateSinglet
>>>>>> ons(DefaultListableBeanFactory.java:778)
>>>>>> at org.springframework.context.su
>>>>>> pport.AbstractApplicationContext.finishBeanFactoryInitializa
>>>>>> tion(AbstractApplicationContext.java:839)
>>>>>> at org.springframework.context.su
>>>>>> pport.AbstractApplicationContext.refresh(AbstractApplication
>>>>>> Context.java:538)
>>>>>> at org.springframework.web.contex
>>>>>> t.ContextLoader.configureAndRefreshWebApplicationContext(Con
>>>>>> textLoader.java:444)
>>>>>> at org.springframework.web.contex
>>>>>> t.ContextLoader.initWebApplicationContext(ContextLoader.java:326)
>>>>>> ......
>>>>>>
>>>>>>
>>>>>> Caused by: org.springframework.beans.BeanInstantiationException:
>>>>>> Failed to instantiate [org.springframework.web.servl
>>>>>> et.mvc.method.annotation.RequestMappingHandlerAdapter]: Constructor
>>>>>> threw exception; nested exception is java.lang.NoSuchMethodError:
>>>>>> org.joda.time.format.DateTimeFormatter.withZoneUTC()Lorg/jod
>>>>>> a/time/format/DateTimeFormatter;
>>>>>> at org.springframework.beans.Bean
>>>>>> Utils.instantiateClass(BeanUtils.java:163)
>>>>>> at org.springframework.beans.fact
>>>>>> ory.support.SimpleInstantiationStrategy.instantiate(SimpleIn
>>>>>> stantiationStrategy.java:89)
>>>>>> at org.springframework.beans.fact
>>>>>> ory.support.AbstractAutowireCapableBeanFactory.instantiateBe
>>>>>> an(AbstractAutowireCapableBeanFactory.java:1102)
>>>>>> ... 26 more
>>>>>>
>>>>>>
>>>>>> SEVERE: Exception sending context initialized event to listener
>>>>>> instance of class org.springframework.web.contex
>>>>>> t.ContextLoaderListener
>>>>>> org.springframework.beans.factory.BeanCreationException: Error
>>>>>> creating bean with name 'org.springframework.web.servl
>>>>>> et.mvc.method.annotation.RequestMappingHandlerAdapter':
>>>>>> Instantiation of bean failed; nested exception is
>>>>>> org.springframework.beans.BeanInstantiationException: Failed to
>>>>>> instantiate [org.springframework.web.servl
>>>>>> et.mvc.method.annotation.RequestMappingHandlerAdapter]: Constructor
>>>>>> threw exception; nested exception is java.lang.NoSuchMethodError:
>>>>>> org.joda.time.format.DateTimeFormatter.withZoneUTC()Lorg/jod
>>>>>> a/time/format/DateTimeFormatter;
>>>>>> at org.springframework.beans.fact
>>>>>> ory.support.AbstractAutowireCapableBeanFactory.instantiateBe
>>>>>> an(AbstractAutowireCapableBeanFactory.java:1109)
>>>>>> at org.springframework.beans.fact
>>>>>> ory.support.AbstractAutowireCapableBeanFactory.createBeanIns
>>>>>> tance(AbstractAutowireCapableBeanFactory.java:1054)
>>>>>> at org.springframework.beans.fact
>>>>>> ory.support.AbstractAutowireCapableBeanFactory.doCreateBean(
>>>>>> AbstractAutowireCapableBeanFactory.java:510)
>>>>>> at org.springframework.beans.fact
>>>>>> ory.support.AbstractAutowireCapableBeanFactory.createBean(Ab
>>>>>> stractAutowireCapableBeanFactory.java:482)
>>>>>> at org.springframework.beans.fact
>>>>>> ory.support.AbstractBeanFactory$1.getObject(AbstractBeanFact
>>>>>> ory.java:306)
>>>>>> at org.springframework.beans.fact
>>>>>> ory.support.DefaultSingletonBeanRegistry.getSingleton(Defaul
>>>>>> tSingletonBeanRegistry.java:230)
>>>>>> at org.springframework.beans.fact
>>>>>> ory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactor
>>>>>> y.java:302)
>>>>>> at org.springframework.beans.fact
>>>>>> ory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
>>>>>> at org.springframework.beans.fact
>>>>>> ory.support.DefaultListableBeanFactory.preInstantiateSinglet
>>>>>> ons(DefaultListableBeanFactory.java:778)
>>>>>> at org.springframework.context.su
>>>>>> pport.AbstractApplicationContext.finishBeanFactoryInitializa
>>>>>> tion(AbstractApplicationContext.java:839)
>>>>>>
>>>>>> *kylin.out*
>>>>>>
>>>>>> There is not keystore files in the folder
>>>>>>
>>>>>> [root@ip-172-31-35-25 conf]# pwd
>>>>>> /etc/apache-kylin-2.1.0-bin-hbase1x/tomcat/conf
>>>>>> [root@ip-172-31-35-25 conf]# ls *.keystore
>>>>>> ls: cannot access *.keystore: No such file or directory
>>>>>>
>>>>>>
>>>>>> SEVERE: Failed to load keystore type JKS with path conf/.keystore due
>>>>>> to /etc/apache-kylin-2.1.0-bin-hbase1x/tomcat/conf/.keystore (No
>>>>>> such file or directory)
>>>>>> java.io.FileNotFoundException: /etc/apache-kylin-2.1.0-bin-hbase1x/tomcat/conf/.keystore
>>>>>> (No such file or directory)
>>>>>> at java.io.FileInputStream.open0(Native Method)
>>>>>> at java.io.FileInputStream.open(FileInputStream.java:195)
>>>>>> at java.io.FileInputStream.<init>(FileInputStream.java:138)
>>>>>> at java.io.FileInputStream.<init>(FileInputStream.java:93)
>>>>>>
>>>>>>
>>>>>> Sep 05, 2017 12:06:38 PM org.apache.catalina.core.StandardService
>>>>>> initInternal
>>>>>> SEVERE: Failed to initialize connector [Connector[HTTP/1.1-7443]]
>>>>>> org.apache.catalina.LifecycleException: Failed to initialize
>>>>>> component [Connector[HTTP/1.1-7443]]
>>>>>> at org.apache.catalina.util.Lifec
>>>>>> ycleBase.init(LifecycleBase.java:106)
>>>>>> at org.apache.catalina.core.Stand
>>>>>> ardService.initInternal(StandardService.java:560)
>>>>>> at org.apache.catalina.util.Lifec
>>>>>> ycleBase.init(LifecycleBase.java:102)
>>>>>> at org.apache.catalina.core.Stand
>>>>>> ardServer.initInternal(StandardServer.java:838)
>>>>>> at org.apache.catalina.util.Lifec
>>>>>> ycleBase.init(LifecycleBase.java:102)
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Best regards,
>>>>>
>>>>> Shaofeng Shi 史少锋
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> Best regards,
>>>
>>> Shaofeng Shi 史少锋
>>>
>>>
>>
>>
>> --
>> Best regards,
>>
>> Shaofeng Shi 史少锋
>>
>>
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>
>
--
Best regards,
Shaofeng Shi 史少锋
回复： Kylin on EMR Installation Error
Posted by 朱真龙 <18...@qq.com>.
Hi, ShaoFeng,
thank you , I'll add the patch to kylin2.1 and retry . thank you again.
------------------ 原始邮件 ------------------
发件人: "ShaoFeng Shi";<sh...@apache.org>;
发送时间: 2017年12月25日(星期一) 中午11:09
收件人: "user"<us...@kylin.apache.org>;
主题: Re: Kylin on EMR Installation Error
Hi zhenlong,
Thanks for the detailed analysis. You're correct, there was some inconsistency in Kylin 2.1, it should have been fixed in 2.2, could you please upgrade Kylin and then verify?
The related JIRA is:
https://issues.apache.org/jira/browse/KYLIN-2865
https://issues.apache.org/jira/browse/KYLIN-2766
2017-12-25 10:56 GMT+08:00 朱真龙 <18...@qq.com>:
thank you for your response.after check https://issues.apache.org/jira/browse/KYLIN-3129 ,I find it's not the same problem as mine and I have found the way to resolved my problem. I used kylin between two hadoop cluster ,like this:
its works well when model is not too big. but some model with many row keys does not . when I configured fifteen columns for rowkey and setted dict encode like this :
when running to "Build Dimension Dictionary" step, I got error as follow( it means after writted snapshot to hdfs, when submit these hdfs file to hbase ,the files and hbaseTable are in different hdfs cluster , one in cluster 1 and one in cluster 2):
through looked into source code the error referred, I found that ,when writting "Dimension Dictionary" to hbase , kylin will judge if your keyvalue is larger than the configured Max_KeyValue size (hbase.client.keyvalue.maxsize is setted in hbase-site.xml),if you not setted ,it will get 10485760(10MB) as default . if your keyValue size is smaller than hbase.client.keyvalue.maxsize ,then kylin
will set your key and value into hbase's put object then submit. if your keyvalue is bigger than hbase.client.keyvalue.maxsize ,it will write snapshot into hdfs first then submit it via hbase table. but I think it's maybe a bug on kylin2.1 when use kylin between two hadoop cluster and build Dimension with keyvalue bigger than hbase.client.keyvalue.maxsize.
when keyvlaue size is bigger than hbase.client.keyvalue.maxsize , writte snapshot into hdfs( it got the hdfs path from hive table instead of hbase, and they are in different hdfs cluster) . so if you are using one hadoop cluster(hive and hbase are on one hadoop cluster ),it will works well . if you are using two hadoop cluster and your keyvalue size is smaller than hbase.client.keyvalue.maxsize ,it will also works well (use hbase's put object). but if you are using two hadoop cluster and your keyvalue is bigger than hbase.client.keyvalue.maxsize , you will get then same error like mine(beause your hive and hbase are in different hdfs cluster) .
I checked my hbase.client.keyvalue.maxsize in hbase-site.xml, and found ambari set 1MB as default ,and kylin set 10MB as default .so I changed this value to 10MB , and now my model works well. If you find a better way to resolve this problem ,please tell me , thank you.
apache-kylin-2.1.0-src\apache-kylin-2.1.0\storage-hbase\src\main\java\org\apache\kylin\storage\hbase\HBaseResourceStore.java
------------------ 原始邮件 ------------------
发件人: "ShaoFeng Shi";<sh...@apache.org>;
发送时间: 2017年12月22日(星期五) 下午5:49
收件人: "user"<us...@kylin.apache.org>;
主题: Re: Kylin on EMR Installation Error
Please check https://issues.apache.org/jira/browse/KYLIN-3129
2017-09-06 22:31 GMT+08:00 ShaoFeng Shi <sh...@apache.org>:
Thanks, I will try EMR 5.8 later;
At this moment, you can try emr 5.5; one thing to pay attention is, you need copy a configuration from hbase-site.xml to KYLIN_HOME/conf/kylin_job_conf.xml (replace the zk quorum name with yours):
<property>
<name>hbase.zookeeper.quorum</name>
<value>ip-nn-nn-nn-nn.ap-northeast-2.compute.internal</value>
</property>
2017-09-06 21:07 GMT+08:00 Mukund Agilisium <mu...@gmail.com>:
Hi
I am using EMR 5.8.0 , the latest one.
Thanks & Regards
Mukund
On Wed, Sep 6, 2017 at 8:44 AM, ShaoFeng Shi <sh...@apache.org> wrote:
Hi Mukund,
What's the EMR version?
Check if there is other version's jar in EMR node, e.g,:
sudo find / -name joda-time*.jar
2017-09-06 1:52 GMT+08:00 Mukund Agilisium <mu...@gmail.com>:
Hi All
I am trying to install Kylin apache-kylin-2.1.0-bin-hbase1x on EMR and getting the following errors.
[root@ip-172-31-35-25 lib]# pwd
/etc/apache-kylin-2.1.0-bin-hbase1x/tomcat/webapps/kylin/WEB-INF/lib
[root@ip-172-31-35-25 lib]# ls -al | grep joda
-rw-r--r-- 1 root root 573912 May 6 2016 joda-time-2.2.jar
Looks like the jar is available inside the webinf/lib as well.
Any help is appreciated!!
kylin.log
2017-09-05 12:06:45,097 ERROR [localhost-startStop-1] context.ContextLoader:351 : Context initialization failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter': Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter]: Constructor threw exception; nested exception is java.lang.NoSuchMethodError: org.joda.time.format.DateTimeFormatter.withZoneUTC()Lorg/joda/time/format/DateTimeFormatter;
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateBean(AbstractAutowireCapableBeanFactory.java:1109)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1054)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:510)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:778)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:839)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:538)
at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:444)
at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:326)
......
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter]: Constructor threw exception; nested exception is java.lang.NoSuchMethodError: org.joda.time.format.DateTimeFormatter.withZoneUTC()Lorg/joda/time/format/DateTimeFormatter;
at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:163)
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:89)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateBean(AbstractAutowireCapableBeanFactory.java:1102)
... 26 more
SEVERE: Exception sending context initialized event to listener instance of class org.springframework.web.context.ContextLoaderListener
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter': Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter]: Constructor threw exception; nested exception is java.lang.NoSuchMethodError: org.joda.time.format.DateTimeFormatter.withZoneUTC()Lorg/joda/time/format/DateTimeFormatter;
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateBean(AbstractAutowireCapableBeanFactory.java:1109)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1054)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:510)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:778)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:839)
kylin.out
There is not keystore files in the folder
[root@ip-172-31-35-25 conf]# pwd
/etc/apache-kylin-2.1.0-bin-hbase1x/tomcat/conf
[root@ip-172-31-35-25 conf]# ls *.keystore
ls: cannot access *.keystore: No such file or directory
SEVERE: Failed to load keystore type JKS with path conf/.keystore due to /etc/apache-kylin-2.1.0-bin-hbase1x/tomcat/conf/.keystore (No such file or directory)
java.io.FileNotFoundException: /etc/apache-kylin-2.1.0-bin-hbase1x/tomcat/conf/.keystore (No such file or directory)
at java.io.FileInputStream.open0(Native Method)
at java.io.FileInputStream.open(FileInputStream.java:195)
at java.io.FileInputStream.<init>(FileInputStream.java:138)
at java.io.FileInputStream.<init>(FileInputStream.java:93)
Sep 05, 2017 12:06:38 PM org.apache.catalina.core.StandardService initInternal
SEVERE: Failed to initialize connector [Connector[HTTP/1.1-7443]]
org.apache.catalina.LifecycleException: Failed to initialize component [Connector[HTTP/1.1-7443]]
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:106)
at org.apache.catalina.core.StandardService.initInternal(StandardService.java:560)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:102)
at org.apache.catalina.core.StandardServer.initInternal(StandardServer.java:838)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:102)
--
Best regards,
Shaofeng Shi 史少锋
--
Best regards,
Shaofeng Shi 史少锋
--
Best regards,
Shaofeng Shi 史少锋
--
Best regards,
Shaofeng Shi 史少锋
Re: Kylin on EMR Installation Error
Posted by ShaoFeng Shi <sh...@apache.org>.
Hi zhenlong,
Thanks for the detailed analysis. You're correct, there was some
inconsistency in Kylin 2.1, it should have been fixed in 2.2, could you
please upgrade Kylin and then verify?
The related JIRA is:
https://issues.apache.org/jira/browse/KYLIN-2865
https://issues.apache.org/jira/browse/KYLIN-2766
2017-12-25 10:56 GMT+08:00 朱真龙 <18...@qq.com>:
> thank you for your response.after check https://issues.apache.
> org/jira/browse/KYLIN-3129 ,I find it's not the same problem as mine and
> I have found the way to resolved my problem. I used kylin between two
> hadoop cluster ,like this:
>
>
>
> its works well when model is not too big. but some model with many row
> keys does not . when I configured fifteen columns for rowkey and setted
> dict encode like this :
> when running to "Build Dimension Dictionary" step, I got error as follow(
> it means after writted snapshot to hdfs, when submit these hdfs file to
> hbase ,the files and hbaseTable are in different hdfs cluster , one in
> cluster 1 and one in cluster 2):
>
>
> through looked into source code the error referred, I found that ,when
> writting "Dimension Dictionary" to hbase , kylin will judge if your
> keyvalue is larger than the configured Max_KeyValue size
> (hbase.client.keyvalue.maxsize is setted in hbase-site.xml),if you not
> setted ,it will get 10485760(10MB) as default . if your keyValue size is
> smaller than hbase.client.keyvalue.maxsize ,then kylin
> will set your key and value into hbase's put object then submit. if your
> keyvalue is bigger than hbase.client.keyvalue.maxsize ,it will write
> snapshot into hdfs first then submit it via hbase table. but I think it's
> maybe a bug on kylin2.1 when use kylin between two hadoop cluster and
> build Dimension with keyvalue bigger than hbase.client.keyvalue.maxsize.
>
> when keyvlaue size is bigger than hbase.client.keyvalue.maxsize , writte
> snapshot into hdfs( it got the hdfs path from hive table instead of hbase,
> and they are in different hdfs cluster) . so if you are using one hadoop
> cluster(hive and hbase are on one hadoop cluster ),it will works well . if
> you are using two hadoop cluster and your keyvalue size is smaller than
> hbase.client.keyvalue.maxsize ,it will also works well (use hbase's put
> object). but if you are using two hadoop cluster and your keyvalue is
> bigger than hbase.client.keyvalue.maxsize , you will get then same error
> like mine(beause your hive and hbase are in different hdfs cluster) .
>
> I checked my hbase.client.keyvalue.maxsize in hbase-site.xml, and
> found ambari set 1MB as default ,and kylin set 10MB as default .so I
> changed this value to 10MB , and now my model works well. If you find a
> better way to resolve this problem ,please tell me , thank you.
>
>
> apache-kylin-2.1.0-src\apache-kylin-2.1.0\storage-hbase\src\
> main\java\org\apache\kylin\storage\hbase\HBaseResourceStore.java
>
>
>
>
> ------------------ 原始邮件 ------------------
> *发件人:* "ShaoFeng Shi";<sh...@apache.org>;
> *发送时间:* 2017年12月22日(星期五) 下午5:49
> *收件人:* "user"<us...@kylin.apache.org>;
> *主题:* Re: Kylin on EMR Installation Error
>
> Please check https://issues.apache.org/jira/browse/KYLIN-3129
>
> 2017-09-06 22:31 GMT+08:00 ShaoFeng Shi <sh...@apache.org>:
>
>> Thanks, I will try EMR 5.8 later;
>>
>> At this moment, you can try emr 5.5; one thing to pay attention is, you
>> need copy a configuration from hbase-site.xml to
>> KYLIN_HOME/conf/kylin_job_conf.xml (replace the zk quorum name with
>> yours):
>>
>> <property>
>> <name>hbase.zookeeper.quorum</name>
>> <value>ip-nn-nn-nn-nn.ap-northeast-2.compute.internal</value>
>> </property>
>>
>>
>> 2017-09-06 21:07 GMT+08:00 Mukund Agilisium <mu...@gmail.com>:
>>
>>> Hi
>>>
>>> I am using EMR 5.8.0 , the latest one.
>>>
>>> Thanks & Regards
>>> Mukund
>>>
>>> On Wed, Sep 6, 2017 at 8:44 AM, ShaoFeng Shi <sh...@apache.org>
>>> wrote:
>>>
>>>> Hi Mukund,
>>>>
>>>> What's the EMR version?
>>>>
>>>> Check if there is other version's jar in EMR node, e.g,:
>>>>
>>>> sudo find / -name joda-time*.jar
>>>>
>>>>
>>>> 2017-09-06 1:52 GMT+08:00 Mukund Agilisium <mu...@gmail.com>
>>>> :
>>>>
>>>>> Hi All
>>>>>
>>>>> I am trying to install Kylin apache-kylin-2.1.0-bin-hbase1x on EMR
>>>>> and getting the following errors.
>>>>>
>>>>> [root@ip-172-31-35-25 lib]# pwd
>>>>> /etc/apache-kylin-2.1.0-bin-hbase1x/tomcat/webapps/kylin/WEB-INF/lib
>>>>> [root@ip-172-31-35-25 lib]# ls -al | grep joda
>>>>> -rw-r--r-- 1 root root 573912 May 6 2016 joda-time-2.2.jar
>>>>>
>>>>> Looks like the jar is available inside the webinf/lib as well.
>>>>>
>>>>> Any help is appreciated!!
>>>>>
>>>>> *kylin.log*
>>>>> 2017-09-05 12:06:45,097 ERROR [localhost-startStop-1]
>>>>> context.ContextLoader:351 : Context initialization failed
>>>>> org.springframework.beans.factory.BeanCreationException: Error
>>>>> creating bean with name 'org.springframework.web.servl
>>>>> et.mvc.method.annotation.RequestMappingHandlerAdapter': Instantiation
>>>>> of bean failed; nested exception is org.springframework.beans.BeanInstantiationException:
>>>>> Failed to instantiate [org.springframework.web.servl
>>>>> et.mvc.method.annotation.RequestMappingHandlerAdapter]: Constructor
>>>>> threw exception; nested exception is java.lang.NoSuchMethodError:
>>>>> org.joda.time.format.DateTimeFormatter.withZoneUTC()Lorg/jod
>>>>> a/time/format/DateTimeFormatter;
>>>>> at org.springframework.beans.fact
>>>>> ory.support.AbstractAutowireCapableBeanFactory.instantiateBe
>>>>> an(AbstractAutowireCapableBeanFactory.java:1109)
>>>>> at org.springframework.beans.fact
>>>>> ory.support.AbstractAutowireCapableBeanFactory.createBeanIns
>>>>> tance(AbstractAutowireCapableBeanFactory.java:1054)
>>>>> at org.springframework.beans.fact
>>>>> ory.support.AbstractAutowireCapableBeanFactory.doCreateBean(
>>>>> AbstractAutowireCapableBeanFactory.java:510)
>>>>> at org.springframework.beans.fact
>>>>> ory.support.AbstractAutowireCapableBeanFactory.createBean(Ab
>>>>> stractAutowireCapableBeanFactory.java:482)
>>>>> at org.springframework.beans.fact
>>>>> ory.support.AbstractBeanFactory$1.getObject(AbstractBeanFact
>>>>> ory.java:306)
>>>>> at org.springframework.beans.fact
>>>>> ory.support.DefaultSingletonBeanRegistry.getSingleton(Defaul
>>>>> tSingletonBeanRegistry.java:230)
>>>>> at org.springframework.beans.fact
>>>>> ory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactor
>>>>> y.java:302)
>>>>> at org.springframework.beans.fact
>>>>> ory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
>>>>> at org.springframework.beans.fact
>>>>> ory.support.DefaultListableBeanFactory.preInstantiateSinglet
>>>>> ons(DefaultListableBeanFactory.java:778)
>>>>> at org.springframework.context.su
>>>>> pport.AbstractApplicationContext.finishBeanFactoryInitializa
>>>>> tion(AbstractApplicationContext.java:839)
>>>>> at org.springframework.context.su
>>>>> pport.AbstractApplicationContext.refresh(AbstractApplication
>>>>> Context.java:538)
>>>>> at org.springframework.web.contex
>>>>> t.ContextLoader.configureAndRefreshWebApplicationContext(Con
>>>>> textLoader.java:444)
>>>>> at org.springframework.web.contex
>>>>> t.ContextLoader.initWebApplicationContext(ContextLoader.java:326)
>>>>> ......
>>>>>
>>>>>
>>>>> Caused by: org.springframework.beans.BeanInstantiationException:
>>>>> Failed to instantiate [org.springframework.web.servl
>>>>> et.mvc.method.annotation.RequestMappingHandlerAdapter]: Constructor
>>>>> threw exception; nested exception is java.lang.NoSuchMethodError:
>>>>> org.joda.time.format.DateTimeFormatter.withZoneUTC()Lorg/jod
>>>>> a/time/format/DateTimeFormatter;
>>>>> at org.springframework.beans.Bean
>>>>> Utils.instantiateClass(BeanUtils.java:163)
>>>>> at org.springframework.beans.fact
>>>>> ory.support.SimpleInstantiationStrategy.instantiate(SimpleIn
>>>>> stantiationStrategy.java:89)
>>>>> at org.springframework.beans.fact
>>>>> ory.support.AbstractAutowireCapableBeanFactory.instantiateBe
>>>>> an(AbstractAutowireCapableBeanFactory.java:1102)
>>>>> ... 26 more
>>>>>
>>>>>
>>>>> SEVERE: Exception sending context initialized event to listener
>>>>> instance of class org.springframework.web.contex
>>>>> t.ContextLoaderListener
>>>>> org.springframework.beans.factory.BeanCreationException: Error
>>>>> creating bean with name 'org.springframework.web.servl
>>>>> et.mvc.method.annotation.RequestMappingHandlerAdapter': Instantiation
>>>>> of bean failed; nested exception is org.springframework.beans.BeanInstantiationException:
>>>>> Failed to instantiate [org.springframework.web.servl
>>>>> et.mvc.method.annotation.RequestMappingHandlerAdapter]: Constructor
>>>>> threw exception; nested exception is java.lang.NoSuchMethodError:
>>>>> org.joda.time.format.DateTimeFormatter.withZoneUTC()Lorg/jod
>>>>> a/time/format/DateTimeFormatter;
>>>>> at org.springframework.beans.fact
>>>>> ory.support.AbstractAutowireCapableBeanFactory.instantiateBe
>>>>> an(AbstractAutowireCapableBeanFactory.java:1109)
>>>>> at org.springframework.beans.fact
>>>>> ory.support.AbstractAutowireCapableBeanFactory.createBeanIns
>>>>> tance(AbstractAutowireCapableBeanFactory.java:1054)
>>>>> at org.springframework.beans.fact
>>>>> ory.support.AbstractAutowireCapableBeanFactory.doCreateBean(
>>>>> AbstractAutowireCapableBeanFactory.java:510)
>>>>> at org.springframework.beans.fact
>>>>> ory.support.AbstractAutowireCapableBeanFactory.createBean(Ab
>>>>> stractAutowireCapableBeanFactory.java:482)
>>>>> at org.springframework.beans.fact
>>>>> ory.support.AbstractBeanFactory$1.getObject(AbstractBeanFact
>>>>> ory.java:306)
>>>>> at org.springframework.beans.fact
>>>>> ory.support.DefaultSingletonBeanRegistry.getSingleton(Defaul
>>>>> tSingletonBeanRegistry.java:230)
>>>>> at org.springframework.beans.fact
>>>>> ory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactor
>>>>> y.java:302)
>>>>> at org.springframework.beans.fact
>>>>> ory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
>>>>> at org.springframework.beans.fact
>>>>> ory.support.DefaultListableBeanFactory.preInstantiateSinglet
>>>>> ons(DefaultListableBeanFactory.java:778)
>>>>> at org.springframework.context.su
>>>>> pport.AbstractApplicationContext.finishBeanFactoryInitializa
>>>>> tion(AbstractApplicationContext.java:839)
>>>>>
>>>>> *kylin.out*
>>>>>
>>>>> There is not keystore files in the folder
>>>>>
>>>>> [root@ip-172-31-35-25 conf]# pwd
>>>>> /etc/apache-kylin-2.1.0-bin-hbase1x/tomcat/conf
>>>>> [root@ip-172-31-35-25 conf]# ls *.keystore
>>>>> ls: cannot access *.keystore: No such file or directory
>>>>>
>>>>>
>>>>> SEVERE: Failed to load keystore type JKS with path conf/.keystore due
>>>>> to /etc/apache-kylin-2.1.0-bin-hbase1x/tomcat/conf/.keystore (No such
>>>>> file or directory)
>>>>> java.io.FileNotFoundException: /etc/apache-kylin-2.1.0-bin-hbase1x/tomcat/conf/.keystore
>>>>> (No such file or directory)
>>>>> at java.io.FileInputStream.open0(Native Method)
>>>>> at java.io.FileInputStream.open(FileInputStream.java:195)
>>>>> at java.io.FileInputStream.<init>(FileInputStream.java:138)
>>>>> at java.io.FileInputStream.<init>(FileInputStream.java:93)
>>>>>
>>>>>
>>>>> Sep 05, 2017 12:06:38 PM org.apache.catalina.core.StandardService
>>>>> initInternal
>>>>> SEVERE: Failed to initialize connector [Connector[HTTP/1.1-7443]]
>>>>> org.apache.catalina.LifecycleException: Failed to initialize
>>>>> component [Connector[HTTP/1.1-7443]]
>>>>> at org.apache.catalina.util.Lifec
>>>>> ycleBase.init(LifecycleBase.java:106)
>>>>> at org.apache.catalina.core.Stand
>>>>> ardService.initInternal(StandardService.java:560)
>>>>> at org.apache.catalina.util.Lifec
>>>>> ycleBase.init(LifecycleBase.java:102)
>>>>> at org.apache.catalina.core.Stand
>>>>> ardServer.initInternal(StandardServer.java:838)
>>>>> at org.apache.catalina.util.Lifec
>>>>> ycleBase.init(LifecycleBase.java:102)
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Best regards,
>>>>
>>>> Shaofeng Shi 史少锋
>>>>
>>>>
>>>
>>
>>
>> --
>> Best regards,
>>
>> Shaofeng Shi 史少锋
>>
>>
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>
>
--
Best regards,
Shaofeng Shi 史少锋
回复： Kylin on EMR Installation Error
Posted by 朱真龙 <18...@qq.com>.
thank you for your response.after check https://issues.apache.org/jira/browse/KYLIN-3129 ,I find it's not the same problem as mine and I have found the way to resolved my problem. I used kylin between two hadoop cluster ,like this:
its works well when model is not too big. but some model with many row keys does not . when I configured fifteen columns for rowkey and setted dict encode like this :
when running to "Build Dimension Dictionary" step, I got error as follow( it means after writted snapshot to hdfs, when submit these hdfs file to hbase ,the files and hbaseTable are in different hdfs cluster , one in cluster 1 and one in cluster 2):
through looked into source code the error referred, I found that ,when writting "Dimension Dictionary" to hbase , kylin will judge if your keyvalue is larger than the configured Max_KeyValue size (hbase.client.keyvalue.maxsize is setted in hbase-site.xml),if you not setted ,it will get 10485760(10MB) as default . if your keyValue size is smaller than hbase.client.keyvalue.maxsize ,then kylin
will set your key and value into hbase's put object then submit. if your keyvalue is bigger than hbase.client.keyvalue.maxsize ,it will write snapshot into hdfs first then submit it via hbase table. but I think it's maybe a bug on kylin2.1 when use kylin between two hadoop cluster and build Dimension with keyvalue bigger than hbase.client.keyvalue.maxsize.
when keyvlaue size is bigger than hbase.client.keyvalue.maxsize , writte snapshot into hdfs( it got the hdfs path from hive table instead of hbase, and they are in different hdfs cluster) . so if you are using one hadoop cluster(hive and hbase are on one hadoop cluster ),it will works well . if you are using two hadoop cluster and your keyvalue size is smaller than hbase.client.keyvalue.maxsize ,it will also works well (use hbase's put object). but if you are using two hadoop cluster and your keyvalue is bigger than hbase.client.keyvalue.maxsize , you will get then same error like mine(beause your hive and hbase are in different hdfs cluster) .
I checked my hbase.client.keyvalue.maxsize in hbase-site.xml, and found ambari set 1MB as default ,and kylin set 10MB as default .so I changed this value to 10MB , and now my model works well. If you find a better way to resolve this problem ,please tell me , thank you.
apache-kylin-2.1.0-src\apache-kylin-2.1.0\storage-hbase\src\main\java\org\apache\kylin\storage\hbase\HBaseResourceStore.java
------------------ 原始邮件 ------------------
发件人: "ShaoFeng Shi";<sh...@apache.org>;
发送时间: 2017年12月22日(星期五) 下午5:49
收件人: "user"<us...@kylin.apache.org>;
主题: Re: Kylin on EMR Installation Error
Please check https://issues.apache.org/jira/browse/KYLIN-3129
2017-09-06 22:31 GMT+08:00 ShaoFeng Shi <sh...@apache.org>:
Thanks, I will try EMR 5.8 later;
At this moment, you can try emr 5.5; one thing to pay attention is, you need copy a configuration from hbase-site.xml to KYLIN_HOME/conf/kylin_job_conf.xml (replace the zk quorum name with yours):
<property>
<name>hbase.zookeeper.quorum</name>
<value>ip-nn-nn-nn-nn.ap-northeast-2.compute.internal</value>
</property>
2017-09-06 21:07 GMT+08:00 Mukund Agilisium <mu...@gmail.com>:
Hi
I am using EMR 5.8.0 , the latest one.
Thanks & Regards
Mukund
On Wed, Sep 6, 2017 at 8:44 AM, ShaoFeng Shi <sh...@apache.org> wrote:
Hi Mukund,
What's the EMR version?
Check if there is other version's jar in EMR node, e.g,:
sudo find / -name joda-time*.jar
2017-09-06 1:52 GMT+08:00 Mukund Agilisium <mu...@gmail.com>:
Hi All
I am trying to install Kylin apache-kylin-2.1.0-bin-hbase1x on EMR and getting the following errors.
[root@ip-172-31-35-25 lib]# pwd
/etc/apache-kylin-2.1.0-bin-hbase1x/tomcat/webapps/kylin/WEB-INF/lib
[root@ip-172-31-35-25 lib]# ls -al | grep joda
-rw-r--r-- 1 root root 573912 May 6 2016 joda-time-2.2.jar
Looks like the jar is available inside the webinf/lib as well.
Any help is appreciated!!
kylin.log
2017-09-05 12:06:45,097 ERROR [localhost-startStop-1] context.ContextLoader:351 : Context initialization failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter': Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter]: Constructor threw exception; nested exception is java.lang.NoSuchMethodError: org.joda.time.format.DateTimeFormatter.withZoneUTC()Lorg/joda/time/format/DateTimeFormatter;
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateBean(AbstractAutowireCapableBeanFactory.java:1109)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1054)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:510)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:778)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:839)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:538)
at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:444)
at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:326)
......
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter]: Constructor threw exception; nested exception is java.lang.NoSuchMethodError: org.joda.time.format.DateTimeFormatter.withZoneUTC()Lorg/joda/time/format/DateTimeFormatter;
at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:163)
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:89)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateBean(AbstractAutowireCapableBeanFactory.java:1102)
... 26 more
SEVERE: Exception sending context initialized event to listener instance of class org.springframework.web.context.ContextLoaderListener
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter': Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter]: Constructor threw exception; nested exception is java.lang.NoSuchMethodError: org.joda.time.format.DateTimeFormatter.withZoneUTC()Lorg/joda/time/format/DateTimeFormatter;
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateBean(AbstractAutowireCapableBeanFactory.java:1109)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1054)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:510)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:778)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:839)
kylin.out
There is not keystore files in the folder
[root@ip-172-31-35-25 conf]# pwd
/etc/apache-kylin-2.1.0-bin-hbase1x/tomcat/conf
[root@ip-172-31-35-25 conf]# ls *.keystore
ls: cannot access *.keystore: No such file or directory
SEVERE: Failed to load keystore type JKS with path conf/.keystore due to /etc/apache-kylin-2.1.0-bin-hbase1x/tomcat/conf/.keystore (No such file or directory)
java.io.FileNotFoundException: /etc/apache-kylin-2.1.0-bin-hbase1x/tomcat/conf/.keystore (No such file or directory)
at java.io.FileInputStream.open0(Native Method)
at java.io.FileInputStream.open(FileInputStream.java:195)
at java.io.FileInputStream.<init>(FileInputStream.java:138)
at java.io.FileInputStream.<init>(FileInputStream.java:93)
Sep 05, 2017 12:06:38 PM org.apache.catalina.core.StandardService initInternal
SEVERE: Failed to initialize connector [Connector[HTTP/1.1-7443]]
org.apache.catalina.LifecycleException: Failed to initialize component [Connector[HTTP/1.1-7443]]
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:106)
at org.apache.catalina.core.StandardService.initInternal(StandardService.java:560)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:102)
at org.apache.catalina.core.StandardServer.initInternal(StandardServer.java:838)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:102)
--
Best regards,
Shaofeng Shi 史少锋
--
Best regards,
Shaofeng Shi 史少锋
--
Best regards,
Shaofeng Shi 史少锋