You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@kylin.apache.org by udana pathirana <ud...@gmail.com> on 2016/09/12 02:58:10 UTC
Kylin UI doesnt show hive databases/tables ?
After lots of trial-and-error I managed to start Kylin Web application
without any exceptions.
Here's the summary in short
* We are running HDP 2.4.2
* HBase server 1.1.2 on Slider
* Kylin installed on an edge node outside the cluster (Kerebros keytab
files are configured on the edge node)
- Kylin node have hbase-client-1.1.2.2.3.4.7-4.jar
- from Kylin node ,I can access hive,hbase CLIs sucessfully
I merged "1.5.x-HBase1.x" into "master" branch and built the binary
package.Managed to solve previous errors related to Spring
initialization/library mismatch..etc
Now I can login to the Kylin UI.However,I cannot see the hive database
list in the UI, When I give the table name manually , it throws the error
"NoSuchObjectException" even though I can see this database/table using
"hive" CLI.
----------------------
java.io.IOException: NoSuchObjectException(message:default.test01 table
not found)
at
org.apache.kylin.source.hive.HiveSourceTableLoader.extractHiveTables(HiveSourceTableLoader.java:102)
at
org.apache.kylin.source.hive.HiveSourceTableLoader.reloadHiveTables(HiveSourceTableLoader.java:74)
at
org.apache.kylin.rest.service.CubeService.reloadHiveTable(CubeService.java:553)
at
org.apache.kylin.rest.service.CubeService$$FastClassBySpringCGLIB$$17a07c0e.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
------------------------------
I see following logs as well :
2016-09-12 02:37:30,263 INFO [http-bio-7070-exec-9]
metastore.HiveMetaStore:756 : 2: get_table : db=default tbl=test01
2016-09-12 02:37:30,264 INFO [http-bio-7070-exec-9]
HiveMetaStore.audit:367 : ugi=udana/ins-kylin102.hadoop.local@C2000.HADOOP
ip=unknown-ip-addr cmd=get_table : db=default tbl=test01
2016-09-12 02:37:30,264 INFO [http-bio-7070-exec-9]
metastore.HiveMetaStore:599 : 2: Opening raw store with implemenation
class:org.apache.hadoop.hive.metastore.ObjectStore
2016-09-12 02:37:30,265 INFO [http-bio-7070-exec-9]
metastore.ObjectStore:294 : ObjectStore, initialize called
2016-09-12 02:37:30,268 INFO [http-bio-7070-exec-9] DataNucleus.Query:77 :
Reading in results for query "org.datanucleus.store.rdbms.query.SQLQuery@0"
since the connection used is closing
2016-09-12 02:37:30,268 INFO [http-bio-7070-exec-9]
metastore.MetaStoreDirectSql:140 : Using direct SQL, underlying DB is DERBY
2016-09-12 02:37:30,268 INFO [http-bio-7070-exec-9]
metastore.ObjectStore:277 : Initialized ObjectStore
1) Why Kylin cannot retrieve Hive databases/tables?
2) Do I have to configure the Kerebros keytab file with Kylin?
Re: Kylin UI doesnt show hive databases/tables ?
Posted by Li Yang <li...@apache.org>.
Note in kylin.sh you can check the actual classpath that used to launch
java.
#debug if encounter NoClassDefError
echo "hbase classpath is:"
hbase classpath
I find it very helpful.
Cheers
Yang
On Tue, Sep 13, 2016 at 11:53 AM, udana pathirana <udana.pathirana@gmail.com
> wrote:
> Thank you.
> There's only one hive-site.xml.But I found a workaround to solve this
> issue.
>
> I noticed that in the UI "System->Server Environment " the CLASSPATH value
> doesn't include the hive/conf folder, even though its detected by the
> "find-hive-dependenct.sh" script.
> So , I created a symbolic link to my "hive-site.xml" in "KYLIN_HOME/conf"
> folder (which is in the CLASSPATH entry list) and now I can see my hive
> databases/tables.
>
>
> On Tue, Sep 13, 2016 at 10:21 AM, ShaoFeng Shi <sh...@apache.org>
> wrote:
>
>> Your analysis is correct; When it failed to locate the proper
>> hive-site.xml, it will use default values which is an embeded metadata
>> store.
>>
>> Please search more widely to see whether there is another hive-site.xml
>> on the classpath. It may hide in hbase's folder or other place.
>>
>> 2016-09-12 14:25 GMT+08:00 udana pathirana <ud...@gmail.com>:
>>
>>> More infomation:
>>>
>>> I have a valid 'hive-site.xml" and the folder is added into the "hive
>>> _dependency" by the "find-hive-dependency.sh" script.
>>> "hive-site.xml" contains valid "hive.metastore.uris" :
>>> thrift://hdp-hive01.hadoop.local:9083,thrift://hdp-hive02.
>>> hadoop.local:9083
>>>
>>> I can see all the databases/tables using hive/beeline client.
>>> I found that HiveMetaStoreClient look for the hite-site.xml in the class
>>> path and if its not found, it will fallback to default settings, which is
>>> DerbDB . but it seems "find-hive-dependency.sh" add the correct folder
>>> to the class path but still why it doesnt work ?
>>>
>>>
>>> On Mon, Sep 12, 2016 at 11:58 AM, udana pathirana <
>>> udana.pathirana@gmail.com> wrote:
>>>
>>>> After lots of trial-and-error I managed to start Kylin Web application
>>>> without any exceptions.
>>>>
>>>> Here's the summary in short
>>>>
>>>> * We are running HDP 2.4.2
>>>> * HBase server 1.1.2 on Slider
>>>> * Kylin installed on an edge node outside the cluster (Kerebros keytab
>>>> files are configured on the edge node)
>>>> - Kylin node have hbase-client-1.1.2.2.3.4.7-4.jar
>>>> - from Kylin node ,I can access hive,hbase CLIs sucessfully
>>>>
>>>>
>>>> I merged "1.5.x-HBase1.x" into "master" branch and built the binary
>>>> package.Managed to solve previous errors related to Spring
>>>> initialization/library mismatch..etc
>>>>
>>>>
>>>> Now I can login to the Kylin UI.However,I cannot see the hive database
>>>> list in the UI, When I give the table name manually , it throws the error
>>>> "NoSuchObjectException" even though I can see this database/table using
>>>> "hive" CLI.
>>>>
>>>> ----------------------
>>>> java.io.IOException: NoSuchObjectException(message:default.test01
>>>> table not found)
>>>> at org.apache.kylin.source.hive.HiveSourceTableLoader.extractHi
>>>> veTables(HiveSourceTableLoader.java:102)
>>>> at org.apache.kylin.source.hive.HiveSourceTableLoader.reloadHiv
>>>> eTables(HiveSourceTableLoader.java:74)
>>>> at org.apache.kylin.rest.service.CubeService.reloadHiveTable(Cu
>>>> beService.java:553)
>>>> at org.apache.kylin.rest.service.CubeService$$FastClassBySpring
>>>> CGLIB$$17a07c0e.invoke(<generated>)
>>>> at org.springframework.cglib.proxy.MethodProxy.invoke(MethodPro
>>>> xy.java:204)
>>>>
>>>> ------------------------------
>>>> I see following logs as well :
>>>> 2016-09-12 02:37:30,263 INFO [http-bio-7070-exec-9]
>>>> metastore.HiveMetaStore:756 : 2: get_table : db=default tbl=test01
>>>> 2016-09-12 02:37:30,264 INFO [http-bio-7070-exec-9]
>>>> HiveMetaStore.audit:367 : ugi=udana/ins-kylin102.hadoop.
>>>> local@C2000.HADOOP ip=unknown-ip-addr cmd=get_table : db=default
>>>> tbl=test01
>>>> 2016-09-12 02:37:30,264 INFO [http-bio-7070-exec-9]
>>>> metastore.HiveMetaStore:599 : 2: Opening raw store with implemenation
>>>> class:org.apache.hadoop.hive.metastore.ObjectStore
>>>> 2016-09-12 02:37:30,265 INFO [http-bio-7070-exec-9]
>>>> metastore.ObjectStore:294 : ObjectStore, initialize called
>>>> 2016-09-12 02:37:30,268 INFO [http-bio-7070-exec-9]
>>>> DataNucleus.Query:77 : Reading in results for query
>>>> "org.datanucleus.store.rdbms.query.SQLQuery@0" since the connection
>>>> used is closing
>>>> 2016-09-12 02:37:30,268 INFO [http-bio-7070-exec-9]
>>>> metastore.MetaStoreDirectSql:140 : Using direct SQL, underlying DB is
>>>> DERBY
>>>> 2016-09-12 02:37:30,268 INFO [http-bio-7070-exec-9]
>>>> metastore.ObjectStore:277 : Initialized ObjectStore
>>>>
>>>>
>>>>
>>>> 1) Why Kylin cannot retrieve Hive databases/tables?
>>>> 2) Do I have to configure the Kerebros keytab file with Kylin?
>>>>
>>>
>>>
>>
>>
>> --
>> Best regards,
>>
>> Shaofeng Shi
>>
>>
>
Re: Kylin UI doesnt show hive databases/tables ?
Posted by udana pathirana <ud...@gmail.com>.
Thank you.
There's only one hive-site.xml.But I found a workaround to solve this issue.
I noticed that in the UI "System->Server Environment " the CLASSPATH value
doesn't include the hive/conf folder, even though its detected by the
"find-hive-dependenct.sh" script.
So , I created a symbolic link to my "hive-site.xml" in "KYLIN_HOME/conf"
folder (which is in the CLASSPATH entry list) and now I can see my hive
databases/tables.
On Tue, Sep 13, 2016 at 10:21 AM, ShaoFeng Shi <sh...@apache.org>
wrote:
> Your analysis is correct; When it failed to locate the proper
> hive-site.xml, it will use default values which is an embeded metadata
> store.
>
> Please search more widely to see whether there is another hive-site.xml on
> the classpath. It may hide in hbase's folder or other place.
>
> 2016-09-12 14:25 GMT+08:00 udana pathirana <ud...@gmail.com>:
>
>> More infomation:
>>
>> I have a valid 'hive-site.xml" and the folder is added into the "hive
>> _dependency" by the "find-hive-dependency.sh" script.
>> "hive-site.xml" contains valid "hive.metastore.uris" :
>> thrift://hdp-hive01.hadoop.local:9083,thrift://hdp-hive02.
>> hadoop.local:9083
>>
>> I can see all the databases/tables using hive/beeline client.
>> I found that HiveMetaStoreClient look for the hite-site.xml in the class
>> path and if its not found, it will fallback to default settings, which is
>> DerbDB . but it seems "find-hive-dependency.sh" add the correct folder
>> to the class path but still why it doesnt work ?
>>
>>
>> On Mon, Sep 12, 2016 at 11:58 AM, udana pathirana <
>> udana.pathirana@gmail.com> wrote:
>>
>>> After lots of trial-and-error I managed to start Kylin Web application
>>> without any exceptions.
>>>
>>> Here's the summary in short
>>>
>>> * We are running HDP 2.4.2
>>> * HBase server 1.1.2 on Slider
>>> * Kylin installed on an edge node outside the cluster (Kerebros keytab
>>> files are configured on the edge node)
>>> - Kylin node have hbase-client-1.1.2.2.3.4.7-4.jar
>>> - from Kylin node ,I can access hive,hbase CLIs sucessfully
>>>
>>>
>>> I merged "1.5.x-HBase1.x" into "master" branch and built the binary
>>> package.Managed to solve previous errors related to Spring
>>> initialization/library mismatch..etc
>>>
>>>
>>> Now I can login to the Kylin UI.However,I cannot see the hive database
>>> list in the UI, When I give the table name manually , it throws the error
>>> "NoSuchObjectException" even though I can see this database/table using
>>> "hive" CLI.
>>>
>>> ----------------------
>>> java.io.IOException: NoSuchObjectException(message:default.test01
>>> table not found)
>>> at org.apache.kylin.source.hive.HiveSourceTableLoader.extractHi
>>> veTables(HiveSourceTableLoader.java:102)
>>> at org.apache.kylin.source.hive.HiveSourceTableLoader.reloadHiv
>>> eTables(HiveSourceTableLoader.java:74)
>>> at org.apache.kylin.rest.service.CubeService.reloadHiveTable(Cu
>>> beService.java:553)
>>> at org.apache.kylin.rest.service.CubeService$$FastClassBySpring
>>> CGLIB$$17a07c0e.invoke(<generated>)
>>> at org.springframework.cglib.proxy.MethodProxy.invoke(MethodPro
>>> xy.java:204)
>>>
>>> ------------------------------
>>> I see following logs as well :
>>> 2016-09-12 02:37:30,263 INFO [http-bio-7070-exec-9]
>>> metastore.HiveMetaStore:756 : 2: get_table : db=default tbl=test01
>>> 2016-09-12 02:37:30,264 INFO [http-bio-7070-exec-9]
>>> HiveMetaStore.audit:367 : ugi=udana/ins-kylin102.hadoop.
>>> local@C2000.HADOOP ip=unknown-ip-addr cmd=get_table : db=default
>>> tbl=test01
>>> 2016-09-12 02:37:30,264 INFO [http-bio-7070-exec-9]
>>> metastore.HiveMetaStore:599 : 2: Opening raw store with implemenation
>>> class:org.apache.hadoop.hive.metastore.ObjectStore
>>> 2016-09-12 02:37:30,265 INFO [http-bio-7070-exec-9]
>>> metastore.ObjectStore:294 : ObjectStore, initialize called
>>> 2016-09-12 02:37:30,268 INFO [http-bio-7070-exec-9]
>>> DataNucleus.Query:77 : Reading in results for query
>>> "org.datanucleus.store.rdbms.query.SQLQuery@0" since the connection
>>> used is closing
>>> 2016-09-12 02:37:30,268 INFO [http-bio-7070-exec-9]
>>> metastore.MetaStoreDirectSql:140 : Using direct SQL, underlying DB is
>>> DERBY
>>> 2016-09-12 02:37:30,268 INFO [http-bio-7070-exec-9]
>>> metastore.ObjectStore:277 : Initialized ObjectStore
>>>
>>>
>>>
>>> 1) Why Kylin cannot retrieve Hive databases/tables?
>>> 2) Do I have to configure the Kerebros keytab file with Kylin?
>>>
>>
>>
>
>
> --
> Best regards,
>
> Shaofeng Shi
>
>
Re: Kylin UI doesnt show hive databases/tables ?
Posted by ShaoFeng Shi <sh...@apache.org>.
Your analysis is correct; When it failed to locate the proper
hive-site.xml, it will use default values which is an embeded metadata
store.
Please search more widely to see whether there is another hive-site.xml on
the classpath. It may hide in hbase's folder or other place.
2016-09-12 14:25 GMT+08:00 udana pathirana <ud...@gmail.com>:
> More infomation:
>
> I have a valid 'hive-site.xml" and the folder is added into the "hive
> _dependency" by the "find-hive-dependency.sh" script.
> "hive-site.xml" contains valid "hive.metastore.uris" :
> thrift://hdp-hive01.hadoop.local:9083,thrift://hdp-
> hive02.hadoop.local:9083
>
> I can see all the databases/tables using hive/beeline client.
> I found that HiveMetaStoreClient look for the hite-site.xml in the class
> path and if its not found, it will fallback to default settings, which is
> DerbDB . but it seems "find-hive-dependency.sh" add the correct folder to
> the class path but still why it doesnt work ?
>
>
> On Mon, Sep 12, 2016 at 11:58 AM, udana pathirana <
> udana.pathirana@gmail.com> wrote:
>
>> After lots of trial-and-error I managed to start Kylin Web application
>> without any exceptions.
>>
>> Here's the summary in short
>>
>> * We are running HDP 2.4.2
>> * HBase server 1.1.2 on Slider
>> * Kylin installed on an edge node outside the cluster (Kerebros keytab
>> files are configured on the edge node)
>> - Kylin node have hbase-client-1.1.2.2.3.4.7-4.jar
>> - from Kylin node ,I can access hive,hbase CLIs sucessfully
>>
>>
>> I merged "1.5.x-HBase1.x" into "master" branch and built the binary
>> package.Managed to solve previous errors related to Spring
>> initialization/library mismatch..etc
>>
>>
>> Now I can login to the Kylin UI.However,I cannot see the hive database
>> list in the UI, When I give the table name manually , it throws the error
>> "NoSuchObjectException" even though I can see this database/table using
>> "hive" CLI.
>>
>> ----------------------
>> java.io.IOException: NoSuchObjectException(message:default.test01 table
>> not found)
>> at org.apache.kylin.source.hive.HiveSourceTableLoader.extractHi
>> veTables(HiveSourceTableLoader.java:102)
>> at org.apache.kylin.source.hive.HiveSourceTableLoader.reloadHiv
>> eTables(HiveSourceTableLoader.java:74)
>> at org.apache.kylin.rest.service.CubeService.reloadHiveTable(Cu
>> beService.java:553)
>> at org.apache.kylin.rest.service.CubeService$$FastClassBySpring
>> CGLIB$$17a07c0e.invoke(<generated>)
>> at org.springframework.cglib.proxy.MethodProxy.invoke(MethodPro
>> xy.java:204)
>>
>> ------------------------------
>> I see following logs as well :
>> 2016-09-12 02:37:30,263 INFO [http-bio-7070-exec-9]
>> metastore.HiveMetaStore:756 : 2: get_table : db=default tbl=test01
>> 2016-09-12 02:37:30,264 INFO [http-bio-7070-exec-9]
>> HiveMetaStore.audit:367 : ugi=udana/ins-kylin102.hadoop.
>> local@C2000.HADOOP ip=unknown-ip-addr cmd=get_table : db=default
>> tbl=test01
>> 2016-09-12 02:37:30,264 INFO [http-bio-7070-exec-9]
>> metastore.HiveMetaStore:599 : 2: Opening raw store with implemenation
>> class:org.apache.hadoop.hive.metastore.ObjectStore
>> 2016-09-12 02:37:30,265 INFO [http-bio-7070-exec-9]
>> metastore.ObjectStore:294 : ObjectStore, initialize called
>> 2016-09-12 02:37:30,268 INFO [http-bio-7070-exec-9] DataNucleus.Query:77
>> : Reading in results for query "org.datanucleus.store.rdbms.q
>> uery.SQLQuery@0" since the connection used is closing
>> 2016-09-12 02:37:30,268 INFO [http-bio-7070-exec-9]
>> metastore.MetaStoreDirectSql:140 : Using direct SQL, underlying DB is
>> DERBY
>> 2016-09-12 02:37:30,268 INFO [http-bio-7070-exec-9]
>> metastore.ObjectStore:277 : Initialized ObjectStore
>>
>>
>>
>> 1) Why Kylin cannot retrieve Hive databases/tables?
>> 2) Do I have to configure the Kerebros keytab file with Kylin?
>>
>
>
--
Best regards,
Shaofeng Shi
Re: Kylin UI doesnt show hive databases/tables ?
Posted by udana pathirana <ud...@gmail.com>.
More infomation:
I have a valid 'hive-site.xml" and the folder is added into the "hive
_dependency" by the "find-hive-dependency.sh" script.
"hive-site.xml" contains valid "hive.metastore.uris" :
thrift://hdp-hive01.hadoop.local:9083,thrift://hdp-hive02.hadoop.local:9083
I can see all the databases/tables using hive/beeline client.
I found that HiveMetaStoreClient look for the hite-site.xml in the class
path and if its not found, it will fallback to default settings, which is
DerbDB . but it seems "find-hive-dependency.sh" add the correct folder to
the class path but still why it doesnt work ?
On Mon, Sep 12, 2016 at 11:58 AM, udana pathirana <udana.pathirana@gmail.com
> wrote:
> After lots of trial-and-error I managed to start Kylin Web application
> without any exceptions.
>
> Here's the summary in short
>
> * We are running HDP 2.4.2
> * HBase server 1.1.2 on Slider
> * Kylin installed on an edge node outside the cluster (Kerebros keytab
> files are configured on the edge node)
> - Kylin node have hbase-client-1.1.2.2.3.4.7-4.jar
> - from Kylin node ,I can access hive,hbase CLIs sucessfully
>
>
> I merged "1.5.x-HBase1.x" into "master" branch and built the binary
> package.Managed to solve previous errors related to Spring
> initialization/library mismatch..etc
>
>
> Now I can login to the Kylin UI.However,I cannot see the hive database
> list in the UI, When I give the table name manually , it throws the error
> "NoSuchObjectException" even though I can see this database/table using
> "hive" CLI.
>
> ----------------------
> java.io.IOException: NoSuchObjectException(message:default.test01 table
> not found)
> at org.apache.kylin.source.hive.HiveSourceTableLoader.extractHiveTables(
> HiveSourceTableLoader.java:102)
> at org.apache.kylin.source.hive.HiveSourceTableLoader.reloadHiveTables(
> HiveSourceTableLoader.java:74)
> at org.apache.kylin.rest.service.CubeService.reloadHiveTable(
> CubeService.java:553)
> at org.apache.kylin.rest.service.CubeService$$FastClassBySpringCGLIB$$
> 17a07c0e.invoke(<generated>)
> at org.springframework.cglib.proxy.MethodProxy.invoke(
> MethodProxy.java:204)
>
> ------------------------------
> I see following logs as well :
> 2016-09-12 02:37:30,263 INFO [http-bio-7070-exec-9]
> metastore.HiveMetaStore:756 : 2: get_table : db=default tbl=test01
> 2016-09-12 02:37:30,264 INFO [http-bio-7070-exec-9]
> HiveMetaStore.audit:367 : ugi=udana/ins-kylin102.hadoop.local@C2000.HADOOP
> ip=unknown-ip-addr cmd=get_table : db=default tbl=test01
> 2016-09-12 02:37:30,264 INFO [http-bio-7070-exec-9]
> metastore.HiveMetaStore:599 : 2: Opening raw store with implemenation
> class:org.apache.hadoop.hive.metastore.ObjectStore
> 2016-09-12 02:37:30,265 INFO [http-bio-7070-exec-9]
> metastore.ObjectStore:294 : ObjectStore, initialize called
> 2016-09-12 02:37:30,268 INFO [http-bio-7070-exec-9] DataNucleus.Query:77
> : Reading in results for query "org.datanucleus.store.rdbms.
> query.SQLQuery@0" since the connection used is closing
> 2016-09-12 02:37:30,268 INFO [http-bio-7070-exec-9]
> metastore.MetaStoreDirectSql:140 : Using direct SQL, underlying DB is
> DERBY
> 2016-09-12 02:37:30,268 INFO [http-bio-7070-exec-9]
> metastore.ObjectStore:277 : Initialized ObjectStore
>
>
>
> 1) Why Kylin cannot retrieve Hive databases/tables?
> 2) Do I have to configure the Kerebros keytab file with Kylin?
>