You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kylin.apache.org by Krishna Bandaru <Kr...@scanbuy.com> on 2019/07/17 22:10:15 UTC
Regarding hive metadata
Hi,
I am not able to sync the table from hive to Kylin data source.
I have a table kylin_cpg_dg in the hive:
[cid:002da0b1-411f-4e83-b8ca-231963faaa22]
I am not able to sync this in Kylin
[cid:50741009-f1ee-46bc-be31-e07cc404b2ee]
here are my logs from three servers in HA mode.
1)
2019-07-17 21:20:16,885 DEBUG [http-bio-7070-exec-76] security.KylinAuthenticationProvider:126 : Authenticated user org.springframework.security.authentication.UsernamePasswordAuthenticationToken@3704b3d2: Principal: org.springframework.security.core.userdetails.User@3b40b2f: Username: ADMIN; Password: [PROTECTED]; Enabled: true; AccountNonExpired: true; credentialsNonExpired: true; AccountNonLocked: true; Granted Authorities: ROLE_ADMIN,ROLE_ANALYST,ROLE_MODELER; Credentials: [PROTECTED]; Authenticated: true; Details: org.springframework.security.web.authentication.WebAuthenticationDetails@ffff8868: RemoteIpAddress: 24.30.233.234; SessionId: null; Granted Authorities: ROLE_ADMIN, ROLE_ANALYST, ROLE_MODELER
2019-07-17 21:20:16,887 DEBUG [http-bio-7070-exec-76] controller.UserController:52 : User login: org.springframework.security.core.userdetails.User@3b40b2f: Username: ADMIN; Password: [PROTECTED]; Enabled: true; AccountNonExpired: true; credentialsNonExpired: true; AccountNonLocked: true; Granted Authorities: ROLE_ADMIN,ROLE_ANALYST,ROLE_MODELER
2019-07-17 21:20:16,889 DEBUG [http-bio-7070-exec-62] cachesync.Broadcaster:247 : Broadcasting UPDATE, user, ADMIN
2019-07-17 21:20:16,890 DEBUG [http-bio-7070-exec-62] cachesync.Broadcaster:281 : Done broadcasting UPDATE, user, ADMIN
2019-07-17 21:20:18,606 INFO [BadQueryDetector] service.BadQueryDetector:147 : Detect bad query.
2019-07-17 21:20:20,900 INFO [FetcherRunner 352123462-50] threadpool.DefaultFetcherRunner:96 : Job Fetcher: 0 should running, 0 actual running, 0 stopped, 0 ready, 0 already succeed, 1 error, 0 discarded, 0 others
2019-07-17 21:20:24,432 DEBUG [http-bio-7070-exec-34] project.ProjectL2Cache:199 : Loading L2 project cache for learn_kylin
2019-07-17 21:20:24,433 WARN [http-bio-7070-exec-34] realization.RealizationRegistry:91 : No provider for realization type INVERTED_INDEX
2019-07-17 21:20:24,433 WARN [http-bio-7070-exec-34] realization.RealizationRegistry:91 : No provider for realization type INVERTED_INDEX
2019-07-17 21:20:24,434 WARN [http-bio-7070-exec-34] realization.RealizationRegistry:91 : No provider for realization type INVERTED_INDEX
2019-07-17 21:20:26,681 INFO [http-bio-7070-exec-34] hive.metastore:407 : Trying to connect to metastore with URI thrift://ip-10-0-0-46.ec2.internal:9083
2019-07-17 21:20:26,682 INFO [http-bio-7070-exec-34] hive.metastore:481 : Opened a connection to metastore, current connections: 4
2019-07-17 21:20:26,684 INFO [http-bio-7070-exec-34] hive.metastore:534 : Connected to metastore.
2019-07-17 21:20:27,833 INFO [http-bio-7070-exec-34] hive.metastore:407 : Trying to connect to metastore with URI thrift://ip-10-0-0-45.ec2.internal:9083
2019-07-17 21:20:27,834 INFO [http-bio-7070-exec-34] hive.metastore:481 : Opened a connection to metastore, current connections: 5
2019-07-17 21:20:27,835 INFO [http-bio-7070-exec-34] hive.metastore:534 : Connected to metastore.
2019-07-17 21:20:50,900 INFO [FetcherRunner 352123462-50] threadpool.DefaultFetcherRunner:96 : Job Fetcher: 0 should running, 0 actual running, 0 stopped, 0 ready, 0 already succeed, 1 error, 0 discarded, 0 others
2019-07-17 21:21:18,606 INFO [BadQueryDetector] service.BadQueryDetector:147 : Detect bad query.
2019-07-17 21:21:20,922 INFO [FetcherRunner 352123462-50] threadpool.DefaultFetcherRunner:96 : Job Fetcher: 0 should running, 0 actual running, 0 stopped, 0 ready, 0 already succeed, 1 error, 0 discarded, 0 others
2019-07-17 21:21:50,900 INFO [FetcherRunner 352123462-50] threadpool.DefaultFetcherRunner:96 : Job Fetcher: 0 should running, 0 actual running, 0 stopped, 0 ready, 0 already succeed, 1 error, 0 discarded, 0 others
2019-07-17 21:21:54,153 DEBUG [http-bio-7070-exec-34] cachesync.Broadcaster:247 : Broadcasting UPDATE, user, ADMIN
2019-07-17 21:21:54,155 DEBUG [http-bio-7070-exec-34] cachesync.Broadcaster:281 : Done broadcasting UPDATE, user, ADMIN
2019-07-17 21:22:18,607 INFO [BadQueryDetector] service.BadQueryDetector:147 : Detect bad query.
2019-07-17 21:22:20,899 INFO [FetcherRunner 352123462-50] threadpool.DefaultFetcherRunner:96 : Job Fetcher: 0 should running, 0 actual running, 0 stopped, 0 ready, 0 already succeed, 1 error, 0 discarded, 0 others
2019-07-17 21:22:50,920 INFO [FetcherRunner 352123462-50] threadpool.DefaultFetcherRunner:96 : Job Fetcher: 0 should running, 0 actual running, 0 stopped, 0 ready, 0 already succeed, 1 error, 0 discarded, 0 others
2019-07-17 21:23:18,607 INFO [BadQueryDetector] service.BadQueryDetector:147 : Detect bad query.
2)
2019-07-17 21:23:16,821 INFO [http-bio-7070-exec-75] common.KylinConfig:332 : Loading kylin-defaults.properties from file:/usr/local/kylin/apache-kylin-2.5.0-bin-hbase1x/tomcat/webapps/kylin/WEB-INF/lib/kylin-core-common-2.5.0.jar!/kylin-defaults.properties
2019-07-17 21:23:16,836 DEBUG [http-bio-7070-exec-75] common.KylinConfig:291 : KYLIN_CONF property was not set, will seek KYLIN_HOME env variable
2019-07-17 21:23:22,976 DEBUG [http-bio-7070-exec-86] badquery.BadQueryHistoryManager:65 : Loaded 0 Bad Query(s)
2019-07-17 21:23:27,717 INFO [BadQueryDetector] service.BadQueryDetector:147 : Detect bad query.
2019-07-17 21:23:30,468 INFO [http-bio-7070-exec-86] hive.metastore:407 : Trying to connect to metastore with URI thrift://ip-10-0-0-46.ec2.internal:9083
2019-07-17 21:23:30,470 INFO [http-bio-7070-exec-86] hive.metastore:481 : Opened a connection to metastore, current connections: 4
2019-07-17 21:23:30,471 INFO [http-bio-7070-exec-86] hive.metastore:534 : Connected to metastore.
2019-07-17 21:23:32,486 INFO [http-bio-7070-exec-68] hive.metastore:407 : Trying to connect to metastore with URI thrift://ip-10-0-0-46.ec2.internal:9083
2019-07-17 21:23:32,487 INFO [http-bio-7070-exec-68] hive.metastore:481 : Opened a connection to metastore, current connections: 5
2019-07-17 21:23:32,488 INFO [http-bio-7070-exec-68] hive.metastore:534 : Connected to metastore.
2019-07-17 21:24:27,717 INFO [BadQueryDetector] service.BadQueryDetector:147 : Detect bad query.
2019-07-17 21:25:27,718 INFO [BadQueryDetector] service.BadQueryDetector:147 : Detect bad query.
2019-07-17 21:26:25,136 INFO [Idle-Rpc-Conn-Sweeper-pool2-t1] ipc.AbstractRpcClient:217 : Cleanup idle connection to ip-10-0-0-54.ec2.internal/10.0.0.54:16020
2019-07-17 21:26:27,718 INFO [BadQueryDetector] service.BadQueryDetector:147 : Detect bad query.
2019-07-17 21:27:27,719 INFO [BadQueryDetector] service.BadQueryDetector:147 : Detect bad query.
2019-07-17 21:28:27,719 INFO [BadQueryDetector] service.BadQueryDetector:147 : Detect bad query.
2019-07-17 21:29:27,720 INFO [BadQueryDetector] service.BadQueryDetector:147 : Detect bad query.
2019-07-17 21:30:27,720 INFO [BadQueryDetector] service.BadQueryDetector:147 : Detect bad query.
2019-07-17 21:31:27,721 INFO [BadQueryDetector] service.BadQueryDetector:147 : Detect bad query.
2019-07-17 21:32:27,721 INFO [BadQueryDetector] service.BadQueryDetector:147 : Detect bad query.
2019-07-17 21:33:27,721 INFO [BadQueryDetector] service.BadQueryDetector:147 : Detect bad query.
2019-07-17 21:34:27,722 INFO [BadQueryDetector] service.BadQueryDetector:147 : Detect bad query.
3)
2019-07-17 20:49:46,875 DEBUG [http-bio-7070-exec-4] security.KylinAuthenticationProvider:57 : User cache [-108, 112, -63, -32, 41, -87, -81, 81, -32, 61, -35, -111, 7, 56, -29, -59] is removed due to EXPLICIT
2019-07-17 20:49:46,973 DEBUG [http-bio-7070-exec-4] security.KylinAuthenticationProvider:113 : User ADMIN authorities : [ROLE_ADMIN, ROLE_ANALYST, ROLE_MODELER]
2019-07-17 20:49:46,973 DEBUG [http-bio-7070-exec-4] cachesync.CachedCrudAssist:190 : Saving ManagedUser at /user/ADMIN
2019-07-17 20:49:46,976 DEBUG [pool-5-thread-1] cachesync.Broadcaster:113 : Servers in the cluster: [ip-10-0-0-45.ec2.internal:7070, ip-10-0-0-46.ec2.internal:7070, ip-10-0-0-51.ec2.internal:7070]
2019-07-17 20:49:46,977 DEBUG [pool-5-thread-1] cachesync.Broadcaster:123 : Announcing new broadcast to all: BroadcastEvent{entity=user, event=update, cacheKey=ADMIN}
2019-07-17 20:49:46,978 DEBUG [http-bio-7070-exec-4] security.KylinAuthenticationProvider:126 : Authenticated user org.springframework.security.authentication.UsernamePasswordAuthenticationToken@37047726: Principal: org.springframework.security.core.userdetails.User@3b40b2f: Username: ADMIN; Password: [PROTECTED]; Enabled: true; AccountNonExpired: true; credentialsNonExpired: true; AccountNonLocked: true; Granted Authorities: ROLE_ADMIN,ROLE_ANALYST,ROLE_MODELER; Credentials: [PROTECTED]; Authenticated: true; Details: org.springframework.security.web.authentication.WebAuthenticationDetails@ffff4c9c: RemoteIpAddress: 10.0.0.59; SessionId: null; Granted Authorities: ROLE_ADMIN, ROLE_ANALYST, ROLE_MODELER
2019-07-17 20:49:46,992 DEBUG [http-bio-7070-exec-18] cachesync.Broadcaster:247 : Broadcasting UPDATE, user, ADMIN
2019-07-17 20:49:46,994 DEBUG [http-bio-7070-exec-18] cachesync.Broadcaster:281 : Done broadcasting UPDATE, user, ADMIN
2019-07-17 20:49:47,051 DEBUG [http-bio-7070-exec-20] cachesync.Broadcaster:247 : Broadcasting UPDATE, user, ADMIN
2019-07-17 20:49:47,053 DEBUG [http-bio-7070-exec-20] cachesync.Broadcaster:281 : Done broadcasting UPDATE, user, ADMIN
2019-07-17 20:50:02,894 DEBUG [http-bio-7070-exec-20] cachesync.Broadcaster:247 : Broadcasting UPDATE, user, ADMIN
2019-07-17 20:50:02,897 DEBUG [http-bio-7070-exec-20] cachesync.Broadcaster:281 : Done broadcasting UPDATE, user, ADMIN
2019-07-17 20:50:03,033 DEBUG [http-bio-7070-exec-9] cachesync.Broadcaster:247 : Broadcasting UPDATE, user, ADMIN
2019-07-17 20:50:03,035 DEBUG [http-bio-7070-exec-9] cachesync.Broadcaster:281 : Done broadcasting UPDATE, user, ADMIN
2019-07-17 20:50:03,159 DEBUG [http-bio-7070-exec-7] cachesync.Broadcaster:247 : Broadcasting UPDATE, user, ADMIN
2019-07-17 20:50:03,161 DEBUG [http-bio-7070-exec-7] cachesync.Broadcaster:281 : Done broadcasting UPDATE, user, ADMIN
I am looking forward to hearing from you.
Thanks,
Krishna
Re: Regarding hive metadata
Posted by Krishna Bandaru <Kr...@scanbuy.com>.
yes, wangweilin
export HBASE_CLASSPATH=$HADOOP_CONF:$HADOOP_HOME/*:$HADOOP_HOME/lib/*:$ZOOKEEPER_HOME/*:$ZOOKEEPER_HOME/lib/*:$HBASE_CLASSPATH
export HBASE_HEAPSIZE=1024
export HBASE_OPTS="$HBASE_OPTS -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=70 -Dsun.net.inetaddr.ttl=60 -Dnetworkaddress.cache.ttl=60"
________________________________
From: wangweilin <sd...@163.com>
Sent: Saturday, July 20, 2019 9:05 AM
To: user@kylin.apache.org <us...@kylin.apache.org>
Subject: Re: Regarding hive metadata
Have you put hbase's classpath in hbase-env.sh?
<https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?ftlId=1&name=wangweilin&uid=sdnuwangweilin%40163.com&iconUrl=http%3A%2F%2Fmail-online.nosdn.127.net%2Fsm34b3cc94dce5f170cfe6377531ec0cf4.jpg&items=%5B%22sdnuwangweilin%40163.com%22%5D>
[http://mail-online.nosdn.127.net/sm34b3cc94dce5f170cfe6377531ec0cf4.jpg]
wangweilin
sdnuwangweilin@163.com
签名由 网易邮箱大师<https://mail.163.com/dashi/dlpro.html?from=mail81> 定制
On 7/18/2019 06:10，Krishna Bandaru<Kr...@scanbuy.com> wrote：
Hi,
I am not able to sync the table from hive to Kylin data source.
I have a table kylin_cpg_dg in the hive:
[cid:7bd0e7d8$1$16c0f7c9b5c$Coremail$sdnuwangweilin$163.com]
I am not able to sync this in Kylin
[cid:5d8ad6c7$2$16c0f7c9b5d$Coremail$sdnuwangweilin$163.com]
here are my logs from three servers in HA mode.
1)
2019-07-17 21:20:16,885 DEBUG [http-bio-7070-exec-76] security.KylinAuthenticationProvider:126 : Authenticated user org.springframework.security.authentication.UsernamePasswordAuthenticationToken@3704b3d2: Principal: org.springframework.security.core.userdetails.User@3b40b2f: Username: ADMIN; Password: [PROTECTED]; Enabled: true; AccountNonExpired: true; credentialsNonExpired: true; AccountNonLocked: true; Granted Authorities: ROLE_ADMIN,ROLE_ANALYST,ROLE_MODELER; Credentials: [PROTECTED]; Authenticated: true; Details: org.springframework.security.web.authentication.WebAuthenticationDetails@ffff8868: RemoteIpAddress: 24.30.233.234; SessionId: null; Granted Authorities: ROLE_ADMIN, ROLE_ANALYST, ROLE_MODELER
2019-07-17 21:20:16,887 DEBUG [http-bio-7070-exec-76] controller.UserController:52 : User login: org.springframework.security.core.userdetails.User@3b40b2f: Username: ADMIN; Password: [PROTECTED]; Enabled: true; AccountNonExpired: true; credentialsNonExpired: true; AccountNonLocked: true; Granted Authorities: ROLE_ADMIN,ROLE_ANALYST,ROLE_MODELER
2019-07-17 21:20:16,889 DEBUG [http-bio-7070-exec-62] cachesync.Broadcaster:247 : Broadcasting UPDATE, user, ADMIN
2019-07-17 21:20:16,890 DEBUG [http-bio-7070-exec-62] cachesync.Broadcaster:281 : Done broadcasting UPDATE, user, ADMIN
2019-07-17 21:20:18,606 INFO [BadQueryDetector] service.BadQueryDetector:147 : Detect bad query.
2019-07-17 21:20:20,900 INFO [FetcherRunner 352123462-50] threadpool.DefaultFetcherRunner:96 : Job Fetcher: 0 should running, 0 actual running, 0 stopped, 0 ready, 0 already succeed, 1 error, 0 discarded, 0 others
2019-07-17 21:20:24,432 DEBUG [http-bio-7070-exec-34] project.ProjectL2Cache:199 : Loading L2 project cache for learn_kylin
2019-07-17 21:20:24,433 WARN [http-bio-7070-exec-34] realization.RealizationRegistry:91 : No provider for realization type INVERTED_INDEX
2019-07-17 21:20:24,433 WARN [http-bio-7070-exec-34] realization.RealizationRegistry:91 : No provider for realization type INVERTED_INDEX
2019-07-17 21:20:24,434 WARN [http-bio-7070-exec-34] realization.RealizationRegistry:91 : No provider for realization type INVERTED_INDEX
2019-07-17 21:20:26,681 INFO [http-bio-7070-exec-34] hive.metastore:407 : Trying to connect to metastore with URI thrift://ip-10-0-0-46.ec2.internal:9083
2019-07-17 21:20:26,682 INFO [http-bio-7070-exec-34] hive.metastore:481 : Opened a connection to metastore, current connections: 4
2019-07-17 21:20:26,684 INFO [http-bio-7070-exec-34] hive.metastore:534 : Connected to metastore.
2019-07-17 21:20:27,833 INFO [http-bio-7070-exec-34] hive.metastore:407 : Trying to connect to metastore with URI thrift://ip-10-0-0-45.ec2.internal:9083
2019-07-17 21:20:27,834 INFO [http-bio-7070-exec-34] hive.metastore:481 : Opened a connection to metastore, current connections: 5
2019-07-17 21:20:27,835 INFO [http-bio-7070-exec-34] hive.metastore:534 : Connected to metastore.
2019-07-17 21:20:50,900 INFO [FetcherRunner 352123462-50] threadpool.DefaultFetcherRunner:96 : Job Fetcher: 0 should running, 0 actual running, 0 stopped, 0 ready, 0 already succeed, 1 error, 0 discarded, 0 others
2019-07-17 21:21:18,606 INFO [BadQueryDetector] service.BadQueryDetector:147 : Detect bad query.
2019-07-17 21:21:20,922 INFO [FetcherRunner 352123462-50] threadpool.DefaultFetcherRunner:96 : Job Fetcher: 0 should running, 0 actual running, 0 stopped, 0 ready, 0 already succeed, 1 error, 0 discarded, 0 others
2019-07-17 21:21:50,900 INFO [FetcherRunner 352123462-50] threadpool.DefaultFetcherRunner:96 : Job Fetcher: 0 should running, 0 actual running, 0 stopped, 0 ready, 0 already succeed, 1 error, 0 discarded, 0 others
2019-07-17 21:21:54,153 DEBUG [http-bio-7070-exec-34] cachesync.Broadcaster:247 : Broadcasting UPDATE, user, ADMIN
2019-07-17 21:21:54,155 DEBUG [http-bio-7070-exec-34] cachesync.Broadcaster:281 : Done broadcasting UPDATE, user, ADMIN
2019-07-17 21:22:18,607 INFO [BadQueryDetector] service.BadQueryDetector:147 : Detect bad query.
2019-07-17 21:22:20,899 INFO [FetcherRunner 352123462-50] threadpool.DefaultFetcherRunner:96 : Job Fetcher: 0 should running, 0 actual running, 0 stopped, 0 ready, 0 already succeed, 1 error, 0 discarded, 0 others
2019-07-17 21:22:50,920 INFO [FetcherRunner 352123462-50] threadpool.DefaultFetcherRunner:96 : Job Fetcher: 0 should running, 0 actual running, 0 stopped, 0 ready, 0 already succeed, 1 error, 0 discarded, 0 others
2019-07-17 21:23:18,607 INFO [BadQueryDetector] service.BadQueryDetector:147 : Detect bad query.
2)
2019-07-17 21:23:16,821 INFO [http-bio-7070-exec-75] common.KylinConfig:332 : Loading kylin-defaults.properties from file:/usr/local/kylin/apache-kylin-2.5.0-bin-hbase1x/tomcat/webapps/kylin/WEB-INF/lib/kylin-core-common-2.5.0.jar!/kylin-defaults.properties
2019-07-17 21:23:16,836 DEBUG [http-bio-7070-exec-75] common.KylinConfig:291 : KYLIN_CONF property was not set, will seek KYLIN_HOME env variable
2019-07-17 21:23:22,976 DEBUG [http-bio-7070-exec-86] badquery.BadQueryHistoryManager:65 : Loaded 0 Bad Query(s)
2019-07-17 21:23:27,717 INFO [BadQueryDetector] service.BadQueryDetector:147 : Detect bad query.
2019-07-17 21:23:30,468 INFO [http-bio-7070-exec-86] hive.metastore:407 : Trying to connect to metastore with URI thrift://ip-10-0-0-46.ec2.internal:9083
2019-07-17 21:23:30,470 INFO [http-bio-7070-exec-86] hive.metastore:481 : Opened a connection to metastore, current connections: 4
2019-07-17 21:23:30,471 INFO [http-bio-7070-exec-86] hive.metastore:534 : Connected to metastore.
2019-07-17 21:23:32,486 INFO [http-bio-7070-exec-68] hive.metastore:407 : Trying to connect to metastore with URI thrift://ip-10-0-0-46.ec2.internal:9083
2019-07-17 21:23:32,487 INFO [http-bio-7070-exec-68] hive.metastore:481 : Opened a connection to metastore, current connections: 5
2019-07-17 21:23:32,488 INFO [http-bio-7070-exec-68] hive.metastore:534 : Connected to metastore.
2019-07-17 21:24:27,717 INFO [BadQueryDetector] service.BadQueryDetector:147 : Detect bad query.
2019-07-17 21:25:27,718 INFO [BadQueryDetector] service.BadQueryDetector:147 : Detect bad query.
2019-07-17 21:26:25,136 INFO [Idle-Rpc-Conn-Sweeper-pool2-t1] ipc.AbstractRpcClient:217 : Cleanup idle connection to ip-10-0-0-54.ec2.internal/10.0.0.54:16020
2019-07-17 21:26:27,718 INFO [BadQueryDetector] service.BadQueryDetector:147 : Detect bad query.
2019-07-17 21:27:27,719 INFO [BadQueryDetector] service.BadQueryDetector:147 : Detect bad query.
2019-07-17 21:28:27,719 INFO [BadQueryDetector] service.BadQueryDetector:147 : Detect bad query.
2019-07-17 21:29:27,720 INFO [BadQueryDetector] service.BadQueryDetector:147 : Detect bad query.
2019-07-17 21:30:27,720 INFO [BadQueryDetector] service.BadQueryDetector:147 : Detect bad query.
2019-07-17 21:31:27,721 INFO [BadQueryDetector] service.BadQueryDetector:147 : Detect bad query.
2019-07-17 21:32:27,721 INFO [BadQueryDetector] service.BadQueryDetector:147 : Detect bad query.
2019-07-17 21:33:27,721 INFO [BadQueryDetector] service.BadQueryDetector:147 : Detect bad query.
2019-07-17 21:34:27,722 INFO [BadQueryDetector] service.BadQueryDetector:147 : Detect bad query.
3)
2019-07-17 20:49:46,875 DEBUG [http-bio-7070-exec-4] security.KylinAuthenticationProvider:57 : User cache [-108, 112, -63, -32, 41, -87, -81, 81, -32, 61, -35, -111, 7, 56, -29, -59] is removed due to EXPLICIT
2019-07-17 20:49:46,973 DEBUG [http-bio-7070-exec-4] security.KylinAuthenticationProvider:113 : User ADMIN authorities : [ROLE_ADMIN, ROLE_ANALYST, ROLE_MODELER]
2019-07-17 20:49:46,973 DEBUG [http-bio-7070-exec-4] cachesync.CachedCrudAssist:190 : Saving ManagedUser at /user/ADMIN
2019-07-17 20:49:46,976 DEBUG [pool-5-thread-1] cachesync.Broadcaster:113 : Servers in the cluster: [ip-10-0-0-45.ec2.internal:7070, ip-10-0-0-46.ec2.internal:7070, ip-10-0-0-51.ec2.internal:7070]
2019-07-17 20:49:46,977 DEBUG [pool-5-thread-1] cachesync.Broadcaster:123 : Announcing new broadcast to all: BroadcastEvent{entity=user, event=update, cacheKey=ADMIN}
2019-07-17 20:49:46,978 DEBUG [http-bio-7070-exec-4] security.KylinAuthenticationProvider:126 : Authenticated user org.springframework.security.authentication.UsernamePasswordAuthenticationToken@37047726: Principal: org.springframework.security.core.userdetails.User@3b40b2f: Username: ADMIN; Password: [PROTECTED]; Enabled: true; AccountNonExpired: true; credentialsNonExpired: true; AccountNonLocked: true; Granted Authorities: ROLE_ADMIN,ROLE_ANALYST,ROLE_MODELER; Credentials: [PROTECTED]; Authenticated: true; Details: org.springframework.security.web.authentication.WebAuthenticationDetails@ffff4c9c: RemoteIpAddress: 10.0.0.59; SessionId: null; Granted Authorities: ROLE_ADMIN, ROLE_ANALYST, ROLE_MODELER
2019-07-17 20:49:46,992 DEBUG [http-bio-7070-exec-18] cachesync.Broadcaster:247 : Broadcasting UPDATE, user, ADMIN
2019-07-17 20:49:46,994 DEBUG [http-bio-7070-exec-18] cachesync.Broadcaster:281 : Done broadcasting UPDATE, user, ADMIN
2019-07-17 20:49:47,051 DEBUG [http-bio-7070-exec-20] cachesync.Broadcaster:247 : Broadcasting UPDATE, user, ADMIN
2019-07-17 20:49:47,053 DEBUG [http-bio-7070-exec-20] cachesync.Broadcaster:281 : Done broadcasting UPDATE, user, ADMIN
2019-07-17 20:50:02,894 DEBUG [http-bio-7070-exec-20] cachesync.Broadcaster:247 : Broadcasting UPDATE, user, ADMIN
2019-07-17 20:50:02,897 DEBUG [http-bio-7070-exec-20] cachesync.Broadcaster:281 : Done broadcasting UPDATE, user, ADMIN
2019-07-17 20:50:03,033 DEBUG [http-bio-7070-exec-9] cachesync.Broadcaster:247 : Broadcasting UPDATE, user, ADMIN
2019-07-17 20:50:03,035 DEBUG [http-bio-7070-exec-9] cachesync.Broadcaster:281 : Done broadcasting UPDATE, user, ADMIN
2019-07-17 20:50:03,159 DEBUG [http-bio-7070-exec-7] cachesync.Broadcaster:247 : Broadcasting UPDATE, user, ADMIN
2019-07-17 20:50:03,161 DEBUG [http-bio-7070-exec-7] cachesync.Broadcaster:281 : Done broadcasting UPDATE, user, ADMIN
I am looking forward to hearing from you.
Thanks,
Krishna
Re: Regarding hive metadata
Posted by wangweilin <sd...@163.com>.
Have you put hbase's classpath in hbase-env.sh?
| |
wangweilin
|
|
sdnuwangweilin@163.com
|
签名由网易邮箱大师定制
On 7/18/2019 06:10，Krishna Bandaru<Kr...@scanbuy.com> wrote：
Hi,
I am not able to sync the table from hive to Kylin data source.
I have a table kylin_cpg_dg in the hive:
I am not able to sync this in Kylin
here are my logs from three servers in HA mode.
1)
2019-07-17 21:20:16,885 DEBUG [http-bio-7070-exec-76] security.KylinAuthenticationProvider:126 : Authenticated user org.springframework.security.authentication.UsernamePasswordAuthenticationToken@3704b3d2: Principal: org.springframework.security.core.userdetails.User@3b40b2f: Username: ADMIN; Password: [PROTECTED]; Enabled: true; AccountNonExpired: true; credentialsNonExpired: true; AccountNonLocked: true; Granted Authorities: ROLE_ADMIN,ROLE_ANALYST,ROLE_MODELER; Credentials: [PROTECTED]; Authenticated: true; Details: org.springframework.security.web.authentication.WebAuthenticationDetails@ffff8868: RemoteIpAddress: 24.30.233.234; SessionId: null; Granted Authorities: ROLE_ADMIN, ROLE_ANALYST, ROLE_MODELER
2019-07-17 21:20:16,887 DEBUG [http-bio-7070-exec-76] controller.UserController:52 : User login: org.springframework.security.core.userdetails.User@3b40b2f: Username: ADMIN; Password: [PROTECTED]; Enabled: true; AccountNonExpired: true; credentialsNonExpired: true; AccountNonLocked: true; Granted Authorities: ROLE_ADMIN,ROLE_ANALYST,ROLE_MODELER
2019-07-17 21:20:16,889 DEBUG [http-bio-7070-exec-62] cachesync.Broadcaster:247 : Broadcasting UPDATE, user, ADMIN
2019-07-17 21:20:16,890 DEBUG [http-bio-7070-exec-62] cachesync.Broadcaster:281 : Done broadcasting UPDATE, user, ADMIN
2019-07-17 21:20:18,606 INFO [BadQueryDetector] service.BadQueryDetector:147 : Detect bad query.
2019-07-17 21:20:20,900 INFO [FetcherRunner 352123462-50] threadpool.DefaultFetcherRunner:96 : Job Fetcher: 0 should running, 0 actual running, 0 stopped, 0 ready, 0 already succeed, 1 error, 0 discarded, 0 others
2019-07-17 21:20:24,432 DEBUG [http-bio-7070-exec-34] project.ProjectL2Cache:199 : Loading L2 project cache for learn_kylin
2019-07-17 21:20:24,433 WARN [http-bio-7070-exec-34] realization.RealizationRegistry:91 : No provider for realization type INVERTED_INDEX
2019-07-17 21:20:24,433 WARN [http-bio-7070-exec-34] realization.RealizationRegistry:91 : No provider for realization type INVERTED_INDEX
2019-07-17 21:20:24,434 WARN [http-bio-7070-exec-34] realization.RealizationRegistry:91 : No provider for realization type INVERTED_INDEX
2019-07-17 21:20:26,681 INFO [http-bio-7070-exec-34] hive.metastore:407 : Trying to connect to metastore with URI thrift://ip-10-0-0-46.ec2.internal:9083
2019-07-17 21:20:26,682 INFO [http-bio-7070-exec-34] hive.metastore:481 : Opened a connection to metastore, current connections: 4
2019-07-17 21:20:26,684 INFO [http-bio-7070-exec-34] hive.metastore:534 : Connected to metastore.
2019-07-17 21:20:27,833 INFO [http-bio-7070-exec-34] hive.metastore:407 : Trying to connect to metastore with URI thrift://ip-10-0-0-45.ec2.internal:9083
2019-07-17 21:20:27,834 INFO [http-bio-7070-exec-34] hive.metastore:481 : Opened a connection to metastore, current connections: 5
2019-07-17 21:20:27,835 INFO [http-bio-7070-exec-34] hive.metastore:534 : Connected to metastore.
2019-07-17 21:20:50,900 INFO [FetcherRunner 352123462-50] threadpool.DefaultFetcherRunner:96 : Job Fetcher: 0 should running, 0 actual running, 0 stopped, 0 ready, 0 already succeed, 1 error, 0 discarded, 0 others
2019-07-17 21:21:18,606 INFO [BadQueryDetector] service.BadQueryDetector:147 : Detect bad query.
2019-07-17 21:21:20,922 INFO [FetcherRunner 352123462-50] threadpool.DefaultFetcherRunner:96 : Job Fetcher: 0 should running, 0 actual running, 0 stopped, 0 ready, 0 already succeed, 1 error, 0 discarded, 0 others
2019-07-17 21:21:50,900 INFO [FetcherRunner 352123462-50] threadpool.DefaultFetcherRunner:96 : Job Fetcher: 0 should running, 0 actual running, 0 stopped, 0 ready, 0 already succeed, 1 error, 0 discarded, 0 others
2019-07-17 21:21:54,153 DEBUG [http-bio-7070-exec-34] cachesync.Broadcaster:247 : Broadcasting UPDATE, user, ADMIN
2019-07-17 21:21:54,155 DEBUG [http-bio-7070-exec-34] cachesync.Broadcaster:281 : Done broadcasting UPDATE, user, ADMIN
2019-07-17 21:22:18,607 INFO [BadQueryDetector] service.BadQueryDetector:147 : Detect bad query.
2019-07-17 21:22:20,899 INFO [FetcherRunner 352123462-50] threadpool.DefaultFetcherRunner:96 : Job Fetcher: 0 should running, 0 actual running, 0 stopped, 0 ready, 0 already succeed, 1 error, 0 discarded, 0 others
2019-07-17 21:22:50,920 INFO [FetcherRunner 352123462-50] threadpool.DefaultFetcherRunner:96 : Job Fetcher: 0 should running, 0 actual running, 0 stopped, 0 ready, 0 already succeed, 1 error, 0 discarded, 0 others
2019-07-17 21:23:18,607 INFO [BadQueryDetector] service.BadQueryDetector:147 : Detect bad query.
2)
2019-07-17 21:23:16,821 INFO [http-bio-7070-exec-75] common.KylinConfig:332 : Loading kylin-defaults.properties from file:/usr/local/kylin/apache-kylin-2.5.0-bin-hbase1x/tomcat/webapps/kylin/WEB-INF/lib/kylin-core-common-2.5.0.jar!/kylin-defaults.properties
2019-07-17 21:23:16,836 DEBUG [http-bio-7070-exec-75] common.KylinConfig:291 : KYLIN_CONF property was not set, will seek KYLIN_HOME env variable
2019-07-17 21:23:22,976 DEBUG [http-bio-7070-exec-86] badquery.BadQueryHistoryManager:65 : Loaded 0 Bad Query(s)
2019-07-17 21:23:27,717 INFO [BadQueryDetector] service.BadQueryDetector:147 : Detect bad query.
2019-07-17 21:23:30,468 INFO [http-bio-7070-exec-86] hive.metastore:407 : Trying to connect to metastore with URI thrift://ip-10-0-0-46.ec2.internal:9083
2019-07-17 21:23:30,470 INFO [http-bio-7070-exec-86] hive.metastore:481 : Opened a connection to metastore, current connections: 4
2019-07-17 21:23:30,471 INFO [http-bio-7070-exec-86] hive.metastore:534 : Connected to metastore.
2019-07-17 21:23:32,486 INFO [http-bio-7070-exec-68] hive.metastore:407 : Trying to connect to metastore with URI thrift://ip-10-0-0-46.ec2.internal:9083
2019-07-17 21:23:32,487 INFO [http-bio-7070-exec-68] hive.metastore:481 : Opened a connection to metastore, current connections: 5
2019-07-17 21:23:32,488 INFO [http-bio-7070-exec-68] hive.metastore:534 : Connected to metastore.
2019-07-17 21:24:27,717 INFO [BadQueryDetector] service.BadQueryDetector:147 : Detect bad query.
2019-07-17 21:25:27,718 INFO [BadQueryDetector] service.BadQueryDetector:147 : Detect bad query.
2019-07-17 21:26:25,136 INFO [Idle-Rpc-Conn-Sweeper-pool2-t1] ipc.AbstractRpcClient:217 : Cleanup idle connection to ip-10-0-0-54.ec2.internal/10.0.0.54:16020
2019-07-17 21:26:27,718 INFO [BadQueryDetector] service.BadQueryDetector:147 : Detect bad query.
2019-07-17 21:27:27,719 INFO [BadQueryDetector] service.BadQueryDetector:147 : Detect bad query.
2019-07-17 21:28:27,719 INFO [BadQueryDetector] service.BadQueryDetector:147 : Detect bad query.
2019-07-17 21:29:27,720 INFO [BadQueryDetector] service.BadQueryDetector:147 : Detect bad query.
2019-07-17 21:30:27,720 INFO [BadQueryDetector] service.BadQueryDetector:147 : Detect bad query.
2019-07-17 21:31:27,721 INFO [BadQueryDetector] service.BadQueryDetector:147 : Detect bad query.
2019-07-17 21:32:27,721 INFO [BadQueryDetector] service.BadQueryDetector:147 : Detect bad query.
2019-07-17 21:33:27,721 INFO [BadQueryDetector] service.BadQueryDetector:147 : Detect bad query.
2019-07-17 21:34:27,722 INFO [BadQueryDetector] service.BadQueryDetector:147 : Detect bad query.
3)
2019-07-17 20:49:46,875 DEBUG [http-bio-7070-exec-4] security.KylinAuthenticationProvider:57 : User cache [-108, 112, -63, -32, 41, -87, -81, 81, -32, 61, -35, -111, 7, 56, -29, -59] is removed due to EXPLICIT
2019-07-17 20:49:46,973 DEBUG [http-bio-7070-exec-4] security.KylinAuthenticationProvider:113 : User ADMIN authorities : [ROLE_ADMIN, ROLE_ANALYST, ROLE_MODELER]
2019-07-17 20:49:46,973 DEBUG [http-bio-7070-exec-4] cachesync.CachedCrudAssist:190 : Saving ManagedUser at /user/ADMIN
2019-07-17 20:49:46,976 DEBUG [pool-5-thread-1] cachesync.Broadcaster:113 : Servers in the cluster: [ip-10-0-0-45.ec2.internal:7070, ip-10-0-0-46.ec2.internal:7070, ip-10-0-0-51.ec2.internal:7070]
2019-07-17 20:49:46,977 DEBUG [pool-5-thread-1] cachesync.Broadcaster:123 : Announcing new broadcast to all: BroadcastEvent{entity=user, event=update, cacheKey=ADMIN}
2019-07-17 20:49:46,978 DEBUG [http-bio-7070-exec-4] security.KylinAuthenticationProvider:126 : Authenticated user org.springframework.security.authentication.UsernamePasswordAuthenticationToken@37047726: Principal: org.springframework.security.core.userdetails.User@3b40b2f: Username: ADMIN; Password: [PROTECTED]; Enabled: true; AccountNonExpired: true; credentialsNonExpired: true; AccountNonLocked: true; Granted Authorities: ROLE_ADMIN,ROLE_ANALYST,ROLE_MODELER; Credentials: [PROTECTED]; Authenticated: true; Details: org.springframework.security.web.authentication.WebAuthenticationDetails@ffff4c9c: RemoteIpAddress: 10.0.0.59; SessionId: null; Granted Authorities: ROLE_ADMIN, ROLE_ANALYST, ROLE_MODELER
2019-07-17 20:49:46,992 DEBUG [http-bio-7070-exec-18] cachesync.Broadcaster:247 : Broadcasting UPDATE, user, ADMIN
2019-07-17 20:49:46,994 DEBUG [http-bio-7070-exec-18] cachesync.Broadcaster:281 : Done broadcasting UPDATE, user, ADMIN
2019-07-17 20:49:47,051 DEBUG [http-bio-7070-exec-20] cachesync.Broadcaster:247 : Broadcasting UPDATE, user, ADMIN
2019-07-17 20:49:47,053 DEBUG [http-bio-7070-exec-20] cachesync.Broadcaster:281 : Done broadcasting UPDATE, user, ADMIN
2019-07-17 20:50:02,894 DEBUG [http-bio-7070-exec-20] cachesync.Broadcaster:247 : Broadcasting UPDATE, user, ADMIN
2019-07-17 20:50:02,897 DEBUG [http-bio-7070-exec-20] cachesync.Broadcaster:281 : Done broadcasting UPDATE, user, ADMIN
2019-07-17 20:50:03,033 DEBUG [http-bio-7070-exec-9] cachesync.Broadcaster:247 : Broadcasting UPDATE, user, ADMIN
2019-07-17 20:50:03,035 DEBUG [http-bio-7070-exec-9] cachesync.Broadcaster:281 : Done broadcasting UPDATE, user, ADMIN
2019-07-17 20:50:03,159 DEBUG [http-bio-7070-exec-7] cachesync.Broadcaster:247 : Broadcasting UPDATE, user, ADMIN
2019-07-17 20:50:03,161 DEBUG [http-bio-7070-exec-7] cachesync.Broadcaster:281 : Done broadcasting UPDATE, user, ADMIN
I am looking forward to hearing from you.
Thanks,
Krishna