Lookup refresh issue for JDBC data source

[Chi Yung Lam <ch...@advanzesolutions.com>]

HI all,

We are using Kylin 3.1 and have issues on using lookup refresh.  

 

Our case: 

- We use Sqoop to transfer data from SQL Server to Kylin

- We use lookup tables 

- The dimensions in lookup tables may change and therefore we would like to
leverage on the "lookup refresh" 

 

We try the following: 

(a) If we use the default lookup table setting in cube definition (i.e.,
using metastore) and then trigger "lookup refresh", we don't see the updated
lookup dimension values.   

(b) We then try using global lookup feature in "Cube -> Advanced Settings ->
Advanced Snapshot Table".  When we build the data cube, it gives an error
when we do the first build.  The error step is at "#7 Step Name: Convert
Lookup Table to HFile:DBO.MASTER_PRODUCT_ACTUALS_V", where
MASTER_PRODUCT_ACTUALS_V is our lookup table.  We can see the table in the
Data Source tab of "Model" page.  

 

Step 7 parameter: 

-conf /usr/local/kylin/conf/kylin_job_conf.xml -cubename
gap_store_sales_merchplan_v01_04_lkup -output
hdfs://node01/kylin/kylin_metadata/kylin-810ac500-c1d6-e69e-b9af-fe75b4e51e4
8/DBO.MASTER_PRODUCT_ACTUALS_V/hfile -tableName DBO.MASTER_PRODUCT_ACTUALS_V
-cubingJobId 810ac500-c1d6-e69e-b9af-fe75b4e51e48 -snapshotID
07a2849e-2571-63ce-340d-54562d9bdb8a -jobname
Kylin_LookupTable_HFile_Generator_DBO.MASTER_PRODUCT_ACTUALS_V_Step

 

The error details: 

java.lang.RuntimeException: java.io.IOException:
NoSuchObjectException(message:DBO.MASTER_PRODUCT_ACTUALS_V table not found)

               at
org.apache.kylin.source.hive.HiveMRInput$HiveTableInputFormat.configureJob(H
iveMRInput.java:83)

               at
org.apache.kylin.storage.hbase.lookup.LookupTableToHFileJob.run(LookupTableT
oHFileJob.java:124)

               at
org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecut
able.java:144)

               at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable
.java:178)

               at
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChaine
dExecutable.java:71)

               at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable
.java:178)

               at
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultS
cheduler.java:114)

               at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11
42)

               at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6
17)

               at java.lang.Thread.run(Thread.java:745)

Caused by: java.io.IOException:
NoSuchObjectException(message:DBO.MASTER_PRODUCT_ACTUALS_V table not found)

               at
org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.
java:97)

               at
org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.
java:51)

               at
org.apache.kylin.source.hive.HiveMRInput$HiveTableInputFormat.configureJob(H
iveMRInput.java:80)

               ... 9 more

Caused by: NoSuchObjectException(message:DBO.MASTER_PRODUCT_ACTUALS_V table
not found)

               at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_core(Hiv
eMetaStore.java:1808)

               at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table(HiveMeta
Store.java:1778)

               at sun.reflect.GeneratedMethodAccessor236.invoke(Unknown
Source)

               at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:43)

               at java.lang.reflect.Method.invoke(Method.java:498)

               at
org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandle
r.java:107)

               at com.sun.proxy.$Proxy105.get_table(Unknown Source)

               at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreC
lient.java:1208)

               at sun.reflect.GeneratedMethodAccessor299.invoke(Unknown
Source)

               at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:43)

               at java.lang.reflect.Method.invoke(Method.java:498)

               at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMeta
StoreClient.java:152)

               at com.sun.proxy.$Proxy108.getTable(Unknown Source)

               at
org.apache.hive.hcatalog.common.HCatUtil.getTable(HCatUtil.java:180)

               at
org.apache.hive.hcatalog.mapreduce.InitializeInput.getInputJobInfo(Initializ
eInput.java:105)

               at
org.apache.hive.hcatalog.mapreduce.InitializeInput.setInput(InitializeInput.
java:86)

               at
org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFormat.
java:95)

               ... 11 more

 

Any help is much appreciated!

 

 

Regards, Chi-yung