You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kylin.apache.org by chouc_mei <ch...@gmail.com> on 2019/03/18 10:36:20 UTC
怎么修复cube，当我丢失了某个dict时
您好：
今天我在构建cube 时，遇到一个问题，就是在构建第四步 Build Dimension Dictionary
，一直都会抛出一个异常。问题是在构建今天早 3点到4点的segment出现的，我把2-3点的重新构建 也不行。怎么可以修复这个cube，其它cube 和
在3点之前的segment 都可以构建
org.apache.kylin.engine.mr.exception.HadoopShellException:
java.lang.RuntimeException: java.io.IOException: Failed to read big resource
/dict/AD_V.IMPRESSION_LOG/SCHEDULEID/cdfa5043-e699-fbd7-e8db-b3646d927a5f.dict
at
org.apache.kylin.dict.DictionaryManager.getDictionaryInfo(DictionaryManager.java:108)
at
org.apache.kylin.dict.DictionaryManager.checkDupByContent(DictionaryManager.java:173)
at
org.apache.kylin.dict.DictionaryManager.trySaveNewDict(DictionaryManager.java:151)
at
org.apache.kylin.dict.DictionaryManager.saveDictionary(DictionaryManager.java:320)
at
org.apache.kylin.cube.CubeManager$DictionaryAssist.saveDictionary(CubeManager.java:1117)
at org.apache.kylin.cube.CubeManager.saveDictionary(CubeManager.java:1079)
at
org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:74)
at
org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:55)
at
org.apache.kylin.engine.mr.steps.CreateDictionaryJob.run(CreateDictionaryJob.java:73)
at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:93)
at
org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:63)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:165)
at
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:70)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:165)
at
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Failed to read big resource
/dict/AD_V.IMPRESSION_LOG/SCHEDULEID/cdfa5043-e699-fbd7-e8db-b3646d927a5f.dict
at
org.apache.kylin.common.persistence.PushdownResourceStore.openPushdown(PushdownResourceStore.java:176)
at
org.apache.kylin.storage.hbase.HBaseResourceStore.getInputStream(HBaseResourceStore.java:256)
at
org.apache.kylin.storage.hbase.HBaseResourceStore.rawResource(HBaseResourceStore.java:226)
at
org.apache.kylin.storage.hbase.HBaseResourceStore.getResourceImpl(HBaseResourceStore.java:276)
at
org.apache.kylin.common.persistence.ResourceStore$3.call(ResourceStore.java:291)
at
org.apache.kylin.common.persistence.ResourceStore$3.call(ResourceStore.java:288)
at
org.apache.kylin.common.persistence.ExponentialBackoffRetry.doWithRetry(ExponentialBackoffRetry.java:52)
at
org.apache.kylin.common.persistence.ResourceStore.getResourceWithRetry(ResourceStore.java:288)
at
org.apache.kylin.common.persistence.ResourceStore.getResource(ResourceStore.java:265)
at
org.apache.kylin.common.persistence.ResourceStore.getResource(ResourceStore.java:256)
at org.apache.kylin.dict.DictionaryManager.load(DictionaryManager.java:397)
at
org.apache.kylin.dict.DictionaryManager$1.load(DictionaryManager.java:80)
at
org.apache.kylin.dict.DictionaryManager$1.load(DictionaryManager.java:77)
at
com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3599)
at
com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2379)
at
com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2342)
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2257)
at com.google.common.cache.LocalCache.get(LocalCache.java:4000)
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4004)
at
com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4874)
at
org.apache.kylin.dict.DictionaryManager.getDictionaryInfo(DictionaryManager.java:101)
... 17 more
Caused by: java.io.FileNotFoundException:
/kylin/kylin_metadata/resources/dict/AD_V.IMPRESSION_LOG/SCHEDULEID/cdfa5043-e699-fbd7-e8db-b3646d927a5f.dict
(FS: DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_1042273900_27, ugi=root
(auth:SIMPLE)]])
at
org.apache.kylin.common.persistence.PushdownResourceStore.openPushdown(PushdownResourceStore.java:173)
... 37 more
result code:2
at
org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:73)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:165)
at
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:70)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:165)
at
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
--
Sent from: http://apache-kylin.74782.x6.nabble.com/
Re:回复：怎么修复cube，当我丢失了某个dict时
Posted by 只爱夏天 <ai...@163.com>.
hi,
Now I also facing the problem,and I can't find the losted dict file in the hdfs path"/kylin/kylin_metadata/resources/dict/projectName.ModelName/DimName/xxxx.dict",
and my kylin version is 2.6.2 cdh57
How I can retrieve the lost dict file? Please help me ,thks!
在 2019-03-22 10:46:48，"Chao Long" <wa...@qq.com> 写道：
>Hi,
> You can run command "$KYLIN_HOME/bin/metastore.sh fetch /dict" and check weather the dict file is exists.
>------------------
>Best Regards,
>Chao Long
>
>
>------------------ 原始邮件 ------------------
>发件人: "chouc_mei"<ch...@gmail.com>;
>发送时间: 2019年3月18日(星期一) 晚上6:36
>收件人: "dev"<de...@kylin.apache.org>;
>
>主题: 怎么修复cube，当我丢失了某个dict时
>
>
>
>您好：
> 今天我在构建cube 时，遇到一个问题，就是在构建第四步 Build Dimension Dictionary
>，一直都会抛出一个异常。问题是在构建今天早 3点到4点的segment出现的，我把2-3点的重新构建 也不行。怎么可以修复这个cube，其它cube 和
>在3点之前的segment 都可以构建
>
>
>org.apache.kylin.engine.mr.exception.HadoopShellException:
>java.lang.RuntimeException: java.io.IOException: Failed to read big resource
>/dict/AD_V.IMPRESSION_LOG/SCHEDULEID/cdfa5043-e699-fbd7-e8db-b3646d927a5f.dict
> at
>org.apache.kylin.dict.DictionaryManager.getDictionaryInfo(DictionaryManager.java:108)
> at
>org.apache.kylin.dict.DictionaryManager.checkDupByContent(DictionaryManager.java:173)
> at
>org.apache.kylin.dict.DictionaryManager.trySaveNewDict(DictionaryManager.java:151)
> at
>org.apache.kylin.dict.DictionaryManager.saveDictionary(DictionaryManager.java:320)
> at
>org.apache.kylin.cube.CubeManager$DictionaryAssist.saveDictionary(CubeManager.java:1117)
> at org.apache.kylin.cube.CubeManager.saveDictionary(CubeManager.java:1079)
> at
>org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:74)
> at
>org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:55)
> at
>org.apache.kylin.engine.mr.steps.CreateDictionaryJob.run(CreateDictionaryJob.java:73)
> at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:93)
> at
>org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:63)
> at
>org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:165)
> at
>org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:70)
> at
>org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:165)
> at
>org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114)
> at
>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>Caused by: java.io.IOException: Failed to read big resource
>/dict/AD_V.IMPRESSION_LOG/SCHEDULEID/cdfa5043-e699-fbd7-e8db-b3646d927a5f.dict
> at
>org.apache.kylin.common.persistence.PushdownResourceStore.openPushdown(PushdownResourceStore.java:176)
> at
>org.apache.kylin.storage.hbase.HBaseResourceStore.getInputStream(HBaseResourceStore.java:256)
> at
>org.apache.kylin.storage.hbase.HBaseResourceStore.rawResource(HBaseResourceStore.java:226)
> at
>org.apache.kylin.storage.hbase.HBaseResourceStore.getResourceImpl(HBaseResourceStore.java:276)
> at
>org.apache.kylin.common.persistence.ResourceStore$3.call(ResourceStore.java:291)
> at
>org.apache.kylin.common.persistence.ResourceStore$3.call(ResourceStore.java:288)
> at
>org.apache.kylin.common.persistence.ExponentialBackoffRetry.doWithRetry(ExponentialBackoffRetry.java:52)
> at
>org.apache.kylin.common.persistence.ResourceStore.getResourceWithRetry(ResourceStore.java:288)
> at
>org.apache.kylin.common.persistence.ResourceStore.getResource(ResourceStore.java:265)
> at
>org.apache.kylin.common.persistence.ResourceStore.getResource(ResourceStore.java:256)
> at org.apache.kylin.dict.DictionaryManager.load(DictionaryManager.java:397)
> at
>org.apache.kylin.dict.DictionaryManager$1.load(DictionaryManager.java:80)
> at
>org.apache.kylin.dict.DictionaryManager$1.load(DictionaryManager.java:77)
> at
>com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3599)
> at
>com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2379)
> at
>com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2342)
> at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2257)
> at com.google.common.cache.LocalCache.get(LocalCache.java:4000)
> at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4004)
> at
>com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4874)
> at
>org.apache.kylin.dict.DictionaryManager.getDictionaryInfo(DictionaryManager.java:101)
> ... 17 more
>Caused by: java.io.FileNotFoundException:
>/kylin/kylin_metadata/resources/dict/AD_V.IMPRESSION_LOG/SCHEDULEID/cdfa5043-e699-fbd7-e8db-b3646d927a5f.dict
>(FS: DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_1042273900_27, ugi=root
>(auth:SIMPLE)]])
> at
>org.apache.kylin.common.persistence.PushdownResourceStore.openPushdown(PushdownResourceStore.java:173)
> ... 37 more
>
>result code:2
> at
>org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:73)
> at
>org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:165)
> at
>org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:70)
> at
>org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:165)
> at
>org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114)
> at
>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
>
>
>--
>Sent from: http://apache-kylin.74782.x6.nabble.com/
回复：怎么修复cube，当我丢失了某个dict时
Posted by Chao Long <wa...@qq.com>.
Hi,
You can run command "$KYLIN_HOME/bin/metastore.sh fetch /dict" and check weather the dict file is exists.
------------------
Best Regards,
Chao Long
------------------ 原始邮件 ------------------
发件人: "chouc_mei"<ch...@gmail.com>;
发送时间: 2019年3月18日(星期一) 晚上6:36
收件人: "dev"<de...@kylin.apache.org>;
主题: 怎么修复cube，当我丢失了某个dict时
您好：
今天我在构建cube 时，遇到一个问题，就是在构建第四步 Build Dimension Dictionary
，一直都会抛出一个异常。问题是在构建今天早 3点到4点的segment出现的，我把2-3点的重新构建 也不行。怎么可以修复这个cube，其它cube 和
在3点之前的segment 都可以构建
org.apache.kylin.engine.mr.exception.HadoopShellException:
java.lang.RuntimeException: java.io.IOException: Failed to read big resource
/dict/AD_V.IMPRESSION_LOG/SCHEDULEID/cdfa5043-e699-fbd7-e8db-b3646d927a5f.dict
at
org.apache.kylin.dict.DictionaryManager.getDictionaryInfo(DictionaryManager.java:108)
at
org.apache.kylin.dict.DictionaryManager.checkDupByContent(DictionaryManager.java:173)
at
org.apache.kylin.dict.DictionaryManager.trySaveNewDict(DictionaryManager.java:151)
at
org.apache.kylin.dict.DictionaryManager.saveDictionary(DictionaryManager.java:320)
at
org.apache.kylin.cube.CubeManager$DictionaryAssist.saveDictionary(CubeManager.java:1117)
at org.apache.kylin.cube.CubeManager.saveDictionary(CubeManager.java:1079)
at
org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:74)
at
org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:55)
at
org.apache.kylin.engine.mr.steps.CreateDictionaryJob.run(CreateDictionaryJob.java:73)
at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:93)
at
org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:63)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:165)
at
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:70)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:165)
at
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Failed to read big resource
/dict/AD_V.IMPRESSION_LOG/SCHEDULEID/cdfa5043-e699-fbd7-e8db-b3646d927a5f.dict
at
org.apache.kylin.common.persistence.PushdownResourceStore.openPushdown(PushdownResourceStore.java:176)
at
org.apache.kylin.storage.hbase.HBaseResourceStore.getInputStream(HBaseResourceStore.java:256)
at
org.apache.kylin.storage.hbase.HBaseResourceStore.rawResource(HBaseResourceStore.java:226)
at
org.apache.kylin.storage.hbase.HBaseResourceStore.getResourceImpl(HBaseResourceStore.java:276)
at
org.apache.kylin.common.persistence.ResourceStore$3.call(ResourceStore.java:291)
at
org.apache.kylin.common.persistence.ResourceStore$3.call(ResourceStore.java:288)
at
org.apache.kylin.common.persistence.ExponentialBackoffRetry.doWithRetry(ExponentialBackoffRetry.java:52)
at
org.apache.kylin.common.persistence.ResourceStore.getResourceWithRetry(ResourceStore.java:288)
at
org.apache.kylin.common.persistence.ResourceStore.getResource(ResourceStore.java:265)
at
org.apache.kylin.common.persistence.ResourceStore.getResource(ResourceStore.java:256)
at org.apache.kylin.dict.DictionaryManager.load(DictionaryManager.java:397)
at
org.apache.kylin.dict.DictionaryManager$1.load(DictionaryManager.java:80)
at
org.apache.kylin.dict.DictionaryManager$1.load(DictionaryManager.java:77)
at
com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3599)
at
com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2379)
at
com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2342)
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2257)
at com.google.common.cache.LocalCache.get(LocalCache.java:4000)
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4004)
at
com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4874)
at
org.apache.kylin.dict.DictionaryManager.getDictionaryInfo(DictionaryManager.java:101)
... 17 more
Caused by: java.io.FileNotFoundException:
/kylin/kylin_metadata/resources/dict/AD_V.IMPRESSION_LOG/SCHEDULEID/cdfa5043-e699-fbd7-e8db-b3646d927a5f.dict
(FS: DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_1042273900_27, ugi=root
(auth:SIMPLE)]])
at
org.apache.kylin.common.persistence.PushdownResourceStore.openPushdown(PushdownResourceStore.java:173)
... 37 more
result code:2
at
org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:73)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:165)
at
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:70)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:165)
at
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
--
Sent from: http://apache-kylin.74782.x6.nabble.com/