You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kylin.apache.org by shylinzhang <sh...@163.com> on 2020/02/16 08:22:14 UTC
Failed to read big resource
/dict/KYLIN.HIVE_METRICS_QUERY_CUBE_QA/CUBE_NAME/xxx.dict
Dear all,
The cube is ok when 2:00 am,but after 4:30 am it always shows error ,it shows ``` #4 Step Name: Build Dimension Dictionary
org.apache.kylin.engine.mr.exception.HadoopShellException: java.lang.RuntimeException: java.io.IOException: Failed to read big resource /dict/KYLIN.HIVE_METRICS_QUERY_CUBE_QA/CUBE_NAME/b9bb85ec-b72e-4df1-9ee7-e8c971ec5c72.dict
at org.apache.kylin.dict.DictionaryManager.getDictionaryInfo(DictionaryManager.java:108)
at org.apache.kylin.dict.DictionaryManager.checkDupByContent(DictionaryManager.java:173)
at org.apache.kylin.dict.DictionaryManager.trySaveNewDict(DictionaryManager.java:151)
at org.apache.kylin.dict.DictionaryManager.saveDictionary(DictionaryManager.java:320)
at org.apache.kylin.cube.CubeManager$DictionaryAssist.saveDictionary(CubeManager.java:1117)
at org.apache.kylin.cube.CubeManager.saveDictionary(CubeManager.java:1079)
at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:74)
at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:55)
at org.apache.kylin.engine.mr.steps.CreateDictionaryJob.run(CreateDictionaryJob.java:73)
at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:93)
at org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:63)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166)
at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166)
at org.apache.kylin.job.impl.threadpool.DistributedScheduler$JobRunner.run(DistributedScheduler.java:110)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
```
What I do:
1. I disable the cube and purge the cube and rebulid, it also show the same error;
2. Because the cube at 2:00 am is ok, so I use the meta_backs in 2:00 am and restore and cube; I try short period to build the cube, 2:00 am~3:00am~4:00am~4:00am~6:00am are ok, but in 6:am~7:00 am it also shows above error;
I found the solution in kylin jira, it tells me to update to release 2.6.4; our kylin clusters are used in 2019-06; just today shows the error.
Here’s the jira url: https://issues.apache.org/jira/browse/KYLIN-4153?jql=summary%20~%20%22Failed%20to%20read%20big%20resource*%22%20OR%20description%20~%20%22Failed%20to%20read%20big%20resource*%22%20ORDER%20BY%20lastViewed%20DESC
Therefore I want to know why it occured and if there is a way to fix it except updating.
I am looking forward for your replay, thank you.
Kylin version: 2.6.1
Hadoop version: 3.0.0+cdh6.0.0
Best Wishes!
Shylin Zhang
| |
shylinzhang
|
|
shylinzhang@163.com
|
签名由网易邮箱大师定制
Re:Failed to read big resource
/dict/KYLIN.HIVE_METRICS_QUERY_CUBE_QA/CUBE_NAME/xxx.dict
Posted by shylinzhang <sh...@163.com>.
Hi Johnson,
我的处理步骤是 discard ERROR系统cube-- ${KYLIN_HIME}/bin/metastore.sh remove ${dict路径}--bulid cube again; 然后还是报那个错，只不过不是报之前的/dict/KYLIN.HIVE_METRICS_QUERY_CUBE_QA/CUBE_NAME/xxx.dict那个，又报一个新的。
Best Regards,
Shylin Zhang
On 08/11/2020 10:00，Johnson<it...@163.com> wrote：
Hi Shylin Zhang，
从你的报错信息来看，这次报错的cube不是上次的那个，说明之前那个cube已经构建成功了。你继续使用这个命令清除一下，然后rebuild，绝对是可以的 ${KYLIN_HIME}/bin/metastore.sh remove ${dict路径}
At 2020-08-10 23:02:26, "shylinzhang" <sh...@163.com> wrote:
Hi ZhangQiang,
I tried the command you given, and build the SYSTEM cube again, it also shows the error.
java.lang.RuntimeException: java.io.IOException: Failed to read big resource /dict/KYLIN.HIVE_METRICS_QUERY_RPC_QA/KTIME_MINUTE/3ad46874-e65a-4116-5c0c-3c9aefea36af.dict
at org.apache.kylin.dict.DictionaryManager.getDictionaryInfo(DictionaryManager.java:108)
at org.apache.kylin.dict.DictionaryManager.checkDupByContent(DictionaryManager.java:173)
at org.apache.kylin.dict.DictionaryManager.trySaveNewDict(DictionaryManager.java:151)
at org.apache.kylin.engine.mr.steps.UpdateDictionaryStep.doWork(UpdateDictionaryStep.java:111)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166)
at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166)
at org.apache.kylin.job.impl.threadpool.DistributedScheduler$JobRunner.run(DistributedScheduler.java:110)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Failed to read big resource /dict/KYLIN.HIVE_METRICS_QUERY_RPC_QA/KTIME_MINUTE/3ad46874-e65a-4116-5c0c-3c9aefea36af.dict
Best Regards,
Shylin Zhang
On 08/10/2020 16:50，Johnson<it...@163.com> wrote：
Hi Shylin Zhang
You can use "${KYLIN_HIME}/bin/metastore.sh remove /dict/KYLIN.HIVE_METRICS_JOB_QA/KUSER/d60c8ffc-c883-24d2-ec6b-c78fc963f8d7.dict" to delete this dict，it should work.
At 2020-08-10 16:02:47, "shylinzhang" <sh...@163.com> wrote:
Hi Xiaoxiang,
I met the issue agin and follow what your said last time, and it didn’t work. Here are the details:
ERROR:
org.apache.kylin.engine.mr.exception.HadoopShellException: java.lang.RuntimeException: java.io.IOException: Failed to read big resource /dict/KYLIN.HIVE_METRICS_JOB_QA/KUSER/d60c8ffc-c883-24d2-ec6b-c78fc963f8d7.dict
Caused by: java.io.FileNotFoundException: /kylin/kylin_metadata/resources/dict/KYLIN.HIVE_METRICS_JOB_QA/KUSER/d60c8ffc-c883-24d2-ec6b-c78fc963f8d7.dict (FS: DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_1184245014_33, ugi=kylin_app (auth:SIMPLE)]])
# 查看是否为空,为空
hbase(main):011:0> get 'kylin_metadata','/cube/KYLIN_HIVE_METRICS_JOB_QA.json','f:HIVE_METRICS_JOB_QA.KUSER'
COLUMN CELL
0 row(s)
Took 0.0129 seconds
# 删除空值
hbase(main):012:0> delete 'kylin_metadata','/cube/KYLIN_HIVE_METRICS_JOB_QA.json','f:HIVE_METRICS_JOB_QA.KUSER'
Took 0.0166 seconds
hbase(main):013:0>
After that I build the cube it also shows the error.
ENV:
Kylin version: 2.6.1
Hadoop version: 3.0.0+cdh6.0.0
Best Regards,
Shylin Zhang
On 02/19/2020 10:52，shylinzhang<sh...@163.com> wrote：
Hi Xiaoxiang,
Thanks for your advices, I will follow them.
Thank you for your help again.
Best Regards,
Shylin Zhang
On 02/18/2020 23:35，Xiaoxiang Yu<xx...@apache.org> wrote：
Hi shylinzhang,
Thank you for your update. I am glad to hear you have fixed your problem finally.
I wish you can attach the some detailed description or some error log if you face some problem in the future; using the simple/misunderstanding sentence "it failed' is not good enough in my side.
Here is some advice provided by apache website:
1. http://www.catb.org/~esr/faqs/smart-questions.html,
2. http://www.apache.org/dev/contrib-email-tips .
--
Best wishes to you !
From ：Xiaoxiang Yu
在 2020-02-18 19:43:15，"shylinzhang" <sh...@163.com> 写道：
Hi Xiaoxiang,
I’m sorry that I misunderstood the meaning of “delete the remaining marker”.I followed what you said and solved one system cube issue;
But I fix anther same issued system cube with same way, it failed，And then I purge and rebuild the cube and it works well.
Thank you for your help.
Best Wishes!
Shylin Zhang
| |
shylinzhang
|
|
shylinzhang@163.com
|
签名由网易邮箱大师定制
On 02/17/2020 19:45，Xiaoxiang Yu<xx...@apache.org> wrote：
Hi,
I don't understand what problem did you face in "I tried to delete the remaining marker, it failed"?
As far as I see, you can open the hbase shell. Use the "get /dict/KYLIN.HIVE_METRICS_QUERY_CUBE_QA/CUBE_NAME/xxx.dict" to get the value of that entry,
and use "delete" to delete that empty entry. Before doing this, I think you should back up your metadata.
--
Best wishes to you !
From ：Xiaoxiang Yu
在 2020-02-17 10:44:23，"shylinzhang" <sh...@163.com> 写道：
Hi George,
Thank you for your reply, I tried to delete the remaining marker, it failed. So I had to update.
Thank you!
Best Wishes!
Shylin Zhang
| |
shylinzhang
|
|
shylinzhang@163.com
|
签名由网易邮箱大师定制
On 02/16/2020 21:05，nichunen<ni...@apache.org> wrote：
Hi Shylin,
I think the description part in https://issues.apache.org/jira/browse/KYLIN-4153 is clear enough, you can find the root cause and workaround there, it happened due to the lack of atomicity during the rollback for the "big resource".
As a workaround, you may delete the remaining marker in your resource store, but it’s more advised to do an upgrade for your Kylin.
Best regards,
Ni Chunen / George
On 02/16/2020 16:22，shylinzhang<sh...@163.com> wrote：
Dear all,
The cube is ok when 2:00 am,but after 4:30 am it always shows error ,it shows ``` #4 Step Name: Build Dimension Dictionary
org.apache.kylin.engine.mr.exception.HadoopShellException: java.lang.RuntimeException: java.io.IOException: Failed to read big resource /dict/KYLIN.HIVE_METRICS_QUERY_CUBE_QA/CUBE_NAME/b9bb85ec-b72e-4df1-9ee7-e8c971ec5c72.dict
at org.apache.kylin.dict.DictionaryManager.getDictionaryInfo(DictionaryManager.java:108)
at org.apache.kylin.dict.DictionaryManager.checkDupByContent(DictionaryManager.java:173)
at org.apache.kylin.dict.DictionaryManager.trySaveNewDict(DictionaryManager.java:151)
at org.apache.kylin.dict.DictionaryManager.saveDictionary(DictionaryManager.java:320)
at org.apache.kylin.cube.CubeManager$DictionaryAssist.saveDictionary(CubeManager.java:1117)
at org.apache.kylin.cube.CubeManager.saveDictionary(CubeManager.java:1079)
at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:74)
at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:55)
at org.apache.kylin.engine.mr.steps.CreateDictionaryJob.run(CreateDictionaryJob.java:73)
at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:93)
at org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:63)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166)
at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166)
at org.apache.kylin.job.impl.threadpool.DistributedScheduler$JobRunner.run(DistributedScheduler.java:110)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
```
What I do:
1. I disable the cube and purge the cube and rebulid, it also show the same error;
2. Because the cube at 2:00 am is ok, so I use the meta_backs in 2:00 am and restore and cube; I try short period to build the cube, 2:00 am~3:00am~4:00am~4:00am~6:00am are ok, but in 6:am~7:00 am it also shows above error;
I found the solution in kylin jira, it tells me to update to release 2.6.4; our kylin clusters are used in 2019-06; just today shows the error.
Here’s the jira url: https://issues.apache.org/jira/browse/KYLIN-4153?jql=summary%20~%20%22Failed%20to%20read%20big%20resource*%22%20OR%20description%20~%20%22Failed%20to%20read%20big%20resource*%22%20ORDER%20BY%20lastViewed%20DESC
Therefore I want to know why it occured and if there is a way to fix it except updating.
I am looking forward for your replay, thank you.
Kylin version: 2.6.1
Hadoop version: 3.0.0+cdh6.0.0
Best Wishes!
Shylin Zhang
| |
shylinzhang
|
|
shylinzhang@163.com
|
签名由网易邮箱大师定制
Re:Re:Failed to read big resource
/dict/KYLIN.HIVE_METRICS_QUERY_CUBE_QA/CUBE_NAME/xxx.dict
Posted by Johnson <it...@163.com>.
Hi Shylin Zhang，
从你的报错信息来看，这次报错的cube不是上次的那个，说明之前那个cube已经构建成功了。你继续使用这个命令清除一下，然后rebuild，绝对是可以的 ${KYLIN_HIME}/bin/metastore.sh remove ${dict路径}
At 2020-08-10 23:02:26, "shylinzhang" <sh...@163.com> wrote:
Hi ZhangQiang,
I tried the command you given, and build the SYSTEM cube again, it also shows the error.
java.lang.RuntimeException: java.io.IOException: Failed to read big resource /dict/KYLIN.HIVE_METRICS_QUERY_RPC_QA/KTIME_MINUTE/3ad46874-e65a-4116-5c0c-3c9aefea36af.dict
at org.apache.kylin.dict.DictionaryManager.getDictionaryInfo(DictionaryManager.java:108)
at org.apache.kylin.dict.DictionaryManager.checkDupByContent(DictionaryManager.java:173)
at org.apache.kylin.dict.DictionaryManager.trySaveNewDict(DictionaryManager.java:151)
at org.apache.kylin.engine.mr.steps.UpdateDictionaryStep.doWork(UpdateDictionaryStep.java:111)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166)
at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166)
at org.apache.kylin.job.impl.threadpool.DistributedScheduler$JobRunner.run(DistributedScheduler.java:110)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Failed to read big resource /dict/KYLIN.HIVE_METRICS_QUERY_RPC_QA/KTIME_MINUTE/3ad46874-e65a-4116-5c0c-3c9aefea36af.dict
Best Regards,
Shylin Zhang
On 08/10/2020 16:50，Johnson<it...@163.com> wrote：
Hi Shylin Zhang
You can use "${KYLIN_HIME}/bin/metastore.sh remove /dict/KYLIN.HIVE_METRICS_JOB_QA/KUSER/d60c8ffc-c883-24d2-ec6b-c78fc963f8d7.dict" to delete this dict，it should work.
At 2020-08-10 16:02:47, "shylinzhang" <sh...@163.com> wrote:
Hi Xiaoxiang,
I met the issue agin and follow what your said last time, and it didn’t work. Here are the details:
ERROR:
org.apache.kylin.engine.mr.exception.HadoopShellException: java.lang.RuntimeException: java.io.IOException: Failed to read big resource /dict/KYLIN.HIVE_METRICS_JOB_QA/KUSER/d60c8ffc-c883-24d2-ec6b-c78fc963f8d7.dict
Caused by: java.io.FileNotFoundException: /kylin/kylin_metadata/resources/dict/KYLIN.HIVE_METRICS_JOB_QA/KUSER/d60c8ffc-c883-24d2-ec6b-c78fc963f8d7.dict (FS: DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_1184245014_33, ugi=kylin_app (auth:SIMPLE)]])
# 查看是否为空,为空
hbase(main):011:0> get 'kylin_metadata','/cube/KYLIN_HIVE_METRICS_JOB_QA.json','f:HIVE_METRICS_JOB_QA.KUSER'
COLUMN CELL
0 row(s)
Took 0.0129 seconds
# 删除空值
hbase(main):012:0> delete 'kylin_metadata','/cube/KYLIN_HIVE_METRICS_JOB_QA.json','f:HIVE_METRICS_JOB_QA.KUSER'
Took 0.0166 seconds
hbase(main):013:0>
After that I build the cube it also shows the error.
ENV:
Kylin version: 2.6.1
Hadoop version: 3.0.0+cdh6.0.0
Best Regards,
Shylin Zhang
On 02/19/2020 10:52，shylinzhang<sh...@163.com> wrote：
Hi Xiaoxiang,
Thanks for your advices, I will follow them.
Thank you for your help again.
Best Regards,
Shylin Zhang
On 02/18/2020 23:35，Xiaoxiang Yu<xx...@apache.org> wrote：
Hi shylinzhang,
Thank you for your update. I am glad to hear you have fixed your problem finally.
I wish you can attach the some detailed description or some error log if you face some problem in the future; using the simple/misunderstanding sentence "it failed' is not good enough in my side.
Here is some advice provided by apache website:
1. http://www.catb.org/~esr/faqs/smart-questions.html,
2. http://www.apache.org/dev/contrib-email-tips .
--
Best wishes to you !
From ：Xiaoxiang Yu
在 2020-02-18 19:43:15，"shylinzhang" <sh...@163.com> 写道：
Hi Xiaoxiang,
I’m sorry that I misunderstood the meaning of “delete the remaining marker”.I followed what you said and solved one system cube issue;
But I fix anther same issued system cube with same way, it failed，And then I purge and rebuild the cube and it works well.
Thank you for your help.
Best Wishes!
Shylin Zhang
| |
shylinzhang
|
|
shylinzhang@163.com
|
签名由网易邮箱大师定制
On 02/17/2020 19:45，Xiaoxiang Yu<xx...@apache.org> wrote：
Hi,
I don't understand what problem did you face in "I tried to delete the remaining marker, it failed"?
As far as I see, you can open the hbase shell. Use the "get /dict/KYLIN.HIVE_METRICS_QUERY_CUBE_QA/CUBE_NAME/xxx.dict" to get the value of that entry,
and use "delete" to delete that empty entry. Before doing this, I think you should back up your metadata.
--
Best wishes to you !
From ：Xiaoxiang Yu
在 2020-02-17 10:44:23，"shylinzhang" <sh...@163.com> 写道：
Hi George,
Thank you for your reply, I tried to delete the remaining marker, it failed. So I had to update.
Thank you!
Best Wishes!
Shylin Zhang
| |
shylinzhang
|
|
shylinzhang@163.com
|
签名由网易邮箱大师定制
On 02/16/2020 21:05，nichunen<ni...@apache.org> wrote：
Hi Shylin,
I think the description part in https://issues.apache.org/jira/browse/KYLIN-4153 is clear enough, you can find the root cause and workaround there, it happened due to the lack of atomicity during the rollback for the "big resource".
As a workaround, you may delete the remaining marker in your resource store, but it’s more advised to do an upgrade for your Kylin.
Best regards,
Ni Chunen / George
On 02/16/2020 16:22，shylinzhang<sh...@163.com> wrote：
Dear all,
The cube is ok when 2:00 am,but after 4:30 am it always shows error ,it shows ``` #4 Step Name: Build Dimension Dictionary
org.apache.kylin.engine.mr.exception.HadoopShellException: java.lang.RuntimeException: java.io.IOException: Failed to read big resource /dict/KYLIN.HIVE_METRICS_QUERY_CUBE_QA/CUBE_NAME/b9bb85ec-b72e-4df1-9ee7-e8c971ec5c72.dict
at org.apache.kylin.dict.DictionaryManager.getDictionaryInfo(DictionaryManager.java:108)
at org.apache.kylin.dict.DictionaryManager.checkDupByContent(DictionaryManager.java:173)
at org.apache.kylin.dict.DictionaryManager.trySaveNewDict(DictionaryManager.java:151)
at org.apache.kylin.dict.DictionaryManager.saveDictionary(DictionaryManager.java:320)
at org.apache.kylin.cube.CubeManager$DictionaryAssist.saveDictionary(CubeManager.java:1117)
at org.apache.kylin.cube.CubeManager.saveDictionary(CubeManager.java:1079)
at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:74)
at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:55)
at org.apache.kylin.engine.mr.steps.CreateDictionaryJob.run(CreateDictionaryJob.java:73)
at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:93)
at org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:63)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166)
at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166)
at org.apache.kylin.job.impl.threadpool.DistributedScheduler$JobRunner.run(DistributedScheduler.java:110)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
```
What I do:
1. I disable the cube and purge the cube and rebulid, it also show the same error;
2. Because the cube at 2:00 am is ok, so I use the meta_backs in 2:00 am and restore and cube; I try short period to build the cube, 2:00 am~3:00am~4:00am~4:00am~6:00am are ok, but in 6:am~7:00 am it also shows above error;
I found the solution in kylin jira, it tells me to update to release 2.6.4; our kylin clusters are used in 2019-06; just today shows the error.
Here’s the jira url: https://issues.apache.org/jira/browse/KYLIN-4153?jql=summary%20~%20%22Failed%20to%20read%20big%20resource*%22%20OR%20description%20~%20%22Failed%20to%20read%20big%20resource*%22%20ORDER%20BY%20lastViewed%20DESC
Therefore I want to know why it occured and if there is a way to fix it except updating.
I am looking forward for your replay, thank you.
Kylin version: 2.6.1
Hadoop version: 3.0.0+cdh6.0.0
Best Wishes!
Shylin Zhang
| |
shylinzhang
|
|
shylinzhang@163.com
|
签名由网易邮箱大师定制
Re:Failed to read big resource
/dict/KYLIN.HIVE_METRICS_QUERY_CUBE_QA/CUBE_NAME/xxx.dict
Posted by shylinzhang <sh...@163.com>.
Hi ZhangQiang,
I tried the command you given, and build the SYSTEM cube again, it also shows the error.
java.lang.RuntimeException: java.io.IOException: Failed to read big resource /dict/KYLIN.HIVE_METRICS_QUERY_RPC_QA/KTIME_MINUTE/3ad46874-e65a-4116-5c0c-3c9aefea36af.dict
at org.apache.kylin.dict.DictionaryManager.getDictionaryInfo(DictionaryManager.java:108)
at org.apache.kylin.dict.DictionaryManager.checkDupByContent(DictionaryManager.java:173)
at org.apache.kylin.dict.DictionaryManager.trySaveNewDict(DictionaryManager.java:151)
at org.apache.kylin.engine.mr.steps.UpdateDictionaryStep.doWork(UpdateDictionaryStep.java:111)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166)
at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166)
at org.apache.kylin.job.impl.threadpool.DistributedScheduler$JobRunner.run(DistributedScheduler.java:110)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Failed to read big resource /dict/KYLIN.HIVE_METRICS_QUERY_RPC_QA/KTIME_MINUTE/3ad46874-e65a-4116-5c0c-3c9aefea36af.dict
Best Regards,
Shylin Zhang
On 08/10/2020 16:50，Johnson<it...@163.com> wrote：
Hi Shylin Zhang
You can use "${KYLIN_HIME}/bin/metastore.sh remove /dict/KYLIN.HIVE_METRICS_JOB_QA/KUSER/d60c8ffc-c883-24d2-ec6b-c78fc963f8d7.dict" to delete this dict，it should work.
At 2020-08-10 16:02:47, "shylinzhang" <sh...@163.com> wrote:
Hi Xiaoxiang,
I met the issue agin and follow what your said last time, and it didn’t work. Here are the details:
ERROR:
org.apache.kylin.engine.mr.exception.HadoopShellException: java.lang.RuntimeException: java.io.IOException: Failed to read big resource /dict/KYLIN.HIVE_METRICS_JOB_QA/KUSER/d60c8ffc-c883-24d2-ec6b-c78fc963f8d7.dict
Caused by: java.io.FileNotFoundException: /kylin/kylin_metadata/resources/dict/KYLIN.HIVE_METRICS_JOB_QA/KUSER/d60c8ffc-c883-24d2-ec6b-c78fc963f8d7.dict (FS: DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_1184245014_33, ugi=kylin_app (auth:SIMPLE)]])
# 查看是否为空,为空
hbase(main):011:0> get 'kylin_metadata','/cube/KYLIN_HIVE_METRICS_JOB_QA.json','f:HIVE_METRICS_JOB_QA.KUSER'
COLUMN CELL
0 row(s)
Took 0.0129 seconds
# 删除空值
hbase(main):012:0> delete 'kylin_metadata','/cube/KYLIN_HIVE_METRICS_JOB_QA.json','f:HIVE_METRICS_JOB_QA.KUSER'
Took 0.0166 seconds
hbase(main):013:0>
After that I build the cube it also shows the error.
ENV:
Kylin version: 2.6.1
Hadoop version: 3.0.0+cdh6.0.0
Best Regards,
Shylin Zhang
On 02/19/2020 10:52，shylinzhang<sh...@163.com> wrote：
Hi Xiaoxiang,
Thanks for your advices, I will follow them.
Thank you for your help again.
Best Regards,
Shylin Zhang
On 02/18/2020 23:35，Xiaoxiang Yu<xx...@apache.org> wrote：
Hi shylinzhang,
Thank you for your update. I am glad to hear you have fixed your problem finally.
I wish you can attach the some detailed description or some error log if you face some problem in the future; using the simple/misunderstanding sentence "it failed' is not good enough in my side.
Here is some advice provided by apache website:
1. http://www.catb.org/~esr/faqs/smart-questions.html,
2. http://www.apache.org/dev/contrib-email-tips .
--
Best wishes to you !
From ：Xiaoxiang Yu
在 2020-02-18 19:43:15，"shylinzhang" <sh...@163.com> 写道：
Hi Xiaoxiang,
I’m sorry that I misunderstood the meaning of “delete the remaining marker”.I followed what you said and solved one system cube issue;
But I fix anther same issued system cube with same way, it failed，And then I purge and rebuild the cube and it works well.
Thank you for your help.
Best Wishes!
Shylin Zhang
| |
shylinzhang
|
|
shylinzhang@163.com
|
签名由网易邮箱大师定制
On 02/17/2020 19:45，Xiaoxiang Yu<xx...@apache.org> wrote：
Hi,
I don't understand what problem did you face in "I tried to delete the remaining marker, it failed"?
As far as I see, you can open the hbase shell. Use the "get /dict/KYLIN.HIVE_METRICS_QUERY_CUBE_QA/CUBE_NAME/xxx.dict" to get the value of that entry,
and use "delete" to delete that empty entry. Before doing this, I think you should back up your metadata.
--
Best wishes to you !
From ：Xiaoxiang Yu
在 2020-02-17 10:44:23，"shylinzhang" <sh...@163.com> 写道：
Hi George,
Thank you for your reply, I tried to delete the remaining marker, it failed. So I had to update.
Thank you!
Best Wishes!
Shylin Zhang
| |
shylinzhang
|
|
shylinzhang@163.com
|
签名由网易邮箱大师定制
On 02/16/2020 21:05，nichunen<ni...@apache.org> wrote：
Hi Shylin,
I think the description part in https://issues.apache.org/jira/browse/KYLIN-4153 is clear enough, you can find the root cause and workaround there, it happened due to the lack of atomicity during the rollback for the "big resource".
As a workaround, you may delete the remaining marker in your resource store, but it’s more advised to do an upgrade for your Kylin.
Best regards,
Ni Chunen / George
On 02/16/2020 16:22，shylinzhang<sh...@163.com> wrote：
Dear all,
The cube is ok when 2:00 am,but after 4:30 am it always shows error ,it shows ``` #4 Step Name: Build Dimension Dictionary
org.apache.kylin.engine.mr.exception.HadoopShellException: java.lang.RuntimeException: java.io.IOException: Failed to read big resource /dict/KYLIN.HIVE_METRICS_QUERY_CUBE_QA/CUBE_NAME/b9bb85ec-b72e-4df1-9ee7-e8c971ec5c72.dict
at org.apache.kylin.dict.DictionaryManager.getDictionaryInfo(DictionaryManager.java:108)
at org.apache.kylin.dict.DictionaryManager.checkDupByContent(DictionaryManager.java:173)
at org.apache.kylin.dict.DictionaryManager.trySaveNewDict(DictionaryManager.java:151)
at org.apache.kylin.dict.DictionaryManager.saveDictionary(DictionaryManager.java:320)
at org.apache.kylin.cube.CubeManager$DictionaryAssist.saveDictionary(CubeManager.java:1117)
at org.apache.kylin.cube.CubeManager.saveDictionary(CubeManager.java:1079)
at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:74)
at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:55)
at org.apache.kylin.engine.mr.steps.CreateDictionaryJob.run(CreateDictionaryJob.java:73)
at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:93)
at org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:63)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166)
at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166)
at org.apache.kylin.job.impl.threadpool.DistributedScheduler$JobRunner.run(DistributedScheduler.java:110)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
```
What I do:
1. I disable the cube and purge the cube and rebulid, it also show the same error;
2. Because the cube at 2:00 am is ok, so I use the meta_backs in 2:00 am and restore and cube; I try short period to build the cube, 2:00 am~3:00am~4:00am~4:00am~6:00am are ok, but in 6:am~7:00 am it also shows above error;
I found the solution in kylin jira, it tells me to update to release 2.6.4; our kylin clusters are used in 2019-06; just today shows the error.
Here’s the jira url: https://issues.apache.org/jira/browse/KYLIN-4153?jql=summary%20~%20%22Failed%20to%20read%20big%20resource*%22%20OR%20description%20~%20%22Failed%20to%20read%20big%20resource*%22%20ORDER%20BY%20lastViewed%20DESC
Therefore I want to know why it occured and if there is a way to fix it except updating.
I am looking forward for your replay, thank you.
Kylin version: 2.6.1
Hadoop version: 3.0.0+cdh6.0.0
Best Wishes!
Shylin Zhang
| |
shylinzhang
|
|
shylinzhang@163.com
|
签名由网易邮箱大师定制
Re:Failed to read big resource
/dict/KYLIN.HIVE_METRICS_QUERY_CUBE_QA/CUBE_NAME/xxx.dict
Posted by Johnson <it...@163.com>.
Hi Shylin Zhang
You can use "${KYLIN_HIME}/bin/metastore.sh remove /dict/KYLIN.HIVE_METRICS_JOB_QA/KUSER/d60c8ffc-c883-24d2-ec6b-c78fc963f8d7.dict" to delete this dict，it should work.
At 2020-08-10 16:02:47, "shylinzhang" <sh...@163.com> wrote:
Hi Xiaoxiang,
I met the issue agin and follow what your said last time, and it didn’t work. Here are the details:
ERROR:
org.apache.kylin.engine.mr.exception.HadoopShellException: java.lang.RuntimeException: java.io.IOException: Failed to read big resource /dict/KYLIN.HIVE_METRICS_JOB_QA/KUSER/d60c8ffc-c883-24d2-ec6b-c78fc963f8d7.dict
Caused by: java.io.FileNotFoundException: /kylin/kylin_metadata/resources/dict/KYLIN.HIVE_METRICS_JOB_QA/KUSER/d60c8ffc-c883-24d2-ec6b-c78fc963f8d7.dict (FS: DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_1184245014_33, ugi=kylin_app (auth:SIMPLE)]])
# 查看是否为空,为空
hbase(main):011:0> get 'kylin_metadata','/cube/KYLIN_HIVE_METRICS_JOB_QA.json','f:HIVE_METRICS_JOB_QA.KUSER'
COLUMN CELL
0 row(s)
Took 0.0129 seconds
# 删除空值
hbase(main):012:0> delete 'kylin_metadata','/cube/KYLIN_HIVE_METRICS_JOB_QA.json','f:HIVE_METRICS_JOB_QA.KUSER'
Took 0.0166 seconds
hbase(main):013:0>
After that I build the cube it also shows the error.
ENV:
Kylin version: 2.6.1
Hadoop version: 3.0.0+cdh6.0.0
Best Regards,
Shylin Zhang
On 02/19/2020 10:52，shylinzhang<sh...@163.com> wrote：
Hi Xiaoxiang,
Thanks for your advices, I will follow them.
Thank you for your help again.
Best Regards,
Shylin Zhang
On 02/18/2020 23:35，Xiaoxiang Yu<xx...@apache.org> wrote：
Hi shylinzhang,
Thank you for your update. I am glad to hear you have fixed your problem finally.
I wish you can attach the some detailed description or some error log if you face some problem in the future; using the simple/misunderstanding sentence "it failed' is not good enough in my side.
Here is some advice provided by apache website:
1. http://www.catb.org/~esr/faqs/smart-questions.html,
2. http://www.apache.org/dev/contrib-email-tips .
--
Best wishes to you !
From ：Xiaoxiang Yu
在 2020-02-18 19:43:15，"shylinzhang" <sh...@163.com> 写道：
Hi Xiaoxiang,
I’m sorry that I misunderstood the meaning of “delete the remaining marker”.I followed what you said and solved one system cube issue;
But I fix anther same issued system cube with same way, it failed，And then I purge and rebuild the cube and it works well.
Thank you for your help.
Best Wishes!
Shylin Zhang
| |
shylinzhang
|
|
shylinzhang@163.com
|
签名由网易邮箱大师定制
On 02/17/2020 19:45，Xiaoxiang Yu<xx...@apache.org> wrote：
Hi,
I don't understand what problem did you face in "I tried to delete the remaining marker, it failed"?
As far as I see, you can open the hbase shell. Use the "get /dict/KYLIN.HIVE_METRICS_QUERY_CUBE_QA/CUBE_NAME/xxx.dict" to get the value of that entry,
and use "delete" to delete that empty entry. Before doing this, I think you should back up your metadata.
--
Best wishes to you !
From ：Xiaoxiang Yu
在 2020-02-17 10:44:23，"shylinzhang" <sh...@163.com> 写道：
Hi George,
Thank you for your reply, I tried to delete the remaining marker, it failed. So I had to update.
Thank you!
Best Wishes!
Shylin Zhang
| |
shylinzhang
|
|
shylinzhang@163.com
|
签名由网易邮箱大师定制
On 02/16/2020 21:05，nichunen<ni...@apache.org> wrote：
Hi Shylin,
I think the description part in https://issues.apache.org/jira/browse/KYLIN-4153 is clear enough, you can find the root cause and workaround there, it happened due to the lack of atomicity during the rollback for the "big resource".
As a workaround, you may delete the remaining marker in your resource store, but it’s more advised to do an upgrade for your Kylin.
Best regards,
Ni Chunen / George
On 02/16/2020 16:22，shylinzhang<sh...@163.com> wrote：
Dear all,
The cube is ok when 2:00 am,but after 4:30 am it always shows error ,it shows ``` #4 Step Name: Build Dimension Dictionary
org.apache.kylin.engine.mr.exception.HadoopShellException: java.lang.RuntimeException: java.io.IOException: Failed to read big resource /dict/KYLIN.HIVE_METRICS_QUERY_CUBE_QA/CUBE_NAME/b9bb85ec-b72e-4df1-9ee7-e8c971ec5c72.dict
at org.apache.kylin.dict.DictionaryManager.getDictionaryInfo(DictionaryManager.java:108)
at org.apache.kylin.dict.DictionaryManager.checkDupByContent(DictionaryManager.java:173)
at org.apache.kylin.dict.DictionaryManager.trySaveNewDict(DictionaryManager.java:151)
at org.apache.kylin.dict.DictionaryManager.saveDictionary(DictionaryManager.java:320)
at org.apache.kylin.cube.CubeManager$DictionaryAssist.saveDictionary(CubeManager.java:1117)
at org.apache.kylin.cube.CubeManager.saveDictionary(CubeManager.java:1079)
at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:74)
at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:55)
at org.apache.kylin.engine.mr.steps.CreateDictionaryJob.run(CreateDictionaryJob.java:73)
at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:93)
at org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:63)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166)
at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166)
at org.apache.kylin.job.impl.threadpool.DistributedScheduler$JobRunner.run(DistributedScheduler.java:110)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
```
What I do:
1. I disable the cube and purge the cube and rebulid, it also show the same error;
2. Because the cube at 2:00 am is ok, so I use the meta_backs in 2:00 am and restore and cube; I try short period to build the cube, 2:00 am~3:00am~4:00am~4:00am~6:00am are ok, but in 6:am~7:00 am it also shows above error;
I found the solution in kylin jira, it tells me to update to release 2.6.4; our kylin clusters are used in 2019-06; just today shows the error.
Here’s the jira url: https://issues.apache.org/jira/browse/KYLIN-4153?jql=summary%20~%20%22Failed%20to%20read%20big%20resource*%22%20OR%20description%20~%20%22Failed%20to%20read%20big%20resource*%22%20ORDER%20BY%20lastViewed%20DESC
Therefore I want to know why it occured and if there is a way to fix it except updating.
I am looking forward for your replay, thank you.
Kylin version: 2.6.1
Hadoop version: 3.0.0+cdh6.0.0
Best Wishes!
Shylin Zhang
| |
shylinzhang
|
|
shylinzhang@163.com
|
签名由网易邮箱大师定制
Failed to read big resource
/dict/KYLIN.HIVE_METRICS_QUERY_CUBE_QA/CUBE_NAME/xxx.dict
Posted by shylinzhang <sh...@163.com>.
Hi Xiaoxiang,
I met the issue agin and follow what your said last time, and it didn’t work. Here are the details:
ERROR:
org.apache.kylin.engine.mr.exception.HadoopShellException: java.lang.RuntimeException: java.io.IOException: Failed to read big resource /dict/KYLIN.HIVE_METRICS_JOB_QA/KUSER/d60c8ffc-c883-24d2-ec6b-c78fc963f8d7.dict
Caused by: java.io.FileNotFoundException: /kylin/kylin_metadata/resources/dict/KYLIN.HIVE_METRICS_JOB_QA/KUSER/d60c8ffc-c883-24d2-ec6b-c78fc963f8d7.dict (FS: DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_1184245014_33, ugi=kylin_app (auth:SIMPLE)]])
# 查看是否为空,为空
hbase(main):011:0> get 'kylin_metadata','/cube/KYLIN_HIVE_METRICS_JOB_QA.json','f:HIVE_METRICS_JOB_QA.KUSER'
COLUMN CELL
0 row(s)
Took 0.0129 seconds
# 删除空值
hbase(main):012:0> delete 'kylin_metadata','/cube/KYLIN_HIVE_METRICS_JOB_QA.json','f:HIVE_METRICS_JOB_QA.KUSER'
Took 0.0166 seconds
hbase(main):013:0>
After that I build the cube it also shows the error.
ENV:
Kylin version: 2.6.1
Hadoop version: 3.0.0+cdh6.0.0
Best Regards,
Shylin Zhang
On 02/19/2020 10:52，shylinzhang<sh...@163.com> wrote：
Hi Xiaoxiang,
Thanks for your advices, I will follow them.
Thank you for your help again.
Best Regards,
Shylin Zhang
On 02/18/2020 23:35，Xiaoxiang Yu<xx...@apache.org> wrote：
Hi shylinzhang,
Thank you for your update. I am glad to hear you have fixed your problem finally.
I wish you can attach the some detailed description or some error log if you face some problem in the future; using the simple/misunderstanding sentence "it failed' is not good enough in my side.
Here is some advice provided by apache website:
1. http://www.catb.org/~esr/faqs/smart-questions.html,
2. http://www.apache.org/dev/contrib-email-tips .
--
Best wishes to you !
From ：Xiaoxiang Yu
在 2020-02-18 19:43:15，"shylinzhang" <sh...@163.com> 写道：
Hi Xiaoxiang,
I’m sorry that I misunderstood the meaning of “delete the remaining marker”.I followed what you said and solved one system cube issue;
But I fix anther same issued system cube with same way, it failed，And then I purge and rebuild the cube and it works well.
Thank you for your help.
Best Wishes!
Shylin Zhang
| |
shylinzhang
|
|
shylinzhang@163.com
|
签名由网易邮箱大师定制
On 02/17/2020 19:45，Xiaoxiang Yu<xx...@apache.org> wrote：
Hi,
I don't understand what problem did you face in "I tried to delete the remaining marker, it failed"?
As far as I see, you can open the hbase shell. Use the "get /dict/KYLIN.HIVE_METRICS_QUERY_CUBE_QA/CUBE_NAME/xxx.dict" to get the value of that entry,
and use "delete" to delete that empty entry. Before doing this, I think you should back up your metadata.
--
Best wishes to you !
From ：Xiaoxiang Yu
在 2020-02-17 10:44:23，"shylinzhang" <sh...@163.com> 写道：
Hi George,
Thank you for your reply, I tried to delete the remaining marker, it failed. So I had to update.
Thank you!
Best Wishes!
Shylin Zhang
| |
shylinzhang
|
|
shylinzhang@163.com
|
签名由网易邮箱大师定制
On 02/16/2020 21:05，nichunen<ni...@apache.org> wrote：
Hi Shylin,
I think the description part in https://issues.apache.org/jira/browse/KYLIN-4153 is clear enough, you can find the root cause and workaround there, it happened due to the lack of atomicity during the rollback for the "big resource".
As a workaround, you may delete the remaining marker in your resource store, but it’s more advised to do an upgrade for your Kylin.
Best regards,
Ni Chunen / George
On 02/16/2020 16:22，shylinzhang<sh...@163.com> wrote：
Dear all,
The cube is ok when 2:00 am,but after 4:30 am it always shows error ,it shows ``` #4 Step Name: Build Dimension Dictionary
org.apache.kylin.engine.mr.exception.HadoopShellException: java.lang.RuntimeException: java.io.IOException: Failed to read big resource /dict/KYLIN.HIVE_METRICS_QUERY_CUBE_QA/CUBE_NAME/b9bb85ec-b72e-4df1-9ee7-e8c971ec5c72.dict
at org.apache.kylin.dict.DictionaryManager.getDictionaryInfo(DictionaryManager.java:108)
at org.apache.kylin.dict.DictionaryManager.checkDupByContent(DictionaryManager.java:173)
at org.apache.kylin.dict.DictionaryManager.trySaveNewDict(DictionaryManager.java:151)
at org.apache.kylin.dict.DictionaryManager.saveDictionary(DictionaryManager.java:320)
at org.apache.kylin.cube.CubeManager$DictionaryAssist.saveDictionary(CubeManager.java:1117)
at org.apache.kylin.cube.CubeManager.saveDictionary(CubeManager.java:1079)
at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:74)
at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:55)
at org.apache.kylin.engine.mr.steps.CreateDictionaryJob.run(CreateDictionaryJob.java:73)
at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:93)
at org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:63)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166)
at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166)
at org.apache.kylin.job.impl.threadpool.DistributedScheduler$JobRunner.run(DistributedScheduler.java:110)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
```
What I do:
1. I disable the cube and purge the cube and rebulid, it also show the same error;
2. Because the cube at 2:00 am is ok, so I use the meta_backs in 2:00 am and restore and cube; I try short period to build the cube, 2:00 am~3:00am~4:00am~4:00am~6:00am are ok, but in 6:am~7:00 am it also shows above error;
I found the solution in kylin jira, it tells me to update to release 2.6.4; our kylin clusters are used in 2019-06; just today shows the error.
Here’s the jira url: https://issues.apache.org/jira/browse/KYLIN-4153?jql=summary%20~%20%22Failed%20to%20read%20big%20resource*%22%20OR%20description%20~%20%22Failed%20to%20read%20big%20resource*%22%20ORDER%20BY%20lastViewed%20DESC
Therefore I want to know why it occured and if there is a way to fix it except updating.
I am looking forward for your replay, thank you.
Kylin version: 2.6.1
Hadoop version: 3.0.0+cdh6.0.0
Best Wishes!
Shylin Zhang
| |
shylinzhang
|
|
shylinzhang@163.com
|
签名由网易邮箱大师定制
Re:Failed to read big resource
/dict/KYLIN.HIVE_METRICS_QUERY_CUBE_QA/CUBE_NAME/xxx.dict
Posted by shylinzhang <sh...@163.com>.
Hi Xiaoxiang,
Thanks for your advices, I will follow them.
Thank you for your help again.
Best Regards,
Shylin Zhang
On 02/18/2020 23:35，Xiaoxiang Yu<xx...@apache.org> wrote：
Hi shylinzhang,
Thank you for your update. I am glad to hear you have fixed your problem finally.
I wish you can attach the some detailed description or some error log if you face some problem in the future; using the simple/misunderstanding sentence "it failed' is not good enough in my side.
Here is some advice provided by apache website:
1. http://www.catb.org/~esr/faqs/smart-questions.html,
2. http://www.apache.org/dev/contrib-email-tips .
--
Best wishes to you !
From ：Xiaoxiang Yu
在 2020-02-18 19:43:15，"shylinzhang" <sh...@163.com> 写道：
Hi Xiaoxiang,
I’m sorry that I misunderstood the meaning of “delete the remaining marker”.I followed what you said and solved one system cube issue;
But I fix anther same issued system cube with same way, it failed，And then I purge and rebuild the cube and it works well.
Thank you for your help.
Best Wishes!
Shylin Zhang
| |
shylinzhang
|
|
shylinzhang@163.com
|
签名由网易邮箱大师定制
On 02/17/2020 19:45，Xiaoxiang Yu<xx...@apache.org> wrote：
Hi,
I don't understand what problem did you face in "I tried to delete the remaining marker, it failed"?
As far as I see, you can open the hbase shell. Use the "get /dict/KYLIN.HIVE_METRICS_QUERY_CUBE_QA/CUBE_NAME/xxx.dict" to get the value of that entry,
and use "delete" to delete that empty entry. Before doing this, I think you should back up your metadata.
--
Best wishes to you !
From ：Xiaoxiang Yu
在 2020-02-17 10:44:23，"shylinzhang" <sh...@163.com> 写道：
Hi George,
Thank you for your reply, I tried to delete the remaining marker, it failed. So I had to update.
Thank you!
Best Wishes!
Shylin Zhang
| |
shylinzhang
|
|
shylinzhang@163.com
|
签名由网易邮箱大师定制
On 02/16/2020 21:05，nichunen<ni...@apache.org> wrote：
Hi Shylin,
I think the description part in https://issues.apache.org/jira/browse/KYLIN-4153 is clear enough, you can find the root cause and workaround there, it happened due to the lack of atomicity during the rollback for the "big resource".
As a workaround, you may delete the remaining marker in your resource store, but it’s more advised to do an upgrade for your Kylin.
Best regards,
Ni Chunen / George
On 02/16/2020 16:22，shylinzhang<sh...@163.com> wrote：
Dear all,
The cube is ok when 2:00 am,but after 4:30 am it always shows error ,it shows ``` #4 Step Name: Build Dimension Dictionary
org.apache.kylin.engine.mr.exception.HadoopShellException: java.lang.RuntimeException: java.io.IOException: Failed to read big resource /dict/KYLIN.HIVE_METRICS_QUERY_CUBE_QA/CUBE_NAME/b9bb85ec-b72e-4df1-9ee7-e8c971ec5c72.dict
at org.apache.kylin.dict.DictionaryManager.getDictionaryInfo(DictionaryManager.java:108)
at org.apache.kylin.dict.DictionaryManager.checkDupByContent(DictionaryManager.java:173)
at org.apache.kylin.dict.DictionaryManager.trySaveNewDict(DictionaryManager.java:151)
at org.apache.kylin.dict.DictionaryManager.saveDictionary(DictionaryManager.java:320)
at org.apache.kylin.cube.CubeManager$DictionaryAssist.saveDictionary(CubeManager.java:1117)
at org.apache.kylin.cube.CubeManager.saveDictionary(CubeManager.java:1079)
at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:74)
at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:55)
at org.apache.kylin.engine.mr.steps.CreateDictionaryJob.run(CreateDictionaryJob.java:73)
at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:93)
at org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:63)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166)
at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166)
at org.apache.kylin.job.impl.threadpool.DistributedScheduler$JobRunner.run(DistributedScheduler.java:110)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
```
What I do:
1. I disable the cube and purge the cube and rebulid, it also show the same error;
2. Because the cube at 2:00 am is ok, so I use the meta_backs in 2:00 am and restore and cube; I try short period to build the cube, 2:00 am~3:00am~4:00am~4:00am~6:00am are ok, but in 6:am~7:00 am it also shows above error;
I found the solution in kylin jira, it tells me to update to release 2.6.4; our kylin clusters are used in 2019-06; just today shows the error.
Here’s the jira url: https://issues.apache.org/jira/browse/KYLIN-4153?jql=summary%20~%20%22Failed%20to%20read%20big%20resource*%22%20OR%20description%20~%20%22Failed%20to%20read%20big%20resource*%22%20ORDER%20BY%20lastViewed%20DESC
Therefore I want to know why it occured and if there is a way to fix it except updating.
I am looking forward for your replay, thank you.
Kylin version: 2.6.1
Hadoop version: 3.0.0+cdh6.0.0
Best Wishes!
Shylin Zhang
| |
shylinzhang
|
|
shylinzhang@163.com
|
签名由网易邮箱大师定制
Re:Failed to read big resource
/dict/KYLIN.HIVE_METRICS_QUERY_CUBE_QA/CUBE_NAME/xxx.dict
Posted by Xiaoxiang Yu <xx...@apache.org>.
Hi shylinzhang,
Thank you for your update. I am glad to hear you have fixed your problem finally.
I wish you can attach the some detailed description or some error log if you face some problem in the future; using the simple/misunderstanding sentence "it failed' is not good enough in my side.
Here is some advice provided by apache website:
1. http://www.catb.org/~esr/faqs/smart-questions.html,
2. http://www.apache.org/dev/contrib-email-tips .
--
Best wishes to you !
From ：Xiaoxiang Yu
在 2020-02-18 19:43:15，"shylinzhang" <sh...@163.com> 写道：
Hi Xiaoxiang,
I’m sorry that I misunderstood the meaning of “delete the remaining marker”.I followed what you said and solved one system cube issue;
But I fix anther same issued system cube with same way, it failed，And then I purge and rebuild the cube and it works well.
Thank you for your help.
Best Wishes!
Shylin Zhang
| |
shylinzhang
|
|
shylinzhang@163.com
|
签名由网易邮箱大师定制
On 02/17/2020 19:45，Xiaoxiang Yu<xx...@apache.org> wrote：
Hi,
I don't understand what problem did you face in "I tried to delete the remaining marker, it failed"?
As far as I see, you can open the hbase shell. Use the "get /dict/KYLIN.HIVE_METRICS_QUERY_CUBE_QA/CUBE_NAME/xxx.dict" to get the value of that entry,
and use "delete" to delete that empty entry. Before doing this, I think you should back up your metadata.
--
Best wishes to you !
From ：Xiaoxiang Yu
在 2020-02-17 10:44:23，"shylinzhang" <sh...@163.com> 写道：
Hi George,
Thank you for your reply, I tried to delete the remaining marker, it failed. So I had to update.
Thank you!
Best Wishes!
Shylin Zhang
| |
shylinzhang
|
|
shylinzhang@163.com
|
签名由网易邮箱大师定制
On 02/16/2020 21:05，nichunen<ni...@apache.org> wrote：
Hi Shylin,
I think the description part in https://issues.apache.org/jira/browse/KYLIN-4153 is clear enough, you can find the root cause and workaround there, it happened due to the lack of atomicity during the rollback for the "big resource".
As a workaround, you may delete the remaining marker in your resource store, but it’s more advised to do an upgrade for your Kylin.
Best regards,
Ni Chunen / George
On 02/16/2020 16:22，shylinzhang<sh...@163.com> wrote：
Dear all,
The cube is ok when 2:00 am,but after 4:30 am it always shows error ,it shows ``` #4 Step Name: Build Dimension Dictionary
org.apache.kylin.engine.mr.exception.HadoopShellException: java.lang.RuntimeException: java.io.IOException: Failed to read big resource /dict/KYLIN.HIVE_METRICS_QUERY_CUBE_QA/CUBE_NAME/b9bb85ec-b72e-4df1-9ee7-e8c971ec5c72.dict
at org.apache.kylin.dict.DictionaryManager.getDictionaryInfo(DictionaryManager.java:108)
at org.apache.kylin.dict.DictionaryManager.checkDupByContent(DictionaryManager.java:173)
at org.apache.kylin.dict.DictionaryManager.trySaveNewDict(DictionaryManager.java:151)
at org.apache.kylin.dict.DictionaryManager.saveDictionary(DictionaryManager.java:320)
at org.apache.kylin.cube.CubeManager$DictionaryAssist.saveDictionary(CubeManager.java:1117)
at org.apache.kylin.cube.CubeManager.saveDictionary(CubeManager.java:1079)
at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:74)
at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:55)
at org.apache.kylin.engine.mr.steps.CreateDictionaryJob.run(CreateDictionaryJob.java:73)
at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:93)
at org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:63)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166)
at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166)
at org.apache.kylin.job.impl.threadpool.DistributedScheduler$JobRunner.run(DistributedScheduler.java:110)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
```
What I do:
1. I disable the cube and purge the cube and rebulid, it also show the same error;
2. Because the cube at 2:00 am is ok, so I use the meta_backs in 2:00 am and restore and cube; I try short period to build the cube, 2:00 am~3:00am~4:00am~4:00am~6:00am are ok, but in 6:am~7:00 am it also shows above error;
I found the solution in kylin jira, it tells me to update to release 2.6.4; our kylin clusters are used in 2019-06; just today shows the error.
Here’s the jira url: https://issues.apache.org/jira/browse/KYLIN-4153?jql=summary%20~%20%22Failed%20to%20read%20big%20resource*%22%20OR%20description%20~%20%22Failed%20to%20read%20big%20resource*%22%20ORDER%20BY%20lastViewed%20DESC
Therefore I want to know why it occured and if there is a way to fix it except updating.
I am looking forward for your replay, thank you.
Kylin version: 2.6.1
Hadoop version: 3.0.0+cdh6.0.0
Best Wishes!
Shylin Zhang
| |
shylinzhang
|
|
shylinzhang@163.com
|
签名由网易邮箱大师定制
Re:Failed to read big resource
/dict/KYLIN.HIVE_METRICS_QUERY_CUBE_QA/CUBE_NAME/xxx.dict
Posted by shylinzhang <sh...@163.com>.
Hi Xiaoxiang,
I’m sorry that I misunderstood the meaning of “delete the remaining marker”.I followed what you said and solved one system cube issue;
But I fix anther same issued system cube with same way, it failed，And then I purge and rebuild the cube and it works well.
Thank you for your help.
Best Wishes!
Shylin Zhang
| |
shylinzhang
|
|
shylinzhang@163.com
|
签名由网易邮箱大师定制
On 02/17/2020 19:45，Xiaoxiang Yu<xx...@apache.org> wrote：
Hi,
I don't understand what problem did you face in "I tried to delete the remaining marker, it failed"?
As far as I see, you can open the hbase shell. Use the "get /dict/KYLIN.HIVE_METRICS_QUERY_CUBE_QA/CUBE_NAME/xxx.dict" to get the value of that entry,
and use "delete" to delete that empty entry. Before doing this, I think you should back up your metadata.
--
Best wishes to you !
From ：Xiaoxiang Yu
在 2020-02-17 10:44:23，"shylinzhang" <sh...@163.com> 写道：
Hi George,
Thank you for your reply, I tried to delete the remaining marker, it failed. So I had to update.
Thank you!
Best Wishes!
Shylin Zhang
| |
shylinzhang
|
|
shylinzhang@163.com
|
签名由网易邮箱大师定制
On 02/16/2020 21:05，nichunen<ni...@apache.org> wrote：
Hi Shylin,
I think the description part in https://issues.apache.org/jira/browse/KYLIN-4153 is clear enough, you can find the root cause and workaround there, it happened due to the lack of atomicity during the rollback for the "big resource".
As a workaround, you may delete the remaining marker in your resource store, but it’s more advised to do an upgrade for your Kylin.
Best regards,
Ni Chunen / George
On 02/16/2020 16:22，shylinzhang<sh...@163.com> wrote：
Dear all,
The cube is ok when 2:00 am,but after 4:30 am it always shows error ,it shows ``` #4 Step Name: Build Dimension Dictionary
org.apache.kylin.engine.mr.exception.HadoopShellException: java.lang.RuntimeException: java.io.IOException: Failed to read big resource /dict/KYLIN.HIVE_METRICS_QUERY_CUBE_QA/CUBE_NAME/b9bb85ec-b72e-4df1-9ee7-e8c971ec5c72.dict
at org.apache.kylin.dict.DictionaryManager.getDictionaryInfo(DictionaryManager.java:108)
at org.apache.kylin.dict.DictionaryManager.checkDupByContent(DictionaryManager.java:173)
at org.apache.kylin.dict.DictionaryManager.trySaveNewDict(DictionaryManager.java:151)
at org.apache.kylin.dict.DictionaryManager.saveDictionary(DictionaryManager.java:320)
at org.apache.kylin.cube.CubeManager$DictionaryAssist.saveDictionary(CubeManager.java:1117)
at org.apache.kylin.cube.CubeManager.saveDictionary(CubeManager.java:1079)
at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:74)
at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:55)
at org.apache.kylin.engine.mr.steps.CreateDictionaryJob.run(CreateDictionaryJob.java:73)
at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:93)
at org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:63)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166)
at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166)
at org.apache.kylin.job.impl.threadpool.DistributedScheduler$JobRunner.run(DistributedScheduler.java:110)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
```
What I do:
1. I disable the cube and purge the cube and rebulid, it also show the same error;
2. Because the cube at 2:00 am is ok, so I use the meta_backs in 2:00 am and restore and cube; I try short period to build the cube, 2:00 am~3:00am~4:00am~4:00am~6:00am are ok, but in 6:am~7:00 am it also shows above error;
I found the solution in kylin jira, it tells me to update to release 2.6.4; our kylin clusters are used in 2019-06; just today shows the error.
Here’s the jira url: https://issues.apache.org/jira/browse/KYLIN-4153?jql=summary%20~%20%22Failed%20to%20read%20big%20resource*%22%20OR%20description%20~%20%22Failed%20to%20read%20big%20resource*%22%20ORDER%20BY%20lastViewed%20DESC
Therefore I want to know why it occured and if there is a way to fix it except updating.
I am looking forward for your replay, thank you.
Kylin version: 2.6.1
Hadoop version: 3.0.0+cdh6.0.0
Best Wishes!
Shylin Zhang
| |
shylinzhang
|
|
shylinzhang@163.com
|
签名由网易邮箱大师定制
Re:Failed to read big resource
/dict/KYLIN.HIVE_METRICS_QUERY_CUBE_QA/CUBE_NAME/xxx.dict
Posted by Xiaoxiang Yu <xx...@apache.org>.
Hi,
I don't understand what problem did you face in "I tried to delete the remaining marker, it failed"?
As far as I see, you can open the hbase shell. Use the "get /dict/KYLIN.HIVE_METRICS_QUERY_CUBE_QA/CUBE_NAME/xxx.dict" to get the value of that entry,
and use "delete" to delete that empty entry. Before doing this, I think you should back up your metadata.
--
Best wishes to you !
From ：Xiaoxiang Yu
在 2020-02-17 10:44:23，"shylinzhang" <sh...@163.com> 写道：
Hi George,
Thank you for your reply, I tried to delete the remaining marker, it failed. So I had to update.
Thank you!
Best Wishes!
Shylin Zhang
| |
shylinzhang
|
|
shylinzhang@163.com
|
签名由网易邮箱大师定制
On 02/16/2020 21:05，nichunen<ni...@apache.org> wrote：
Hi Shylin,
I think the description part in https://issues.apache.org/jira/browse/KYLIN-4153 is clear enough, you can find the root cause and workaround there, it happened due to the lack of atomicity during the rollback for the "big resource".
As a workaround, you may delete the remaining marker in your resource store, but it’s more advised to do an upgrade for your Kylin.
Best regards,
Ni Chunen / George
On 02/16/2020 16:22，shylinzhang<sh...@163.com> wrote：
Dear all,
The cube is ok when 2:00 am,but after 4:30 am it always shows error ,it shows ``` #4 Step Name: Build Dimension Dictionary
org.apache.kylin.engine.mr.exception.HadoopShellException: java.lang.RuntimeException: java.io.IOException: Failed to read big resource /dict/KYLIN.HIVE_METRICS_QUERY_CUBE_QA/CUBE_NAME/b9bb85ec-b72e-4df1-9ee7-e8c971ec5c72.dict
at org.apache.kylin.dict.DictionaryManager.getDictionaryInfo(DictionaryManager.java:108)
at org.apache.kylin.dict.DictionaryManager.checkDupByContent(DictionaryManager.java:173)
at org.apache.kylin.dict.DictionaryManager.trySaveNewDict(DictionaryManager.java:151)
at org.apache.kylin.dict.DictionaryManager.saveDictionary(DictionaryManager.java:320)
at org.apache.kylin.cube.CubeManager$DictionaryAssist.saveDictionary(CubeManager.java:1117)
at org.apache.kylin.cube.CubeManager.saveDictionary(CubeManager.java:1079)
at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:74)
at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:55)
at org.apache.kylin.engine.mr.steps.CreateDictionaryJob.run(CreateDictionaryJob.java:73)
at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:93)
at org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:63)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166)
at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166)
at org.apache.kylin.job.impl.threadpool.DistributedScheduler$JobRunner.run(DistributedScheduler.java:110)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
```
What I do:
1. I disable the cube and purge the cube and rebulid, it also show the same error;
2. Because the cube at 2:00 am is ok, so I use the meta_backs in 2:00 am and restore and cube; I try short period to build the cube, 2:00 am~3:00am~4:00am~4:00am~6:00am are ok, but in 6:am~7:00 am it also shows above error;
I found the solution in kylin jira, it tells me to update to release 2.6.4; our kylin clusters are used in 2019-06; just today shows the error.
Here’s the jira url: https://issues.apache.org/jira/browse/KYLIN-4153?jql=summary%20~%20%22Failed%20to%20read%20big%20resource*%22%20OR%20description%20~%20%22Failed%20to%20read%20big%20resource*%22%20ORDER%20BY%20lastViewed%20DESC
Therefore I want to know why it occured and if there is a way to fix it except updating.
I am looking forward for your replay, thank you.
Kylin version: 2.6.1
Hadoop version: 3.0.0+cdh6.0.0
Best Wishes!
Shylin Zhang
| |
shylinzhang
|
|
shylinzhang@163.com
|
签名由网易邮箱大师定制
Re:Failed to read big resource
/dict/KYLIN.HIVE_METRICS_QUERY_CUBE_QA/CUBE_NAME/xxx.dict
Posted by shylinzhang <sh...@163.com>.
Hi George,
Thank you for your reply, I tried to delete the remaining marker, it failed. So I had to update.
Thank you!
Best Wishes!
Shylin Zhang
| |
shylinzhang
|
|
shylinzhang@163.com
|
签名由网易邮箱大师定制
On 02/16/2020 21:05，nichunen<ni...@apache.org> wrote：
Hi Shylin,
I think the description part in https://issues.apache.org/jira/browse/KYLIN-4153 is clear enough, you can find the root cause and workaround there, it happened due to the lack of atomicity during the rollback for the "big resource".
As a workaround, you may delete the remaining marker in your resource store, but it’s more advised to do an upgrade for your Kylin.
Best regards,
Ni Chunen / George
On 02/16/2020 16:22，shylinzhang<sh...@163.com> wrote：
Dear all,
The cube is ok when 2:00 am,but after 4:30 am it always shows error ,it shows ``` #4 Step Name: Build Dimension Dictionary
org.apache.kylin.engine.mr.exception.HadoopShellException: java.lang.RuntimeException: java.io.IOException: Failed to read big resource /dict/KYLIN.HIVE_METRICS_QUERY_CUBE_QA/CUBE_NAME/b9bb85ec-b72e-4df1-9ee7-e8c971ec5c72.dict
at org.apache.kylin.dict.DictionaryManager.getDictionaryInfo(DictionaryManager.java:108)
at org.apache.kylin.dict.DictionaryManager.checkDupByContent(DictionaryManager.java:173)
at org.apache.kylin.dict.DictionaryManager.trySaveNewDict(DictionaryManager.java:151)
at org.apache.kylin.dict.DictionaryManager.saveDictionary(DictionaryManager.java:320)
at org.apache.kylin.cube.CubeManager$DictionaryAssist.saveDictionary(CubeManager.java:1117)
at org.apache.kylin.cube.CubeManager.saveDictionary(CubeManager.java:1079)
at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:74)
at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:55)
at org.apache.kylin.engine.mr.steps.CreateDictionaryJob.run(CreateDictionaryJob.java:73)
at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:93)
at org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:63)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166)
at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166)
at org.apache.kylin.job.impl.threadpool.DistributedScheduler$JobRunner.run(DistributedScheduler.java:110)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
```
What I do:
1. I disable the cube and purge the cube and rebulid, it also show the same error;
2. Because the cube at 2:00 am is ok, so I use the meta_backs in 2:00 am and restore and cube; I try short period to build the cube, 2:00 am~3:00am~4:00am~4:00am~6:00am are ok, but in 6:am~7:00 am it also shows above error;
I found the solution in kylin jira, it tells me to update to release 2.6.4; our kylin clusters are used in 2019-06; just today shows the error.
Here’s the jira url: https://issues.apache.org/jira/browse/KYLIN-4153?jql=summary%20~%20%22Failed%20to%20read%20big%20resource*%22%20OR%20description%20~%20%22Failed%20to%20read%20big%20resource*%22%20ORDER%20BY%20lastViewed%20DESC
Therefore I want to know why it occured and if there is a way to fix it except updating.
I am looking forward for your replay, thank you.
Kylin version: 2.6.1
Hadoop version: 3.0.0+cdh6.0.0
Best Wishes!
Shylin Zhang
| |
shylinzhang
|
|
shylinzhang@163.com
|
签名由网易邮箱大师定制
Re:Failed to read big resource
/dict/KYLIN.HIVE_METRICS_QUERY_CUBE_QA/CUBE_NAME/xxx.dict
Posted by nichunen <ni...@apache.org>.
Hi Shylin,
I think the description part in https://issues.apache.org/jira/browse/KYLIN-4153 is clear enough, you can find the root cause and workaround there, it happened due to the lack of atomicity during the rollback for the "big resource".
As a workaround, you may delete the remaining marker in your resource store, but it’s more advised to do an upgrade for your Kylin.
Best regards,
Ni Chunen / George
On 02/16/2020 16:22，shylinzhang<sh...@163.com> wrote：
Dear all,
The cube is ok when 2:00 am,but after 4:30 am it always shows error ,it shows ``` #4 Step Name: Build Dimension Dictionary
org.apache.kylin.engine.mr.exception.HadoopShellException: java.lang.RuntimeException: java.io.IOException: Failed to read big resource /dict/KYLIN.HIVE_METRICS_QUERY_CUBE_QA/CUBE_NAME/b9bb85ec-b72e-4df1-9ee7-e8c971ec5c72.dict
at org.apache.kylin.dict.DictionaryManager.getDictionaryInfo(DictionaryManager.java:108)
at org.apache.kylin.dict.DictionaryManager.checkDupByContent(DictionaryManager.java:173)
at org.apache.kylin.dict.DictionaryManager.trySaveNewDict(DictionaryManager.java:151)
at org.apache.kylin.dict.DictionaryManager.saveDictionary(DictionaryManager.java:320)
at org.apache.kylin.cube.CubeManager$DictionaryAssist.saveDictionary(CubeManager.java:1117)
at org.apache.kylin.cube.CubeManager.saveDictionary(CubeManager.java:1079)
at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:74)
at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:55)
at org.apache.kylin.engine.mr.steps.CreateDictionaryJob.run(CreateDictionaryJob.java:73)
at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:93)
at org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:63)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166)
at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166)
at org.apache.kylin.job.impl.threadpool.DistributedScheduler$JobRunner.run(DistributedScheduler.java:110)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
```
What I do:
1. I disable the cube and purge the cube and rebulid, it also show the same error;
2. Because the cube at 2:00 am is ok, so I use the meta_backs in 2:00 am and restore and cube; I try short period to build the cube, 2:00 am~3:00am~4:00am~4:00am~6:00am are ok, but in 6:am~7:00 am it also shows above error;
I found the solution in kylin jira, it tells me to update to release 2.6.4; our kylin clusters are used in 2019-06; just today shows the error.
Here’s the jira url: https://issues.apache.org/jira/browse/KYLIN-4153?jql=summary%20~%20%22Failed%20to%20read%20big%20resource*%22%20OR%20description%20~%20%22Failed%20to%20read%20big%20resource*%22%20ORDER%20BY%20lastViewed%20DESC
Therefore I want to know why it occured and if there is a way to fix it except updating.
I am looking forward for your replay, thank you.
Kylin version: 2.6.1
Hadoop version: 3.0.0+cdh6.0.0
Best Wishes!
Shylin Zhang
| |
shylinzhang
|
|
shylinzhang@163.com
|
签名由网易邮箱大师定制
Re:Failed to read big resource
/dict/KYLIN.HIVE_METRICS_QUERY_CUBE_QA/CUBE_NAME/xxx.dict
Posted by nichunen <ni...@apache.org>.
Hi Shylin,
I think the description part in https://issues.apache.org/jira/browse/KYLIN-4153 is clear enough, you can find the root cause and workaround there, it happened due to the lack of atomicity during the rollback for the "big resource".
As a workaround, you may delete the remaining marker in your resource store, but it’s more advised to do an upgrade for your Kylin.
Best regards,
Ni Chunen / George
On 02/16/2020 16:22，shylinzhang<sh...@163.com> wrote：
Dear all,
The cube is ok when 2:00 am,but after 4:30 am it always shows error ,it shows ``` #4 Step Name: Build Dimension Dictionary
org.apache.kylin.engine.mr.exception.HadoopShellException: java.lang.RuntimeException: java.io.IOException: Failed to read big resource /dict/KYLIN.HIVE_METRICS_QUERY_CUBE_QA/CUBE_NAME/b9bb85ec-b72e-4df1-9ee7-e8c971ec5c72.dict
at org.apache.kylin.dict.DictionaryManager.getDictionaryInfo(DictionaryManager.java:108)
at org.apache.kylin.dict.DictionaryManager.checkDupByContent(DictionaryManager.java:173)
at org.apache.kylin.dict.DictionaryManager.trySaveNewDict(DictionaryManager.java:151)
at org.apache.kylin.dict.DictionaryManager.saveDictionary(DictionaryManager.java:320)
at org.apache.kylin.cube.CubeManager$DictionaryAssist.saveDictionary(CubeManager.java:1117)
at org.apache.kylin.cube.CubeManager.saveDictionary(CubeManager.java:1079)
at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:74)
at org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:55)
at org.apache.kylin.engine.mr.steps.CreateDictionaryJob.run(CreateDictionaryJob.java:73)
at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:93)
at org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:63)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166)
at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166)
at org.apache.kylin.job.impl.threadpool.DistributedScheduler$JobRunner.run(DistributedScheduler.java:110)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
```
What I do:
1. I disable the cube and purge the cube and rebulid, it also show the same error;
2. Because the cube at 2:00 am is ok, so I use the meta_backs in 2:00 am and restore and cube; I try short period to build the cube, 2:00 am~3:00am~4:00am~4:00am~6:00am are ok, but in 6:am~7:00 am it also shows above error;
I found the solution in kylin jira, it tells me to update to release 2.6.4; our kylin clusters are used in 2019-06; just today shows the error.
Here’s the jira url: https://issues.apache.org/jira/browse/KYLIN-4153?jql=summary%20~%20%22Failed%20to%20read%20big%20resource*%22%20OR%20description%20~%20%22Failed%20to%20read%20big%20resource*%22%20ORDER%20BY%20lastViewed%20DESC
Therefore I want to know why it occured and if there is a way to fix it except updating.
I am looking forward for your replay, thank you.
Kylin version: 2.6.1
Hadoop version: 3.0.0+cdh6.0.0
Best Wishes!
Shylin Zhang
| |
shylinzhang
|
|
shylinzhang@163.com
|
签名由网易邮箱大师定制