You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@kylin.apache.org by Sonny Heer <so...@gmail.com> on 2018/04/12 04:35:32 UTC
job failures
we have a daily job that builds cubes. It works fine for some number of
days, but at times fails with this:
Error Log: org.apache.kylin.job.exception.ExecuteException:
java.lang.IllegalStateException: Overwriting conflict
/execute_output/a1052507-e3bc-4302-ac73-bbc169a597ff-07, expect old TS
1523504910859, but it is 1523504920864 at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:134)
at
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:57)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:113)
at
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:136)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745) Caused by:
java.lang.IllegalStateException: Overwriting conflict
/execute_output/a1052507-e3bc-4302-ac73-bbc169a597ff-07, expect old TS
1523504910859, but it is 1523504920864 at
org.apache.kylin.storage.hbase.HBaseResourceStore.checkAndPutResourceImpl(HBaseResourceStore.java:265)
at
org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:237)
at
org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:217)
at
org.apache.kylin.job.dao.ExecutableDao.writeJobOutputResource(ExecutableDao.java:97)
at
org.apache.kylin.job.dao.ExecutableDao.updateJobOutput(ExecutableDao.java:212)
at
org.apache.kylin.job.manager.ExecutableManager.addJobInfo(ExecutableManager.java:322)
at
org.apache.kylin.job.manager.ExecutableManager.addJobInfo(ExecutableManager.java:332)
at
org.apache.kylin.job.execution.AbstractExecutable.addExtraInfo(AbstractExecutable.java:337)
at
org.apache.kylin.job.execution.AbstractExecutable.setEndTime(AbstractExecutable.java:349)
at
org.apache.kylin.job.execution.AbstractExecutable.onExecuteFinished(AbstractExecutable.java:73)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:126)
... 6 more
Re: job failures
Posted by ShaoFeng Shi <sh...@apache.org>.
From the error, it seems there was some other 'guy' modified the data in
the between, then Kylin's data is dirty (its old timestamp not equal to the
value in meta store) .
If you sure there is only one "all" node, check whether there is another
active process, for example, run "ps -ef |grep kylin". Just take more
investigation, such error not always occur.
2018-04-13 1:29 GMT+08:00 Sonny Heer <so...@gmail.com>:
> 3 nodes are set to "query" and 1 is set to "all". kylin 1.6
>
> kylin.server.mode=query
>
> kylin.server.mode=all
>
>
> On Wed, Apr 11, 2018 at 10:13 PM, ShaoFeng Shi <sh...@apache.org>
> wrote:
>
>> Do you have multiple Kylin "job" instances?
>>
>> 2018-04-12 12:35 GMT+08:00 Sonny Heer <so...@gmail.com>:
>>
>>> we have a daily job that builds cubes. It works fine for some number of
>>> days, but at times fails with this:
>>>
>>>
>>> Error Log: org.apache.kylin.job.exception.ExecuteException:
>>> java.lang.IllegalStateException: Overwriting conflict
>>> /execute_output/a1052507-e3bc-4302-ac73-bbc169a597ff-07, expect old TS
>>> 1523504910859, but it is 1523504920864 at org.apache.kylin.job.execution
>>> .AbstractExecutable.execute(AbstractExecutable.java:134) at
>>> org.apache.kylin.job.execution.DefaultChainedExecutable.doWo
>>> rk(DefaultChainedExecutable.java:57) at org.apache.kylin.job.execution
>>> .AbstractExecutable.execute(AbstractExecutable.java:113) at
>>> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRun
>>> ner.run(DefaultScheduler.java:136) at java.util.concurrent.ThreadPoo
>>> lExecutor.runWorker(ThreadPoolExecutor.java:1142) at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> at java.lang.Thread.run(Thread.java:745) Caused by:
>>> java.lang.IllegalStateException: Overwriting conflict
>>> /execute_output/a1052507-e3bc-4302-ac73-bbc169a597ff-07, expect old TS
>>> 1523504910859, but it is 1523504920864 at org.apache.kylin.storage.hbase
>>> .HBaseResourceStore.checkAndPutResourceImpl(HBaseResourceStore.java:265)
>>> at org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:237)
>>> at org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:217)
>>> at org.apache.kylin.job.dao.ExecutableDao.writeJobOutputResource(ExecutableDao.java:97)
>>> at org.apache.kylin.job.dao.ExecutableDao.updateJobOutput(ExecutableDao.java:212)
>>> at org.apache.kylin.job.manager.ExecutableManager.addJobInfo(ExecutableManager.java:322)
>>> at org.apache.kylin.job.manager.ExecutableManager.addJobInfo(ExecutableManager.java:332)
>>> at org.apache.kylin.job.execution.AbstractExecutable.addExtraIn
>>> fo(AbstractExecutable.java:337) at org.apache.kylin.job.execution
>>> .AbstractExecutable.setEndTime(AbstractExecutable.java:349) at
>>> org.apache.kylin.job.execution.AbstractExecutable.onExecuteF
>>> inished(AbstractExecutable.java:73) at org.apache.kylin.job.execution
>>> .AbstractExecutable.execute(AbstractExecutable.java:126) ... 6 more
>>>
>>>
>>>
>>
>>
>> --
>> Best regards,
>>
>> Shaofeng Shi 史少锋
>>
>>
>
--
Best regards,
Shaofeng Shi 史少锋
Re: job failures
Posted by Sonny Heer <so...@gmail.com>.
3 nodes are set to "query" and 1 is set to "all". kylin 1.6
kylin.server.mode=query
kylin.server.mode=all
On Wed, Apr 11, 2018 at 10:13 PM, ShaoFeng Shi <sh...@apache.org>
wrote:
> Do you have multiple Kylin "job" instances?
>
> 2018-04-12 12:35 GMT+08:00 Sonny Heer <so...@gmail.com>:
>
>> we have a daily job that builds cubes. It works fine for some number of
>> days, but at times fails with this:
>>
>>
>> Error Log: org.apache.kylin.job.exception.ExecuteException:
>> java.lang.IllegalStateException: Overwriting conflict
>> /execute_output/a1052507-e3bc-4302-ac73-bbc169a597ff-07, expect old TS
>> 1523504910859, but it is 1523504920864 at org.apache.kylin.job.execution
>> .AbstractExecutable.execute(AbstractExecutable.java:134) at
>> org.apache.kylin.job.execution.DefaultChainedExecutable.doWo
>> rk(DefaultChainedExecutable.java:57) at org.apache.kylin.job.execution
>> .AbstractExecutable.execute(AbstractExecutable.java:113) at
>> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRun
>> ner.run(DefaultScheduler.java:136) at java.util.concurrent.ThreadPoo
>> lExecutor.runWorker(ThreadPoolExecutor.java:1142) at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745) Caused by:
>> java.lang.IllegalStateException: Overwriting conflict
>> /execute_output/a1052507-e3bc-4302-ac73-bbc169a597ff-07, expect old TS
>> 1523504910859, but it is 1523504920864 at org.apache.kylin.storage.hbase
>> .HBaseResourceStore.checkAndPutResourceImpl(HBaseResourceStore.java:265)
>> at org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:237)
>> at org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:217)
>> at org.apache.kylin.job.dao.ExecutableDao.writeJobOutputResource(ExecutableDao.java:97)
>> at org.apache.kylin.job.dao.ExecutableDao.updateJobOutput(ExecutableDao.java:212)
>> at org.apache.kylin.job.manager.ExecutableManager.addJobInfo(ExecutableManager.java:322)
>> at org.apache.kylin.job.manager.ExecutableManager.addJobInfo(ExecutableManager.java:332)
>> at org.apache.kylin.job.execution.AbstractExecutable.addExtraIn
>> fo(AbstractExecutable.java:337) at org.apache.kylin.job.execution
>> .AbstractExecutable.setEndTime(AbstractExecutable.java:349) at
>> org.apache.kylin.job.execution.AbstractExecutable.onExecuteF
>> inished(AbstractExecutable.java:73) at org.apache.kylin.job.execution
>> .AbstractExecutable.execute(AbstractExecutable.java:126) ... 6 more
>>
>>
>>
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>
>
Re: job failures
Posted by ShaoFeng Shi <sh...@apache.org>.
Do you have multiple Kylin "job" instances?
2018-04-12 12:35 GMT+08:00 Sonny Heer <so...@gmail.com>:
> we have a daily job that builds cubes. It works fine for some number of
> days, but at times fails with this:
>
>
> Error Log: org.apache.kylin.job.exception.ExecuteException: java.lang.IllegalStateException:
> Overwriting conflict /execute_output/a1052507-e3bc-4302-ac73-bbc169a597ff-07,
> expect old TS 1523504910859, but it is 1523504920864 at
> org.apache.kylin.job.execution.AbstractExecutable.
> execute(AbstractExecutable.java:134) at org.apache.kylin.job.execution.
> DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:57) at
> org.apache.kylin.job.execution.AbstractExecutable.
> execute(AbstractExecutable.java:113) at org.apache.kylin.job.impl.
> threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:136) at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.IllegalStateException:
> Overwriting conflict /execute_output/a1052507-e3bc-4302-ac73-bbc169a597ff-07,
> expect old TS 1523504910859, but it is 1523504920864 at
> org.apache.kylin.storage.hbase.HBaseResourceStore.checkAndPutResourceImpl(HBaseResourceStore.java:265)
> at org.apache.kylin.common.persistence.ResourceStore.
> putResource(ResourceStore.java:237) at org.apache.kylin.common.
> persistence.ResourceStore.putResource(ResourceStore.java:217) at
> org.apache.kylin.job.dao.ExecutableDao.writeJobOutputResource(ExecutableDao.java:97)
> at org.apache.kylin.job.dao.ExecutableDao.updateJobOutput(ExecutableDao.java:212)
> at org.apache.kylin.job.manager.ExecutableManager.addJobInfo(ExecutableManager.java:322)
> at org.apache.kylin.job.manager.ExecutableManager.addJobInfo(ExecutableManager.java:332)
> at org.apache.kylin.job.execution.AbstractExecutable.addExtraInfo(AbstractExecutable.java:337)
> at org.apache.kylin.job.execution.AbstractExecutable.
> setEndTime(AbstractExecutable.java:349) at org.apache.kylin.job.
> execution.AbstractExecutable.onExecuteFinished(AbstractExecutable.java:73)
> at org.apache.kylin.job.execution.AbstractExecutable.
> execute(AbstractExecutable.java:126) ... 6 more
>
>
>
--
Best regards,
Shaofeng Shi 史少锋