Re: kylin intermediate tables in Hive

[hongbin ma <ma...@apache.org>]

it might have something to do with the bug described in
https://issues.apache.org/jira/browse/KYLIN-1828, I'll cross check

On Tue, Jun 21, 2016 at 10:36 PM, ShaoFeng Shi <sh...@apache.org>
wrote:

> Hi Jie, would you mind to report a JIRA with this problem you found? If you
> can attache the json of this job and a couple of screen shot, that would be
> great for analysis. Thank you!
>
> 2016-06-21 16:49 GMT+08:00 Jie Tao <ji...@gameforge.com>:
>
> > actually the jobs have been disgarded. Maybe it is a bug that the status
> > of the job is still "ERROR". I have a look of the jobs  at the Kylin Web
> UI
> > and found that the job was not marked with black colors like other
> > discarded jobs althouth the "action" button only has one choice
> > "diagonose". My Kylin is 1.5.2.1.
> >
> > Cheers,
> >
> > Jie
> >
> >
> > Am 17.06.2016 um 11:05 schrieb ShaoFeng Shi:
> >
> >> by default the web UI only shows the jobs in LAST ONE WEEK, pls have a
> >> check.
> >>
> >> 2016-06-17 16:58 GMT+08:00 Jie Tao <ji...@gameforge.com>:
> >>
> >> actually I discarded all jobs and I do not see any ERROR job in the
> >>> Monitor view of Kylin UI.
> >>>
> >>> Where can I see these error jobs?
> >>>
> >>> Jie
> >>>
> >>>
> >>> Am 17.06.2016 um 10:31 schrieb ShaoFeng Shi:
> >>>
> >>> Hi Jie,
> >>>>
> >>>> If a job is "ERROR", the intermediate hive table of it will not be
> >>>> dropped,
> >>>> as "ERROR" is not a final state; User can resume an "Error" job at any
> >>>> time, so Kylin skipped to cleanup for that.
> >>>>
> >>>> If you discard these error jobs, and re-run the cleanup, the
> >>>> intermediate
> >>>> hive table will be dropped.
> >>>>
> >>>> The message here is not clear, will change the wording...
> >>>>
> >>>> 2016-06-17 15:48 GMT+08:00 Jie Tao <ji...@gameforge.com>:
> >>>>
> >>>> You are correct, the intermediate tables are left by fail-building. I
> do
> >>>>
> >>>>> clean up storage based on the linked guide. Intermediate data in HDFS
> >>>>> and
> >>>>> Hbase are deleted, but the intermediate tables in Hive not. The
> command
> >>>>> shows the tables but do not drop them. I donot have a lookup table
> but
> >>>>> my
> >>>>> fact table is a view.
> >>>>>
> >>>>> As I run the cleanup command,
> >>>>> kylin_intermediate_logout_full_cube_19700101000000_20151001000000
> >>>>> kylin_intermediate_logout_full_cube_19700101000000_20160529010500
> >>>>> kylin_intermediate_logout_full_cube_19700101000000_20160608000000
> >>>>> kylin_intermediate_logout_full_cube_19700101000000_20160608010500
> >>>>> kylin_intermediate_logout_full_cube_19700101000000_20160609010500
> >>>>> kylin_intermediate_logout_full_cube_19700101000000_20160615000000
> >>>>> kylin_intermediate_logout_full_cube_19700101000000_20160626000000
> >>>>> kylin_intermediate_logout_full_cube_19700101000000_20160626042000
> >>>>> kylin_intermediate_test_cube_19700101000000_20151201010500
> >>>>> kylin_intermediate_test_cube_19700101000000_20151231234000
> >>>>> kylin_intermediate_test_cube_19700101000000_20160302063000
> >>>>> kylin_intermediate_test_cube_19700101000000_20160626000000
> >>>>> kylin_intermediate_test_cube_19700101000000_20160626042000
> >>>>> kylin_intermediate_test_cube_19700101000000_20160704082000
> >>>>> Time taken: 0.189 seconds, Fetched: 14 row(s)
> >>>>> 2016-06-17 09:37:12,645 INFO  [main StorageCleanupJob:262]: Remove
> >>>>> intermediate hive table with job id
> >>>>> 493fd20b-3074-403e-9963-fe4fb7ff7c65
> >>>>> with job status ERROR
> >>>>> 2016-06-17 09:37:12,648 INFO  [main StorageCleanupJob:262]: Remove
> >>>>> intermediate hive table with job id
> >>>>> 8a377e30-e3ba-4fe2-be12-e7d412afec5e
> >>>>> with job status ERROR
> >>>>>
> >>>>> Best regards,
> >>>>>
> >>>>> Jie
> >>>>>
> >>>>>
> >>>>> Am 17.06.2016 um 09:16 schrieb ShaoFeng Shi:
> >>>>>
> >>>>> BTW, are you using a view as lookup table?
> >>>>>
> >>>>>> 2016-06-17 15:15 GMT+08:00 ShaoFeng Shi <sh...@apache.org>:
> >>>>>>
> >>>>>> This is common; If you have a job failed in between, and you discard
> >>>>>> that
> >>>>>>
> >>>>>> job, the "Garbage collection" step will not be executed, so the
> >>>>>>> garbages
> >>>>>>> will be left there.
> >>>>>>>
> >>>>>>> This is why we still recommend user to run offline cleanup every
> some
> >>>>>>> period; It is not perfert, but be good for most scenarios:
> >>>>>>> https://kylin.apache.org/docs15/howto/howto_cleanup_storage.html
> >>>>>>>
> >>>>>>> 2016-06-17 15:00 GMT+08:00 Li Yang <li...@apache.org>:
> >>>>>>>
> >>>>>>> Woo... something new to me. Anybody knows?
> >>>>>>>
> >>>>>>> On Tue, Jun 14, 2016 at 6:57 PM, Jie Tao <ji...@gameforge.com>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>> Kylin actually drops useless intermediate tables after cube
> >>>>>>>> building,
> >>>>>>>> but
> >>>>>>>>
> >>>>>>>> I still see one "kylin_intermediate_cubename_searchdata" table for
> >>>>>>>>
> >>>>>>>>> each
> >>>>>>>>> cube building in Hive. Are these tables still usefull for Kylin?
> I
> >>>>>>>>> use
> >>>>>>>>> Kylin 1.5.2.1.
> >>>>>>>>>
> >>>>>>>>> Cheers,
> >>>>>>>>>
> >>>>>>>>> Jie
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>>
> >>>>>>>> Best regards,
> >>>>>>>
> >>>>>>> Shaofeng Shi
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>
> >
>
>
> --
> Best regards,
>
> Shaofeng Shi
>



-- 
Regards,

*Bin Mahone | 马洪宾*