You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kylin.apache.org by Li Yang <li...@apache.org> on 2015/11/05 03:30:40 UTC
Re: NPE when building cube by Kylin 1.2(Hbase 1.1)
fixed on 1.x-HBase1.1.3
On Fri, Oct 30, 2015 at 9:55 PM, Luke Han <lu...@gmail.com> wrote:
> Do you have quick fix on this? Would love to have community help on HBase
> 1.x testing and bug fixes.
>
> Thanks.
>
>
> Best Regards!
> ---------------------
>
> Luke Han
>
> On Fri, Oct 30, 2015 at 8:59 AM, 王晓雨 <wa...@jd.com> wrote:
>
> > I got the same exception in 1.x-staging branch.
> >
> > > 在 2015年10月29日,22:48,Yiming Liu <li...@foxmail.com> 写道:
> > >
> > > Hello, Kylin experts:
> > >
> > >
> > > I have issues when using the sample.sh, and has reported to
> > https://issues.apache.org/jira/browse/KYLIN-1103
> > >
> > > Now I tried to build the sample manually, but met the NPE in step 2.
> > From the log, it seems Kylin MR job could not load the cube definition
> > correctly. Could you give me some help?
> > >
> > >
> > > The kylin jobs details has parameter:
> > > -conf /opt/kylin/conf/kylin_job_conf.xml -cubename
> > kylin_sales_cube_simple -output
> >
> /kylin/kylin_metadata/kylin-d2f99e2d-146c-4fb7-bcbf-5c317af143dd/kylin_sales_cube_simple/fact_distinct_columns
> > -jobname Kylin_Fact_Distinct_Columns_kylin_sales_cube_simple_Step
> > -tablename
> >
> default.kylin_intermediate_kylin_sales_cube_simple_19700101000000_20150927000000_d2f99e2d_146c_4fb7_bcbf_5c317af143dd
> > >
> > >
> > > the log:
> > > 2015-10-29 22:30:17.685 - State of Hadoop job:
> > job_1445989444324_0021:ACCEPTED - UNDEFINED 2015-10-29 22:30:27.704 -
> State
> > of Hadoop job: job_1445989444324_0021:ACCEPTED - UNDEFINED 2015-10-29
> > 22:30:37.721 - State of Hadoop job: job_1445989444324_0021:RUNNING -
> > UNDEFINED 2015-10-29 22:30:47.751 - State of Hadoop job:
> > job_1445989444324_0021:RUNNING - UNDEFINED 2015-10-29 22:30:57.768 -
> State
> > of Hadoop job: job_1445989444324_0021:RUNNING - UNDEFINED 2015-10-29
> > 22:31:07.786 - State of Hadoop job: job_1445989444324_0021:RUNNING -
> > UNDEFINED 2015-10-29 22:31:17.805 - State of Hadoop job:
> > job_1445989444324_0021:FINISHED - FAILED no counters for job
> > job_1445989444324_0021
> > > And the MR jobs has the exception log:
> > > Error: java.lang.NullPointerException at
> >
> org.apache.kylin.job.hadoop.cube.FactDistinctColumnsMapper.setup(FactDistinctColumnsMapper.java:73)
> > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143) at
> > org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787) at
> > org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at
> > org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) at
> > java.security.AccessController.doPrivileged(Native Method) at
> > javax.security.auth.Subject.doAs(Subject.java:422) at
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> > >
> > >
> > > Dip into the code, it seems the NPE means the "cube" is null. From the
> > code, we could tell the CubeManager could not find my cube.
> > > 72 cube = CubeManager.getInstance(config).getCube(cubeName);
> > > 73 cubeDesc = cube.getDescriptor();
> > >
> > >
> > >
> > > Then I looked into HBase, the kylin_metadata has the following two
> > records:
> > > /cube/kylin_sales_cube_simple.json
> > > {
> > > "uuid" : "3a271b47-9363-4434-8bd0-02c2a2a31c2b",
> > > "name" : "kylin_sales_cube_simple",
> > > "owner" : "ADMIN",
> > > "version" : null,
> > > "cost" : 50,
> > > "status" : "DISABLED",
> > > "segments" : [ {
> > > "uuid" : "05a31b8b-ac94-45df-ab41-1f425b64e27f",
> > > "name" : "19700101000000_20150927000000",
> > > "status" : "NEW",
> > > "dictionaries" : null,
> > > "snapshots" : null,
> > > "storage_location_identifier" : "KYLIN_AIKMGGR2BN",
> > > "date_range_start" : 0,
> > > "date_range_end" : 1443312000000,
> > > "size_kb" : 0,
> > > "input_records" : 0,
> > > "input_records_size" : 0,
> > > "last_build_time" : 0,
> > > "last_build_job_id" : null,
> > > "create_time_utc" : 1446128917265,
> > > "binary_signature" : null
> > > } ],
> > > "last_modified" : 1446128917267,
> > > "descriptor" : "kylin_sales_cube_simple",
> > > "create_time_utc" : 1446128903052,
> > > "size_kb" : 0,
> > > "input_records_count" : 0,
> > > "input_records_size" : 0
> > > }
> > >
> > > and /cube_desc/kylin_sales_cube_simple.json
> > > {
> > > "uuid" : "8d759e58-0a97-4913-a7b5-fa3f99fdf67d",
> > > "name" : "kylin_sales_cube_simple",
> > > "description" : "",
> > > "dimensions" : [ {
> > > "id" : 1,
> > > "name" : "DEFAULT.KYLIN_SALES.LSTG_FORMAT_NAME",
> > > "table" : "DEFAULT.KYLIN_SALES",
> > > "column" : [ "LSTG_FORMAT_NAME" ],
> > > "derived" : null,
> > > "hierarchy" : false
> > > }, {
> > > "id" : 2,
> > > "name" : "DEFAULT.KYLIN_SALES.LEAF_CATEG_ID",
> > > "table" : "DEFAULT.KYLIN_SALES",
> > > "column" : [ "LEAF_CATEG_ID" ],
> > > "derived" : null,
> > > "hierarchy" : false
> > > }, {
> > > "id" : 3,
> > > "name" : "DEFAULT.KYLIN_SALES.LSTG_SITE_ID",
> > > "table" : "DEFAULT.KYLIN_SALES",
> > > "column" : [ "LSTG_SITE_ID" ],
> > > "derived" : null,
> > > "hierarchy" : false
> > > } ],
> > > "measures" : [ {
> > > "id" : 1,
> > > "name" : "_COUNT_",
> > > "function" : {
> > > "expression" : "COUNT",
> > > "parameter" : {
> > > "type" : "constant",
> > > "value" : "1"
> > > },
> > > "returntype" : "bigint"
> > > },
> > > "dependent_measure_ref" : null
> > > }, {
> > > "id" : 2,
> > > "name" : "SUM_PRICE",
> > > "function" : {
> > > "expression" : "SUM",
> > > "parameter" : {
> > > "type" : "column",
> > > "value" : "PRICE"
> > > },
> > > "returntype" : "decimal(19,4)"
> > > },
> > > "dependent_measure_ref" : null
> > > } ],
> > > "rowkey" : {
> > > "rowkey_columns" : [ {
> > > "column" : "LSTG_FORMAT_NAME",
> > > "length" : 0,
> > > "dictionary" : "true",
> > > "mandatory" : false
> > > }, {
> > > "column" : "LEAF_CATEG_ID",
> > > "length" : 0,
> > > "dictionary" : "true",
> > > "mandatory" : false
> > > }, {
> > > "column" : "LSTG_SITE_ID",
> > > "length" : 0,
> > > "dictionary" : "true",
> > > "mandatory" : false
> > > } ],
> > > "aggregation_groups" : [ [ "LSTG_FORMAT_NAME", "LEAF_CATEG_ID",
> > "LSTG_SITE_ID" ] ]
> > > },
> > > "signature" : "0FAthGOvTI9p4/K0cTMJ6Q==",
> > > "last_modified" : 1446128903023,
> > > "model_name" : "kylin_sales_cube_simple",
> > > "null_string" : null,
> > > "hbase_mapping" : {
> > > "column_family" : [ {
> > > "name" : "F1",
> > > "columns" : [ {
> > > "qualifier" : "M",
> > > "measure_refs" : [ "_COUNT_", "SUM_PRICE" ]
> > > } ]
> > > } ]
> > > },
> > > "notify_list" : [ ],
> > > "auto_merge_time_ranges" : [ 604800000, 2419200000 ],
> > > "retention_range" : 0
> > > }
> >
> >
>