You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kylin.apache.org by "Shaofeng SHI (JIRA)" <ji...@apache.org> on 2017/03/13 03:49:04 UTC
[jira] [Assigned] (KYLIN-2502) "Create flat table" and
"redistribute table" steps don't show YARN application link
[ https://issues.apache.org/jira/browse/KYLIN-2502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Shaofeng SHI reassigned KYLIN-2502:
-----------------------------------
Assignee: Shaofeng SHI (was: Dong Li)
> "Create flat table" and "redistribute table" steps don't show YARN application link
> -----------------------------------------------------------------------------------
>
> Key: KYLIN-2502
> URL: https://issues.apache.org/jira/browse/KYLIN-2502
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
> Affects Versions: v1.6.0
> Reporter: Shaofeng SHI
> Assignee: Shaofeng SHI
> Fix For: v2.0.0
>
>
> From some version, Kylin doesn't show the tracking url for the hive steps. Need take a look.
> A sample output is:
> {code}
> Create and distribute table, cmd:
> hive -e "SET dfs.replication=2;
> SET hive.exec.compress.output=true;
> SET hive.auto.convert.join=true;
> SET hive.auto.convert.join.noconditionaltask=true;
> SET hive.auto.convert.join.noconditionaltask.size=100000000;
> SET mapreduce.job.split.metainfo.maxsize=-1;
> SET hive.stats.autogather=true;
> SET hive.merge.mapfiles=false;
> SET hive.merge.mapredfiles=false;
> USE default;
> DROP TABLE IF EXISTS kylin_intermediate_kylin_sales_cube_555c4d32_40bb_457d_909a_1bb017bf2d9e;
> CREATE EXTERNAL TABLE IF NOT EXISTS kylin_intermediate_kylin_sales_cube_555c4d32_40bb_457d_909a_1bb017bf2d9e
> (
> KYLIN_SALES_TRANS_ID bigint
> ,KYLIN_SALES_PART_DT date
> ,KYLIN_SALES_LEAF_CATEG_ID bigint
> ,KYLIN_SALES_LSTG_SITE_ID int
> ,KYLIN_CATEGORY_GROUPINGS_META_CATEG_NAME string
> ,KYLIN_CATEGORY_GROUPINGS_CATEG_LVL2_NAME string
> ,KYLIN_CATEGORY_GROUPINGS_CATEG_LVL3_NAME string
> ,KYLIN_SALES_LSTG_FORMAT_NAME string
> ,KYLIN_SALES_SELLER_ID bigint
> ,KYLIN_SALES_BUYER_ID bigint
> ,BUYER_ACCOUNT_ACCOUNT_BUYER_LEVEL int
> ,SELLER_ACCOUNT_ACCOUNT_SELLER_LEVEL int
> ,BUYER_ACCOUNT_ACCOUNT_COUNTRY string
> ,SELLER_ACCOUNT_ACCOUNT_COUNTRY string
> ,BUYER_COUNTRY_NAME string
> ,SELLER_COUNTRY_NAME string
> ,KYLIN_SALES_OPS_USER_ID string
> ,KYLIN_SALES_OPS_REGION string
> ,KYLIN_SALES_PRICE decimal(19,4)
> )
> STORED AS SEQUENCEFILE
> LOCATION 'hdfs:///kylin/kylin_metadata/kylin-2d5c1178-c6f6-4b50-8937-8e5e3b39227e/kylin_intermediate_kylin_sales_cube_555c4d32_40bb_457d_909a_1bb017bf2d9e';
> SET dfs.replication=2;
> SET hive.exec.compress.output=true;
> SET hive.auto.convert.join=true;
> SET hive.auto.convert.join.noconditionaltask=true;
> SET hive.auto.convert.join.noconditionaltask.size=100000000;
> SET mapreduce.job.split.metainfo.maxsize=-1;
> SET hive.stats.autogather=true;
> SET hive.merge.mapfiles=false;
> SET hive.merge.mapredfiles=false;
> INSERT OVERWRITE TABLE kylin_intermediate_kylin_sales_cube_555c4d32_40bb_457d_909a_1bb017bf2d9e SELECT
> KYLIN_SALES.TRANS_ID
> ,KYLIN_SALES.PART_DT
> ,KYLIN_SALES.LEAF_CATEG_ID
> ,KYLIN_SALES.LSTG_SITE_ID
> ,KYLIN_CATEGORY_GROUPINGS.META_CATEG_NAME
> ,KYLIN_CATEGORY_GROUPINGS.CATEG_LVL2_NAME
> ,KYLIN_CATEGORY_GROUPINGS.CATEG_LVL3_NAME
> ,KYLIN_SALES.LSTG_FORMAT_NAME
> ,KYLIN_SALES.SELLER_ID
> ,KYLIN_SALES.BUYER_ID
> ,BUYER_ACCOUNT.ACCOUNT_BUYER_LEVEL
> ,SELLER_ACCOUNT.ACCOUNT_SELLER_LEVEL
> ,BUYER_ACCOUNT.ACCOUNT_COUNTRY
> ,SELLER_ACCOUNT.ACCOUNT_COUNTRY
> ,BUYER_COUNTRY.NAME
> ,SELLER_COUNTRY.NAME
> ,KYLIN_SALES.OPS_USER_ID
> ,KYLIN_SALES.OPS_REGION
> ,KYLIN_SALES.PRICE
> FROM DEFAULT.KYLIN_SALES as KYLIN_SALES
> INNER JOIN DEFAULT.KYLIN_CAL_DT as KYLIN_CAL_DT
> ON KYLIN_SALES.PART_DT = KYLIN_CAL_DT.CAL_DT
> INNER JOIN DEFAULT.KYLIN_CATEGORY_GROUPINGS as KYLIN_CATEGORY_GROUPINGS
> ON KYLIN_SALES.LEAF_CATEG_ID = KYLIN_CATEGORY_GROUPINGS.LEAF_CATEG_ID AND KYLIN_SALES.LSTG_SITE_ID = KYLIN_CATEGORY_GROUPINGS.SITE_ID
> INNER JOIN DEFAULT.KYLIN_ACCOUNT as BUYER_ACCOUNT
> ON KYLIN_SALES.BUYER_ID = BUYER_ACCOUNT.ACCOUNT_ID
> INNER JOIN DEFAULT.KYLIN_ACCOUNT as SELLER_ACCOUNT
> ON KYLIN_SALES.SELLER_ID = SELLER_ACCOUNT.ACCOUNT_ID
> INNER JOIN DEFAULT.KYLIN_COUNTRY as BUYER_COUNTRY
> ON BUYER_ACCOUNT.ACCOUNT_COUNTRY = BUYER_COUNTRY.COUNTRY
> INNER JOIN DEFAULT.KYLIN_COUNTRY as SELLER_COUNTRY
> ON SELLER_ACCOUNT.ACCOUNT_COUNTRY = SELLER_COUNTRY.COUNTRY
> WHERE (KYLIN_SALES.PART_DT >= '2012-01-01' AND KYLIN_SALES.PART_DT < '2017-03-01')
> ;
> "
> WARNING: Use "yarn jar" to launch YARN applications.
> Logging initialized using configuration in file:/etc/hive/2.4.0.0-169/0/hive-log4j.properties
> OK
> Time taken: 1.941 seconds
> OK
> Time taken: 5.394 seconds
> OK
> Time taken: 4.008 seconds
> Query ID = root_20170306144241_724ee8c1-fb9e-4cb5-b37e-04a87fc114e5
> Total jobs = 1
> Launching Job 1 out of 1
> Status: Running (Executing on YARN cluster with App id application_1488805575687_0013)
> Map 1: -/- Map 2: -/- Map 3: -/- Map 4: -/- Map 5: -/- Map 6: -/- Map 7: -/-
> Map 1: 0/1 Map 2: 0/1 Map 3: 0/1 Map 4: 0/1 Map 5: 0/1 Map 6: 0/1 Map 7: 0/1
> Map 1: 0/1 Map 2: 0/1 Map 3: 0/1 Map 4: 0/1 Map 5: 0/1 Map 6: 0/1 Map 7: 0/1
> Map 1: 0/1 Map 2: 0/1 Map 3: 0(+1)/1 Map 4: 0/1 Map 5: 0(+1)/1 Map 6: 0/1 Map 7: 0/1
> Map 1: 0/1 Map 2: 0/1 Map 3: 0(+1)/1 Map 4: 0(+1)/1 Map 5: 0(+1)/1 Map 6: 0(+1)/1 Map 7: 0/1
> Map 1: 0/1 Map 2: 0/1 Map 3: 0(+1)/1 Map 4: 0(+1)/1 Map 5: 0(+1)/1 Map 6: 0(+1)/1 Map 7: 0/1
> Map 1: 0/1 Map 2: 0/1 Map 3: 1/1 Map 4: 0(+1)/1 Map 5: 0(+1)/1 Map 6: 0(+1)/1 Map 7: 0/1
> Map 1: 0(+1)/1 Map 2: 0/1 Map 3: 1/1 Map 4: 0(+1)/1 Map 5: 0(+1)/1 Map 6: 1/1 Map 7: 0(+1)/1
> Map 1: 0(+1)/1 Map 2: 0(+1)/1 Map 3: 1/1 Map 4: 0(+1)/1 Map 5: 0(+1)/1 Map 6: 1/1 Map 7: 1/1
> Map 1: 0(+1)/1 Map 2: 0(+1)/1 Map 3: 1/1 Map 4: 1/1 Map 5: 1/1 Map 6: 1/1 Map 7: 1/1
> Map 1: 1/1 Map 2: 0(+1)/1 Map 3: 1/1 Map 4: 1/1 Map 5: 1/1 Map 6: 1/1 Map 7: 1/1
> Map 1: 1/1 Map 2: 1/1 Map 3: 1/1 Map 4: 1/1 Map 5: 1/1 Map 6: 1/1 Map 7: 1/1
> Loading data to table default.kylin_intermediate_kylin_sales_cube_555c4d32_40bb_457d_909a_1bb017bf2d9e
> Table default.kylin_intermediate_kylin_sales_cube_555c4d32_40bb_457d_909a_1bb017bf2d9e stats: [numFiles=1, numRows=10000, totalSize=302827, rawDataSize=1519831]
> OK
> Time taken: 26.551 seconds
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)