You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kylin.apache.org by "XiaoXiang Yu (JIRA)" <ji...@apache.org> on 2019/03/11 05:00:00 UTC
[jira] [Comment Edited] (KYLIN-3857) add parameter to change sql
quoting character for sqoop
[ https://issues.apache.org/jira/browse/KYLIN-3857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16789136#comment-16789136 ]
XiaoXiang Yu edited comment on KYLIN-3857 at 3/11/19 4:59 AM:
--------------------------------------------------------------
Logs in My DEV env (with quote turn off):
KYLIN [ INFO ] 03-11 12:53:39.114 org.apache.kylin.common.util.BufferedLogger.log(BufferedLogger.java:38) from Scheduler 1705744661 Job 6d64fe74-ea50-76a6-bbd1-f85210984416-129
> Create and distribute table, cmd:
KYLIN [ INFO ] 03-11 12:53:39.115 org.apache.kylin.common.util.BufferedLogger.log(BufferedLogger.java:38) from Scheduler 1705744661 Job 6d64fe74-ea50-76a6-bbd1-f85210984416-129
> hive -e "USE lacus;
DROP TABLE IF EXISTS kylin_intermediate_samplecube_e1c8bdcc_d55b_3807_25e2_b8d2b6fccabe;
CREATE EXTERNAL TABLE IF NOT EXISTS kylin_intermediate_samplecube_e1c8bdcc_d55b_3807_25e2_b8d2b6fccabe
(
KYLIN_SALES_LSTG_FORMAT_NAME string
,KYLIN_SALES_LSTG_SITE_ID int
,KYLIN_SALES_SELLER_ID bigint
,KYLIN_ACCOUNT_ACCOUNT_COUNTRY string
,KYLIN_SALES_PRICE decimal(19,4)
,KYLIN_SALES_ITEM_COUNT bigint
)
STORED AS SEQUENCEFILE
LOCATION 'hdfs://sandbox.hortonworks.com:8020/kylin/lacus/lacus-kylin_metadata/kylin-6d64fe74-ea50-76a6-bbd1-f85210984416/kylin_intermediate_samplecube_e1c8bdcc_d55b_3807_25e2_b8d2b6fccabe';
ALTER TABLE kylin_intermediate_samplecube_e1c8bdcc_d55b_3807_25e2_b8d2b6fccabe SET TBLPROPERTIES('auto.purge'='true');
INSERT OVERWRITE TABLE kylin_intermediate_samplecube_e1c8bdcc_d55b_3807_25e2_b8d2b6fccabe SELECT
KYLIN_SALES.LSTG_FORMAT_NAME as KYLIN_SALES_LSTG_FORMAT_NAME
,KYLIN_SALES.LSTG_SITE_ID as KYLIN_SALES_LSTG_SITE_ID
,KYLIN_SALES.SELLER_ID as KYLIN_SALES_SELLER_ID
,KYLIN_ACCOUNT.ACCOUNT_COUNTRY as KYLIN_ACCOUNT_ACCOUNT_COUNTRY
,KYLIN_SALES.PRICE as KYLIN_SALES_PRICE
,KYLIN_SALES.ITEM_COUNT as KYLIN_SALES_ITEM_COUNT
FROM LACUS.KYLIN_SALES as KYLIN_SALES
LEFT JOIN LACUS.KYLIN_ACCOUNT as KYLIN_ACCOUNT
ON KYLIN_SALES.SELLER_ID = KYLIN_ACCOUNT.ACCOUNT_ID
LEFT JOIN LACUS.KYLIN_COUNTRY as KYLIN_COUNTRY
ON KYLIN_ACCOUNT.ACCOUNT_COUNTRY = KYLIN_COUNTRY.COUNTRY
WHERE 1=1;
" --hiveconf mapreduce.map.output.compress.codec=org.apache.hadoop.io.compress.SnappyCodec --hiveconf dfs.replication=2 --hiveconf hive.exec.compress.output=true
KYLIN [ INFO ] 03-11 12:53:39.115 org.apache.kylin.common.util.SSHClient.execCommand(SSHClient.java:262) from Scheduler 1705744661 Job 6d64fe74-ea50-76a6-bbd1-f85210984416-129
> [root@sandbox] Execute command: hive -e "USE lacus;
DROP TABLE IF EXISTS kylin_intermediate_samplecube_e1c8bdcc_d55b_3807_25e2_b8d2b6fccabe;
CREATE EXTERNAL TABLE IF NOT EXISTS kylin_intermediate_samplecube_e1c8bdcc_d55b_3807_25e2_b8d2b6fccabe
(
KYLIN_SALES_LSTG_FORMAT_NAME string
,KYLIN_SALES_LSTG_SITE_ID int
,KYLIN_SALES_SELLER_ID bigint
,KYLIN_ACCOUNT_ACCOUNT_COUNTRY string
,KYLIN_SALES_PRICE decimal(19,4)
,KYLIN_SALES_ITEM_COUNT bigint
)
STORED AS SEQUENCEFILE
LOCATION 'hdfs://sandbox.hortonworks.com:8020/kylin/lacus/lacus-kylin_metadata/kylin-6d64fe74-ea50-76a6-bbd1-f85210984416/kylin_intermediate_samplecube_e1c8bdcc_d55b_3807_25e2_b8d2b6fccabe';
ALTER TABLE kylin_intermediate_samplecube_e1c8bdcc_d55b_3807_25e2_b8d2b6fccabe SET TBLPROPERTIES('auto.purge'='true');
INSERT OVERWRITE TABLE kylin_intermediate_samplecube_e1c8bdcc_d55b_3807_25e2_b8d2b6fccabe SELECT
KYLIN_SALES.LSTG_FORMAT_NAME as KYLIN_SALES_LSTG_FORMAT_NAME
,KYLIN_SALES.LSTG_SITE_ID as KYLIN_SALES_LSTG_SITE_ID
,KYLIN_SALES.SELLER_ID as KYLIN_SALES_SELLER_ID
,KYLIN_ACCOUNT.ACCOUNT_COUNTRY as KYLIN_ACCOUNT_ACCOUNT_COUNTRY
,KYLIN_SALES.PRICE as KYLIN_SALES_PRICE
,KYLIN_SALES.ITEM_COUNT as KYLIN_SALES_ITEM_COUNT
FROM LACUS.KYLIN_SALES as KYLIN_SALES
LEFT JOIN LACUS.KYLIN_ACCOUNT as KYLIN_ACCOUNT
ON KYLIN_SALES.SELLER_ID = KYLIN_ACCOUNT.ACCOUNT_ID
LEFT JOIN LACUS.KYLIN_COUNTRY as KYLIN_COUNTRY
ON KYLIN_ACCOUNT.ACCOUNT_COUNTRY = KYLIN_COUNTRY.COUNTRY
WHERE 1=1;
" --hiveconf mapreduce.map.output.compress.codec=org.apache.hadoop.io.compress.SnappyCodec --hiveconf dfs.replication=2 --hiveconf hive.exec.compress.output=true
KYLIN [ INFO ] 03-11 12:53:42.237 org.apache.kylin.common.util.BufferedLogger.log(BufferedLogger.java:38) from Scheduler 1705744661 Job 6d64fe74-ea50-76a6-bbd1-f85210984416-129
> WARNING: Use "yarn jar" to launch YARN applications.
KYLIN [ INFO ] 03-11 12:53:43.241 org.apache.kylin.common.util.BufferedLogger.log(BufferedLogger.java:38) from Scheduler 1705744661 Job 6d64fe74-ea50-76a6-bbd1-f85210984416-129
>
Logging initialized using configuration in file:/etc/hive/2.4.0.0-169/0/hive-log4j.properties
KYLIN [ INFO ] 03-11 12:56:07.045 org.apache.kylin.common.util.SSHClient.execCommand(SSHClient.java:262) from Scheduler 1705744661 Job 6d64fe74-ea50-76a6-bbd1-f85210984416-221
> [root@sandbox] Execute command: hive -e "USE lacus;
DROP TABLE IF EXISTS lacus.kylin_intermediate_samplecube_e1c8bdcc_d55b_3807_25e2_b8d2b6fccabe;
" --hiveconf mapreduce.map.output.compress.codec=org.apache.hadoop.io.compress.SnappyCodec --hiveconf dfs.replication=2 --hiveconf hive.exec.compress.output=true
was (Author: hit_lacus):
Screenshots in My DEV env (with quote turn off):
> add parameter to change sql quoting character for sqoop
> -------------------------------------------------------
>
> Key: KYLIN-3857
> URL: https://issues.apache.org/jira/browse/KYLIN-3857
> Project: Kylin
> Issue Type: Bug
> Reporter: Davide Malagoli
> Assignee: XiaoXiang Yu
> Priority: Major
> Attachments: image-2019-03-11-11-46-05-946.png, image-2019-03-11-11-46-14-305.png, image-2019-03-11-11-46-20-664.png, image-2019-03-11-11-46-31-966.png, sqoop-error.log, sqoop-script.sh
>
>
> I've set up Kylin to access a SQL server using JDBC and do the ETL step as described in
> [http://kylin.apache.org/docs/tutorial/setup_jdbc_datasource.html]
> but when the "Sqoop To Flat Hive Table" runs it terminates with an error shown in the attached log.
> It seems that the root cause is the presence of the '`' character in the query, which has no meaning for sql server.
>
> My problem seems to related to this piece of code
>
> public class FlatTableSqlQuoteUtils {
> public static final String QUOTE = "`";
> /**
> * Quote identifier by default quote `
> * @param identifier
> * @return
> */
> public static String quoteIdentifier(String identifier){
> return QUOTE + identifier + QUOTE;
> }
>
> The solution may be to add a parameter to change this character used for quoting.
>
> Just to be sure I made a copy of the original "sqoop" command to "sqoop.orig" and replaced the original with a little script of mine (attached).
> This little script removes the "`" character from the original command and the call "sqoop.orig" with the cleaned arguments.
> It works, but it is still an ugly workaround.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)