You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sqoop.apache.org by "Abraham Elmahrek (JIRA)" <ji...@apache.org> on 2015/06/16 12:30:01 UTC
[jira] [Comment Edited] (SQOOP-2103) Not able define Decimal(n,p)
data type in map-column-hive option
[ https://issues.apache.org/jira/browse/SQOOP-2103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14587812#comment-14587812 ]
Abraham Elmahrek edited comment on SQOOP-2103 at 6/16/15 10:29 AM:
-------------------------------------------------------------------
This is because the parsing logic is very basic:
# Split by ","
# Split by "="
Let's improve this a bit by allowing URL encoded keys and values:
# Split by ","
# Split by "="
# URL decode left and right side of "="
Examples:
* DECIMAL(1%2C1) = DECIMAL(1,1)
* NUMERIC(1%2C%202) = NUMERIC(1, 2)
was (Author: abec):
This is because the parsing logic is very basic:
# Split by ","
# Split by "="
Let's improve this a bit by allowing URL encoded keys and values:
# Split by ","
# Split by "="
# URL decode left and right side of "="
> Not able define Decimal(n,p) data type in map-column-hive option
> ----------------------------------------------------------------
>
> Key: SQOOP-2103
> URL: https://issues.apache.org/jira/browse/SQOOP-2103
> Project: Sqoop
> Issue Type: Bug
> Components: sqoop2-build
> Affects Versions: 1.4.5
> Environment: Unix OS
> Reporter: Anitha
> Assignee: Abraham Elmahrek
> Labels: map-column-hive,decimal,sqoop
> Fix For: 1.4.7
>
>
> Not able to define any column as decimal(n,p) while sqooping data and creating hive table using --map-column-hive option
> I tried different syntax (with single quotes,double quotes,backslash..) but the command errors out.
> Below is the command and error:
> time sqoop import -D oraoop.disabled=true --options-file /home/sqoopuser/configfiles/c360_db.config --query "SELECT DLRS.*,'C360' as SOURCE,TO_CHAR(current_timestamp,'YYYY-MM-DD HH24:MI:SS.FF') AS LOAD_TS FROM DEALERS DLRS WHERE \$CONDITIONS" --split-by DEALER_NUMBER --create-hive-table --hive-table "c360dev.dealers_test" --hive-import --map-column-hive DEALER_NUMBER="VARCHAR(20)",DEALER_NAME="VARCHAR(50)",GEO_CODE="VARCHAR(50)",REGIONCODE=INT,REGIONNAME="VARCHAR(30)",DISTRICTCODE="VARCHAR(4)",21_INDICATOR="VARCHAR(1)",ADDRESS_TYPE="VARCHAR(25)",LINEONE="VARCHAR(50)",CITY="VARCHAR(25)",STATE="VARCHAR(2)",ZIPCODE="VARCHAR(5)",EFFECTIVE_DATE=TIMESTAMP,TERMINATION_CODE="VARCHAR(1)",PHONE_NUMBER="VARCHAR(25)",ENTITY_NAME="VARCHAR(50)",DEALERGROUP_NAME="VARCHAR(50)",DEALERGROUP_1COUNT=SMALLINT,DEALERGROUP_2COUNT=SMALLINT,DEALER_CLASSCODE="DECIMAL(22,7)",REGIONAREA_NAME="VARCHAR(50)",REGIONAREA_SHORTNAME="VARCHAR(50)",REGIONAREA_ABBREVATION="VARCHAR(5)",REGIONAREA_AREACCODE=SMALLINT,REGIONAREA_REGIONCODE=SMALLINT,PMACODE=SMALLINT,MARKETCODE=INT,MARKETNAME="VARCHAR(50)",MARKETTYPE="VARCHAR(50)",DUAL_INDICATOR="VARCHAR(5)",DEALERSHIP_TYPE="VARCHAR(5)",CREATION_DATE_TIMESTAMP=TIMESTAMP,CREATED_BY="VARCHAR(20)",LAST_UPDATE_DATE=TIMESTAMP,LAST_UPDATED_BY="VARCHAR(20)",TXM_DEALER="VARCHAR(50)",TFS_ECONTRACTING="VARCHAR(50)",YEARS_OF_PRESS_AWARDS=SMALLINT,IMAGE_USA="VARCHAR(50)",IMAGE_USA_EFFECTIVE_DATE=TIMESTAMP,TERMINATION_DATE=TIMESTAMP,3_DEALERNAME="VARCHAR(50)",IMAGE_USA_COMMENT="VARCHAR(100)",SINGLE_COMBINED_INDICATOR="VARCHAR(10)",1_URLNAME="VARCHAR(25)",1_URLVALUE="VARCHAR(100)",3_URLNAME="VARCHAR(25)",3_URLVALUE="VARCHAR(100)",2_URLNAME="VARCHAR(25)",2_URLVALUE="VARCHAR(100)" --hive-delims-replacement " " --null-string '\\N' --null-non-string '\\N' --target-dir "/user/hive/warehouse/c360dev.db/dealers_test" --verbose > dealers_test.log 2>&1
> real 0m0.876s
> user 0m1.062s
> sys 0m0.112s
> [sqoopuser@*****09 ~]$ more dealers_test.log
> Warning: /usr/lib/sqoop/../accumulo does not exist! Accumulo imports will fail.
> Please set $ACCUMULO_HOME to the root of your Accumulo installation.
> 15/02/12 15:19:26 INFO sqoop.Sqoop: Running Sqoop version: 1.4.5-cdh5.3.1
> 15/02/12 15:19:26 DEBUG tool.BaseSqoopTool: Enabled debug logging.
> 15/02/12 15:19:26 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
> 15/02/12 15:19:26 DEBUG sqoop.Sqoop: 1
> java.lang.ArrayIndexOutOfBoundsException: 1
> at org.apache.sqoop.SqoopOptions.parseColumnMapping(SqoopOptions.java:1200)
> at org.apache.sqoop.SqoopOptions.setMapColumnHive(SqoopOptions.java:1205)
> at org.apache.sqoop.tool.BaseSqoopTool.applyHiveOptions(BaseSqoopTool.java:1112)
> at org.apache.sqoop.tool.ImportTool.applyOptions(ImportTool.java:973)
> at org.apache.sqoop.tool.SqoopTool.parseArguments(SqoopTool.java:435)
> at org.apache.sqoop.Sqoop.run(Sqoop.java:131)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
> at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
> 1
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)