[jira] [Commented] (SQOOP-3011) sqoop import to HIVE external table based on file system other than HDFS

["Hui Cao (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SQOOP-3011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15486722#comment-15486722 ] 

Hui Cao commented on SQOOP-3011:
--------------------------------

took a further look which found out, that the sqoop import Hive command translates into two separate steps, 
1, import the data into HDFS,
2, call hive command as below
LOAD DATA INPATH 'hdfs://<hostname>:8020/user/hdfs/FOO' INTO TABLE `FOO`
with HIVE which either can load from local or same distributed file system, 
Guessing this is the cause of the error, as object store and hdfs are two different distributed systems.
Recommendation of fix:
1, When sqoop importing data into HDFS, add a logic for it to check if the table is on other distributed system than the one defined in $defaultDS, if not, put the import the temp data to the external system
2, Or, add an option for the command import, to specify the external system location. 

> sqoop import to HIVE external table based on file system other than HDFS
> ------------------------------------------------------------------------
>
>                 Key: SQOOP-3011
>                 URL: https://issues.apache.org/jira/browse/SQOOP-3011
>             Project: Sqoop
>          Issue Type: Bug
>    Affects Versions: 1.4.6
>            Reporter: Hui Cao
>
> 1, Create external Hive table using swift URI in Hive shell
> 	CREATE TABLE foo(id INT, msg STRING, INT) ROW FORMAT
> 	DELIMITED FIELDS TERMINATED BY ','
> 	LINES TERMINATED BY '\n' 
> 	STORED AS TEXTFILE
> 	LOCATION ’swift://swift.location/’;
>   This table is created on an external file system instead of hdfs, in this case, I’m using swift Object Store
> 2, Use sqoop to insert data to this table, 
> $SQOOP_PATH/sqoop import --driver com.ibm.db2.jcc.DB2Driver --connect jdbc:db://<dbhostname>:<port>/<db> --username <username> --password <password> --table FOO --hive-import --hive-home hive
> the process shows following error:
> FAILED: SemanticException [Error 10028]: Line 2:17 Path is not legal ''hdfs://<hostname>:8020/user/hdfs/FOO'': Move from: hdfs://<hostname>:8020/user/hdfs/FOO to: swift://swift.location/ is not valid. Please check that values for params "default.fs.name" and "hive.metastore.warehouse.dir" do not conflict.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)