You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@sqoop.apache.org by Kevin Zhang <zh...@yahoo.com> on 2013/11/19 19:06:40 UTC
sqoop and hive
Hi,
I'm new to Sqoop. Can anybody please tell me how to import mysql database table into hive database with Sqoop?
I'm using sqoop-1.99.3-bin-hadoop100.tar.gz . I searched but could not find the answer. Right now I can import mysql database table to hadoop text files. For 1.99.3, I cannot find any hive related docs.
Thanks a lot,
Kevin
Re: cannot see hive table after import
Posted by Kevin Zhang <zh...@yahoo.com>.
It turns out that hive uses the current directory as the metadata store location.
Running the sqoop command and hive command from the same location fixed my problem.
________________________________
From: "phil@insufficient-light.com" <ph...@insufficient-light.com>
To: user@sqoop.apache.org
Sent: Tuesday, November 19, 2013 2:44 PM
Subject: Re: cannot see hive table after import
It looks like the answer lies here:
13/11/19 15:02:16 WARN tool.BaseSqoopTool: It seems that you've specified
at least one of following:
13/11/19 15:02:16 WARN tool.BaseSqoopTool: --hive-home
13/11/19 15:02:16 WARN tool.BaseSqoopTool: --hive-overwrite
13/11/19 15:02:16 WARN tool.BaseSqoopTool: --create-hive-table
13/11/19 15:02:16 WARN tool.BaseSqoopTool: --hive-table
13/11/19 15:02:16 WARN tool.BaseSqoopTool: --hive-partition-key
13/11/19 15:02:16 WARN tool.BaseSqoopTool: --hive-partition-value
13/11/19 15:02:16 WARN tool.BaseSqoopTool: --map-column-hive
13/11/19 15:02:16 WARN tool.BaseSqoopTool: Without specifying parameter
--hive-import. Please note that
13/11/19 15:02:16 WARN tool.BaseSqoopTool: those arguments will not be
used in this session. Either
13/11/19 15:02:16 WARN tool.BaseSqoopTool: specify --hive-import to apply
them correctly or remove them
13/11/19 15:02:16 WARN tool.BaseSqoopTool: from command line to remove
this warning.
Perhaps you should try re-running the command you used, adding the
--hive-import parameter.
> Just started to use Hadoop, Hive, Sqoop today. Sorry for any stupid
> questions.
>
> I managed to run a sqoop-import command. The command output says "Hive
> import complete". But I cannot see the table within hive. Hive command
> "show tables" show nothing.
>
> My Sqoop command is:
> /bin/sqoop-import --connect jdbc:mysql://xxx.com/yyyy --table user_entity
> --username uuuuuu --password ppppp --hive-table
> h_user_entity --create-hive-table
>>
> My Hive command is:
> show tables;
> Sqoop command output is:
> Warning: /usr/lib/hbase does not exist! HBase imports will fail.
> Please set $HBASE_HOME to the root of your HBase installation.
>>Warning: /usr/lib/hcatalog does not exist! HCatalog jobs will fail.
>>Please set $HCAT_HOME to the root of your HCatalog installation.
>>Warning: $HADOOP_HOME is deprecated.
>>
>>
>>13/11/19 15:02:16 WARN tool.BaseSqoopTool: Setting your password on the
>> command-line is insecure. Consider using -P instead.
>>13/11/19 15:02:16 INFO tool.BaseSqoopTool: Using Hive-specific delimiters
>> for output. You can override
>>13/11/19 15:02:16 INFO tool.BaseSqoopTool: delimiters with
>> --fields-terminated-by, etc.
>>13/11/19 15:02:16 WARN tool.BaseSqoopTool: It seems that you've specified
>> at least one of following:
>>13/11/19 15:02:16 WARN tool.BaseSqoopTool: --hive-home
>>13/11/19 15:02:16 WARN tool.BaseSqoopTool: --hive-overwrite
>>13/11/19 15:02:16 WARN tool.BaseSqoopTool: --create-hive-table
>>13/11/19 15:02:16 WARN tool.BaseSqoopTool: --hive-table
>>13/11/19 15:02:16 WARN tool.BaseSqoopTool: --hive-partition-key
>>13/11/19 15:02:16 WARN tool.BaseSqoopTool: --hive-partition-value
>>13/11/19 15:02:16 WARN tool.BaseSqoopTool: --map-column-hive
>>13/11/19 15:02:16 WARN tool.BaseSqoopTool: Without specifying parameter
>> --hive-import. Please note that
>>13/11/19 15:02:16 WARN tool.BaseSqoopTool: those arguments will not be
>> used in this session. Either
>>13/11/19 15:02:16 WARN tool.BaseSqoopTool: specify --hive-import to apply
>> them correctly or remove them
>>13/11/19 15:02:16 WARN tool.BaseSqoopTool: from command line to remove
>> this warning.
>>13/11/19 15:02:16 INFO tool.BaseSqoopTool: Please note that --hive-home,
>> --hive-partition-key,
>>13/11/19 15:02:16 INFO tool.BaseSqoopTool: hive-partition-value and
>> --map-column-hive options are
>>13/11/19 15:02:16 INFO tool.BaseSqoopTool: are also valid for HCatalog
>> imports and exports
>>13/11/19 15:02:16 INFO manager.MySQLManager: Preparing to use a MySQL
>> streaming resultset.
>>13/11/19 15:02:16 INFO tool.CodeGenTool: Beginning code generation
>>13/11/19 15:02:16 INFO manager.SqlManager: Executing SQL statement:
>> SELECT t.* FROM `user_entity` AS t LIMIT 1
>>13/11/19 15:02:16 INFO manager.SqlManager: Executing SQL statement:
>> SELECT t.* FROM `user_entity` AS t LIMIT 1
>>13/11/19 15:02:16 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is
>> /data/hadoop/current
>>Note:
>> /tmp/sqoop-root/compile/39e8d314afcc1b81c1cc50a6c3d396b1/user_entity.java
>> uses or overrides a deprecated API.
>>Note: Recompile with -Xlint:deprecation for details.
>>13/11/19 15:02:17 INFO orm.CompilationManager: Writing jar file:
>> /tmp/sqoop-root/compile/39e8d314afcc1b81c1cc50a6c3d396b1/user_entity.jar
>>13/11/19 15:02:17 WARN manager.MySQLManager: It looks like you are
>> importing from mysql.
>>13/11/19 15:02:17 WARN manager.MySQLManager: This transfer can be faster!
>> Use the --direct
>>13/11/19 15:02:17 WARN manager.MySQLManager: option to exercise a
>> MySQL-specific fast path.
>>13/11/19 15:02:17 INFO manager.MySQLManager: Setting zero DATETIME
>> behavior to convertToNull (mysql)
>>13/11/19 15:02:17 INFO mapreduce.ImportJobBase: Beginning import of
>> user_entity
>>13/11/19 15:02:18 INFO db.DataDrivenDBInputFormat: BoundingValsQuery:
>> SELECT MIN(`id`), MAX(`id`) FROM `user_entity`
>>13/11/19 15:02:18 WARN db.TextSplitter: Generating splits for a textual
>> index column.
>>13/11/19 15:02:18 WARN db.TextSplitter: If your database sorts in a
>> case-insensitive order, this may result in a partial import or duplicate
>> records.
>>13/11/19 15:02:18 WARN db.TextSplitter: You are strongly encouraged to
>> choose an integral split column.
>>13/11/19 15:02:19 INFO mapred.JobClient: Running job:
>> job_201311181910_0007
>>13/11/19 15:02:20 INFO mapred.JobClient: map 0% reduce 0%
>>13/11/19 15:02:29 INFO mapred.JobClient: map 16% reduce 0%
>>13/11/19 15:02:32 INFO mapred.JobClient: map 33% reduce 0%
>>13/11/19 15:02:35 INFO mapred.JobClient: map 50% reduce 0%
>>13/11/19 15:02:39 INFO mapred.JobClient: map 66% reduce 0%
>>13/11/19 15:02:42 INFO mapred.JobClient: map 83% reduce 0%
>>13/11/19 15:02:43 INFO mapred.JobClient: map 100% reduce 0%
>>13/11/19 15:02:44 INFO mapred.JobClient: Job complete:
>> job_201311181910_0007
>>13/11/19 15:02:44 INFO mapred.JobClient: Counters: 18
>>13/11/19 15:02:44 INFO mapred.JobClient: Job Counters
>>13/11/19 15:02:44 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=38087
>>13/11/19 15:02:44 INFO mapred.JobClient: Total time spent by all
>> reduces waiting after reserving slots (ms)=0
>>13/11/19 15:02:44 INFO mapred.JobClient: Total time spent by all maps
>> waiting after reserving slots (ms)=0
>>13/11/19 15:02:44 INFO mapred.JobClient: Launched map tasks=6
>>13/11/19 15:02:44 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0
>>13/11/19 15:02:44 INFO mapred.JobClient: File Output Format Counters
>>13/11/19 15:02:44 INFO mapred.JobClient: Bytes Written=85103032
>>13/11/19 15:02:44 INFO mapred.JobClient: FileSystemCounters
>>13/11/19 15:02:44 INFO mapred.JobClient: HDFS_BYTES_READ=825
>>13/11/19 15:02:44 INFO mapred.JobClient: FILE_BYTES_WRITTEN=389460
>>13/11/19 15:02:44 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=85103032
>>13/11/19 15:02:44 INFO mapred.JobClient: File Input Format Counters
>>13/11/19 15:02:44 INFO mapred.JobClient: Bytes Read=0
>>13/11/19 15:02:44 INFO mapred.JobClient: Map-Reduce Framework
>>13/11/19 15:02:44 INFO mapred.JobClient: Map input records=387423
>>13/11/19 15:02:44 INFO mapred.JobClient: Physical memory (bytes)
>> snapshot=787578880
>>13/11/19 15:02:44 INFO mapred.JobClient: Spilled Records=0
>>13/11/19 15:02:44 INFO mapred.JobClient: CPU time spent (ms)=22420
>>13/11/19 15:02:44 INFO mapred.JobClient: Total committed heap usage
>> (bytes)=1205403648
>>13/11/19 15:02:44 INFO mapred.JobClient: Virtual memory (bytes)
>> snapshot=6980870144
>>13/11/19 15:02:44 INFO mapred.JobClient: Map output records=387423
>>13/11/19 15:02:44 INFO mapred.JobClient: SPLIT_RAW_BYTES=825
>>13/11/19 15:02:44 INFO mapreduce.ImportJobBase: Transferred 81.1606 MB in
>> 26.1394 seconds (3.1049 MB/sec)
>>13/11/19 15:02:44 INFO mapreduce.ImportJobBase: Retrieved 387423 records.
>>13/11/19 15:02:44 INFO manager.SqlManager: Executing SQL statement:
>> SELECT t.* FROM `user_entity` AS t LIMIT 1
>>13/11/19 15:02:44 INFO hive.HiveImport: Removing temporary files from
>> import process: hdfs://localhost:9000/user/root/user_entity/_logs
>>13/11/19 15:02:44 INFO hive.HiveImport: Loading uploaded data into Hive
>>13/11/19 15:02:46 INFO hive.HiveImport:
>>13/11/19 15:02:46 INFO hive.HiveImport: Logging initialized using
>> configuration in
>> jar:file:/data/hive/hive-0.12.0/lib/hive-common-0.12.0.jar!/hive-log4j.properties
>>13/11/19 15:02:51 INFO hive.HiveImport: OK
>>13/11/19 15:02:51 INFO hive.HiveImport: Time taken: 5.221 seconds
>>13/11/19 15:02:51 INFO hive.HiveImport: Loading data to table
>> default.h_user_entity
>>13/11/19 15:02:52 INFO hive.HiveImport: Table default.h_user_entity
>> stats: [num_partitions: 0, num_files: 7, num_rows: 0, total_size:
>> 85103032, raw_data_size: 0]
>>13/11/19 15:02:52 INFO hive.HiveImport: OK
>>13/11/19 15:02:52 INFO hive.HiveImport: Time taken: 0.636 seconds
>>13/11/19 15:02:52 INFO hive.HiveImport: Hive import complete.
>>13/11/19 15:02:52 INFO hive.HiveImport: Export directory is empty,
>> removing it.
>
> Any help is highly appreciated.
>
> -Kevin
Re: cannot see hive table after import
Posted by ph...@insufficient-light.com.
It looks like the answer lies here:
13/11/19 15:02:16 WARN tool.BaseSqoopTool: It seems that you've specified
at least one of following:
13/11/19 15:02:16 WARN tool.BaseSqoopTool: --hive-home
13/11/19 15:02:16 WARN tool.BaseSqoopTool: --hive-overwrite
13/11/19 15:02:16 WARN tool.BaseSqoopTool: --create-hive-table
13/11/19 15:02:16 WARN tool.BaseSqoopTool: --hive-table
13/11/19 15:02:16 WARN tool.BaseSqoopTool: --hive-partition-key
13/11/19 15:02:16 WARN tool.BaseSqoopTool: --hive-partition-value
13/11/19 15:02:16 WARN tool.BaseSqoopTool: --map-column-hive
13/11/19 15:02:16 WARN tool.BaseSqoopTool: Without specifying parameter
--hive-import. Please note that
13/11/19 15:02:16 WARN tool.BaseSqoopTool: those arguments will not be
used in this session. Either
13/11/19 15:02:16 WARN tool.BaseSqoopTool: specify --hive-import to apply
them correctly or remove them
13/11/19 15:02:16 WARN tool.BaseSqoopTool: from command line to remove
this warning.
Perhaps you should try re-running the command you used, adding the
--hive-import parameter.
> Just started to use Hadoop, Hive, Sqoop today. Sorry for any stupid
> questions.
>
> I managed to run a sqoop-import command. The command output says "Hive
> import complete". But I cannot see the table within hive. Hive command
> "show tables" show nothing.
>
> My Sqoop command is:
> /bin/sqoop-import --connect jdbc:mysql://xxx.com/yyyy --table user_entity
> --username uuuuuu --password ppppp --hive-table
> h_user_entity --create-hive-table
>>
> My Hive command is:
> show tables;
> Sqoop command output is:
> Warning: /usr/lib/hbase does not exist! HBase imports will fail.
> Please set $HBASE_HOME to the root of your HBase installation.
>>Warning: /usr/lib/hcatalog does not exist! HCatalog jobs will fail.
>>Please set $HCAT_HOME to the root of your HCatalog installation.
>>Warning: $HADOOP_HOME is deprecated.
>>
>>
>>13/11/19 15:02:16 WARN tool.BaseSqoopTool: Setting your password on the
>> command-line is insecure. Consider using -P instead.
>>13/11/19 15:02:16 INFO tool.BaseSqoopTool: Using Hive-specific delimiters
>> for output. You can override
>>13/11/19 15:02:16 INFO tool.BaseSqoopTool: delimiters with
>> --fields-terminated-by, etc.
>>13/11/19 15:02:16 WARN tool.BaseSqoopTool: It seems that you've specified
>> at least one of following:
>>13/11/19 15:02:16 WARN tool.BaseSqoopTool: --hive-home
>>13/11/19 15:02:16 WARN tool.BaseSqoopTool: --hive-overwrite
>>13/11/19 15:02:16 WARN tool.BaseSqoopTool: --create-hive-table
>>13/11/19 15:02:16 WARN tool.BaseSqoopTool: --hive-table
>>13/11/19 15:02:16 WARN tool.BaseSqoopTool: --hive-partition-key
>>13/11/19 15:02:16 WARN tool.BaseSqoopTool: --hive-partition-value
>>13/11/19 15:02:16 WARN tool.BaseSqoopTool: --map-column-hive
>>13/11/19 15:02:16 WARN tool.BaseSqoopTool: Without specifying parameter
>> --hive-import. Please note that
>>13/11/19 15:02:16 WARN tool.BaseSqoopTool: those arguments will not be
>> used in this session. Either
>>13/11/19 15:02:16 WARN tool.BaseSqoopTool: specify --hive-import to apply
>> them correctly or remove them
>>13/11/19 15:02:16 WARN tool.BaseSqoopTool: from command line to remove
>> this warning.
>>13/11/19 15:02:16 INFO tool.BaseSqoopTool: Please note that --hive-home,
>> --hive-partition-key,
>>13/11/19 15:02:16 INFO tool.BaseSqoopTool: hive-partition-value and
>> --map-column-hive options are
>>13/11/19 15:02:16 INFO tool.BaseSqoopTool: are also valid for HCatalog
>> imports and exports
>>13/11/19 15:02:16 INFO manager.MySQLManager: Preparing to use a MySQL
>> streaming resultset.
>>13/11/19 15:02:16 INFO tool.CodeGenTool: Beginning code generation
>>13/11/19 15:02:16 INFO manager.SqlManager: Executing SQL statement:
>> SELECT t.* FROM `user_entity` AS t LIMIT 1
>>13/11/19 15:02:16 INFO manager.SqlManager: Executing SQL statement:
>> SELECT t.* FROM `user_entity` AS t LIMIT 1
>>13/11/19 15:02:16 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is
>> /data/hadoop/current
>>Note:
>> /tmp/sqoop-root/compile/39e8d314afcc1b81c1cc50a6c3d396b1/user_entity.java
>> uses or overrides a deprecated API.
>>Note: Recompile with -Xlint:deprecation for details.
>>13/11/19 15:02:17 INFO orm.CompilationManager: Writing jar file:
>> /tmp/sqoop-root/compile/39e8d314afcc1b81c1cc50a6c3d396b1/user_entity.jar
>>13/11/19 15:02:17 WARN manager.MySQLManager: It looks like you are
>> importing from mysql.
>>13/11/19 15:02:17 WARN manager.MySQLManager: This transfer can be faster!
>> Use the --direct
>>13/11/19 15:02:17 WARN manager.MySQLManager: option to exercise a
>> MySQL-specific fast path.
>>13/11/19 15:02:17 INFO manager.MySQLManager: Setting zero DATETIME
>> behavior to convertToNull (mysql)
>>13/11/19 15:02:17 INFO mapreduce.ImportJobBase: Beginning import of
>> user_entity
>>13/11/19 15:02:18 INFO db.DataDrivenDBInputFormat: BoundingValsQuery:
>> SELECT MIN(`id`), MAX(`id`) FROM `user_entity`
>>13/11/19 15:02:18 WARN db.TextSplitter: Generating splits for a textual
>> index column.
>>13/11/19 15:02:18 WARN db.TextSplitter: If your database sorts in a
>> case-insensitive order, this may result in a partial import or duplicate
>> records.
>>13/11/19 15:02:18 WARN db.TextSplitter: You are strongly encouraged to
>> choose an integral split column.
>>13/11/19 15:02:19 INFO mapred.JobClient: Running job:
>> job_201311181910_0007
>>13/11/19 15:02:20 INFO mapred.JobClient: map 0% reduce 0%
>>13/11/19 15:02:29 INFO mapred.JobClient: map 16% reduce 0%
>>13/11/19 15:02:32 INFO mapred.JobClient: map 33% reduce 0%
>>13/11/19 15:02:35 INFO mapred.JobClient: map 50% reduce 0%
>>13/11/19 15:02:39 INFO mapred.JobClient: map 66% reduce 0%
>>13/11/19 15:02:42 INFO mapred.JobClient: map 83% reduce 0%
>>13/11/19 15:02:43 INFO mapred.JobClient: map 100% reduce 0%
>>13/11/19 15:02:44 INFO mapred.JobClient: Job complete:
>> job_201311181910_0007
>>13/11/19 15:02:44 INFO mapred.JobClient: Counters: 18
>>13/11/19 15:02:44 INFO mapred.JobClient: Job Counters
>>13/11/19 15:02:44 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=38087
>>13/11/19 15:02:44 INFO mapred.JobClient: Total time spent by all
>> reduces waiting after reserving slots (ms)=0
>>13/11/19 15:02:44 INFO mapred.JobClient: Total time spent by all maps
>> waiting after reserving slots (ms)=0
>>13/11/19 15:02:44 INFO mapred.JobClient: Launched map tasks=6
>>13/11/19 15:02:44 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0
>>13/11/19 15:02:44 INFO mapred.JobClient: File Output Format Counters
>>13/11/19 15:02:44 INFO mapred.JobClient: Bytes Written=85103032
>>13/11/19 15:02:44 INFO mapred.JobClient: FileSystemCounters
>>13/11/19 15:02:44 INFO mapred.JobClient: HDFS_BYTES_READ=825
>>13/11/19 15:02:44 INFO mapred.JobClient: FILE_BYTES_WRITTEN=389460
>>13/11/19 15:02:44 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=85103032
>>13/11/19 15:02:44 INFO mapred.JobClient: File Input Format Counters
>>13/11/19 15:02:44 INFO mapred.JobClient: Bytes Read=0
>>13/11/19 15:02:44 INFO mapred.JobClient: Map-Reduce Framework
>>13/11/19 15:02:44 INFO mapred.JobClient: Map input records=387423
>>13/11/19 15:02:44 INFO mapred.JobClient: Physical memory (bytes)
>> snapshot=787578880
>>13/11/19 15:02:44 INFO mapred.JobClient: Spilled Records=0
>>13/11/19 15:02:44 INFO mapred.JobClient: CPU time spent (ms)=22420
>>13/11/19 15:02:44 INFO mapred.JobClient: Total committed heap usage
>> (bytes)=1205403648
>>13/11/19 15:02:44 INFO mapred.JobClient: Virtual memory (bytes)
>> snapshot=6980870144
>>13/11/19 15:02:44 INFO mapred.JobClient: Map output records=387423
>>13/11/19 15:02:44 INFO mapred.JobClient: SPLIT_RAW_BYTES=825
>>13/11/19 15:02:44 INFO mapreduce.ImportJobBase: Transferred 81.1606 MB in
>> 26.1394 seconds (3.1049 MB/sec)
>>13/11/19 15:02:44 INFO mapreduce.ImportJobBase: Retrieved 387423 records.
>>13/11/19 15:02:44 INFO manager.SqlManager: Executing SQL statement:
>> SELECT t.* FROM `user_entity` AS t LIMIT 1
>>13/11/19 15:02:44 INFO hive.HiveImport: Removing temporary files from
>> import process: hdfs://localhost:9000/user/root/user_entity/_logs
>>13/11/19 15:02:44 INFO hive.HiveImport: Loading uploaded data into Hive
>>13/11/19 15:02:46 INFO hive.HiveImport:
>>13/11/19 15:02:46 INFO hive.HiveImport: Logging initialized using
>> configuration in
>> jar:file:/data/hive/hive-0.12.0/lib/hive-common-0.12.0.jar!/hive-log4j.properties
>>13/11/19 15:02:51 INFO hive.HiveImport: OK
>>13/11/19 15:02:51 INFO hive.HiveImport: Time taken: 5.221 seconds
>>13/11/19 15:02:51 INFO hive.HiveImport: Loading data to table
>> default.h_user_entity
>>13/11/19 15:02:52 INFO hive.HiveImport: Table default.h_user_entity
>> stats: [num_partitions: 0, num_files: 7, num_rows: 0, total_size:
>> 85103032, raw_data_size: 0]
>>13/11/19 15:02:52 INFO hive.HiveImport: OK
>>13/11/19 15:02:52 INFO hive.HiveImport: Time taken: 0.636 seconds
>>13/11/19 15:02:52 INFO hive.HiveImport: Hive import complete.
>>13/11/19 15:02:52 INFO hive.HiveImport: Export directory is empty,
>> removing it.
>
> Any help is highly appreciated.
>
> -Kevin
cannot see hive table after import
Posted by Kevin Zhang <zh...@yahoo.com>.
Just started to use Hadoop, Hive, Sqoop today. Sorry for any stupid questions.
I managed to run a sqoop-import command. The command output says "Hive import complete". But I cannot see the table within hive. Hive command "show tables" show nothing.
My Sqoop command is:
/bin/sqoop-import --connect jdbc:mysql://xxx.com/yyyy --table user_entity --username uuuuuu --password ppppp --hive-table h_user_entity --create-hive-table
>
My Hive command is:
show tables;
Sqoop command output is:
Warning: /usr/lib/hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
>Warning: /usr/lib/hcatalog does not exist! HCatalog jobs will fail.
>Please set $HCAT_HOME to the root of your HCatalog installation.
>Warning: $HADOOP_HOME is deprecated.
>
>
>13/11/19 15:02:16 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
>13/11/19 15:02:16 INFO tool.BaseSqoopTool: Using Hive-specific delimiters for output. You can override
>13/11/19 15:02:16 INFO tool.BaseSqoopTool: delimiters with --fields-terminated-by, etc.
>13/11/19 15:02:16 WARN tool.BaseSqoopTool: It seems that you've specified at least one of following:
>13/11/19 15:02:16 WARN tool.BaseSqoopTool: --hive-home
>13/11/19 15:02:16 WARN tool.BaseSqoopTool: --hive-overwrite
>13/11/19 15:02:16 WARN tool.BaseSqoopTool: --create-hive-table
>13/11/19 15:02:16 WARN tool.BaseSqoopTool: --hive-table
>13/11/19 15:02:16 WARN tool.BaseSqoopTool: --hive-partition-key
>13/11/19 15:02:16 WARN tool.BaseSqoopTool: --hive-partition-value
>13/11/19 15:02:16 WARN tool.BaseSqoopTool: --map-column-hive
>13/11/19 15:02:16 WARN tool.BaseSqoopTool: Without specifying parameter --hive-import. Please note that
>13/11/19 15:02:16 WARN tool.BaseSqoopTool: those arguments will not be used in this session. Either
>13/11/19 15:02:16 WARN tool.BaseSqoopTool: specify --hive-import to apply them correctly or remove them
>13/11/19 15:02:16 WARN tool.BaseSqoopTool: from command line to remove this warning.
>13/11/19 15:02:16 INFO tool.BaseSqoopTool: Please note that --hive-home, --hive-partition-key,
>13/11/19 15:02:16 INFO tool.BaseSqoopTool: hive-partition-value and --map-column-hive options are
>13/11/19 15:02:16 INFO tool.BaseSqoopTool: are also valid for HCatalog imports and exports
>13/11/19 15:02:16 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
>13/11/19 15:02:16 INFO tool.CodeGenTool: Beginning code generation
>13/11/19 15:02:16 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `user_entity` AS t LIMIT 1
>13/11/19 15:02:16 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `user_entity` AS t LIMIT 1
>13/11/19 15:02:16 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /data/hadoop/current
>Note: /tmp/sqoop-root/compile/39e8d314afcc1b81c1cc50a6c3d396b1/user_entity.java uses or overrides a deprecated API.
>Note: Recompile with -Xlint:deprecation for details.
>13/11/19 15:02:17 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-root/compile/39e8d314afcc1b81c1cc50a6c3d396b1/user_entity.jar
>13/11/19 15:02:17 WARN manager.MySQLManager: It looks like you are importing from mysql.
>13/11/19 15:02:17 WARN manager.MySQLManager: This transfer can be faster! Use the --direct
>13/11/19 15:02:17 WARN manager.MySQLManager: option to exercise a MySQL-specific fast path.
>13/11/19 15:02:17 INFO manager.MySQLManager: Setting zero DATETIME behavior to convertToNull (mysql)
>13/11/19 15:02:17 INFO mapreduce.ImportJobBase: Beginning import of user_entity
>13/11/19 15:02:18 INFO db.DataDrivenDBInputFormat: BoundingValsQuery: SELECT MIN(`id`), MAX(`id`) FROM `user_entity`
>13/11/19 15:02:18 WARN db.TextSplitter: Generating splits for a textual index column.
>13/11/19 15:02:18 WARN db.TextSplitter: If your database sorts in a case-insensitive order, this may result in a partial import or duplicate records.
>13/11/19 15:02:18 WARN db.TextSplitter: You are strongly encouraged to choose an integral split column.
>13/11/19 15:02:19 INFO mapred.JobClient: Running job: job_201311181910_0007
>13/11/19 15:02:20 INFO mapred.JobClient: map 0% reduce 0%
>13/11/19 15:02:29 INFO mapred.JobClient: map 16% reduce 0%
>13/11/19 15:02:32 INFO mapred.JobClient: map 33% reduce 0%
>13/11/19 15:02:35 INFO mapred.JobClient: map 50% reduce 0%
>13/11/19 15:02:39 INFO mapred.JobClient: map 66% reduce 0%
>13/11/19 15:02:42 INFO mapred.JobClient: map 83% reduce 0%
>13/11/19 15:02:43 INFO mapred.JobClient: map 100% reduce 0%
>13/11/19 15:02:44 INFO mapred.JobClient: Job complete: job_201311181910_0007
>13/11/19 15:02:44 INFO mapred.JobClient: Counters: 18
>13/11/19 15:02:44 INFO mapred.JobClient: Job Counters
>13/11/19 15:02:44 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=38087
>13/11/19 15:02:44 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0
>13/11/19 15:02:44 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0
>13/11/19 15:02:44 INFO mapred.JobClient: Launched map tasks=6
>13/11/19 15:02:44 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0
>13/11/19 15:02:44 INFO mapred.JobClient: File Output Format Counters
>13/11/19 15:02:44 INFO mapred.JobClient: Bytes Written=85103032
>13/11/19 15:02:44 INFO mapred.JobClient: FileSystemCounters
>13/11/19 15:02:44 INFO mapred.JobClient: HDFS_BYTES_READ=825
>13/11/19 15:02:44 INFO mapred.JobClient: FILE_BYTES_WRITTEN=389460
>13/11/19 15:02:44 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=85103032
>13/11/19 15:02:44 INFO mapred.JobClient: File Input Format Counters
>13/11/19 15:02:44 INFO mapred.JobClient: Bytes Read=0
>13/11/19 15:02:44 INFO mapred.JobClient: Map-Reduce Framework
>13/11/19 15:02:44 INFO mapred.JobClient: Map input records=387423
>13/11/19 15:02:44 INFO mapred.JobClient: Physical memory (bytes) snapshot=787578880
>13/11/19 15:02:44 INFO mapred.JobClient: Spilled Records=0
>13/11/19 15:02:44 INFO mapred.JobClient: CPU time spent (ms)=22420
>13/11/19 15:02:44 INFO mapred.JobClient: Total committed heap usage (bytes)=1205403648
>13/11/19 15:02:44 INFO mapred.JobClient: Virtual memory (bytes) snapshot=6980870144
>13/11/19 15:02:44 INFO mapred.JobClient: Map output records=387423
>13/11/19 15:02:44 INFO mapred.JobClient: SPLIT_RAW_BYTES=825
>13/11/19 15:02:44 INFO mapreduce.ImportJobBase: Transferred 81.1606 MB in 26.1394 seconds (3.1049 MB/sec)
>13/11/19 15:02:44 INFO mapreduce.ImportJobBase: Retrieved 387423 records.
>13/11/19 15:02:44 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `user_entity` AS t LIMIT 1
>13/11/19 15:02:44 INFO hive.HiveImport: Removing temporary files from import process: hdfs://localhost:9000/user/root/user_entity/_logs
>13/11/19 15:02:44 INFO hive.HiveImport: Loading uploaded data into Hive
>13/11/19 15:02:46 INFO hive.HiveImport:
>13/11/19 15:02:46 INFO hive.HiveImport: Logging initialized using configuration in jar:file:/data/hive/hive-0.12.0/lib/hive-common-0.12.0.jar!/hive-log4j.properties
>13/11/19 15:02:51 INFO hive.HiveImport: OK
>13/11/19 15:02:51 INFO hive.HiveImport: Time taken: 5.221 seconds
>13/11/19 15:02:51 INFO hive.HiveImport: Loading data to table default.h_user_entity
>13/11/19 15:02:52 INFO hive.HiveImport: Table default.h_user_entity stats: [num_partitions: 0, num_files: 7, num_rows: 0, total_size: 85103032, raw_data_size: 0]
>13/11/19 15:02:52 INFO hive.HiveImport: OK
>13/11/19 15:02:52 INFO hive.HiveImport: Time taken: 0.636 seconds
>13/11/19 15:02:52 INFO hive.HiveImport: Hive import complete.
>13/11/19 15:02:52 INFO hive.HiveImport: Export directory is empty, removing it.
Any help is highly appreciated.
-Kevin
Re: sqoop and hive
Posted by Venkat Ranganathan <vr...@hortonworks.com>.
Currently hive import is not supported with Sqoop 1.99.3. Please use
Sqoop 1.4.4 which support hive imports
Thanks
Venkat
On Tue, Nov 19, 2013 at 10:06 AM, Kevin Zhang <zh...@yahoo.com>wrote:
> Hi,
>
> I'm new to Sqoop. Can anybody please tell me how to import mysql database
> table into hive database with Sqoop?
>
> I'm using sqoop-1.99.3-bin-hadoop100.tar.gz . I searched but could not
> find the answer. Right now I can import mysql database table to hadoop text
> files. For 1.99.3, I cannot find any hive related docs.
>
> Thanks a lot,
>
> Kevin
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.