You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@sqoop.apache.org by K Makeev <k....@gmail.com> on 2019/06/04 11:15:47 UTC
hive CLI not started while using Sqoop import from oracle to hive
Hi
Trying to import simple test table from oracle to hive using sqoop.
Sqoop successfully fetches oracle data, puts them to HDFS, but when it
comes to hive it doesn't seem to work. Sqoop writes no errors or warnings
to output. It jus quits.
Here is my command line
sqoop import --connect jdbc:oracle:thin:@host:1521:SID --username username
--password password --delete-target-dir --table TEST_SQOOP --hive-import
--hive-database db -m 1 -verbose
Here is the end of verbose output
19/06/03 18:28:02 DEBUG hive.TableDefWriter: Create statement: CREATE TABLE
IF NOT EXISTS `db`.`TEST_SQOOP` ( `A` DOUBLE, `B` STRING, `D` STRING)
COMMENT 'Imported by sqoop on 2019/06/03 18:28:02' ROW FORMAT DELIMITED
FIELDS TERMINATED BY '\001' LINES TERMINATED BY '\012' STORED AS TEXTFILE
19/06/03 18:28:02 DEBUG hive.TableDefWriter: Load statement: LOAD DATA
INPATH 'hdfs://hadoop/user/airflow/TEST_SQOOP' INTO TABLE `db`.`TEST_SQOOP`
19/06/03 18:28:02 DEBUG ipc.Client: The ping interval is 60000 ms.
19/06/03 18:28:02 DEBUG ipc.Client: Connecting to hadoop-m1/ip:8020
19/06/03 18:28:02 DEBUG ipc.Client: IPC Client (2071364564) connection to
hadoop-m1/ip:8020 from airflow: starting, having connections 2
19/06/03 18:28:02 DEBUG ipc.Client: IPC Client (2071364564) connection to
hadoop-m1/ip:8020 from airflow sending #779
19/06/03 18:28:02 DEBUG ipc.Client: IPC Client (2071364564) connection to
hadoop-m1/ip:8020 from airflow got value #779
19/06/03 18:28:02 DEBUG ipc.ProtobufRpcEngine: Call: getFileInfo took 69ms
19/06/03 18:28:02 INFO hive.HiveImport: Loading uploaded data into Hive
19/06/03 18:28:02 DEBUG hive.HiveImport: Using in-process Hive instance.
19/06/03 18:28:02 DEBUG util.SubprocessSecurityManager: Installing
subprocess security manager
Logging initialized using configuration in
jar:file:/usr/hdp/2.5.3.0-37/hive/lib/hive-common-1.2.1000.2.5.3.0-37.jar!/hive-log4j.properties
Nothing happens in hive
If i run hive manually through hive command i'm able to execute Create
Table/LOAD DATA INPATH statements without problems.
Why sqoop doesn't do its hive job?
Re: hive CLI not started while using Sqoop import from oracle to hive
Posted by Suresh Kumar Sethuramaswamy <ro...@gmail.com>.
Glad that it worked.
Thanks for the update Kirill
On Tue, Jun 4, 2019, 11:20 AM K Makeev <k.makeev@gmail.com wrote:
> Suresh,
>
> copying hive-site.xml to current dir worked well. Now sqoop hive import
> works properly.
>
> Thanks a lot
>
> going to find out why sqoop doesn't see hive configuration. Setting
> --hive-home or HIVE_CONF_DIR doesn't helps.
>
> Kirill
>
>
>
> On Tue, Jun 4, 2019 at 5:37 PM Suresh Kumar Sethuramaswamy <
> rockssk@gmail.com> wrote:
>
>> If you are using Cloudera Manager or Ambari,try deploying client
>> configuration.
>>
>> If not copy the hive-site.xml and place it in the current working
>> directory
>>
>> On Tue, Jun 4, 2019, 10:14 AM K Makeev <k.makeev@gmail.com wrote:
>>
>>> Hi, Suresh
>>>
>>> hive-metastore seems to have no info about sqoop trying to do anything.
>>> Found nothing.
>>>
>>> when using just hive command to create table - hive-metastore has all
>>> logs, SQL statement, usernames and etc.
>>>
>>> Kirill
>>>
>>> On Tue, Jun 4, 2019 at 2:57 PM Suresh Kumar Sethuramaswamy <
>>> rockssk@gmail.com> wrote:
>>>
>>>> Couple of things to check
>>>>
>>>> 1. Check if there is a hive-site.xml placed mistakenly in the current
>>>> working directory, if so remove it, else it might overwrite system config
>>>>
>>>> 2. Get the hive-metastore log for any clues while sqoop import happens
>>>>
>>>> Regards
>>>> Suresh
>>>>
>>>> On Tue, Jun 4, 2019, 7:16 AM K Makeev <k.makeev@gmail.com wrote:
>>>>
>>>>> Hi
>>>>>
>>>>> Trying to import simple test table from oracle to hive using sqoop.
>>>>>
>>>>> Sqoop successfully fetches oracle data, puts them to HDFS, but when it
>>>>> comes to hive it doesn't seem to work. Sqoop writes no errors or warnings
>>>>> to output. It jus quits.
>>>>>
>>>>> Here is my command line
>>>>>
>>>>> sqoop import --connect jdbc:oracle:thin:@host:1521:SID --username
>>>>> username --password password --delete-target-dir --table TEST_SQOOP
>>>>> --hive-import --hive-database db -m 1 -verbose
>>>>>
>>>>> Here is the end of verbose output
>>>>>
>>>>> 19/06/03 18:28:02 DEBUG hive.TableDefWriter: Create statement: CREATE
>>>>> TABLE IF NOT EXISTS `db`.`TEST_SQOOP` ( `A` DOUBLE, `B` STRING, `D` STRING)
>>>>> COMMENT 'Imported by sqoop on 2019/06/03 18:28:02' ROW FORMAT DELIMITED
>>>>> FIELDS TERMINATED BY '\001' LINES TERMINATED BY '\012' STORED AS TEXTFILE
>>>>> 19/06/03 18:28:02 DEBUG hive.TableDefWriter: Load statement: LOAD DATA
>>>>> INPATH 'hdfs://hadoop/user/airflow/TEST_SQOOP' INTO TABLE `db`.`TEST_SQOOP`
>>>>> 19/06/03 18:28:02 DEBUG ipc.Client: The ping interval is 60000 ms.
>>>>> 19/06/03 18:28:02 DEBUG ipc.Client: Connecting to hadoop-m1/ip:8020
>>>>> 19/06/03 18:28:02 DEBUG ipc.Client: IPC Client (2071364564) connection
>>>>> to hadoop-m1/ip:8020 from airflow: starting, having connections 2
>>>>> 19/06/03 18:28:02 DEBUG ipc.Client: IPC Client (2071364564) connection
>>>>> to hadoop-m1/ip:8020 from airflow sending #779
>>>>> 19/06/03 18:28:02 DEBUG ipc.Client: IPC Client (2071364564) connection
>>>>> to hadoop-m1/ip:8020 from airflow got value #779
>>>>> 19/06/03 18:28:02 DEBUG ipc.ProtobufRpcEngine: Call: getFileInfo took
>>>>> 69ms
>>>>> 19/06/03 18:28:02 INFO hive.HiveImport: Loading uploaded data into Hive
>>>>> 19/06/03 18:28:02 DEBUG hive.HiveImport: Using in-process Hive
>>>>> instance.
>>>>> 19/06/03 18:28:02 DEBUG util.SubprocessSecurityManager: Installing
>>>>> subprocess security manager
>>>>>
>>>>> Logging initialized using configuration in
>>>>> jar:file:/usr/hdp/2.5.3.0-37/hive/lib/hive-common-1.2.1000.2.5.3.0-37.jar!/hive-log4j.properties
>>>>>
>>>>> Nothing happens in hive
>>>>>
>>>>> If i run hive manually through hive command i'm able to execute Create
>>>>> Table/LOAD DATA INPATH statements without problems.
>>>>>
>>>>> Why sqoop doesn't do its hive job?
>>>>>
>>>>>
Re: hive CLI not started while using Sqoop import from oracle to hive
Posted by K Makeev <k....@gmail.com>.
Hi, Suresh
hive-metastore seems to have no info about sqoop trying to do anything.
Found nothing.
when using just hive command to create table - hive-metastore has all logs,
SQL statement, usernames and etc.
Kirill
On Tue, Jun 4, 2019 at 2:57 PM Suresh Kumar Sethuramaswamy <
rockssk@gmail.com> wrote:
> Couple of things to check
>
> 1. Check if there is a hive-site.xml placed mistakenly in the current
> working directory, if so remove it, else it might overwrite system config
>
> 2. Get the hive-metastore log for any clues while sqoop import happens
>
> Regards
> Suresh
>
> On Tue, Jun 4, 2019, 7:16 AM K Makeev <k.makeev@gmail.com wrote:
>
>> Hi
>>
>> Trying to import simple test table from oracle to hive using sqoop.
>>
>> Sqoop successfully fetches oracle data, puts them to HDFS, but when it
>> comes to hive it doesn't seem to work. Sqoop writes no errors or warnings
>> to output. It jus quits.
>>
>> Here is my command line
>>
>> sqoop import --connect jdbc:oracle:thin:@host:1521:SID --username
>> username --password password --delete-target-dir --table TEST_SQOOP
>> --hive-import --hive-database db -m 1 -verbose
>>
>> Here is the end of verbose output
>>
>> 19/06/03 18:28:02 DEBUG hive.TableDefWriter: Create statement: CREATE
>> TABLE IF NOT EXISTS `db`.`TEST_SQOOP` ( `A` DOUBLE, `B` STRING, `D` STRING)
>> COMMENT 'Imported by sqoop on 2019/06/03 18:28:02' ROW FORMAT DELIMITED
>> FIELDS TERMINATED BY '\001' LINES TERMINATED BY '\012' STORED AS TEXTFILE
>> 19/06/03 18:28:02 DEBUG hive.TableDefWriter: Load statement: LOAD DATA
>> INPATH 'hdfs://hadoop/user/airflow/TEST_SQOOP' INTO TABLE `db`.`TEST_SQOOP`
>> 19/06/03 18:28:02 DEBUG ipc.Client: The ping interval is 60000 ms.
>> 19/06/03 18:28:02 DEBUG ipc.Client: Connecting to hadoop-m1/ip:8020
>> 19/06/03 18:28:02 DEBUG ipc.Client: IPC Client (2071364564) connection to
>> hadoop-m1/ip:8020 from airflow: starting, having connections 2
>> 19/06/03 18:28:02 DEBUG ipc.Client: IPC Client (2071364564) connection to
>> hadoop-m1/ip:8020 from airflow sending #779
>> 19/06/03 18:28:02 DEBUG ipc.Client: IPC Client (2071364564) connection to
>> hadoop-m1/ip:8020 from airflow got value #779
>> 19/06/03 18:28:02 DEBUG ipc.ProtobufRpcEngine: Call: getFileInfo took 69ms
>> 19/06/03 18:28:02 INFO hive.HiveImport: Loading uploaded data into Hive
>> 19/06/03 18:28:02 DEBUG hive.HiveImport: Using in-process Hive instance.
>> 19/06/03 18:28:02 DEBUG util.SubprocessSecurityManager: Installing
>> subprocess security manager
>>
>> Logging initialized using configuration in
>> jar:file:/usr/hdp/2.5.3.0-37/hive/lib/hive-common-1.2.1000.2.5.3.0-37.jar!/hive-log4j.properties
>>
>> Nothing happens in hive
>>
>> If i run hive manually through hive command i'm able to execute Create
>> Table/LOAD DATA INPATH statements without problems.
>>
>> Why sqoop doesn't do its hive job?
>>
>>
Re: hive CLI not started while using Sqoop import from oracle to hive
Posted by Suresh Kumar Sethuramaswamy <ro...@gmail.com>.
Couple of things to check
1. Check if there is a hive-site.xml placed mistakenly in the current
working directory, if so remove it, else it might overwrite system config
2. Get the hive-metastore log for any clues while sqoop import happens
Regards
Suresh
On Tue, Jun 4, 2019, 7:16 AM K Makeev <k.makeev@gmail.com wrote:
> Hi
>
> Trying to import simple test table from oracle to hive using sqoop.
>
> Sqoop successfully fetches oracle data, puts them to HDFS, but when it
> comes to hive it doesn't seem to work. Sqoop writes no errors or warnings
> to output. It jus quits.
>
> Here is my command line
>
> sqoop import --connect jdbc:oracle:thin:@host:1521:SID --username
> username --password password --delete-target-dir --table TEST_SQOOP
> --hive-import --hive-database db -m 1 -verbose
>
> Here is the end of verbose output
>
> 19/06/03 18:28:02 DEBUG hive.TableDefWriter: Create statement: CREATE
> TABLE IF NOT EXISTS `db`.`TEST_SQOOP` ( `A` DOUBLE, `B` STRING, `D` STRING)
> COMMENT 'Imported by sqoop on 2019/06/03 18:28:02' ROW FORMAT DELIMITED
> FIELDS TERMINATED BY '\001' LINES TERMINATED BY '\012' STORED AS TEXTFILE
> 19/06/03 18:28:02 DEBUG hive.TableDefWriter: Load statement: LOAD DATA
> INPATH 'hdfs://hadoop/user/airflow/TEST_SQOOP' INTO TABLE `db`.`TEST_SQOOP`
> 19/06/03 18:28:02 DEBUG ipc.Client: The ping interval is 60000 ms.
> 19/06/03 18:28:02 DEBUG ipc.Client: Connecting to hadoop-m1/ip:8020
> 19/06/03 18:28:02 DEBUG ipc.Client: IPC Client (2071364564) connection to
> hadoop-m1/ip:8020 from airflow: starting, having connections 2
> 19/06/03 18:28:02 DEBUG ipc.Client: IPC Client (2071364564) connection to
> hadoop-m1/ip:8020 from airflow sending #779
> 19/06/03 18:28:02 DEBUG ipc.Client: IPC Client (2071364564) connection to
> hadoop-m1/ip:8020 from airflow got value #779
> 19/06/03 18:28:02 DEBUG ipc.ProtobufRpcEngine: Call: getFileInfo took 69ms
> 19/06/03 18:28:02 INFO hive.HiveImport: Loading uploaded data into Hive
> 19/06/03 18:28:02 DEBUG hive.HiveImport: Using in-process Hive instance.
> 19/06/03 18:28:02 DEBUG util.SubprocessSecurityManager: Installing
> subprocess security manager
>
> Logging initialized using configuration in
> jar:file:/usr/hdp/2.5.3.0-37/hive/lib/hive-common-1.2.1000.2.5.3.0-37.jar!/hive-log4j.properties
>
> Nothing happens in hive
>
> If i run hive manually through hive command i'm able to execute Create
> Table/LOAD DATA INPATH statements without problems.
>
> Why sqoop doesn't do its hive job?
>
>