You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Aaron Kimball (JIRA)" <ji...@apache.org> on 2009/05/21 23:23:45 UTC
[jira] Created: (HADOOP-5887) Sqoop should create tables in Hive
metastore after importing to HDFS
Sqoop should create tables in Hive metastore after importing to HDFS
--------------------------------------------------------------------
Key: HADOOP-5887
URL: https://issues.apache.org/jira/browse/HADOOP-5887
Project: Hadoop Core
Issue Type: New Feature
Reporter: Aaron Kimball
Assignee: Aaron Kimball
Attachments: HADOOP-5887.patch
Sqoop (HADOOP-5815) imports tables into HDFS; it is a straightforward enhancement to then generate a Hive DDL statement to recreate the table definition in the Hive metastore and move the imported table into the Hive warehouse directory from its upload target.
This feature enhancement makes this process automatic. An import is performed with sqoop in the usual way; providing the argument "--hive-import" will cause it to then issue a CREATE TABLE .. LOAD DATA INTO statement to a Hive shell. It generates a script file and then attempts to run "$HIVE_HOME/bin/hive" on it, or failing that, any "hive" on the $PATH; $HIVE_HOME can be overridden with --hive-home. As a result, no direct linking against Hive is necessary.
The unit tests provided with this enhancement use a mock implementation of 'bin/hive' that compares the script it's fed with one from a directory full of "expected" scripts. The exact script file referenced is controlled via an environment variable. It doesn't actually load into a proper Hive metastore, but manual testing has shown that this process works in practice, so the mock implementation is a reasonable unit testing tool.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5887) Sqoop should create tables in Hive
metastore after importing to HDFS
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated HADOOP-5887:
----------------------------------
Status: Patch Available (was: Open)
> Sqoop should create tables in Hive metastore after importing to HDFS
> --------------------------------------------------------------------
>
> Key: HADOOP-5887
> URL: https://issues.apache.org/jira/browse/HADOOP-5887
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: HADOOP-5887.patch
>
>
> Sqoop (HADOOP-5815) imports tables into HDFS; it is a straightforward enhancement to then generate a Hive DDL statement to recreate the table definition in the Hive metastore and move the imported table into the Hive warehouse directory from its upload target.
> This feature enhancement makes this process automatic. An import is performed with sqoop in the usual way; providing the argument "--hive-import" will cause it to then issue a CREATE TABLE .. LOAD DATA INTO statement to a Hive shell. It generates a script file and then attempts to run "$HIVE_HOME/bin/hive" on it, or failing that, any "hive" on the $PATH; $HIVE_HOME can be overridden with --hive-home. As a result, no direct linking against Hive is necessary.
> The unit tests provided with this enhancement use a mock implementation of 'bin/hive' that compares the script it's fed with one from a directory full of "expected" scripts. The exact script file referenced is controlled via an environment variable. It doesn't actually load into a proper Hive metastore, but manual testing has shown that this process works in practice, so the mock implementation is a reasonable unit testing tool.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5887) Sqoop should create tables in Hive
metastore after importing to HDFS
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated HADOOP-5887:
----------------------------------
Status: Patch Available (was: Open)
> Sqoop should create tables in Hive metastore after importing to HDFS
> --------------------------------------------------------------------
>
> Key: HADOOP-5887
> URL: https://issues.apache.org/jira/browse/HADOOP-5887
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: HADOOP-5887.2.patch, HADOOP-5887.patch
>
>
> Sqoop (HADOOP-5815) imports tables into HDFS; it is a straightforward enhancement to then generate a Hive DDL statement to recreate the table definition in the Hive metastore and move the imported table into the Hive warehouse directory from its upload target.
> This feature enhancement makes this process automatic. An import is performed with sqoop in the usual way; providing the argument "--hive-import" will cause it to then issue a CREATE TABLE .. LOAD DATA INTO statement to a Hive shell. It generates a script file and then attempts to run "$HIVE_HOME/bin/hive" on it, or failing that, any "hive" on the $PATH; $HIVE_HOME can be overridden with --hive-home. As a result, no direct linking against Hive is necessary.
> The unit tests provided with this enhancement use a mock implementation of 'bin/hive' that compares the script it's fed with one from a directory full of "expected" scripts. The exact script file referenced is controlled via an environment variable. It doesn't actually load into a proper Hive metastore, but manual testing has shown that this process works in practice, so the mock implementation is a reasonable unit testing tool.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5887) Sqoop should create tables in Hive
metastore after importing to HDFS
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12711853#action_12711853 ]
Aaron Kimball commented on HADOOP-5887:
---------------------------------------
True. Though in Hive, LOAD DATA INPATH is implemented as an in-HDFS move, not a copy, so it neither performs (much) faster nor saves space to do this. That having been said, a reasonable improvement for the future would be to add a flag to suppress the move into the "public" warehouse dir and leave it in the user's home directory. It'd be nice if any Hive mavens would comment on which use cases they use external vs. internal tables for. As I see it, there's not a huge amount of difference.
> Sqoop should create tables in Hive metastore after importing to HDFS
> --------------------------------------------------------------------
>
> Key: HADOOP-5887
> URL: https://issues.apache.org/jira/browse/HADOOP-5887
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: HADOOP-5887.patch
>
>
> Sqoop (HADOOP-5815) imports tables into HDFS; it is a straightforward enhancement to then generate a Hive DDL statement to recreate the table definition in the Hive metastore and move the imported table into the Hive warehouse directory from its upload target.
> This feature enhancement makes this process automatic. An import is performed with sqoop in the usual way; providing the argument "--hive-import" will cause it to then issue a CREATE TABLE .. LOAD DATA INTO statement to a Hive shell. It generates a script file and then attempts to run "$HIVE_HOME/bin/hive" on it, or failing that, any "hive" on the $PATH; $HIVE_HOME can be overridden with --hive-home. As a result, no direct linking against Hive is necessary.
> The unit tests provided with this enhancement use a mock implementation of 'bin/hive' that compares the script it's fed with one from a directory full of "expected" scripts. The exact script file referenced is controlled via an environment variable. It doesn't actually load into a proper Hive metastore, but manual testing has shown that this process works in practice, so the mock implementation is a reasonable unit testing tool.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5887) Sqoop should create tables in Hive
metastore after importing to HDFS
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White updated HADOOP-5887:
------------------------------
Resolution: Fixed
Fix Version/s: 0.21.0
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
+1
I've just committed this. Thanks Aaron!
> Sqoop should create tables in Hive metastore after importing to HDFS
> --------------------------------------------------------------------
>
> Key: HADOOP-5887
> URL: https://issues.apache.org/jira/browse/HADOOP-5887
> Project: Hadoop Common
> Issue Type: New Feature
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Fix For: 0.21.0
>
> Attachments: HADOOP-5887.2.patch, HADOOP-5887.patch
>
>
> Sqoop (HADOOP-5815) imports tables into HDFS; it is a straightforward enhancement to then generate a Hive DDL statement to recreate the table definition in the Hive metastore and move the imported table into the Hive warehouse directory from its upload target.
> This feature enhancement makes this process automatic. An import is performed with sqoop in the usual way; providing the argument "--hive-import" will cause it to then issue a CREATE TABLE .. LOAD DATA INTO statement to a Hive shell. It generates a script file and then attempts to run "$HIVE_HOME/bin/hive" on it, or failing that, any "hive" on the $PATH; $HIVE_HOME can be overridden with --hive-home. As a result, no direct linking against Hive is necessary.
> The unit tests provided with this enhancement use a mock implementation of 'bin/hive' that compares the script it's fed with one from a directory full of "expected" scripts. The exact script file referenced is controlled via an environment variable. It doesn't actually load into a proper Hive metastore, but manual testing has shown that this process works in practice, so the mock implementation is a reasonable unit testing tool.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5887) Sqoop should create tables in Hive
metastore after importing to HDFS
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12722011#action_12722011 ]
Aaron Kimball commented on HADOOP-5887:
---------------------------------------
Test failures are unrelated.
The four release audit warnings that this added should be added to the exceptions list in Hudson. Sqoop auto-generates Hive scripts (.q files). The scripts in the testdata/ directory are used as known-good results to compare the generated results against. As such, they can't easily include boilerplate headers.
> Sqoop should create tables in Hive metastore after importing to HDFS
> --------------------------------------------------------------------
>
> Key: HADOOP-5887
> URL: https://issues.apache.org/jira/browse/HADOOP-5887
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: HADOOP-5887.2.patch, HADOOP-5887.patch
>
>
> Sqoop (HADOOP-5815) imports tables into HDFS; it is a straightforward enhancement to then generate a Hive DDL statement to recreate the table definition in the Hive metastore and move the imported table into the Hive warehouse directory from its upload target.
> This feature enhancement makes this process automatic. An import is performed with sqoop in the usual way; providing the argument "--hive-import" will cause it to then issue a CREATE TABLE .. LOAD DATA INTO statement to a Hive shell. It generates a script file and then attempts to run "$HIVE_HOME/bin/hive" on it, or failing that, any "hive" on the $PATH; $HIVE_HOME can be overridden with --hive-home. As a result, no direct linking against Hive is necessary.
> The unit tests provided with this enhancement use a mock implementation of 'bin/hive' that compares the script it's fed with one from a directory full of "expected" scripts. The exact script file referenced is controlled via an environment variable. It doesn't actually load into a proper Hive metastore, but manual testing has shown that this process works in practice, so the mock implementation is a reasonable unit testing tool.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5887) Sqoop should create tables in Hive
metastore after importing to HDFS
Posted by "Amr Awadallah (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12711851#action_12711851 ]
Amr Awadallah commented on HADOOP-5887:
---------------------------------------
alternatively you can also just point to the original HDFS files as an external table and avoid the copy.
e.g.
CREATE EXTERNAL TABLE mytable( schema )
ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n'
STORED AS TEXTFILE
LOCATION '<hdfs_path>';
-- amr
> Sqoop should create tables in Hive metastore after importing to HDFS
> --------------------------------------------------------------------
>
> Key: HADOOP-5887
> URL: https://issues.apache.org/jira/browse/HADOOP-5887
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: HADOOP-5887.patch
>
>
> Sqoop (HADOOP-5815) imports tables into HDFS; it is a straightforward enhancement to then generate a Hive DDL statement to recreate the table definition in the Hive metastore and move the imported table into the Hive warehouse directory from its upload target.
> This feature enhancement makes this process automatic. An import is performed with sqoop in the usual way; providing the argument "--hive-import" will cause it to then issue a CREATE TABLE .. LOAD DATA INTO statement to a Hive shell. It generates a script file and then attempts to run "$HIVE_HOME/bin/hive" on it, or failing that, any "hive" on the $PATH; $HIVE_HOME can be overridden with --hive-home. As a result, no direct linking against Hive is necessary.
> The unit tests provided with this enhancement use a mock implementation of 'bin/hive' that compares the script it's fed with one from a directory full of "expected" scripts. The exact script file referenced is controlled via an environment variable. It doesn't actually load into a proper Hive metastore, but manual testing has shown that this process works in practice, so the mock implementation is a reasonable unit testing tool.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5887) Sqoop should create tables in Hive
metastore after importing to HDFS
Posted by "Prasad Chakka (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12711877#action_12711877 ]
Prasad Chakka commented on HADOOP-5887:
---------------------------------------
use external if the data is not movable or if data has to reside in non-default file system (NFS mounted or a different HDFS or S3 etc). in any other case, use internal tables.
> Sqoop should create tables in Hive metastore after importing to HDFS
> --------------------------------------------------------------------
>
> Key: HADOOP-5887
> URL: https://issues.apache.org/jira/browse/HADOOP-5887
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: HADOOP-5887.patch
>
>
> Sqoop (HADOOP-5815) imports tables into HDFS; it is a straightforward enhancement to then generate a Hive DDL statement to recreate the table definition in the Hive metastore and move the imported table into the Hive warehouse directory from its upload target.
> This feature enhancement makes this process automatic. An import is performed with sqoop in the usual way; providing the argument "--hive-import" will cause it to then issue a CREATE TABLE .. LOAD DATA INTO statement to a Hive shell. It generates a script file and then attempts to run "$HIVE_HOME/bin/hive" on it, or failing that, any "hive" on the $PATH; $HIVE_HOME can be overridden with --hive-home. As a result, no direct linking against Hive is necessary.
> The unit tests provided with this enhancement use a mock implementation of 'bin/hive' that compares the script it's fed with one from a directory full of "expected" scripts. The exact script file referenced is controlled via an environment variable. It doesn't actually load into a proper Hive metastore, but manual testing has shown that this process works in practice, so the mock implementation is a reasonable unit testing tool.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5887) Sqoop should create tables in Hive
metastore after importing to HDFS
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12712526#action_12712526 ]
Hadoop QA commented on HADOOP-5887:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12408740/HADOOP-5887.patch
against trunk revision 777761.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 19 new or modified tests.
-1 patch. The patch command could not apply the patch.
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/391/console
This message is automatically generated.
> Sqoop should create tables in Hive metastore after importing to HDFS
> --------------------------------------------------------------------
>
> Key: HADOOP-5887
> URL: https://issues.apache.org/jira/browse/HADOOP-5887
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: HADOOP-5887.patch
>
>
> Sqoop (HADOOP-5815) imports tables into HDFS; it is a straightforward enhancement to then generate a Hive DDL statement to recreate the table definition in the Hive metastore and move the imported table into the Hive warehouse directory from its upload target.
> This feature enhancement makes this process automatic. An import is performed with sqoop in the usual way; providing the argument "--hive-import" will cause it to then issue a CREATE TABLE .. LOAD DATA INTO statement to a Hive shell. It generates a script file and then attempts to run "$HIVE_HOME/bin/hive" on it, or failing that, any "hive" on the $PATH; $HIVE_HOME can be overridden with --hive-home. As a result, no direct linking against Hive is necessary.
> The unit tests provided with this enhancement use a mock implementation of 'bin/hive' that compares the script it's fed with one from a directory full of "expected" scripts. The exact script file referenced is controlled via an environment variable. It doesn't actually load into a proper Hive metastore, but manual testing has shown that this process works in practice, so the mock implementation is a reasonable unit testing tool.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5887) Sqoop should create tables in Hive
metastore after importing to HDFS
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated HADOOP-5887:
----------------------------------
Attachment: HADOOP-5887.patch
> Sqoop should create tables in Hive metastore after importing to HDFS
> --------------------------------------------------------------------
>
> Key: HADOOP-5887
> URL: https://issues.apache.org/jira/browse/HADOOP-5887
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: HADOOP-5887.patch
>
>
> Sqoop (HADOOP-5815) imports tables into HDFS; it is a straightforward enhancement to then generate a Hive DDL statement to recreate the table definition in the Hive metastore and move the imported table into the Hive warehouse directory from its upload target.
> This feature enhancement makes this process automatic. An import is performed with sqoop in the usual way; providing the argument "--hive-import" will cause it to then issue a CREATE TABLE .. LOAD DATA INTO statement to a Hive shell. It generates a script file and then attempts to run "$HIVE_HOME/bin/hive" on it, or failing that, any "hive" on the $PATH; $HIVE_HOME can be overridden with --hive-home. As a result, no direct linking against Hive is necessary.
> The unit tests provided with this enhancement use a mock implementation of 'bin/hive' that compares the script it's fed with one from a directory full of "expected" scripts. The exact script file referenced is controlled via an environment variable. It doesn't actually load into a proper Hive metastore, but manual testing has shown that this process works in practice, so the mock implementation is a reasonable unit testing tool.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5887) Sqoop should create tables in Hive
metastore after importing to HDFS
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12716771#action_12716771 ]
Hadoop QA commented on HADOOP-5887:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12409797/HADOOP-5887.2.patch
against trunk revision 782083.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 19 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 Eclipse classpath. The patch retains Eclipse classpath integrity.
-1 release audit. The applied patch generated 496 release audit warnings (more than the trunk's current 492 warnings).
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/469/testReport/
Release audit warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/469/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/469/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/469/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/469/console
This message is automatically generated.
> Sqoop should create tables in Hive metastore after importing to HDFS
> --------------------------------------------------------------------
>
> Key: HADOOP-5887
> URL: https://issues.apache.org/jira/browse/HADOOP-5887
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: HADOOP-5887.2.patch, HADOOP-5887.patch
>
>
> Sqoop (HADOOP-5815) imports tables into HDFS; it is a straightforward enhancement to then generate a Hive DDL statement to recreate the table definition in the Hive metastore and move the imported table into the Hive warehouse directory from its upload target.
> This feature enhancement makes this process automatic. An import is performed with sqoop in the usual way; providing the argument "--hive-import" will cause it to then issue a CREATE TABLE .. LOAD DATA INTO statement to a Hive shell. It generates a script file and then attempts to run "$HIVE_HOME/bin/hive" on it, or failing that, any "hive" on the $PATH; $HIVE_HOME can be overridden with --hive-home. As a result, no direct linking against Hive is necessary.
> The unit tests provided with this enhancement use a mock implementation of 'bin/hive' that compares the script it's fed with one from a directory full of "expected" scripts. The exact script file referenced is controlled via an environment variable. It doesn't actually load into a proper Hive metastore, but manual testing has shown that this process works in practice, so the mock implementation is a reasonable unit testing tool.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5887) Sqoop should create tables in Hive
metastore after importing to HDFS
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated HADOOP-5887:
----------------------------------
Attachment: HADOOP-5887.2.patch
New patch to handle merge conflicts with HADOOP-5844
> Sqoop should create tables in Hive metastore after importing to HDFS
> --------------------------------------------------------------------
>
> Key: HADOOP-5887
> URL: https://issues.apache.org/jira/browse/HADOOP-5887
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: HADOOP-5887.2.patch, HADOOP-5887.patch
>
>
> Sqoop (HADOOP-5815) imports tables into HDFS; it is a straightforward enhancement to then generate a Hive DDL statement to recreate the table definition in the Hive metastore and move the imported table into the Hive warehouse directory from its upload target.
> This feature enhancement makes this process automatic. An import is performed with sqoop in the usual way; providing the argument "--hive-import" will cause it to then issue a CREATE TABLE .. LOAD DATA INTO statement to a Hive shell. It generates a script file and then attempts to run "$HIVE_HOME/bin/hive" on it, or failing that, any "hive" on the $PATH; $HIVE_HOME can be overridden with --hive-home. As a result, no direct linking against Hive is necessary.
> The unit tests provided with this enhancement use a mock implementation of 'bin/hive' that compares the script it's fed with one from a directory full of "expected" scripts. The exact script file referenced is controlled via an environment variable. It doesn't actually load into a proper Hive metastore, but manual testing has shown that this process works in practice, so the mock implementation is a reasonable unit testing tool.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5887) Sqoop should create tables in Hive
metastore after importing to HDFS
Posted by "Aaron Kimball (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Kimball updated HADOOP-5887:
----------------------------------
Status: Open (was: Patch Available)
> Sqoop should create tables in Hive metastore after importing to HDFS
> --------------------------------------------------------------------
>
> Key: HADOOP-5887
> URL: https://issues.apache.org/jira/browse/HADOOP-5887
> Project: Hadoop Core
> Issue Type: New Feature
> Reporter: Aaron Kimball
> Assignee: Aaron Kimball
> Attachments: HADOOP-5887.2.patch, HADOOP-5887.patch
>
>
> Sqoop (HADOOP-5815) imports tables into HDFS; it is a straightforward enhancement to then generate a Hive DDL statement to recreate the table definition in the Hive metastore and move the imported table into the Hive warehouse directory from its upload target.
> This feature enhancement makes this process automatic. An import is performed with sqoop in the usual way; providing the argument "--hive-import" will cause it to then issue a CREATE TABLE .. LOAD DATA INTO statement to a Hive shell. It generates a script file and then attempts to run "$HIVE_HOME/bin/hive" on it, or failing that, any "hive" on the $PATH; $HIVE_HOME can be overridden with --hive-home. As a result, no direct linking against Hive is necessary.
> The unit tests provided with this enhancement use a mock implementation of 'bin/hive' that compares the script it's fed with one from a directory full of "expected" scripts. The exact script file referenced is controlled via an environment variable. It doesn't actually load into a proper Hive metastore, but manual testing has shown that this process works in practice, so the mock implementation is a reasonable unit testing tool.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.