You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sqoop.apache.org by "Porati Sébastien (Created JIRA)" <ji...@apache.org> on 2012/04/04 16:35:23 UTC
[jira] [Created] (SQOOP-475) Unable to import into external hive
table located on S3
Unable to import into external hive table located on S3
-------------------------------------------------------
Key: SQOOP-475
URL: https://issues.apache.org/jira/browse/SQOOP-475
Project: Sqoop
Issue Type: Bug
Components: hive-integration
Affects Versions: 1.4.1-incubating
Environment: Amazon EMR
Hadoop 0.20.205
Hive 0.7.1
Sqoop 1.4.1-incubating
Reporter: Porati Sébastien
When i try to import into an hive table located on an S3 bucket, i got the following error message :
FAILED: Error in semantic analysis: Line 2:17 Path is not legal 'hdfs://10.48.189.XX:9000/user/hadoop/client': Move from: hdfs://10.48.189.XX:9000/user/hadoop/client to: s3://some-bucket/sqoop-test/hive/client is not valid. Please check that values for params "default.fs.name" and "hive.metastore.warehouse.dir" do not conflict.
Hive table creation Script :
CREATE DATABASE IF NOT EXISTS sqoop_test;
USE sqoop_test;
CREATE EXTERNAL TABLE IF NOT EXISTS client (
id INT,
email STRING,
cookie_uid STRING,
is_blacklisted TINYINT
)
LOCATION 's3://some-bucket/sqoop-test/hive/client';
Sqoop command :
sqoop import --connect jdbc:mysql://my.domain.com/mydb --username myuser --password XXXX --table client --hive-import --hive-overwrite --hive-table sqoop_test.client
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-475) Unable to import into external hive
table located on S3
Posted by "Georgy B Abraham (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402187#comment-13402187 ]
Georgy B Abraham commented on SQOOP-475:
----------------------------------------
the options.setHiveInput(true) is not commented then only the hive import option is selected then this error occoured
> Unable to import into external hive table located on S3
> -------------------------------------------------------
>
> Key: SQOOP-475
> URL: https://issues.apache.org/jira/browse/SQOOP-475
> Project: Sqoop
> Issue Type: Bug
> Components: hive-integration
> Affects Versions: 1.4.1-incubating
> Environment: Amazon EMR
> Hadoop 0.20.205
> Hive 0.7.1
> Sqoop 1.4.1-incubating
> Reporter: Porati Sébastien
>
> When i try to import into an hive table located on an S3 bucket, i got the following error message :
> FAILED: Error in semantic analysis: Line 2:17 Path is not legal 'hdfs://10.48.189.XX:9000/user/hadoop/client': Move from: hdfs://10.48.189.XX:9000/user/hadoop/client to: s3://some-bucket/sqoop-test/hive/client is not valid. Please check that values for params "default.fs.name" and "hive.metastore.warehouse.dir" do not conflict.
> Hive table creation Script :
> CREATE DATABASE IF NOT EXISTS sqoop_test;
> USE sqoop_test;
> CREATE EXTERNAL TABLE IF NOT EXISTS client (
> id INT,
> email STRING,
> cookie_uid STRING,
> is_blacklisted TINYINT
> )
> LOCATION 's3://some-bucket/sqoop-test/hive/client';
> Sqoop command :
> sqoop import --connect jdbc:mysql://my.domain.com/mydb --username myuser --password XXXX --table client --hive-import --hive-overwrite --hive-table sqoop_test.client
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-475) Unable to import into external hive
table located on S3
Posted by "Jarek Jarcec Cecho (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13422373#comment-13422373 ]
Jarek Jarcec Cecho commented on SQOOP-475:
------------------------------------------
Hi sir,
I'm afraid that you're using Sqoop in not supported use case and therefore you're more or less on your own with the question how to get those two paths in sync.
But based on the error message, it seems to me that it's hive that is using the local disk instead of HDFS. Your hive-site.xml seems to be correct, so the question is - are you sure that it gets properly loaded by hive during sqoop import?
Jarcec
> Unable to import into external hive table located on S3
> -------------------------------------------------------
>
> Key: SQOOP-475
> URL: https://issues.apache.org/jira/browse/SQOOP-475
> Project: Sqoop
> Issue Type: Bug
> Components: hive-integration
> Affects Versions: 1.4.1-incubating
> Environment: Amazon EMR
> Hadoop 0.20.205
> Hive 0.7.1
> Sqoop 1.4.1-incubating
> Reporter: Porati Sébastien
>
> When i try to import into an hive table located on an S3 bucket, i got the following error message :
> FAILED: Error in semantic analysis: Line 2:17 Path is not legal 'hdfs://10.48.189.XX:9000/user/hadoop/client': Move from: hdfs://10.48.189.XX:9000/user/hadoop/client to: s3://some-bucket/sqoop-test/hive/client is not valid. Please check that values for params "default.fs.name" and "hive.metastore.warehouse.dir" do not conflict.
> Hive table creation Script :
> CREATE DATABASE IF NOT EXISTS sqoop_test;
> USE sqoop_test;
> CREATE EXTERNAL TABLE IF NOT EXISTS client (
> id INT,
> email STRING,
> cookie_uid STRING,
> is_blacklisted TINYINT
> )
> LOCATION 's3://some-bucket/sqoop-test/hive/client';
> Sqoop command :
> sqoop import --connect jdbc:mysql://my.domain.com/mydb --username myuser --password XXXX --table client --hive-import --hive-overwrite --hive-table sqoop_test.client
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-475) Unable to import into external hive
table located on S3
Posted by "Georgy B Abraham (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402202#comment-13402202 ]
Georgy B Abraham commented on SQOOP-475:
----------------------------------------
Sqoop has the function to directly import into a hive table when running in the terminal and all is fine there . But when trying from eclipse the file is imported into the package as a temp file(in eclipse workspace) and then only its moving to hive but while moving this error strikes ...
> Unable to import into external hive table located on S3
> -------------------------------------------------------
>
> Key: SQOOP-475
> URL: https://issues.apache.org/jira/browse/SQOOP-475
> Project: Sqoop
> Issue Type: Bug
> Components: hive-integration
> Affects Versions: 1.4.1-incubating
> Environment: Amazon EMR
> Hadoop 0.20.205
> Hive 0.7.1
> Sqoop 1.4.1-incubating
> Reporter: Porati Sébastien
>
> When i try to import into an hive table located on an S3 bucket, i got the following error message :
> FAILED: Error in semantic analysis: Line 2:17 Path is not legal 'hdfs://10.48.189.XX:9000/user/hadoop/client': Move from: hdfs://10.48.189.XX:9000/user/hadoop/client to: s3://some-bucket/sqoop-test/hive/client is not valid. Please check that values for params "default.fs.name" and "hive.metastore.warehouse.dir" do not conflict.
> Hive table creation Script :
> CREATE DATABASE IF NOT EXISTS sqoop_test;
> USE sqoop_test;
> CREATE EXTERNAL TABLE IF NOT EXISTS client (
> id INT,
> email STRING,
> cookie_uid STRING,
> is_blacklisted TINYINT
> )
> LOCATION 's3://some-bucket/sqoop-test/hive/client';
> Sqoop command :
> sqoop import --connect jdbc:mysql://my.domain.com/mydb --username myuser --password XXXX --table client --hive-import --hive-overwrite --hive-table sqoop_test.client
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-475) Unable to import into external hive
table located on S3
Posted by "sachin pawar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13416458#comment-13416458 ]
sachin pawar commented on SQOOP-475:
------------------------------------
Hi,
I am also facing the same issue..can you please help..
let me know if you need more info
i am running following code
String sqoopArgs[] = new String[]
{
"-D", "fs.default.name=hdfs://<hostname>:8020/",
"-D", "mapred.job.tracker=<hostname>:8021",
"-D", "hive.metastore.warehouse.dir=hdfs://<hostname>:8020/user/hive/warehouse",
"-D", "hadoop.security.authorization=false",
"-D", "dfs.replication=1",
"-D", "dfs.data.dir=/home/sacpawar/hadoop_storage/data",
"-D", "dfs.name.dir=/home/sacpawar/hadoop_storage/name",
"-D", "hadoop.tmp.dir=/home/haiyee/hadoop_storage/tmp",
"-D", "mapreduce.jobtracker.staging.root.dir=/user/sacpawar/",
"--connect", "jdbc:sqlserver://<dbhostname>:1433;databaseName=SqoopTestDB",
"--username", "sa",
"--password", "sa",
"--table", "bizunit",
"--hive-home", "/opt/dev/hive-0.9.0",
"--hive-import"
};
// ====================================
ImportTool iTool = new ImportTool();
Sqoop sqoop = new Sqoop(iTool);
try {
System.out.println("Started....");
ToolRunner.run(sqoop, sqoopArgs);
System.out.println("=======DONE==========");
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
standalone hadoop
hadoop sqoop and hive installed on same box
HADOOP_HOME, SQOOP_HOME, HIVE_HOME set to respective install dirs
no hive-site.xml is used ..using default derby...
I can run following form command line on hadoop/sqoop/hive machine and it ran fine and created table
sqoop create-hive-table --connect "jdbc:sqlserver://<dbhostname>:1433;databaseName=SqoopTestDB" --username sa --password sa --table bizunit --hive-table bizunit_test
trying this code from different machine in the network...provided all the required jars
data is loaded in HDFS through sqoop import process fine.
end up getting following error during hive import process
FAILED: Error in semantic analysis: Line 2:17 Path is not legal ''hdfs://172.25.6.96:8020/user/sacpawar/bizunit'':
Move from: hdfs://172.25.6.96:8020/user/sacpawar/bizunit to: file:/user/hive/warehouse/bizunit is not valid.
Please check that values for params "default.fs.name" and "hive.metastore.warehouse.dir" do not conflict.
please help!!!!
Sachin
> Unable to import into external hive table located on S3
> -------------------------------------------------------
>
> Key: SQOOP-475
> URL: https://issues.apache.org/jira/browse/SQOOP-475
> Project: Sqoop
> Issue Type: Bug
> Components: hive-integration
> Affects Versions: 1.4.1-incubating
> Environment: Amazon EMR
> Hadoop 0.20.205
> Hive 0.7.1
> Sqoop 1.4.1-incubating
> Reporter: Porati Sébastien
>
> When i try to import into an hive table located on an S3 bucket, i got the following error message :
> FAILED: Error in semantic analysis: Line 2:17 Path is not legal 'hdfs://10.48.189.XX:9000/user/hadoop/client': Move from: hdfs://10.48.189.XX:9000/user/hadoop/client to: s3://some-bucket/sqoop-test/hive/client is not valid. Please check that values for params "default.fs.name" and "hive.metastore.warehouse.dir" do not conflict.
> Hive table creation Script :
> CREATE DATABASE IF NOT EXISTS sqoop_test;
> USE sqoop_test;
> CREATE EXTERNAL TABLE IF NOT EXISTS client (
> id INT,
> email STRING,
> cookie_uid STRING,
> is_blacklisted TINYINT
> )
> LOCATION 's3://some-bucket/sqoop-test/hive/client';
> Sqoop command :
> sqoop import --connect jdbc:mysql://my.domain.com/mydb --username myuser --password XXXX --table client --hive-import --hive-overwrite --hive-table sqoop_test.client
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-475) Unable to import into external hive
table located on S3
Posted by "Georgy B Abraham (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13406293#comment-13406293 ]
Georgy B Abraham commented on SQOOP-475:
----------------------------------------
Hi Jarek,
Yes .. i changed the hive metastore to the package where the java pgm is executed still dint work
> Unable to import into external hive table located on S3
> -------------------------------------------------------
>
> Key: SQOOP-475
> URL: https://issues.apache.org/jira/browse/SQOOP-475
> Project: Sqoop
> Issue Type: Bug
> Components: hive-integration
> Affects Versions: 1.4.1-incubating
> Environment: Amazon EMR
> Hadoop 0.20.205
> Hive 0.7.1
> Sqoop 1.4.1-incubating
> Reporter: Porati Sébastien
>
> When i try to import into an hive table located on an S3 bucket, i got the following error message :
> FAILED: Error in semantic analysis: Line 2:17 Path is not legal 'hdfs://10.48.189.XX:9000/user/hadoop/client': Move from: hdfs://10.48.189.XX:9000/user/hadoop/client to: s3://some-bucket/sqoop-test/hive/client is not valid. Please check that values for params "default.fs.name" and "hive.metastore.warehouse.dir" do not conflict.
> Hive table creation Script :
> CREATE DATABASE IF NOT EXISTS sqoop_test;
> USE sqoop_test;
> CREATE EXTERNAL TABLE IF NOT EXISTS client (
> id INT,
> email STRING,
> cookie_uid STRING,
> is_blacklisted TINYINT
> )
> LOCATION 's3://some-bucket/sqoop-test/hive/client';
> Sqoop command :
> sqoop import --connect jdbc:mysql://my.domain.com/mydb --username myuser --password XXXX --table client --hive-import --hive-overwrite --hive-table sqoop_test.client
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-475) Unable to import into external hive
table located on S3
Posted by "Afsal Thaj (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402822#comment-13402822 ]
Afsal Thaj commented on SQOOP-475:
----------------------------------
I have done it using another method.Following was the code line used to execute the sqoop command from a java client program
Sqoop.runTool(new String[]{"import","--connect","jdbc:mysql://localhost/dbname","--username","root","--password","****","--table","tablename","-m","1","--hive-import","--hive-home","/home/549570/hive-0.7.1"})
And I expected the temporary file to be created in hdfs but it got transfered temorarily to eclipse workspace.Later the transfer from eclipse workspace to hdfs://localhost:9000/hive/warehouse/tablename is said to be invalid.
Is there any possibility I can have this temporary file in hdfs???
The intention is to use Sqoop in a client program.We are not facing any issue in transfering data to hdfs but problem arise only when we try to import to hive warehouse as table.
> Unable to import into external hive table located on S3
> -------------------------------------------------------
>
> Key: SQOOP-475
> URL: https://issues.apache.org/jira/browse/SQOOP-475
> Project: Sqoop
> Issue Type: Bug
> Components: hive-integration
> Affects Versions: 1.4.1-incubating
> Environment: Amazon EMR
> Hadoop 0.20.205
> Hive 0.7.1
> Sqoop 1.4.1-incubating
> Reporter: Porati Sébastien
>
> When i try to import into an hive table located on an S3 bucket, i got the following error message :
> FAILED: Error in semantic analysis: Line 2:17 Path is not legal 'hdfs://10.48.189.XX:9000/user/hadoop/client': Move from: hdfs://10.48.189.XX:9000/user/hadoop/client to: s3://some-bucket/sqoop-test/hive/client is not valid. Please check that values for params "default.fs.name" and "hive.metastore.warehouse.dir" do not conflict.
> Hive table creation Script :
> CREATE DATABASE IF NOT EXISTS sqoop_test;
> USE sqoop_test;
> CREATE EXTERNAL TABLE IF NOT EXISTS client (
> id INT,
> email STRING,
> cookie_uid STRING,
> is_blacklisted TINYINT
> )
> LOCATION 's3://some-bucket/sqoop-test/hive/client';
> Sqoop command :
> sqoop import --connect jdbc:mysql://my.domain.com/mydb --username myuser --password XXXX --table client --hive-import --hive-overwrite --hive-table sqoop_test.client
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-475) Unable to import into external hive
table located on S3
Posted by "Georgy B Abraham (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402849#comment-13402849 ]
Georgy B Abraham commented on SQOOP-475:
----------------------------------------
Hi Jarek ,
Did the thing by first importing the table to hdfs
then by another command of create hive table
and then again by jdbc connection to hive imported the data into the table
and atlast the table in mysql was in HIVE using Java
Thanks a lot from me and Afsal
> Unable to import into external hive table located on S3
> -------------------------------------------------------
>
> Key: SQOOP-475
> URL: https://issues.apache.org/jira/browse/SQOOP-475
> Project: Sqoop
> Issue Type: Bug
> Components: hive-integration
> Affects Versions: 1.4.1-incubating
> Environment: Amazon EMR
> Hadoop 0.20.205
> Hive 0.7.1
> Sqoop 1.4.1-incubating
> Reporter: Porati Sébastien
>
> When i try to import into an hive table located on an S3 bucket, i got the following error message :
> FAILED: Error in semantic analysis: Line 2:17 Path is not legal 'hdfs://10.48.189.XX:9000/user/hadoop/client': Move from: hdfs://10.48.189.XX:9000/user/hadoop/client to: s3://some-bucket/sqoop-test/hive/client is not valid. Please check that values for params "default.fs.name" and "hive.metastore.warehouse.dir" do not conflict.
> Hive table creation Script :
> CREATE DATABASE IF NOT EXISTS sqoop_test;
> USE sqoop_test;
> CREATE EXTERNAL TABLE IF NOT EXISTS client (
> id INT,
> email STRING,
> cookie_uid STRING,
> is_blacklisted TINYINT
> )
> LOCATION 's3://some-bucket/sqoop-test/hive/client';
> Sqoop command :
> sqoop import --connect jdbc:mysql://my.domain.com/mydb --username myuser --password XXXX --table client --hive-import --hive-overwrite --hive-table sqoop_test.client
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-475) Unable to import into external hive
table located on S3
Posted by "Jarek Jarcec Cecho (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13491504#comment-13491504 ]
Jarek Jarcec Cecho commented on SQOOP-475:
------------------------------------------
Hi Shirley,
I'm afraid that using hive import is officially not support from within Oozie. Recommended workaround is to firstly get data to HDFS using Sqoop in one Sqoop action followed by second Hive action that will load the data to Hive.
About your particular issue with port appended - Sqoop is relying on underlying hadoop configuration files, so I would recommend to verify your configuration (oozie namenode tag, hadoop configuration files, ...).
Jarcec
> Unable to import into external hive table located on S3
> -------------------------------------------------------
>
> Key: SQOOP-475
> URL: https://issues.apache.org/jira/browse/SQOOP-475
> Project: Sqoop
> Issue Type: Bug
> Components: hive-integration
> Affects Versions: 1.4.1-incubating
> Environment: Amazon EMR
> Hadoop 0.20.205
> Hive 0.7.1
> Sqoop 1.4.1-incubating
> Reporter: Porati Sébastien
>
> When i try to import into an hive table located on an S3 bucket, i got the following error message :
> FAILED: Error in semantic analysis: Line 2:17 Path is not legal 'hdfs://10.48.189.XX:9000/user/hadoop/client': Move from: hdfs://10.48.189.XX:9000/user/hadoop/client to: s3://some-bucket/sqoop-test/hive/client is not valid. Please check that values for params "default.fs.name" and "hive.metastore.warehouse.dir" do not conflict.
> Hive table creation Script :
> CREATE DATABASE IF NOT EXISTS sqoop_test;
> USE sqoop_test;
> CREATE EXTERNAL TABLE IF NOT EXISTS client (
> id INT,
> email STRING,
> cookie_uid STRING,
> is_blacklisted TINYINT
> )
> LOCATION 's3://some-bucket/sqoop-test/hive/client';
> Sqoop command :
> sqoop import --connect jdbc:mysql://my.domain.com/mydb --username myuser --password XXXX --table client --hive-import --hive-overwrite --hive-table sqoop_test.client
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-475) Unable to import into external hive
table located on S3
Posted by "Georgy B Abraham (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402166#comment-13402166 ]
Georgy B Abraham commented on SQOOP-475:
----------------------------------------
@Jarek i am a colleague of afsal ,
We were just trying to import a table from mysql Db to hive by writing the code in java. the pgm could fetch the data from mysql into the eclipse's package. the java file generated by sqoop is also stored in the package but the pgm couldnt write the table into hive ..
when using command line everything is ok , but when we use java it shows the error mentioned above
the bit of code we used
@SuppressWarnings("deprecation")
com.cloudera.sqoop.SqoopOptions options = new com.cloudera.sqoop.SqoopOptions();
options.setConnectString("jdbc:mysql://localhost/sebin");
options.setTableName("afsal");
options.setUsername("root");
options.setPassword("6397");
System.out.println(options.getConf());
options.setHadoopHome("/home/556940/hadoop-0.20.2-cdh3u2/bin");
System.out.println(options.getHadoopHome());
options.setNumMappers(1);
options.setClearStagingTable(true);
//options.setHiveImport(true);
System.out.println(options.getHadoopHome());
String targetDir="afsal";
//options.setTargetDir("hdfs://localhost:9000/user/hive/warehouse/"+targetDir);
options.setHiveHome("/home/556940/hive-0.7.1/");
System.out.println(options.doHiveImport());
System.out.println(options.getConf());
new ImportTool().run(options);
> Unable to import into external hive table located on S3
> -------------------------------------------------------
>
> Key: SQOOP-475
> URL: https://issues.apache.org/jira/browse/SQOOP-475
> Project: Sqoop
> Issue Type: Bug
> Components: hive-integration
> Affects Versions: 1.4.1-incubating
> Environment: Amazon EMR
> Hadoop 0.20.205
> Hive 0.7.1
> Sqoop 1.4.1-incubating
> Reporter: Porati Sébastien
>
> When i try to import into an hive table located on an S3 bucket, i got the following error message :
> FAILED: Error in semantic analysis: Line 2:17 Path is not legal 'hdfs://10.48.189.XX:9000/user/hadoop/client': Move from: hdfs://10.48.189.XX:9000/user/hadoop/client to: s3://some-bucket/sqoop-test/hive/client is not valid. Please check that values for params "default.fs.name" and "hive.metastore.warehouse.dir" do not conflict.
> Hive table creation Script :
> CREATE DATABASE IF NOT EXISTS sqoop_test;
> USE sqoop_test;
> CREATE EXTERNAL TABLE IF NOT EXISTS client (
> id INT,
> email STRING,
> cookie_uid STRING,
> is_blacklisted TINYINT
> )
> LOCATION 's3://some-bucket/sqoop-test/hive/client';
> Sqoop command :
> sqoop import --connect jdbc:mysql://my.domain.com/mydb --username myuser --password XXXX --table client --hive-import --hive-overwrite --hive-table sqoop_test.client
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-475) Unable to import into external hive
table located on S3
Posted by "Afsal Thaj (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402821#comment-13402821 ]
Afsal Thaj commented on SQOOP-475:
----------------------------------
I have done it using another method.Following was the code line used to execute the sqoop command from a java client program
Sqoop.runTool(new String[]{"import","--connect","jdbc:mysql://localhost/dbname","--username","root","--password","****","--table","tablename","-m","1","--hive-import","--hive-home","/home/549570/hive-0.7.1"})
And I expected the temporary file to be created in hdfs but it got transfered temorarily to eclipse workspace.Later the transfer from eclipse workspace to hdfs://localhost:9000/hive/warehouse/tablename is said to be invalid.
Is there any possibility I can have this temporary file in hdfs???
The intention is to use Sqoop in a client program.We are not facing any issue in transfering data to hdfs but problem arise only when we to import to hive warehouse as table.
> Unable to import into external hive table located on S3
> -------------------------------------------------------
>
> Key: SQOOP-475
> URL: https://issues.apache.org/jira/browse/SQOOP-475
> Project: Sqoop
> Issue Type: Bug
> Components: hive-integration
> Affects Versions: 1.4.1-incubating
> Environment: Amazon EMR
> Hadoop 0.20.205
> Hive 0.7.1
> Sqoop 1.4.1-incubating
> Reporter: Porati Sébastien
>
> When i try to import into an hive table located on an S3 bucket, i got the following error message :
> FAILED: Error in semantic analysis: Line 2:17 Path is not legal 'hdfs://10.48.189.XX:9000/user/hadoop/client': Move from: hdfs://10.48.189.XX:9000/user/hadoop/client to: s3://some-bucket/sqoop-test/hive/client is not valid. Please check that values for params "default.fs.name" and "hive.metastore.warehouse.dir" do not conflict.
> Hive table creation Script :
> CREATE DATABASE IF NOT EXISTS sqoop_test;
> USE sqoop_test;
> CREATE EXTERNAL TABLE IF NOT EXISTS client (
> id INT,
> email STRING,
> cookie_uid STRING,
> is_blacklisted TINYINT
> )
> LOCATION 's3://some-bucket/sqoop-test/hive/client';
> Sqoop command :
> sqoop import --connect jdbc:mysql://my.domain.com/mydb --username myuser --password XXXX --table client --hive-import --hive-overwrite --hive-table sqoop_test.client
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-475) Unable to import into external hive
table located on S3
Posted by "Jarek Jarcec Cecho (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13406290#comment-13406290 ]
Jarek Jarcec Cecho commented on SQOOP-475:
------------------------------------------
But you do have the local "cluster" running on your local box right?
Jarcec
> Unable to import into external hive table located on S3
> -------------------------------------------------------
>
> Key: SQOOP-475
> URL: https://issues.apache.org/jira/browse/SQOOP-475
> Project: Sqoop
> Issue Type: Bug
> Components: hive-integration
> Affects Versions: 1.4.1-incubating
> Environment: Amazon EMR
> Hadoop 0.20.205
> Hive 0.7.1
> Sqoop 1.4.1-incubating
> Reporter: Porati Sébastien
>
> When i try to import into an hive table located on an S3 bucket, i got the following error message :
> FAILED: Error in semantic analysis: Line 2:17 Path is not legal 'hdfs://10.48.189.XX:9000/user/hadoop/client': Move from: hdfs://10.48.189.XX:9000/user/hadoop/client to: s3://some-bucket/sqoop-test/hive/client is not valid. Please check that values for params "default.fs.name" and "hive.metastore.warehouse.dir" do not conflict.
> Hive table creation Script :
> CREATE DATABASE IF NOT EXISTS sqoop_test;
> USE sqoop_test;
> CREATE EXTERNAL TABLE IF NOT EXISTS client (
> id INT,
> email STRING,
> cookie_uid STRING,
> is_blacklisted TINYINT
> )
> LOCATION 's3://some-bucket/sqoop-test/hive/client';
> Sqoop command :
> sqoop import --connect jdbc:mysql://my.domain.com/mydb --username myuser --password XXXX --table client --hive-import --hive-overwrite --hive-table sqoop_test.client
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-475) Unable to import into external hive
table located on S3
Posted by "Afsal Thaj (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13406286#comment-13406286 ]
Afsal Thaj commented on SQOOP-475:
----------------------------------
Hi Jarek,
I tried it in the same way as georgy mentioned.I used create-hive-table tool from eclipse and the table directory is created in hive/warehouse/ but with no schema.Running "show tables" in hive doesnot show this table in the list of tables.The metadata.db is created in workspace itself.What could be the possible solution for this?
> Unable to import into external hive table located on S3
> -------------------------------------------------------
>
> Key: SQOOP-475
> URL: https://issues.apache.org/jira/browse/SQOOP-475
> Project: Sqoop
> Issue Type: Bug
> Components: hive-integration
> Affects Versions: 1.4.1-incubating
> Environment: Amazon EMR
> Hadoop 0.20.205
> Hive 0.7.1
> Sqoop 1.4.1-incubating
> Reporter: Porati Sébastien
>
> When i try to import into an hive table located on an S3 bucket, i got the following error message :
> FAILED: Error in semantic analysis: Line 2:17 Path is not legal 'hdfs://10.48.189.XX:9000/user/hadoop/client': Move from: hdfs://10.48.189.XX:9000/user/hadoop/client to: s3://some-bucket/sqoop-test/hive/client is not valid. Please check that values for params "default.fs.name" and "hive.metastore.warehouse.dir" do not conflict.
> Hive table creation Script :
> CREATE DATABASE IF NOT EXISTS sqoop_test;
> USE sqoop_test;
> CREATE EXTERNAL TABLE IF NOT EXISTS client (
> id INT,
> email STRING,
> cookie_uid STRING,
> is_blacklisted TINYINT
> )
> LOCATION 's3://some-bucket/sqoop-test/hive/client';
> Sqoop command :
> sqoop import --connect jdbc:mysql://my.domain.com/mydb --username myuser --password XXXX --table client --hive-import --hive-overwrite --hive-table sqoop_test.client
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-475) Unable to import into external hive
table located on S3
Posted by "Jarek Jarcec Cecho (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402122#comment-13402122 ]
Jarek Jarcec Cecho commented on SQOOP-475:
------------------------------------------
Generally speaking, this errors occurs in HIVE in case that you're trying to load files from different HDFS than the HIVE is using or when there is discrepancy in configuration somewhere.
Afsal, could you describe your environment with sqoop command that you're trying to execute?
Jarcec
> Unable to import into external hive table located on S3
> -------------------------------------------------------
>
> Key: SQOOP-475
> URL: https://issues.apache.org/jira/browse/SQOOP-475
> Project: Sqoop
> Issue Type: Bug
> Components: hive-integration
> Affects Versions: 1.4.1-incubating
> Environment: Amazon EMR
> Hadoop 0.20.205
> Hive 0.7.1
> Sqoop 1.4.1-incubating
> Reporter: Porati Sébastien
>
> When i try to import into an hive table located on an S3 bucket, i got the following error message :
> FAILED: Error in semantic analysis: Line 2:17 Path is not legal 'hdfs://10.48.189.XX:9000/user/hadoop/client': Move from: hdfs://10.48.189.XX:9000/user/hadoop/client to: s3://some-bucket/sqoop-test/hive/client is not valid. Please check that values for params "default.fs.name" and "hive.metastore.warehouse.dir" do not conflict.
> Hive table creation Script :
> CREATE DATABASE IF NOT EXISTS sqoop_test;
> USE sqoop_test;
> CREATE EXTERNAL TABLE IF NOT EXISTS client (
> id INT,
> email STRING,
> cookie_uid STRING,
> is_blacklisted TINYINT
> )
> LOCATION 's3://some-bucket/sqoop-test/hive/client';
> Sqoop command :
> sqoop import --connect jdbc:mysql://my.domain.com/mydb --username myuser --password XXXX --table client --hive-import --hive-overwrite --hive-table sqoop_test.client
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (SQOOP-475) Unable to import into external hive
table located on S3
Posted by "Afsal Thaj (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Afsal Thaj updated SQOOP-475:
-----------------------------
Comment: was deleted
(was: I have done it using another method.Following was the code line used to execute the sqoop command from a java client program
Sqoop.runTool(new String[]{"import","--connect","jdbc:mysql://localhost/dbname","--username","root","--password","****","--table","tablename","-m","1","--hive-import","--hive-home","/home/549570/hive-0.7.1"})
And I expected the temporary file to be created in hdfs but it got transfered temorarily to eclipse workspace.Later the transfer from eclipse workspace to hdfs://localhost:9000/hive/warehouse/tablename is said to be invalid.
Is there any possibility I can have this temporary file in hdfs???
The intention is to use Sqoop in a client program.We are not facing any issue in transfering data to hdfs but problem arise only when we to import to hive warehouse as table.)
> Unable to import into external hive table located on S3
> -------------------------------------------------------
>
> Key: SQOOP-475
> URL: https://issues.apache.org/jira/browse/SQOOP-475
> Project: Sqoop
> Issue Type: Bug
> Components: hive-integration
> Affects Versions: 1.4.1-incubating
> Environment: Amazon EMR
> Hadoop 0.20.205
> Hive 0.7.1
> Sqoop 1.4.1-incubating
> Reporter: Porati Sébastien
>
> When i try to import into an hive table located on an S3 bucket, i got the following error message :
> FAILED: Error in semantic analysis: Line 2:17 Path is not legal 'hdfs://10.48.189.XX:9000/user/hadoop/client': Move from: hdfs://10.48.189.XX:9000/user/hadoop/client to: s3://some-bucket/sqoop-test/hive/client is not valid. Please check that values for params "default.fs.name" and "hive.metastore.warehouse.dir" do not conflict.
> Hive table creation Script :
> CREATE DATABASE IF NOT EXISTS sqoop_test;
> USE sqoop_test;
> CREATE EXTERNAL TABLE IF NOT EXISTS client (
> id INT,
> email STRING,
> cookie_uid STRING,
> is_blacklisted TINYINT
> )
> LOCATION 's3://some-bucket/sqoop-test/hive/client';
> Sqoop command :
> sqoop import --connect jdbc:mysql://my.domain.com/mydb --username myuser --password XXXX --table client --hive-import --hive-overwrite --hive-table sqoop_test.client
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-475) Unable to import into external hive
table located on S3
Posted by "Jarek Jarcec Cecho (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402224#comment-13402224 ]
Jarek Jarcec Cecho commented on SQOOP-475:
------------------------------------------
Sqoop is not supporting direct HIVE import (we were never supporting that btw). We are firstly importing data to temporal directory on HDFS and then using HIVE standard queries (LOAD DATA) we're moving data into appropriate hive folders. This behavior is intentional and is causing your issue as you seem to be doing import to local disk whereas your hive is configured for HDFS.
Jarcec
> Unable to import into external hive table located on S3
> -------------------------------------------------------
>
> Key: SQOOP-475
> URL: https://issues.apache.org/jira/browse/SQOOP-475
> Project: Sqoop
> Issue Type: Bug
> Components: hive-integration
> Affects Versions: 1.4.1-incubating
> Environment: Amazon EMR
> Hadoop 0.20.205
> Hive 0.7.1
> Sqoop 1.4.1-incubating
> Reporter: Porati Sébastien
>
> When i try to import into an hive table located on an S3 bucket, i got the following error message :
> FAILED: Error in semantic analysis: Line 2:17 Path is not legal 'hdfs://10.48.189.XX:9000/user/hadoop/client': Move from: hdfs://10.48.189.XX:9000/user/hadoop/client to: s3://some-bucket/sqoop-test/hive/client is not valid. Please check that values for params "default.fs.name" and "hive.metastore.warehouse.dir" do not conflict.
> Hive table creation Script :
> CREATE DATABASE IF NOT EXISTS sqoop_test;
> USE sqoop_test;
> CREATE EXTERNAL TABLE IF NOT EXISTS client (
> id INT,
> email STRING,
> cookie_uid STRING,
> is_blacklisted TINYINT
> )
> LOCATION 's3://some-bucket/sqoop-test/hive/client';
> Sqoop command :
> sqoop import --connect jdbc:mysql://my.domain.com/mydb --username myuser --password XXXX --table client --hive-import --hive-overwrite --hive-table sqoop_test.client
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-475) Unable to import into external hive
table located on S3
Posted by "Afsal Thaj (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402098#comment-13402098 ]
Afsal Thaj commented on SQOOP-475:
----------------------------------
i was trying from java to move a mysql table from local to hive .. the pgm was sucessfull in moving the table from mysql to hdfs but it couldnt get it into hive and it showed the above error
we were not using external table just sqoop import into hive
> Unable to import into external hive table located on S3
> -------------------------------------------------------
>
> Key: SQOOP-475
> URL: https://issues.apache.org/jira/browse/SQOOP-475
> Project: Sqoop
> Issue Type: Bug
> Components: hive-integration
> Affects Versions: 1.4.1-incubating
> Environment: Amazon EMR
> Hadoop 0.20.205
> Hive 0.7.1
> Sqoop 1.4.1-incubating
> Reporter: Porati Sébastien
>
> When i try to import into an hive table located on an S3 bucket, i got the following error message :
> FAILED: Error in semantic analysis: Line 2:17 Path is not legal 'hdfs://10.48.189.XX:9000/user/hadoop/client': Move from: hdfs://10.48.189.XX:9000/user/hadoop/client to: s3://some-bucket/sqoop-test/hive/client is not valid. Please check that values for params "default.fs.name" and "hive.metastore.warehouse.dir" do not conflict.
> Hive table creation Script :
> CREATE DATABASE IF NOT EXISTS sqoop_test;
> USE sqoop_test;
> CREATE EXTERNAL TABLE IF NOT EXISTS client (
> id INT,
> email STRING,
> cookie_uid STRING,
> is_blacklisted TINYINT
> )
> LOCATION 's3://some-bucket/sqoop-test/hive/client';
> Sqoop command :
> sqoop import --connect jdbc:mysql://my.domain.com/mydb --username myuser --password XXXX --table client --hive-import --hive-overwrite --hive-table sqoop_test.client
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-475) Unable to import into external hive
table located on S3
Posted by "Georgy B Abraham (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402185#comment-13402185 ]
Georgy B Abraham commented on SQOOP-475:
----------------------------------------
The specific Error
Jun 28, 2012 12:40:04 AM org.apache.sqoop.util.LoggingAsyncSink$LoggingThread run
INFO: FAILED: Error in semantic analysis: Line 2:17 Path is not legal 'file:/home/556940/Eclipse/Workspace/SqoopGUI/SqoopGUI/afsal': Move from: file:/home/556940/Eclipse/Workspace/SqoopGUI/SqoopGUI/afsal to: hdfs://localhost:9000/user/hive/warehouse/afsal is not valid. Please check that values for params "default.fs.name" and "hive.metastore.warehouse.dir" do not conflict.
Jun 28, 2012 12:40:04 AM org.apache.sqoop.tool.ImportTool run
SEVERE: Encountered IOException running import job: java.io.IOException: Hive exited with status 10
at org.apache.sqoop.hive.HiveImport.executeExternalHiveScript(HiveImport.java:347)
at org.apache.sqoop.hive.HiveImport.executeScript(HiveImport.java:297)
at org.apache.sqoop.hive.HiveImport.importTable(HiveImport.java:239)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:392)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:453)
at xcc.main(xcc.java:28)
> Unable to import into external hive table located on S3
> -------------------------------------------------------
>
> Key: SQOOP-475
> URL: https://issues.apache.org/jira/browse/SQOOP-475
> Project: Sqoop
> Issue Type: Bug
> Components: hive-integration
> Affects Versions: 1.4.1-incubating
> Environment: Amazon EMR
> Hadoop 0.20.205
> Hive 0.7.1
> Sqoop 1.4.1-incubating
> Reporter: Porati Sébastien
>
> When i try to import into an hive table located on an S3 bucket, i got the following error message :
> FAILED: Error in semantic analysis: Line 2:17 Path is not legal 'hdfs://10.48.189.XX:9000/user/hadoop/client': Move from: hdfs://10.48.189.XX:9000/user/hadoop/client to: s3://some-bucket/sqoop-test/hive/client is not valid. Please check that values for params "default.fs.name" and "hive.metastore.warehouse.dir" do not conflict.
> Hive table creation Script :
> CREATE DATABASE IF NOT EXISTS sqoop_test;
> USE sqoop_test;
> CREATE EXTERNAL TABLE IF NOT EXISTS client (
> id INT,
> email STRING,
> cookie_uid STRING,
> is_blacklisted TINYINT
> )
> LOCATION 's3://some-bucket/sqoop-test/hive/client';
> Sqoop command :
> sqoop import --connect jdbc:mysql://my.domain.com/mydb --username myuser --password XXXX --table client --hive-import --hive-overwrite --hive-table sqoop_test.client
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (SQOOP-475) Unable to import into external hive
table located on S3
Posted by "Afsal Thaj (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Afsal Thaj updated SQOOP-475:
-----------------------------
Comment: was deleted
(was: I have done it using another method.Following was the code line used to execute the sqoop command from a java client program
Sqoop.runTool(new String[]{"import","--connect","jdbc:mysql://localhost/dbname","--username","root","--password","****","--table","tablename","-m","1","--hive-import","--hive-home","/home/549570/hive-0.7.1"})
As shown,I have tried a direct hive import.And I expected the temporary file to be created in hdfs but it got transffered temorarily to eclipse workspace.Later the transfer from eclipse workspace to hdfs://localhost:9000/hive/warehouse/tablename is said to be invalid.
Is there any possibility I can have this temporary file in hdfs???
)
> Unable to import into external hive table located on S3
> -------------------------------------------------------
>
> Key: SQOOP-475
> URL: https://issues.apache.org/jira/browse/SQOOP-475
> Project: Sqoop
> Issue Type: Bug
> Components: hive-integration
> Affects Versions: 1.4.1-incubating
> Environment: Amazon EMR
> Hadoop 0.20.205
> Hive 0.7.1
> Sqoop 1.4.1-incubating
> Reporter: Porati Sébastien
>
> When i try to import into an hive table located on an S3 bucket, i got the following error message :
> FAILED: Error in semantic analysis: Line 2:17 Path is not legal 'hdfs://10.48.189.XX:9000/user/hadoop/client': Move from: hdfs://10.48.189.XX:9000/user/hadoop/client to: s3://some-bucket/sqoop-test/hive/client is not valid. Please check that values for params "default.fs.name" and "hive.metastore.warehouse.dir" do not conflict.
> Hive table creation Script :
> CREATE DATABASE IF NOT EXISTS sqoop_test;
> USE sqoop_test;
> CREATE EXTERNAL TABLE IF NOT EXISTS client (
> id INT,
> email STRING,
> cookie_uid STRING,
> is_blacklisted TINYINT
> )
> LOCATION 's3://some-bucket/sqoop-test/hive/client';
> Sqoop command :
> sqoop import --connect jdbc:mysql://my.domain.com/mydb --username myuser --password XXXX --table client --hive-import --hive-overwrite --hive-table sqoop_test.client
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-475) Unable to import into external hive
table located on S3
Posted by "sachin pawar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13417169#comment-13417169 ]
sachin pawar commented on SQOOP-475:
------------------------------------
can you please explain what you mean by "in sync".
I tried setting both to same hdfs://172.25.6.96:8020/ in code ...but ran into same error.
How do I make hive use hdfs path not local file system?.
I thought I was doing this with my hive-site.xml. see below.
its a very basic set up.
hadoop/ hive installed by same user ...running under same user.
sqoop job run from remote machine is also same user
/user/hive has write permissions to all just to be on safe side.
following is my hive-site.xml
<property>
<name>hive.metastore.local</name>
<value>true</value>
</property>
<property>
<name>hive.metastore.warehouse.dir</name>
<value>hdfs://172.25.6.96:8020/user/hive/warehouse</value>
</property>
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://localhost/metastore_db?createDatabaseIfNotExist=true</value>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
</property>
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>uname</value>
</property>
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>pwd</value>
</property>
thanks
> Unable to import into external hive table located on S3
> -------------------------------------------------------
>
> Key: SQOOP-475
> URL: https://issues.apache.org/jira/browse/SQOOP-475
> Project: Sqoop
> Issue Type: Bug
> Components: hive-integration
> Affects Versions: 1.4.1-incubating
> Environment: Amazon EMR
> Hadoop 0.20.205
> Hive 0.7.1
> Sqoop 1.4.1-incubating
> Reporter: Porati Sébastien
>
> When i try to import into an hive table located on an S3 bucket, i got the following error message :
> FAILED: Error in semantic analysis: Line 2:17 Path is not legal 'hdfs://10.48.189.XX:9000/user/hadoop/client': Move from: hdfs://10.48.189.XX:9000/user/hadoop/client to: s3://some-bucket/sqoop-test/hive/client is not valid. Please check that values for params "default.fs.name" and "hive.metastore.warehouse.dir" do not conflict.
> Hive table creation Script :
> CREATE DATABASE IF NOT EXISTS sqoop_test;
> USE sqoop_test;
> CREATE EXTERNAL TABLE IF NOT EXISTS client (
> id INT,
> email STRING,
> cookie_uid STRING,
> is_blacklisted TINYINT
> )
> LOCATION 's3://some-bucket/sqoop-test/hive/client';
> Sqoop command :
> sqoop import --connect jdbc:mysql://my.domain.com/mydb --username myuser --password XXXX --table client --hive-import --hive-overwrite --hive-table sqoop_test.client
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-475) Unable to import into external hive
table located on S3
Posted by "Jarek Jarcec Cecho (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13416957#comment-13416957 ]
Jarek Jarcec Cecho commented on SQOOP-475:
------------------------------------------
Hi Saschin,
I believe that the problem is described in your error message:
FAILED: Error in semantic analysis: Line 2:17 Path is not legal ''hdfs://172.25.6.96:8020/user/sacpawar/bizunit'':
Move from: hdfs://172.25.6.96:8020/user/sacpawar/bizunit to: file:/user/hive/warehouse/bizunit is not valid.
Please check that values for params "default.fs.name" and "hive.metastore.warehouse.dir" do not conflict.
It seems that sqoop has imported data to hdfs instance running at 172.25.6.96:8020, however your hive seems to be configured to use local filesystem (file:/user/hive/warehouse/) as a warehouse dir. You need to get in sync properties default.fs.name (HDFS location) and hive.metastore.warehouse.dir (warehouse directory).
Jarcec
> Unable to import into external hive table located on S3
> -------------------------------------------------------
>
> Key: SQOOP-475
> URL: https://issues.apache.org/jira/browse/SQOOP-475
> Project: Sqoop
> Issue Type: Bug
> Components: hive-integration
> Affects Versions: 1.4.1-incubating
> Environment: Amazon EMR
> Hadoop 0.20.205
> Hive 0.7.1
> Sqoop 1.4.1-incubating
> Reporter: Porati Sébastien
>
> When i try to import into an hive table located on an S3 bucket, i got the following error message :
> FAILED: Error in semantic analysis: Line 2:17 Path is not legal 'hdfs://10.48.189.XX:9000/user/hadoop/client': Move from: hdfs://10.48.189.XX:9000/user/hadoop/client to: s3://some-bucket/sqoop-test/hive/client is not valid. Please check that values for params "default.fs.name" and "hive.metastore.warehouse.dir" do not conflict.
> Hive table creation Script :
> CREATE DATABASE IF NOT EXISTS sqoop_test;
> USE sqoop_test;
> CREATE EXTERNAL TABLE IF NOT EXISTS client (
> id INT,
> email STRING,
> cookie_uid STRING,
> is_blacklisted TINYINT
> )
> LOCATION 's3://some-bucket/sqoop-test/hive/client';
> Sqoop command :
> sqoop import --connect jdbc:mysql://my.domain.com/mydb --username myuser --password XXXX --table client --hive-import --hive-overwrite --hive-table sqoop_test.client
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-475) Unable to import into external hive
table located on S3
Posted by "Jarek Jarcec Cecho (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13406287#comment-13406287 ]
Jarek Jarcec Cecho commented on SQOOP-475:
------------------------------------------
Fact that create-hive-table tool has created table in workspace itself is actually helpful. I would guess that hive is using default configuration file and not your cluster wise config. Could you load your cluster hive-site.xml and then call sqoop to see what happens?
Jarcec
> Unable to import into external hive table located on S3
> -------------------------------------------------------
>
> Key: SQOOP-475
> URL: https://issues.apache.org/jira/browse/SQOOP-475
> Project: Sqoop
> Issue Type: Bug
> Components: hive-integration
> Affects Versions: 1.4.1-incubating
> Environment: Amazon EMR
> Hadoop 0.20.205
> Hive 0.7.1
> Sqoop 1.4.1-incubating
> Reporter: Porati Sébastien
>
> When i try to import into an hive table located on an S3 bucket, i got the following error message :
> FAILED: Error in semantic analysis: Line 2:17 Path is not legal 'hdfs://10.48.189.XX:9000/user/hadoop/client': Move from: hdfs://10.48.189.XX:9000/user/hadoop/client to: s3://some-bucket/sqoop-test/hive/client is not valid. Please check that values for params "default.fs.name" and "hive.metastore.warehouse.dir" do not conflict.
> Hive table creation Script :
> CREATE DATABASE IF NOT EXISTS sqoop_test;
> USE sqoop_test;
> CREATE EXTERNAL TABLE IF NOT EXISTS client (
> id INT,
> email STRING,
> cookie_uid STRING,
> is_blacklisted TINYINT
> )
> LOCATION 's3://some-bucket/sqoop-test/hive/client';
> Sqoop command :
> sqoop import --connect jdbc:mysql://my.domain.com/mydb --username myuser --password XXXX --table client --hive-import --hive-overwrite --hive-table sqoop_test.client
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-475) Unable to import into external hive
table located on S3
Posted by "Afsal Thaj (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402817#comment-13402817 ]
Afsal Thaj commented on SQOOP-475:
----------------------------------
I have done it using another method.Following was the code line used to execute the sqoop command from a java client program
Sqoop.runTool(new String[]{"import","--connect","jdbc:mysql://localhost/dbname","--username","root","--password","****","--table","tablename","-m","1","--hive-import","--hive-home","/home/549570/hive-0.7.1"})
And I expected the temporary file to be created in hdfs but it got transfered temorarily to eclipse workspace.Later the transfer from eclipse workspace to hdfs://localhost:9000/hive/warehouse/tablename is said to be invalid.
Is there any possibility I can have this temporary file in hdfs???
> Unable to import into external hive table located on S3
> -------------------------------------------------------
>
> Key: SQOOP-475
> URL: https://issues.apache.org/jira/browse/SQOOP-475
> Project: Sqoop
> Issue Type: Bug
> Components: hive-integration
> Affects Versions: 1.4.1-incubating
> Environment: Amazon EMR
> Hadoop 0.20.205
> Hive 0.7.1
> Sqoop 1.4.1-incubating
> Reporter: Porati Sébastien
>
> When i try to import into an hive table located on an S3 bucket, i got the following error message :
> FAILED: Error in semantic analysis: Line 2:17 Path is not legal 'hdfs://10.48.189.XX:9000/user/hadoop/client': Move from: hdfs://10.48.189.XX:9000/user/hadoop/client to: s3://some-bucket/sqoop-test/hive/client is not valid. Please check that values for params "default.fs.name" and "hive.metastore.warehouse.dir" do not conflict.
> Hive table creation Script :
> CREATE DATABASE IF NOT EXISTS sqoop_test;
> USE sqoop_test;
> CREATE EXTERNAL TABLE IF NOT EXISTS client (
> id INT,
> email STRING,
> cookie_uid STRING,
> is_blacklisted TINYINT
> )
> LOCATION 's3://some-bucket/sqoop-test/hive/client';
> Sqoop command :
> sqoop import --connect jdbc:mysql://my.domain.com/mydb --username myuser --password XXXX --table client --hive-import --hive-overwrite --hive-table sqoop_test.client
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-475) Unable to import into external hive
table located on S3
Posted by "Jarek Jarcec Cecho (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402096#comment-13402096 ]
Jarek Jarcec Cecho commented on SQOOP-475:
------------------------------------------
Hi Afsal,
as far as I know, we're currently supporting only HIVE internal tables in sqoop.
Jarcec
> Unable to import into external hive table located on S3
> -------------------------------------------------------
>
> Key: SQOOP-475
> URL: https://issues.apache.org/jira/browse/SQOOP-475
> Project: Sqoop
> Issue Type: Bug
> Components: hive-integration
> Affects Versions: 1.4.1-incubating
> Environment: Amazon EMR
> Hadoop 0.20.205
> Hive 0.7.1
> Sqoop 1.4.1-incubating
> Reporter: Porati Sébastien
>
> When i try to import into an hive table located on an S3 bucket, i got the following error message :
> FAILED: Error in semantic analysis: Line 2:17 Path is not legal 'hdfs://10.48.189.XX:9000/user/hadoop/client': Move from: hdfs://10.48.189.XX:9000/user/hadoop/client to: s3://some-bucket/sqoop-test/hive/client is not valid. Please check that values for params "default.fs.name" and "hive.metastore.warehouse.dir" do not conflict.
> Hive table creation Script :
> CREATE DATABASE IF NOT EXISTS sqoop_test;
> USE sqoop_test;
> CREATE EXTERNAL TABLE IF NOT EXISTS client (
> id INT,
> email STRING,
> cookie_uid STRING,
> is_blacklisted TINYINT
> )
> LOCATION 's3://some-bucket/sqoop-test/hive/client';
> Sqoop command :
> sqoop import --connect jdbc:mysql://my.domain.com/mydb --username myuser --password XXXX --table client --hive-import --hive-overwrite --hive-table sqoop_test.client
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (SQOOP-475) Unable to import into external hive
table located on S3
Posted by "Afsal Thaj (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Afsal Thaj updated SQOOP-475:
-----------------------------
Comment: was deleted
(was: I have done it using another method.Following was the code line used to execute the sqoop command from a java client program
Sqoop.runTool(new String[]{"import","--connect","jdbc:mysql://localhost/sebin","--username","root","--password","6397","--table","afsal","-m","1","--hive-import","--hive-home","/home/556940/hive-0.7.1"})
As shown,I have tried a direct hive import.And I expected the temporary file to be created in hdfs but it got transffered temorarily to eclipse workspace.Later the transfer from eclipse workspace to hdfs://localhost:9000/hive/warehouse/tablename is said to be invalid.
Is there any possibility I can have this temporary file in hdfs??? )
> Unable to import into external hive table located on S3
> -------------------------------------------------------
>
> Key: SQOOP-475
> URL: https://issues.apache.org/jira/browse/SQOOP-475
> Project: Sqoop
> Issue Type: Bug
> Components: hive-integration
> Affects Versions: 1.4.1-incubating
> Environment: Amazon EMR
> Hadoop 0.20.205
> Hive 0.7.1
> Sqoop 1.4.1-incubating
> Reporter: Porati Sébastien
>
> When i try to import into an hive table located on an S3 bucket, i got the following error message :
> FAILED: Error in semantic analysis: Line 2:17 Path is not legal 'hdfs://10.48.189.XX:9000/user/hadoop/client': Move from: hdfs://10.48.189.XX:9000/user/hadoop/client to: s3://some-bucket/sqoop-test/hive/client is not valid. Please check that values for params "default.fs.name" and "hive.metastore.warehouse.dir" do not conflict.
> Hive table creation Script :
> CREATE DATABASE IF NOT EXISTS sqoop_test;
> USE sqoop_test;
> CREATE EXTERNAL TABLE IF NOT EXISTS client (
> id INT,
> email STRING,
> cookie_uid STRING,
> is_blacklisted TINYINT
> )
> LOCATION 's3://some-bucket/sqoop-test/hive/client';
> Sqoop command :
> sqoop import --connect jdbc:mysql://my.domain.com/mydb --username myuser --password XXXX --table client --hive-import --hive-overwrite --hive-table sqoop_test.client
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-475) Unable to import into external hive
table located on S3
Posted by "Afsal Thaj (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402813#comment-13402813 ]
Afsal Thaj commented on SQOOP-475:
----------------------------------
I have done it using another method.Following was the code line used to execute the sqoop command from a java client program
Sqoop.runTool(new String[]{"import","--connect","jdbc:mysql://localhost/sebin","--username","root","--password","6397","--table","afsal","-m","1","--hive-import","--hive-home","/home/556940/hive-0.7.1"})
As shown,I have tried a direct hive import.And I expected the temporary file to be created in hdfs but it got transffered temorarily to eclipse workspace.Later the transfer from eclipse workspace to hdfs://localhost:9000/hive/warehouse/tablename is said to be invalid.
Is there any possibility I can have this temporary file in hdfs???
> Unable to import into external hive table located on S3
> -------------------------------------------------------
>
> Key: SQOOP-475
> URL: https://issues.apache.org/jira/browse/SQOOP-475
> Project: Sqoop
> Issue Type: Bug
> Components: hive-integration
> Affects Versions: 1.4.1-incubating
> Environment: Amazon EMR
> Hadoop 0.20.205
> Hive 0.7.1
> Sqoop 1.4.1-incubating
> Reporter: Porati Sébastien
>
> When i try to import into an hive table located on an S3 bucket, i got the following error message :
> FAILED: Error in semantic analysis: Line 2:17 Path is not legal 'hdfs://10.48.189.XX:9000/user/hadoop/client': Move from: hdfs://10.48.189.XX:9000/user/hadoop/client to: s3://some-bucket/sqoop-test/hive/client is not valid. Please check that values for params "default.fs.name" and "hive.metastore.warehouse.dir" do not conflict.
> Hive table creation Script :
> CREATE DATABASE IF NOT EXISTS sqoop_test;
> USE sqoop_test;
> CREATE EXTERNAL TABLE IF NOT EXISTS client (
> id INT,
> email STRING,
> cookie_uid STRING,
> is_blacklisted TINYINT
> )
> LOCATION 's3://some-bucket/sqoop-test/hive/client';
> Sqoop command :
> sqoop import --connect jdbc:mysql://my.domain.com/mydb --username myuser --password XXXX --table client --hive-import --hive-overwrite --hive-table sqoop_test.client
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (SQOOP-475) Unable to import into external hive
table located on S3
Posted by "Jarek Jarcec Cecho (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jarek Jarcec Cecho resolved SQOOP-475.
--------------------------------------
Resolution: Invalid
I'm afraid that this is not supported Sqoop use case and there is no activity for quite some time, so I'm closing this. Please feel free to reopen if needed!
Jarcec
> Unable to import into external hive table located on S3
> -------------------------------------------------------
>
> Key: SQOOP-475
> URL: https://issues.apache.org/jira/browse/SQOOP-475
> Project: Sqoop
> Issue Type: Bug
> Components: hive-integration
> Affects Versions: 1.4.1-incubating
> Environment: Amazon EMR
> Hadoop 0.20.205
> Hive 0.7.1
> Sqoop 1.4.1-incubating
> Reporter: Porati Sébastien
>
> When i try to import into an hive table located on an S3 bucket, i got the following error message :
> FAILED: Error in semantic analysis: Line 2:17 Path is not legal 'hdfs://10.48.189.XX:9000/user/hadoop/client': Move from: hdfs://10.48.189.XX:9000/user/hadoop/client to: s3://some-bucket/sqoop-test/hive/client is not valid. Please check that values for params "default.fs.name" and "hive.metastore.warehouse.dir" do not conflict.
> Hive table creation Script :
> CREATE DATABASE IF NOT EXISTS sqoop_test;
> USE sqoop_test;
> CREATE EXTERNAL TABLE IF NOT EXISTS client (
> id INT,
> email STRING,
> cookie_uid STRING,
> is_blacklisted TINYINT
> )
> LOCATION 's3://some-bucket/sqoop-test/hive/client';
> Sqoop command :
> sqoop import --connect jdbc:mysql://my.domain.com/mydb --username myuser --password XXXX --table client --hive-import --hive-overwrite --hive-table sqoop_test.client
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-475) Unable to import into external hive
table located on S3
Posted by "Afsal Thaj (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13406288#comment-13406288 ]
Afsal Thaj commented on SQOOP-475:
----------------------------------
Jarek
We are not working in cluster.We havent made any change in hive configuration file too.
> Unable to import into external hive table located on S3
> -------------------------------------------------------
>
> Key: SQOOP-475
> URL: https://issues.apache.org/jira/browse/SQOOP-475
> Project: Sqoop
> Issue Type: Bug
> Components: hive-integration
> Affects Versions: 1.4.1-incubating
> Environment: Amazon EMR
> Hadoop 0.20.205
> Hive 0.7.1
> Sqoop 1.4.1-incubating
> Reporter: Porati Sébastien
>
> When i try to import into an hive table located on an S3 bucket, i got the following error message :
> FAILED: Error in semantic analysis: Line 2:17 Path is not legal 'hdfs://10.48.189.XX:9000/user/hadoop/client': Move from: hdfs://10.48.189.XX:9000/user/hadoop/client to: s3://some-bucket/sqoop-test/hive/client is not valid. Please check that values for params "default.fs.name" and "hive.metastore.warehouse.dir" do not conflict.
> Hive table creation Script :
> CREATE DATABASE IF NOT EXISTS sqoop_test;
> USE sqoop_test;
> CREATE EXTERNAL TABLE IF NOT EXISTS client (
> id INT,
> email STRING,
> cookie_uid STRING,
> is_blacklisted TINYINT
> )
> LOCATION 's3://some-bucket/sqoop-test/hive/client';
> Sqoop command :
> sqoop import --connect jdbc:mysql://my.domain.com/mydb --username myuser --password XXXX --table client --hive-import --hive-overwrite --hive-table sqoop_test.client
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-475) Unable to import into external hive
table located on S3
Posted by "Georgy B Abraham (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402852#comment-13402852 ]
Georgy B Abraham commented on SQOOP-475:
----------------------------------------
afsal is trying it now ... we will update soon ..
> Unable to import into external hive table located on S3
> -------------------------------------------------------
>
> Key: SQOOP-475
> URL: https://issues.apache.org/jira/browse/SQOOP-475
> Project: Sqoop
> Issue Type: Bug
> Components: hive-integration
> Affects Versions: 1.4.1-incubating
> Environment: Amazon EMR
> Hadoop 0.20.205
> Hive 0.7.1
> Sqoop 1.4.1-incubating
> Reporter: Porati Sébastien
>
> When i try to import into an hive table located on an S3 bucket, i got the following error message :
> FAILED: Error in semantic analysis: Line 2:17 Path is not legal 'hdfs://10.48.189.XX:9000/user/hadoop/client': Move from: hdfs://10.48.189.XX:9000/user/hadoop/client to: s3://some-bucket/sqoop-test/hive/client is not valid. Please check that values for params "default.fs.name" and "hive.metastore.warehouse.dir" do not conflict.
> Hive table creation Script :
> CREATE DATABASE IF NOT EXISTS sqoop_test;
> USE sqoop_test;
> CREATE EXTERNAL TABLE IF NOT EXISTS client (
> id INT,
> email STRING,
> cookie_uid STRING,
> is_blacklisted TINYINT
> )
> LOCATION 's3://some-bucket/sqoop-test/hive/client';
> Sqoop command :
> sqoop import --connect jdbc:mysql://my.domain.com/mydb --username myuser --password XXXX --table client --hive-import --hive-overwrite --hive-table sqoop_test.client
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-475) Unable to import into external hive
table located on S3
Posted by "Afsal Thaj (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402815#comment-13402815 ]
Afsal Thaj commented on SQOOP-475:
----------------------------------
I have done it using another method.Following was the code line used to execute the sqoop command from a java client program
Sqoop.runTool(new String[]{"import","--connect","jdbc:mysql://localhost/dbname","--username","root","--password","****","--table","tablename","-m","1","--hive-import","--hive-home","/home/549570/hive-0.7.1"})
As shown,I have tried a direct hive import.And I expected the temporary file to be created in hdfs but it got transffered temorarily to eclipse workspace.Later the transfer from eclipse workspace to hdfs://localhost:9000/hive/warehouse/tablename is said to be invalid.
Is there any possibility I can have this temporary file in hdfs???
> Unable to import into external hive table located on S3
> -------------------------------------------------------
>
> Key: SQOOP-475
> URL: https://issues.apache.org/jira/browse/SQOOP-475
> Project: Sqoop
> Issue Type: Bug
> Components: hive-integration
> Affects Versions: 1.4.1-incubating
> Environment: Amazon EMR
> Hadoop 0.20.205
> Hive 0.7.1
> Sqoop 1.4.1-incubating
> Reporter: Porati Sébastien
>
> When i try to import into an hive table located on an S3 bucket, i got the following error message :
> FAILED: Error in semantic analysis: Line 2:17 Path is not legal 'hdfs://10.48.189.XX:9000/user/hadoop/client': Move from: hdfs://10.48.189.XX:9000/user/hadoop/client to: s3://some-bucket/sqoop-test/hive/client is not valid. Please check that values for params "default.fs.name" and "hive.metastore.warehouse.dir" do not conflict.
> Hive table creation Script :
> CREATE DATABASE IF NOT EXISTS sqoop_test;
> USE sqoop_test;
> CREATE EXTERNAL TABLE IF NOT EXISTS client (
> id INT,
> email STRING,
> cookie_uid STRING,
> is_blacklisted TINYINT
> )
> LOCATION 's3://some-bucket/sqoop-test/hive/client';
> Sqoop command :
> sqoop import --connect jdbc:mysql://my.domain.com/mydb --username myuser --password XXXX --table client --hive-import --hive-overwrite --hive-table sqoop_test.client
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (SQOOP-475) Unable to import into external hive
table located on S3
Posted by "Afsal Thaj (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Afsal Thaj updated SQOOP-475:
-----------------------------
Comment: was deleted
(was: I have done it using another method.Following was the code line used to execute the sqoop command from a java client program
Sqoop.runTool(new String[]{"import","--connect","jdbc:mysql://localhost/dbname","--username","root","--password","****","--table","tablename","-m","1","--hive-import","--hive-home","/home/549570/hive-0.7.1"})
And I expected the temporary file to be created in hdfs but it got transfered temorarily to eclipse workspace.Later the transfer from eclipse workspace to hdfs://localhost:9000/hive/warehouse/tablename is said to be invalid.
Is there any possibility I can have this temporary file in hdfs???
)
> Unable to import into external hive table located on S3
> -------------------------------------------------------
>
> Key: SQOOP-475
> URL: https://issues.apache.org/jira/browse/SQOOP-475
> Project: Sqoop
> Issue Type: Bug
> Components: hive-integration
> Affects Versions: 1.4.1-incubating
> Environment: Amazon EMR
> Hadoop 0.20.205
> Hive 0.7.1
> Sqoop 1.4.1-incubating
> Reporter: Porati Sébastien
>
> When i try to import into an hive table located on an S3 bucket, i got the following error message :
> FAILED: Error in semantic analysis: Line 2:17 Path is not legal 'hdfs://10.48.189.XX:9000/user/hadoop/client': Move from: hdfs://10.48.189.XX:9000/user/hadoop/client to: s3://some-bucket/sqoop-test/hive/client is not valid. Please check that values for params "default.fs.name" and "hive.metastore.warehouse.dir" do not conflict.
> Hive table creation Script :
> CREATE DATABASE IF NOT EXISTS sqoop_test;
> USE sqoop_test;
> CREATE EXTERNAL TABLE IF NOT EXISTS client (
> id INT,
> email STRING,
> cookie_uid STRING,
> is_blacklisted TINYINT
> )
> LOCATION 's3://some-bucket/sqoop-test/hive/client';
> Sqoop command :
> sqoop import --connect jdbc:mysql://my.domain.com/mydb --username myuser --password XXXX --table client --hive-import --hive-overwrite --hive-table sqoop_test.client
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-475) Unable to import into external hive
table located on S3
Posted by "Afsal Thaj (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402093#comment-13402093 ]
Afsal Thaj commented on SQOOP-475:
----------------------------------
Got the same error while trying to move from mysql to hdfs to hive
> Unable to import into external hive table located on S3
> -------------------------------------------------------
>
> Key: SQOOP-475
> URL: https://issues.apache.org/jira/browse/SQOOP-475
> Project: Sqoop
> Issue Type: Bug
> Components: hive-integration
> Affects Versions: 1.4.1-incubating
> Environment: Amazon EMR
> Hadoop 0.20.205
> Hive 0.7.1
> Sqoop 1.4.1-incubating
> Reporter: Porati Sébastien
>
> When i try to import into an hive table located on an S3 bucket, i got the following error message :
> FAILED: Error in semantic analysis: Line 2:17 Path is not legal 'hdfs://10.48.189.XX:9000/user/hadoop/client': Move from: hdfs://10.48.189.XX:9000/user/hadoop/client to: s3://some-bucket/sqoop-test/hive/client is not valid. Please check that values for params "default.fs.name" and "hive.metastore.warehouse.dir" do not conflict.
> Hive table creation Script :
> CREATE DATABASE IF NOT EXISTS sqoop_test;
> USE sqoop_test;
> CREATE EXTERNAL TABLE IF NOT EXISTS client (
> id INT,
> email STRING,
> cookie_uid STRING,
> is_blacklisted TINYINT
> )
> LOCATION 's3://some-bucket/sqoop-test/hive/client';
> Sqoop command :
> sqoop import --connect jdbc:mysql://my.domain.com/mydb --username myuser --password XXXX --table client --hive-import --hive-overwrite --hive-table sqoop_test.client
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-475) Unable to import into external hive
table located on S3
Posted by "Jarek Jarcec Cecho (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13406306#comment-13406306 ]
Jarek Jarcec Cecho commented on SQOOP-475:
------------------------------------------
Could you be more precise what has happened sir? Is your configuration file applied? Did HIVE modified correct metadata storage? If not, than you probably have to provide proper configuration there (regarding metadata back end).
Jarcec
> Unable to import into external hive table located on S3
> -------------------------------------------------------
>
> Key: SQOOP-475
> URL: https://issues.apache.org/jira/browse/SQOOP-475
> Project: Sqoop
> Issue Type: Bug
> Components: hive-integration
> Affects Versions: 1.4.1-incubating
> Environment: Amazon EMR
> Hadoop 0.20.205
> Hive 0.7.1
> Sqoop 1.4.1-incubating
> Reporter: Porati Sébastien
>
> When i try to import into an hive table located on an S3 bucket, i got the following error message :
> FAILED: Error in semantic analysis: Line 2:17 Path is not legal 'hdfs://10.48.189.XX:9000/user/hadoop/client': Move from: hdfs://10.48.189.XX:9000/user/hadoop/client to: s3://some-bucket/sqoop-test/hive/client is not valid. Please check that values for params "default.fs.name" and "hive.metastore.warehouse.dir" do not conflict.
> Hive table creation Script :
> CREATE DATABASE IF NOT EXISTS sqoop_test;
> USE sqoop_test;
> CREATE EXTERNAL TABLE IF NOT EXISTS client (
> id INT,
> email STRING,
> cookie_uid STRING,
> is_blacklisted TINYINT
> )
> LOCATION 's3://some-bucket/sqoop-test/hive/client';
> Sqoop command :
> sqoop import --connect jdbc:mysql://my.domain.com/mydb --username myuser --password XXXX --table client --hive-import --hive-overwrite --hive-table sqoop_test.client
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-475) Unable to import into external hive
table located on S3
Posted by "Jarek Jarcec Cecho (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402193#comment-13402193 ]
Jarek Jarcec Cecho commented on SQOOP-475:
------------------------------------------
Hi George,
I'm still not sure what pgm is or how exactly are you executing the sqoop.
It appears that your execution is storing imported data directly on the disk (path file://), however your hive instance is configured to have warehouse directory on hdfs (hdfs://localhost:9000). Those two locations needs to be in sync, so you either need to store the import on your local HDFS or reconfigure hive to use your local disk (I'm not sure whether it's supported).
Jarcec
> Unable to import into external hive table located on S3
> -------------------------------------------------------
>
> Key: SQOOP-475
> URL: https://issues.apache.org/jira/browse/SQOOP-475
> Project: Sqoop
> Issue Type: Bug
> Components: hive-integration
> Affects Versions: 1.4.1-incubating
> Environment: Amazon EMR
> Hadoop 0.20.205
> Hive 0.7.1
> Sqoop 1.4.1-incubating
> Reporter: Porati Sébastien
>
> When i try to import into an hive table located on an S3 bucket, i got the following error message :
> FAILED: Error in semantic analysis: Line 2:17 Path is not legal 'hdfs://10.48.189.XX:9000/user/hadoop/client': Move from: hdfs://10.48.189.XX:9000/user/hadoop/client to: s3://some-bucket/sqoop-test/hive/client is not valid. Please check that values for params "default.fs.name" and "hive.metastore.warehouse.dir" do not conflict.
> Hive table creation Script :
> CREATE DATABASE IF NOT EXISTS sqoop_test;
> USE sqoop_test;
> CREATE EXTERNAL TABLE IF NOT EXISTS client (
> id INT,
> email STRING,
> cookie_uid STRING,
> is_blacklisted TINYINT
> )
> LOCATION 's3://some-bucket/sqoop-test/hive/client';
> Sqoop command :
> sqoop import --connect jdbc:mysql://my.domain.com/mydb --username myuser --password XXXX --table client --hive-import --hive-overwrite --hive-table sqoop_test.client
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-475) Unable to import into external hive
table located on S3
Posted by "Jarek Jarcec Cecho (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402177#comment-13402177 ]
Jarek Jarcec Cecho commented on SQOOP-475:
------------------------------------------
Hi Georgy,
could you advise what pgm stands for?
Your approach is unusual, but probably should be working. Could you share precise error message you're getting?
Jarcec
> Unable to import into external hive table located on S3
> -------------------------------------------------------
>
> Key: SQOOP-475
> URL: https://issues.apache.org/jira/browse/SQOOP-475
> Project: Sqoop
> Issue Type: Bug
> Components: hive-integration
> Affects Versions: 1.4.1-incubating
> Environment: Amazon EMR
> Hadoop 0.20.205
> Hive 0.7.1
> Sqoop 1.4.1-incubating
> Reporter: Porati Sébastien
>
> When i try to import into an hive table located on an S3 bucket, i got the following error message :
> FAILED: Error in semantic analysis: Line 2:17 Path is not legal 'hdfs://10.48.189.XX:9000/user/hadoop/client': Move from: hdfs://10.48.189.XX:9000/user/hadoop/client to: s3://some-bucket/sqoop-test/hive/client is not valid. Please check that values for params "default.fs.name" and "hive.metastore.warehouse.dir" do not conflict.
> Hive table creation Script :
> CREATE DATABASE IF NOT EXISTS sqoop_test;
> USE sqoop_test;
> CREATE EXTERNAL TABLE IF NOT EXISTS client (
> id INT,
> email STRING,
> cookie_uid STRING,
> is_blacklisted TINYINT
> )
> LOCATION 's3://some-bucket/sqoop-test/hive/client';
> Sqoop command :
> sqoop import --connect jdbc:mysql://my.domain.com/mydb --username myuser --password XXXX --table client --hive-import --hive-overwrite --hive-table sqoop_test.client
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-475) Unable to import into external hive
table located on S3
Posted by "shirley gracelyn (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13491473#comment-13491473 ]
shirley gracelyn commented on SQOOP-475:
----------------------------------------
I am facing the same error. This is what I see:
2012-11-05 20:12:57,711 ERROR org.apache.hadoop.hive.ql.Driver: FAILED: Error in semantic analysis: Line 2:17 Path is not legal 'hdfs://host:8020/user/root/table_name': Move from: hdfs://host:8020/user/root/table_name to: hdfs://host/user/hive/warehouse/some.db/othr_tbl is not valid. Please check that values for params "default.fs.name" and "hive.metastore.warehouse.dir" do not conflict.
I see this error when executing a sqoop import action from oozie: 2012-11-05 20:12:57,667 INFO hive.ql.parse.ParseDriver: Parsing command:
LOAD DATA INPATH 'hdfs://host:8020/user/root/table_name' OVERWRITE INTO TABLE `some.other_tbl`
The LOAD DATA IN PATH succeeds in hive, when it does not have the port number 8020 in the hdfs path. I confirmed this in hive. But problem is that, sqoop import somehow appends the 8020 into the source URI in the LOAD statement, and causes the failure. Is there a way to stop sqoop from appending the port number in the hdfs path?
> Unable to import into external hive table located on S3
> -------------------------------------------------------
>
> Key: SQOOP-475
> URL: https://issues.apache.org/jira/browse/SQOOP-475
> Project: Sqoop
> Issue Type: Bug
> Components: hive-integration
> Affects Versions: 1.4.1-incubating
> Environment: Amazon EMR
> Hadoop 0.20.205
> Hive 0.7.1
> Sqoop 1.4.1-incubating
> Reporter: Porati Sébastien
>
> When i try to import into an hive table located on an S3 bucket, i got the following error message :
> FAILED: Error in semantic analysis: Line 2:17 Path is not legal 'hdfs://10.48.189.XX:9000/user/hadoop/client': Move from: hdfs://10.48.189.XX:9000/user/hadoop/client to: s3://some-bucket/sqoop-test/hive/client is not valid. Please check that values for params "default.fs.name" and "hive.metastore.warehouse.dir" do not conflict.
> Hive table creation Script :
> CREATE DATABASE IF NOT EXISTS sqoop_test;
> USE sqoop_test;
> CREATE EXTERNAL TABLE IF NOT EXISTS client (
> id INT,
> email STRING,
> cookie_uid STRING,
> is_blacklisted TINYINT
> )
> LOCATION 's3://some-bucket/sqoop-test/hive/client';
> Sqoop command :
> sqoop import --connect jdbc:mysql://my.domain.com/mydb --username myuser --password XXXX --table client --hive-import --hive-overwrite --hive-table sqoop_test.client
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira