You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sqoop.apache.org by "Sonya Ling (JIRA)" <ji...@apache.org> on 2013/08/15 02:10:48 UTC
[jira] [Commented] (SQOOP-931) Integrate HCatalog with Sqoop
[ https://issues.apache.org/jira/browse/SQOOP-931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13740468#comment-13740468 ]
Sonya Ling commented on SQOOP-931:
----------------------------------
I tried sqoop-1.4.4 for this feature today. I have Hive 0.11.0 that has hcatalog merged into it. I have Hadoop 2.0.0-cdh4.3.0. I run statement similar to the followings:
sqoop import --connect jdbc:mysql://<host>/<database> --username <user> --password <password> --table transaction --hcatalog-table transaction --create-hcatalog-table --where "date >= 08-01-2012" --hive-partition-key date
It went through and started hadoop job but halt due the following error:
Exception in thread "main" java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.JobContext, but class was expected
at org.apache.hcatalog.mapreduce.HCatBaseOutputFormat.getJobInfo(HCatBaseOutputFormat.java:94)
at org.apache.hcatalog.mapreduce.HCatBaseOutputFormat.getOutputFormat(HCatBaseOutputFormat.java:82)
at org.apache.hcatalog.mapreduce.HCatBaseOutputFormat.checkOutputSpecs(HCatBaseOutputFormat.java:72)
at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:417)
I know this is due to Hadoop version conflict issue. HCatBaseOutputFormat is expecting Hadoop 1.0.x but I have Hadoop 2.0.x. I saw similar error when I ran Ozzie. I could get around by add mapred.mapper.new-api and mapred.reducer.new-api to true in workflow.xml of ozzie. I added the same properties to proto-hive-site.xml in hcatalog/etc. It did not work. I checked hcatalog 0.50 source codes. It does not use those properties.
How can I get around this issue? Please advise. Thanks.
> Integrate HCatalog with Sqoop
> -----------------------------
>
> Key: SQOOP-931
> URL: https://issues.apache.org/jira/browse/SQOOP-931
> Project: Sqoop
> Issue Type: New Feature
> Affects Versions: 1.4.2, 1.4.3
> Environment: All 1.x sqoop version
> Reporter: Venkat Ranganathan
> Assignee: Venkat Ranganathan
> Fix For: 1.4.4
>
> Attachments: SQOOP-931.patch, SQOOP-931.patch.14, SQOOP HCatalog Integration - 2.pdf, SQOOP HCatalog Integration - 3.pdf, SQOOP HCatalog Integration.pdf
>
>
> Apache HCatalog is a table and storage management service that provides a shared schema, data types and table abstraction freeing users from being concerned about where or how their data is stored. It provides interoperability across Pig, Map Reduce, and Hive.
> A sqoop hcatalog connector will help in supporting storage formats that are abstracted by HCatalog.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira