You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sqoop.apache.org by "Jarek Jarcec Cecho (Created) (JIRA)" <ji...@apache.org> on 2011/12/19 09:06:30 UTC
[jira] [Created] (SQOOP-412) Create our own implementation of
org.apache.hadoop.conf.Configuration.getInstances
Create our own implementation of org.apache.hadoop.conf.Configuration.getInstances
----------------------------------------------------------------------------------
Key: SQOOP-412
URL: https://issues.apache.org/jira/browse/SQOOP-412
Project: Sqoop
Issue Type: Sub-task
Reporter: Jarek Jarcec Cecho
Priority: Minor
We need to port functionality of org.apache.hadoop.conf.Configuration.getInstances for supporting hadoop releases before 0.21 (or before CDH3).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-412) Create our own implementation of
org.apache.hadoop.conf.Configuration.getInstances
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13173314#comment-13173314 ]
jiraposter@reviews.apache.org commented on SQOOP-412:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3272/
-----------------------------------------------------------
Review request for Sqoop and Arvind Prabhakar.
Summary
-------
I've copypasted all four files from CHD3 release and put them into sqoop mapreduce module (I altered files to conform our checkstyle policy). I've also changed all references in sqoop to use this new for classes instead of the default ones present in hadoop-0.21.
This addresses bug SQOOP-412.
https://issues.apache.org/jira/browse/SQOOP-412
Diffs
-----
/src/java/com/cloudera/sqoop/mapreduce/CombineShimRecordReader.java 1221303
/src/java/com/cloudera/sqoop/mapreduce/db/DBInputFormat.java 1221303
/src/java/com/cloudera/sqoop/mapreduce/db/DBOutputFormat.java 1221303
/src/java/com/cloudera/sqoop/mapreduce/db/DBRecordReader.java 1221303
/src/java/com/cloudera/sqoop/mapreduce/db/DataDrivenDBInputFormat.java 1221303
/src/java/com/cloudera/sqoop/mapreduce/db/DataDrivenDBRecordReader.java 1221303
/src/java/com/cloudera/sqoop/mapreduce/db/OracleDBRecordReader.java 1221303
/src/java/com/cloudera/sqoop/mapreduce/db/OracleDataDrivenDBInputFormat.java 1221303
/src/java/com/cloudera/sqoop/mapreduce/db/OracleDataDrivenDBRecordReader.java 1221303
/src/java/org/apache/sqoop/config/ConfigurationHelper.java 1221303
/src/java/org/apache/sqoop/lib/SqoopRecord.java 1221303
/src/java/org/apache/sqoop/mapreduce/CombineFileInputFormat.java PRE-CREATION
/src/java/org/apache/sqoop/mapreduce/CombineFileRecordReader.java PRE-CREATION
/src/java/org/apache/sqoop/mapreduce/CombineFileSplit.java PRE-CREATION
/src/java/org/apache/sqoop/mapreduce/CombineShimRecordReader.java 1221303
/src/java/org/apache/sqoop/mapreduce/DBWritable.java PRE-CREATION
/src/java/org/apache/sqoop/mapreduce/DataDrivenImportJob.java 1221303
/src/java/org/apache/sqoop/mapreduce/ExportInputFormat.java 1221303
/src/java/org/apache/sqoop/mapreduce/MySQLDumpImportJob.java 1221303
/src/java/org/apache/sqoop/mapreduce/MySQLExportJob.java 1221303
/src/java/org/apache/sqoop/mapreduce/db/DBConfiguration.java 1221303
/src/java/org/apache/sqoop/mapreduce/db/DBInputFormat.java 1221303
/src/java/org/apache/sqoop/mapreduce/db/DBOutputFormat.java 1221303
/src/java/org/apache/sqoop/mapreduce/db/DBRecordReader.java 1221303
/src/java/org/apache/sqoop/mapreduce/db/DataDrivenDBInputFormat.java 1221303
/src/java/org/apache/sqoop/mapreduce/db/DataDrivenDBRecordReader.java 1221303
/src/java/org/apache/sqoop/mapreduce/db/OracleDBRecordReader.java 1221303
/src/java/org/apache/sqoop/mapreduce/db/OracleDataDrivenDBInputFormat.java 1221303
/src/java/org/apache/sqoop/mapreduce/db/OracleDataDrivenDBRecordReader.java 1221303
/src/test/com/cloudera/sqoop/mapreduce/db/TestDataDrivenDBInputFormat.java 1221303
Diff: https://reviews.apache.org/r/3272/diff
Testing
-------
I've tested with both -Dhadopoversion=20 and -Dhadoopversion=23.
Thanks,
Jarek
> Create our own implementation of org.apache.hadoop.conf.Configuration.getInstances
> ----------------------------------------------------------------------------------
>
> Key: SQOOP-412
> URL: https://issues.apache.org/jira/browse/SQOOP-412
> Project: Sqoop
> Issue Type: Sub-task
> Components: codegen
> Affects Versions: 1.3.0, 1.4.0-incubating
> Reporter: Jarek Jarcec Cecho
> Assignee: Eric Wadsworth
> Priority: Minor
> Labels: patch
> Fix For: 1.4.1-incubating
>
> Attachments: try_6_ConfigurationHelper_getInstances.patch
>
>
> We need to port functionality of org.apache.hadoop.conf.Configuration.getInstances for supporting hadoop releases before 0.21 (or before CDH3).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (SQOOP-412) Create our own implementation of
org.apache.hadoop.conf.Configuration.getInstances
Posted by "Eric Wadsworth (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Wadsworth updated SQOOP-412:
---------------------------------
Labels: patch (was: )
Affects Version/s: 1.3.0
1.4.0-incubating
Status: Patch Available (was: Open)
This provides the getInstances method for use by the apache sqoop code.
> Create our own implementation of org.apache.hadoop.conf.Configuration.getInstances
> ----------------------------------------------------------------------------------
>
> Key: SQOOP-412
> URL: https://issues.apache.org/jira/browse/SQOOP-412
> Project: Sqoop
> Issue Type: Sub-task
> Components: codegen
> Affects Versions: 1.4.0-incubating, 1.3.0
> Reporter: Jarek Jarcec Cecho
> Priority: Minor
> Labels: patch
> Fix For: 1.4.1-incubating
>
>
> We need to port functionality of org.apache.hadoop.conf.Configuration.getInstances for supporting hadoop releases before 0.21 (or before CDH3).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-412) Create our own implementation of
org.apache.hadoop.conf.Configuration.getInstances
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13173014#comment-13173014 ]
Hudson commented on SQOOP-412:
------------------------------
Integrated in Sqoop-ant-jdk-1.6 #70 (See [https://builds.apache.org/job/Sqoop-ant-jdk-1.6/70/])
SQOOP-412. Create our own implementation of org.apache.hadoop.conf.Configuration.getInstances
(Eric Wadsworth via Jarek Jarcec Cecho)
jarcec : http://svn.apache.org/viewvc/?view=rev&rev=1221127
Files :
* /incubator/sqoop/trunk/src/java/org/apache/sqoop/config/ConfigurationHelper.java
* /incubator/sqoop/trunk/src/java/org/apache/sqoop/metastore/JobStorageFactory.java
* /incubator/sqoop/trunk/src/java/org/apache/sqoop/tool/SqoopTool.java
> Create our own implementation of org.apache.hadoop.conf.Configuration.getInstances
> ----------------------------------------------------------------------------------
>
> Key: SQOOP-412
> URL: https://issues.apache.org/jira/browse/SQOOP-412
> Project: Sqoop
> Issue Type: Sub-task
> Components: codegen
> Affects Versions: 1.3.0, 1.4.0-incubating
> Reporter: Jarek Jarcec Cecho
> Assignee: Eric Wadsworth
> Priority: Minor
> Labels: patch
> Fix For: 1.4.1-incubating
>
> Attachments: try_6_ConfigurationHelper_getInstances.patch
>
>
> We need to port functionality of org.apache.hadoop.conf.Configuration.getInstances for supporting hadoop releases before 0.21 (or before CDH3).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-412) Create our own implementation of
org.apache.hadoop.conf.Configuration.getInstances
Posted by "Jarek Jarcec Cecho (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13172998#comment-13172998 ]
Jarek Jarcec Cecho commented on SQOOP-412:
------------------------------------------
Patch committed,
thank you very much Eric!
Jarcec
> Create our own implementation of org.apache.hadoop.conf.Configuration.getInstances
> ----------------------------------------------------------------------------------
>
> Key: SQOOP-412
> URL: https://issues.apache.org/jira/browse/SQOOP-412
> Project: Sqoop
> Issue Type: Sub-task
> Components: codegen
> Affects Versions: 1.3.0, 1.4.0-incubating
> Reporter: Jarek Jarcec Cecho
> Assignee: Eric Wadsworth
> Priority: Minor
> Labels: patch
> Fix For: 1.4.1-incubating
>
> Attachments: try_6_ConfigurationHelper_getInstances.patch
>
>
> We need to port functionality of org.apache.hadoop.conf.Configuration.getInstances for supporting hadoop releases before 0.21 (or before CDH3).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-412) Create our own implementation of
org.apache.hadoop.conf.Configuration.getInstances
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13173727#comment-13173727 ]
jiraposter@reviews.apache.org commented on SQOOP-412:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3272/#review4028
-----------------------------------------------------------
Ship it!
+1
Thanks for the patch Jarcec. The changes look good. I tested basic import against MySQL and Hadoop 0.20.205 and that seems to work.
Please attach the patch to the JIRA and also update the "Bugs" field to SQOOP-413 as that is the relevant JIRA for this issue.
- Arvind
On 2011-12-20 17:09:54, Jarek Cecho wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/3272/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-12-20 17:09:54)
bq.
bq.
bq. Review request for Sqoop and Arvind Prabhakar.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. I've copypasted all four files from CHD3 release and put them into sqoop mapreduce module (I altered files to conform our checkstyle policy). I've also changed all references in sqoop to use this new for classes instead of the default ones present in hadoop-0.21.
bq.
bq.
bq. This addresses bug SQOOP-412.
bq. https://issues.apache.org/jira/browse/SQOOP-412
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. /src/java/com/cloudera/sqoop/mapreduce/CombineShimRecordReader.java 1221303
bq. /src/java/com/cloudera/sqoop/mapreduce/db/DBInputFormat.java 1221303
bq. /src/java/com/cloudera/sqoop/mapreduce/db/DBOutputFormat.java 1221303
bq. /src/java/com/cloudera/sqoop/mapreduce/db/DBRecordReader.java 1221303
bq. /src/java/com/cloudera/sqoop/mapreduce/db/DataDrivenDBInputFormat.java 1221303
bq. /src/java/com/cloudera/sqoop/mapreduce/db/DataDrivenDBRecordReader.java 1221303
bq. /src/java/com/cloudera/sqoop/mapreduce/db/OracleDBRecordReader.java 1221303
bq. /src/java/com/cloudera/sqoop/mapreduce/db/OracleDataDrivenDBInputFormat.java 1221303
bq. /src/java/com/cloudera/sqoop/mapreduce/db/OracleDataDrivenDBRecordReader.java 1221303
bq. /src/java/org/apache/sqoop/config/ConfigurationHelper.java 1221303
bq. /src/java/org/apache/sqoop/lib/SqoopRecord.java 1221303
bq. /src/java/org/apache/sqoop/mapreduce/CombineFileInputFormat.java PRE-CREATION
bq. /src/java/org/apache/sqoop/mapreduce/CombineFileRecordReader.java PRE-CREATION
bq. /src/java/org/apache/sqoop/mapreduce/CombineFileSplit.java PRE-CREATION
bq. /src/java/org/apache/sqoop/mapreduce/CombineShimRecordReader.java 1221303
bq. /src/java/org/apache/sqoop/mapreduce/DBWritable.java PRE-CREATION
bq. /src/java/org/apache/sqoop/mapreduce/DataDrivenImportJob.java 1221303
bq. /src/java/org/apache/sqoop/mapreduce/ExportInputFormat.java 1221303
bq. /src/java/org/apache/sqoop/mapreduce/MySQLDumpImportJob.java 1221303
bq. /src/java/org/apache/sqoop/mapreduce/MySQLExportJob.java 1221303
bq. /src/java/org/apache/sqoop/mapreduce/db/DBConfiguration.java 1221303
bq. /src/java/org/apache/sqoop/mapreduce/db/DBInputFormat.java 1221303
bq. /src/java/org/apache/sqoop/mapreduce/db/DBOutputFormat.java 1221303
bq. /src/java/org/apache/sqoop/mapreduce/db/DBRecordReader.java 1221303
bq. /src/java/org/apache/sqoop/mapreduce/db/DataDrivenDBInputFormat.java 1221303
bq. /src/java/org/apache/sqoop/mapreduce/db/DataDrivenDBRecordReader.java 1221303
bq. /src/java/org/apache/sqoop/mapreduce/db/OracleDBRecordReader.java 1221303
bq. /src/java/org/apache/sqoop/mapreduce/db/OracleDataDrivenDBInputFormat.java 1221303
bq. /src/java/org/apache/sqoop/mapreduce/db/OracleDataDrivenDBRecordReader.java 1221303
bq. /src/test/com/cloudera/sqoop/mapreduce/db/TestDataDrivenDBInputFormat.java 1221303
bq.
bq. Diff: https://reviews.apache.org/r/3272/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. I've tested with both -Dhadopoversion=20 and -Dhadoopversion=23.
bq.
bq.
bq. Thanks,
bq.
bq. Jarek
bq.
bq.
> Create our own implementation of org.apache.hadoop.conf.Configuration.getInstances
> ----------------------------------------------------------------------------------
>
> Key: SQOOP-412
> URL: https://issues.apache.org/jira/browse/SQOOP-412
> Project: Sqoop
> Issue Type: Sub-task
> Components: codegen
> Affects Versions: 1.3.0, 1.4.0-incubating
> Reporter: Jarek Jarcec Cecho
> Assignee: Eric Wadsworth
> Priority: Minor
> Labels: patch
> Fix For: 1.4.1-incubating
>
> Attachments: try_6_ConfigurationHelper_getInstances.patch
>
>
> We need to port functionality of org.apache.hadoop.conf.Configuration.getInstances for supporting hadoop releases before 0.21 (or before CDH3).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (SQOOP-412) Create our own implementation of
org.apache.hadoop.conf.Configuration.getInstances
Posted by "Eric Wadsworth (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Wadsworth updated SQOOP-412:
---------------------------------
Attachment: try_6_ConfigurationHelper_getInstances.patch
This is the patch.
> Create our own implementation of org.apache.hadoop.conf.Configuration.getInstances
> ----------------------------------------------------------------------------------
>
> Key: SQOOP-412
> URL: https://issues.apache.org/jira/browse/SQOOP-412
> Project: Sqoop
> Issue Type: Sub-task
> Components: codegen
> Affects Versions: 1.3.0, 1.4.0-incubating
> Reporter: Jarek Jarcec Cecho
> Priority: Minor
> Labels: patch
> Fix For: 1.4.1-incubating
>
> Attachments: try_6_ConfigurationHelper_getInstances.patch
>
>
> We need to port functionality of org.apache.hadoop.conf.Configuration.getInstances for supporting hadoop releases before 0.21 (or before CDH3).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (SQOOP-412) Create our own implementation of
org.apache.hadoop.conf.Configuration.getInstances
Posted by "Arvind Prabhakar (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arvind Prabhakar reassigned SQOOP-412:
--------------------------------------
Assignee: Eric Wadsworth
> Create our own implementation of org.apache.hadoop.conf.Configuration.getInstances
> ----------------------------------------------------------------------------------
>
> Key: SQOOP-412
> URL: https://issues.apache.org/jira/browse/SQOOP-412
> Project: Sqoop
> Issue Type: Sub-task
> Components: codegen
> Affects Versions: 1.3.0, 1.4.0-incubating
> Reporter: Jarek Jarcec Cecho
> Assignee: Eric Wadsworth
> Priority: Minor
> Labels: patch
> Fix For: 1.4.1-incubating
>
> Attachments: try_6_ConfigurationHelper_getInstances.patch
>
>
> We need to port functionality of org.apache.hadoop.conf.Configuration.getInstances for supporting hadoop releases before 0.21 (or before CDH3).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (SQOOP-412) Create our own implementation of
org.apache.hadoop.conf.Configuration.getInstances
Posted by "Eric Wadsworth (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Wadsworth updated SQOOP-412:
---------------------------------
Comment: was deleted
(was: This provides the getInstances method for use by the apache sqoop code.)
> Create our own implementation of org.apache.hadoop.conf.Configuration.getInstances
> ----------------------------------------------------------------------------------
>
> Key: SQOOP-412
> URL: https://issues.apache.org/jira/browse/SQOOP-412
> Project: Sqoop
> Issue Type: Sub-task
> Components: codegen
> Affects Versions: 1.3.0, 1.4.0-incubating
> Reporter: Jarek Jarcec Cecho
> Priority: Minor
> Labels: patch
> Fix For: 1.4.1-incubating
>
> Attachments: try_6_ConfigurationHelper_getInstances.patch
>
>
> We need to port functionality of org.apache.hadoop.conf.Configuration.getInstances for supporting hadoop releases before 0.21 (or before CDH3).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (SQOOP-412) Create our own implementation of
org.apache.hadoop.conf.Configuration.getInstances
Posted by "Jarek Jarcec Cecho (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jarek Jarcec Cecho updated SQOOP-412:
-------------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Create our own implementation of org.apache.hadoop.conf.Configuration.getInstances
> ----------------------------------------------------------------------------------
>
> Key: SQOOP-412
> URL: https://issues.apache.org/jira/browse/SQOOP-412
> Project: Sqoop
> Issue Type: Sub-task
> Components: codegen
> Affects Versions: 1.3.0, 1.4.0-incubating
> Reporter: Jarek Jarcec Cecho
> Assignee: Eric Wadsworth
> Priority: Minor
> Labels: patch
> Fix For: 1.4.1-incubating
>
> Attachments: try_6_ConfigurationHelper_getInstances.patch
>
>
> We need to port functionality of org.apache.hadoop.conf.Configuration.getInstances for supporting hadoop releases before 0.21 (or before CDH3).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira