You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sqoop.apache.org by Venkat Ranganathan <n....@live.com> on 2014/11/19 08:24:36 UTC
Review Request 28220: SQOOP-1627: Fix Hadoop100 and Hadoop20 profile
builds
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28220/
-----------------------------------------------------------
Review request for Sqoop.
Bugs: SQOOP-1627
https://issues.apache.org/jira/browse/SQOOP-1627
Repository: sqoop-trunk
Description
-------
With the last few changes builds are broken with Hadoop100 and Hadoop20 builds. There are a variety of issues.
We are using classes that are only in Hadoop 2 (TaskAtttemptContextImpl) - This is not even public and stable. We should change it later!
We used Hadoop 2 specific methods because Hadoop 1 methods are deprecated (Job.getInstance)
Method that is only available in later HBase versions (merge) for example
Diffs
-----
build.xml 21fd406
src/java/org/apache/sqoop/mapreduce/HBaseImportJob.java a93114f
src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeDatasetFTPRecordReader.java 613ee7a
src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeDatasetInputFormat.java 70958e0
src/test/org/apache/sqoop/mapreduce/sqlserver/SqlServerUpsertOutputFormatTest.java b8c4538
Diff: https://reviews.apache.org/r/28220/diff/
Testing
-------
Ran with the profiles affected as well as 200 profile
Thanks,
Venkat Ranganathan
Re: Review Request 28220: SQOOP-1627: Fix Hadoop100 and Hadoop20
profile builds
Posted by Abraham Elmahrek <ab...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28220/#review62869
-----------------------------------------------------------
Ship it!
Cool! Can you attach a patch to the Jira?
- Abraham Elmahrek
On Nov. 21, 2014, 1:38 a.m., Venkat Ranganathan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/28220/
> -----------------------------------------------------------
>
> (Updated Nov. 21, 2014, 1:38 a.m.)
>
>
> Review request for Sqoop.
>
>
> Bugs: SQOOP-1627
> https://issues.apache.org/jira/browse/SQOOP-1627
>
>
> Repository: sqoop-trunk
>
>
> Description
> -------
>
> With the last few changes builds are broken with Hadoop100 and Hadoop20 builds. There are a variety of issues.
> We are using classes that are only in Hadoop 2 (TaskAtttemptContextImpl) - This is not even public and stable. We should change it later!
> We used Hadoop 2 specific methods because Hadoop 1 methods are deprecated (Job.getInstance)
> Method that is only available in later HBase versions (merge) for example
>
> I have updated the hadoop version to 1.0.4 to address the MutipleOutputs not being present in Hadoop 1.0.0 and also updated hadoop20 profile to the latest update which allows tests and code to compile (cdh3u5)
>
>
> Diffs
> -----
>
> build.xml 21fd406
> src/java/org/apache/sqoop/mapreduce/HBaseImportJob.java a93114f
> src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeDatasetFTPRecordReader.java 613ee7a
> src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeDatasetInputFormat.java 70958e0
> src/test/org/apache/sqoop/mapreduce/sqlserver/SqlServerUpsertOutputFormatTest.java b8c4538
>
> Diff: https://reviews.apache.org/r/28220/diff/
>
>
> Testing
> -------
>
> Ran with the profiles affected as well as 200 profile. With hadoop20 profile we have 4 test suite failures in Parquet Import/Export and Accumulo import/export and with hadoop100 profile, we have no failures.
>
>
> Thanks,
>
> Venkat Ranganathan
>
>
Re: Review Request 28220: SQOOP-1627: Fix Hadoop100 and Hadoop20
profile builds
Posted by Venkat Ranganathan <n....@live.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28220/
-----------------------------------------------------------
(Updated Nov. 21, 2014, 1:38 a.m.)
Review request for Sqoop.
Changes
-------
Fixed whitespace issues
Bugs: SQOOP-1627
https://issues.apache.org/jira/browse/SQOOP-1627
Repository: sqoop-trunk
Description
-------
With the last few changes builds are broken with Hadoop100 and Hadoop20 builds. There are a variety of issues.
We are using classes that are only in Hadoop 2 (TaskAtttemptContextImpl) - This is not even public and stable. We should change it later!
We used Hadoop 2 specific methods because Hadoop 1 methods are deprecated (Job.getInstance)
Method that is only available in later HBase versions (merge) for example
I have updated the hadoop version to 1.0.4 to address the MutipleOutputs not being present in Hadoop 1.0.0 and also updated hadoop20 profile to the latest update which allows tests and code to compile (cdh3u5)
Diffs (updated)
-----
build.xml 21fd406
src/java/org/apache/sqoop/mapreduce/HBaseImportJob.java a93114f
src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeDatasetFTPRecordReader.java 613ee7a
src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeDatasetInputFormat.java 70958e0
src/test/org/apache/sqoop/mapreduce/sqlserver/SqlServerUpsertOutputFormatTest.java b8c4538
Diff: https://reviews.apache.org/r/28220/diff/
Testing
-------
Ran with the profiles affected as well as 200 profile. With hadoop20 profile we have 4 test suite failures in Parquet Import/Export and Accumulo import/export and with hadoop100 profile, we have no failures.
Thanks,
Venkat Ranganathan
Re: Review Request 28220: SQOOP-1627: Fix Hadoop100 and Hadoop20
profile builds
Posted by Venkat Ranganathan <n....@live.com>.
> On Nov. 20, 2014, 7:41 a.m., Abraham Elmahrek wrote:
> > Good stuff. Watch out for whitespace as usual.
Thanks Abe. Will fix it.
> On Nov. 20, 2014, 7:41 a.m., Abraham Elmahrek wrote:
> > src/java/org/apache/sqoop/mapreduce/HBaseImportJob.java, lines 158-176
> > <https://reviews.apache.org/r/28220/diff/1/?file=768862#file768862line158>
> >
> > Can't 'create' just be called? It seems like 'create(Configuration)' is doing pretty much the same work as 'merge(Configuration, Configuration)'.
> >
> > http://grepcode.com/file/repo1.maven.org/maven2/org.apache.hbase/hbase/0.92.0/org/apache/hadoop/hbase/HBaseConfiguration.java#HBaseConfiguration.create%28org.apache.hadoop.hbase.Configuration%29
Good point. I did think about it but given we are talking about some varying versions here, I thought restoring code is what we need to do. Enhancement can be followed up separately.
- Venkat
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28220/#review62329
-----------------------------------------------------------
On Nov. 19, 2014, 4:03 p.m., Venkat Ranganathan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/28220/
> -----------------------------------------------------------
>
> (Updated Nov. 19, 2014, 4:03 p.m.)
>
>
> Review request for Sqoop.
>
>
> Bugs: SQOOP-1627
> https://issues.apache.org/jira/browse/SQOOP-1627
>
>
> Repository: sqoop-trunk
>
>
> Description
> -------
>
> With the last few changes builds are broken with Hadoop100 and Hadoop20 builds. There are a variety of issues.
> We are using classes that are only in Hadoop 2 (TaskAtttemptContextImpl) - This is not even public and stable. We should change it later!
> We used Hadoop 2 specific methods because Hadoop 1 methods are deprecated (Job.getInstance)
> Method that is only available in later HBase versions (merge) for example
>
> I have updated the hadoop version to 1.0.4 to address the MutipleOutputs not being present in Hadoop 1.0.0 and also updated hadoop20 profile to the latest update which allows tests and code to compile (cdh3u5)
>
>
> Diffs
> -----
>
> build.xml 21fd406
> src/java/org/apache/sqoop/mapreduce/HBaseImportJob.java a93114f
> src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeDatasetFTPRecordReader.java 613ee7a
> src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeDatasetInputFormat.java 70958e0
> src/test/org/apache/sqoop/mapreduce/sqlserver/SqlServerUpsertOutputFormatTest.java b8c4538
>
> Diff: https://reviews.apache.org/r/28220/diff/
>
>
> Testing
> -------
>
> Ran with the profiles affected as well as 200 profile. With hadoop20 profile we have 4 test suite failures in Parquet Import/Export and Accumulo import/export and with hadoop100 profile, we have no failures.
>
>
> Thanks,
>
> Venkat Ranganathan
>
>
Re: Review Request 28220: SQOOP-1627: Fix Hadoop100 and Hadoop20
profile builds
Posted by Abraham Elmahrek <ab...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28220/#review62329
-----------------------------------------------------------
Ship it!
Good stuff. Watch out for whitespace as usual.
src/java/org/apache/sqoop/mapreduce/HBaseImportJob.java
<https://reviews.apache.org/r/28220/#comment104360>
Can't 'create' just be called? It seems like 'create(Configuration)' is doing pretty much the same work as 'merge(Configuration, Configuration)'.
http://grepcode.com/file/repo1.maven.org/maven2/org.apache.hbase/hbase/0.92.0/org/apache/hadoop/hbase/HBaseConfiguration.java#HBaseConfiguration.create%28org.apache.hadoop.hbase.Configuration%29
- Abraham Elmahrek
On Nov. 19, 2014, 4:03 p.m., Venkat Ranganathan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/28220/
> -----------------------------------------------------------
>
> (Updated Nov. 19, 2014, 4:03 p.m.)
>
>
> Review request for Sqoop.
>
>
> Bugs: SQOOP-1627
> https://issues.apache.org/jira/browse/SQOOP-1627
>
>
> Repository: sqoop-trunk
>
>
> Description
> -------
>
> With the last few changes builds are broken with Hadoop100 and Hadoop20 builds. There are a variety of issues.
> We are using classes that are only in Hadoop 2 (TaskAtttemptContextImpl) - This is not even public and stable. We should change it later!
> We used Hadoop 2 specific methods because Hadoop 1 methods are deprecated (Job.getInstance)
> Method that is only available in later HBase versions (merge) for example
>
> I have updated the hadoop version to 1.0.4 to address the MutipleOutputs not being present in Hadoop 1.0.0 and also updated hadoop20 profile to the latest update which allows tests and code to compile (cdh3u5)
>
>
> Diffs
> -----
>
> build.xml 21fd406
> src/java/org/apache/sqoop/mapreduce/HBaseImportJob.java a93114f
> src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeDatasetFTPRecordReader.java 613ee7a
> src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeDatasetInputFormat.java 70958e0
> src/test/org/apache/sqoop/mapreduce/sqlserver/SqlServerUpsertOutputFormatTest.java b8c4538
>
> Diff: https://reviews.apache.org/r/28220/diff/
>
>
> Testing
> -------
>
> Ran with the profiles affected as well as 200 profile. With hadoop20 profile we have 4 test suite failures in Parquet Import/Export and Accumulo import/export and with hadoop100 profile, we have no failures.
>
>
> Thanks,
>
> Venkat Ranganathan
>
>
Re: Review Request 28220: SQOOP-1627: Fix Hadoop100 and Hadoop20
profile builds
Posted by Venkat Ranganathan <n....@live.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28220/
-----------------------------------------------------------
(Updated Nov. 19, 2014, 4:03 p.m.)
Review request for Sqoop.
Changes
-------
Updated failed tests in hadoop20 profile
Bugs: SQOOP-1627
https://issues.apache.org/jira/browse/SQOOP-1627
Repository: sqoop-trunk
Description (updated)
-------
With the last few changes builds are broken with Hadoop100 and Hadoop20 builds. There are a variety of issues.
We are using classes that are only in Hadoop 2 (TaskAtttemptContextImpl) - This is not even public and stable. We should change it later!
We used Hadoop 2 specific methods because Hadoop 1 methods are deprecated (Job.getInstance)
Method that is only available in later HBase versions (merge) for example
I have updated the hadoop version to 1.0.4 to address the MutipleOutputs not being present in Hadoop 1.0.0 and also updated hadoop20 profile to the latest update which allows tests and code to compile (cdh3u5)
Diffs
-----
build.xml 21fd406
src/java/org/apache/sqoop/mapreduce/HBaseImportJob.java a93114f
src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeDatasetFTPRecordReader.java 613ee7a
src/test/org/apache/sqoop/mapreduce/mainframe/TestMainframeDatasetInputFormat.java 70958e0
src/test/org/apache/sqoop/mapreduce/sqlserver/SqlServerUpsertOutputFormatTest.java b8c4538
Diff: https://reviews.apache.org/r/28220/diff/
Testing (updated)
-------
Ran with the profiles affected as well as 200 profile. With hadoop20 profile we have 4 test suite failures in Parquet Import/Export and Accumulo import/export and with hadoop100 profile, we have no failures.
Thanks,
Venkat Ranganathan