You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sqoop.apache.org by Abraham Elmahrek <ab...@cloudera.com> on 2013/08/16 02:17:08 UTC
Re: Review Request 10964: SQOOP-927 Sqoop2: Integration: Mapreduce specific
tests should be running on MiniCluster
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10964/
-----------------------------------------------------------
(Updated Aug. 16, 2013, 12:17 a.m.)
Review request for Sqoop and Jarek Cecho.
Changes
-------
Rebased
Bugs: SQOOP-927
https://issues.apache.org/jira/browse/SQOOP-927
Repository: sqoop-sqoop2
Description
-------
commit ff7dba55a09dd7789a34136233000c625759e583
Author: Abraham Elmahrek <ab...@elmahrek.com>
Date: Fri Apr 26 15:10:24 2013 -0700
SQOOP-927 Sqoop2: Integration: Mapreduce specific tests should be running on MiniCluster
Handle MiniDFSCluster and MiniMRClientCluster on own.
Set yarn.application.classpath to get over classpath errors.
Set to use fair scheduler.
:100644 100644 0abbb18... f09704b... M pom.xml
:100644 100644 6eb3184... a4c2a5b... M test/src/test/java/org/apache/sqoop/integration/TomcatTestCase.java
:100644 100644 0f48a8b... 758c885... M test/src/test/java/org/apache/sqoop/integration/server/VersionTest.java
Diffs (updated)
-----
dist/src/main/server/conf/catalina.properties 5a47295d6e1bfa2114f86d620bb1452bc576df71
pom.xml 513b6d0570ee6f1ec32f8c35a403becc1cfa156a
test/pom.xml 8001fcec29bf5a7d9c3119b9f6ef548f678e1ebe
test/src/main/java/org/apache/sqoop/test/asserts/HdfsAsserts.java c17527285167104994d78f5919b7f11c0c31cd15
test/src/main/java/org/apache/sqoop/test/hadoop/HadoopLocalRunner.java PRE-CREATION
test/src/main/java/org/apache/sqoop/test/hadoop/HadoopMiniClusterRunner.java PRE-CREATION
test/src/main/java/org/apache/sqoop/test/hadoop/HadoopRunner.java PRE-CREATION
test/src/main/java/org/apache/sqoop/test/hadoop/HadoopRunnerFactory.java PRE-CREATION
test/src/main/java/org/apache/sqoop/test/testcases/ConnectorTestCase.java d10b94268babeb3dc7b1d2450776af5fd8449fd3
test/src/main/java/org/apache/sqoop/test/testcases/TomcatTestCase.java efdfed4e2065c0c01c16da88cf55705ab13d7bcf
test/src/main/java/org/apache/sqoop/test/utils/HdfsUtils.java 95dd177a11c5d822e35ca54e5d93785f0a40fbfc
Diff: https://reviews.apache.org/r/10964/diff/
Testing
-------
Ran integration tests without and with miniclusters.
Currently need to use both miniclusters or neither for tests to work.
Thanks,
Abraham Elmahrek
Re: Review Request 10964: SQOOP-927 Sqoop2: Integration: Mapreduce specific
tests should be running on MiniCluster
Posted by Jarek Cecho <ja...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10964/#review27347
-----------------------------------------------------------
Ship it!
Ship It!
- Jarek Cecho
On Oct. 18, 2013, 2 a.m., Abraham Elmahrek wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/10964/
> -----------------------------------------------------------
>
> (Updated Oct. 18, 2013, 2 a.m.)
>
>
> Review request for Sqoop and Jarek Cecho.
>
>
> Bugs: SQOOP-927
> https://issues.apache.org/jira/browse/SQOOP-927
>
>
> Repository: sqoop-sqoop2
>
>
> Description
> -------
>
> commit 2cc666f2aa1ac679a58099819a6985a19dfb5de9
> Author: Abraham Elmahrek <ab...@elmahrek.com>
> Date: Tue Jul 2 11:53:57 2013 -0700
>
> SQOOP-927 Sqoop2: Integration: Mapreduce specific tests should be running on MiniCluster
>
> Minicluster test output set to <temp dir>/minicluster.
>
> Set yarn.application.classpath to get over classpath errors.
> Set to use fair scheduler.
>
> :100644 100644 513b6d0... 524117e... M pom.xml
> :100644 100644 c175272... d8f2b8d... M test/src/main/java/org/apache/sqoop/test/asserts/HdfsAsserts.java
> :000000 100644 0000000... 44465b4... A test/src/main/java/org/apache/sqoop/test/hadoop/HadoopLocalRunner.java
> :000000 100644 0000000... b06dcab... A test/src/main/java/org/apache/sqoop/test/hadoop/HadoopMiniClusterRunner.java
> :000000 100644 0000000... 2516ff1... A test/src/main/java/org/apache/sqoop/test/hadoop/HadoopRunner.java
> :000000 100644 0000000... 020fa3f... A test/src/main/java/org/apache/sqoop/test/hadoop/HadoopRunnerFactory.java
> :100644 100644 d10b942... 5ec4fa4... M test/src/main/java/org/apache/sqoop/test/testcases/ConnectorTestCase.java
> :100644 100644 efdfed4... 7e2558f... M test/src/main/java/org/apache/sqoop/test/testcases/TomcatTestCase.java
> :100644 100644 95dd177... 59c5f15... M test/src/main/java/org/apache/sqoop/test/utils/HdfsUtils.java
>
>
> Diffs
> -----
>
> pom.xml 513b6d0570ee6f1ec32f8c35a403becc1cfa156a
> test/src/main/java/org/apache/sqoop/test/asserts/HdfsAsserts.java c17527285167104994d78f5919b7f11c0c31cd15
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopLocalRunner.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopMiniClusterRunner.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopRunner.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopRunnerFactory.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/testcases/ConnectorTestCase.java d10b94268babeb3dc7b1d2450776af5fd8449fd3
> test/src/main/java/org/apache/sqoop/test/testcases/TomcatTestCase.java efdfed4e2065c0c01c16da88cf55705ab13d7bcf
> test/src/main/java/org/apache/sqoop/test/utils/HdfsUtils.java 95dd177a11c5d822e35ca54e5d93785f0a40fbfc
>
> Diff: https://reviews.apache.org/r/10964/diff/
>
>
> Testing
> -------
>
> mvn clean integration-test -DfailIfNoTests=false -X -Phadoop100 -Dhadoop.profile=100 -Dsurefire.useFile=false -Dmaven.test.redirectTestOutputToFile=false
> mvn clean integration-test -DfailIfNoTests=false -X -Dsurefire.useFile=false -Dmaven.test.redirectTestOutputToFile=false
> mvn clean integration-test -DfailIfNoTests=false -X -Phadoop100 -Dhadoop.profile=100 -Dsurefire.useFile=false -Dmaven.test.redirectTestOutputToFile=false -Dsqoop.hadoop.runner.class=org.apache.sqoop.test.hadoop.HadoopLocalRunner
> mvn clean integration-test -DfailIfNoTests=false -X -Dsurefire.useFile=false -Dmaven.test.redirectTestOutputToFile=false -Dsqoop.hadoop.runner.class=org.apache.sqoop.test.hadoop.HadoopLocalRunner
>
>
> Thanks,
>
> Abraham Elmahrek
>
>
Re: Review Request 10964: SQOOP-927 Sqoop2: Integration: Mapreduce specific
tests should be running on MiniCluster
Posted by Abraham Elmahrek <ab...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10964/
-----------------------------------------------------------
(Updated Oct. 18, 2013, 2 a.m.)
Review request for Sqoop and Jarek Cecho.
Bugs: SQOOP-927
https://issues.apache.org/jira/browse/SQOOP-927
Repository: sqoop-sqoop2
Description (updated)
-------
commit 2cc666f2aa1ac679a58099819a6985a19dfb5de9
Author: Abraham Elmahrek <ab...@elmahrek.com>
Date: Tue Jul 2 11:53:57 2013 -0700
SQOOP-927 Sqoop2: Integration: Mapreduce specific tests should be running on MiniCluster
Minicluster test output set to <temp dir>/minicluster.
Set yarn.application.classpath to get over classpath errors.
Set to use fair scheduler.
:100644 100644 513b6d0... 524117e... M pom.xml
:100644 100644 c175272... d8f2b8d... M test/src/main/java/org/apache/sqoop/test/asserts/HdfsAsserts.java
:000000 100644 0000000... 44465b4... A test/src/main/java/org/apache/sqoop/test/hadoop/HadoopLocalRunner.java
:000000 100644 0000000... b06dcab... A test/src/main/java/org/apache/sqoop/test/hadoop/HadoopMiniClusterRunner.java
:000000 100644 0000000... 2516ff1... A test/src/main/java/org/apache/sqoop/test/hadoop/HadoopRunner.java
:000000 100644 0000000... 020fa3f... A test/src/main/java/org/apache/sqoop/test/hadoop/HadoopRunnerFactory.java
:100644 100644 d10b942... 5ec4fa4... M test/src/main/java/org/apache/sqoop/test/testcases/ConnectorTestCase.java
:100644 100644 efdfed4... 7e2558f... M test/src/main/java/org/apache/sqoop/test/testcases/TomcatTestCase.java
:100644 100644 95dd177... 59c5f15... M test/src/main/java/org/apache/sqoop/test/utils/HdfsUtils.java
Diffs (updated)
-----
pom.xml 513b6d0570ee6f1ec32f8c35a403becc1cfa156a
test/src/main/java/org/apache/sqoop/test/asserts/HdfsAsserts.java c17527285167104994d78f5919b7f11c0c31cd15
test/src/main/java/org/apache/sqoop/test/hadoop/HadoopLocalRunner.java PRE-CREATION
test/src/main/java/org/apache/sqoop/test/hadoop/HadoopMiniClusterRunner.java PRE-CREATION
test/src/main/java/org/apache/sqoop/test/hadoop/HadoopRunner.java PRE-CREATION
test/src/main/java/org/apache/sqoop/test/hadoop/HadoopRunnerFactory.java PRE-CREATION
test/src/main/java/org/apache/sqoop/test/testcases/ConnectorTestCase.java d10b94268babeb3dc7b1d2450776af5fd8449fd3
test/src/main/java/org/apache/sqoop/test/testcases/TomcatTestCase.java efdfed4e2065c0c01c16da88cf55705ab13d7bcf
test/src/main/java/org/apache/sqoop/test/utils/HdfsUtils.java 95dd177a11c5d822e35ca54e5d93785f0a40fbfc
Diff: https://reviews.apache.org/r/10964/diff/
Testing (updated)
-------
mvn clean integration-test -DfailIfNoTests=false -X -Phadoop100 -Dhadoop.profile=100 -Dsurefire.useFile=false -Dmaven.test.redirectTestOutputToFile=false
mvn clean integration-test -DfailIfNoTests=false -X -Dsurefire.useFile=false -Dmaven.test.redirectTestOutputToFile=false
mvn clean integration-test -DfailIfNoTests=false -X -Phadoop100 -Dhadoop.profile=100 -Dsurefire.useFile=false -Dmaven.test.redirectTestOutputToFile=false -Dsqoop.hadoop.runner.class=org.apache.sqoop.test.hadoop.HadoopLocalRunner
mvn clean integration-test -DfailIfNoTests=false -X -Dsurefire.useFile=false -Dmaven.test.redirectTestOutputToFile=false -Dsqoop.hadoop.runner.class=org.apache.sqoop.test.hadoop.HadoopLocalRunner
Thanks,
Abraham Elmahrek
Re: Review Request 10964: SQOOP-927 Sqoop2: Integration: Mapreduce specific
tests should be running on MiniCluster
Posted by Jarek Cecho <ja...@apache.org>.
On Oct. 16, 2013, 10:27 p.m., Abraham Elmahrek wrote:
> > Jarcec
>
> Abraham Elmahrek wrote:
> Unfortunately, there is a test case that can take up to 30 minutes I've found. The PartitionerTest uses the parameterized runner and can take a while to finish.
I see, I guess that I have too powerful machine as it needed only 6 minutes :-)
- Jarek
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10964/#review27093
-----------------------------------------------------------
On Aug. 26, 2013, 5:24 p.m., Abraham Elmahrek wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/10964/
> -----------------------------------------------------------
>
> (Updated Aug. 26, 2013, 5:24 p.m.)
>
>
> Review request for Sqoop and Jarek Cecho.
>
>
> Bugs: SQOOP-927
> https://issues.apache.org/jira/browse/SQOOP-927
>
>
> Repository: sqoop-sqoop2
>
>
> Description
> -------
>
> commit ff7dba55a09dd7789a34136233000c625759e583
> Author: Abraham Elmahrek <ab...@elmahrek.com>
> Date: Fri Apr 26 15:10:24 2013 -0700
>
> SQOOP-927 Sqoop2: Integration: Mapreduce specific tests should be running on MiniCluster
>
> Handle MiniDFSCluster and MiniMRClientCluster on own.
>
> Set yarn.application.classpath to get over classpath errors.
> Set to use fair scheduler.
>
> :100644 100644 0abbb18... f09704b... M pom.xml
> :100644 100644 6eb3184... a4c2a5b... M test/src/test/java/org/apache/sqoop/integration/TomcatTestCase.java
> :100644 100644 0f48a8b... 758c885... M test/src/test/java/org/apache/sqoop/integration/server/VersionTest.java
>
>
> Diffs
> -----
>
> pom.xml 513b6d0570ee6f1ec32f8c35a403becc1cfa156a
> test/pom.xml 8001fcec29bf5a7d9c3119b9f6ef548f678e1ebe
> test/src/main/java/org/apache/sqoop/test/asserts/HdfsAsserts.java c17527285167104994d78f5919b7f11c0c31cd15
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopLocalRunner.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopMiniClusterRunner.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopRunner.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopRunnerFactory.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/testcases/ConnectorTestCase.java d10b94268babeb3dc7b1d2450776af5fd8449fd3
> test/src/main/java/org/apache/sqoop/test/testcases/TomcatTestCase.java efdfed4e2065c0c01c16da88cf55705ab13d7bcf
> test/src/main/java/org/apache/sqoop/test/utils/HdfsUtils.java 95dd177a11c5d822e35ca54e5d93785f0a40fbfc
>
> Diff: https://reviews.apache.org/r/10964/diff/
>
>
> Testing
> -------
>
> Ran integration tests without and with miniclusters.
> Currently need to use both miniclusters or neither for tests to work.
>
>
> Thanks,
>
> Abraham Elmahrek
>
>
Re: Review Request 10964: SQOOP-927 Sqoop2: Integration: Mapreduce specific
tests should be running on MiniCluster
Posted by Abraham Elmahrek <ab...@cloudera.com>.
> On Oct. 16, 2013, 10:27 p.m., Jarek Cecho wrote:
> > Hi Abe,
> > thank you very much for working on this patch! Please accept my apologies for late review! I'm having difficulties in Hadoop 100 profile - getting NullPointerExceptions in the cluster initialization. I was wondering if this patch version is not yet ready for Hadoop 1 or whether I should dig into my environment to see what is going on.
> >
> >
> >
Jarcec, great points. I'm looking into what's going wrong with the Hadoop 100 profile right now. As always, thank you for the review!
> On Oct. 16, 2013, 10:27 p.m., Jarek Cecho wrote:
> > pom.xml, line 99
> > <https://reviews.apache.org/r/10964/diff/8/?file=344782#file344782line99>
> >
> > Can we wait until SQOOP-1218 will get committed? That will upgrade the version even further to 2.2.0 (first GA release of Hadoop 2).
Indeed.
> On Oct. 16, 2013, 10:27 p.m., Jarek Cecho wrote:
> > test/pom.xml, line 120
> > <https://reviews.apache.org/r/10964/diff/8/?file=344783#file344783line120>
> >
> > One hour long timeout seems too much, what about 900 seconds (15 minutes)?
Unfortunately, there is a test case that can take up to 30 minutes I've found. The PartitionerTest uses the parameterized runner and can take a while to finish.
On Oct. 16, 2013, 10:27 p.m., Abraham Elmahrek wrote:
> > Jarcec
Unfortunately, there is a test case that can take up to 30 minutes I've found. The PartitionerTest uses the parameterized runner and can take a while to finish.
- Abraham
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10964/#review27093
-----------------------------------------------------------
On Aug. 26, 2013, 5:24 p.m., Abraham Elmahrek wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/10964/
> -----------------------------------------------------------
>
> (Updated Aug. 26, 2013, 5:24 p.m.)
>
>
> Review request for Sqoop and Jarek Cecho.
>
>
> Bugs: SQOOP-927
> https://issues.apache.org/jira/browse/SQOOP-927
>
>
> Repository: sqoop-sqoop2
>
>
> Description
> -------
>
> commit ff7dba55a09dd7789a34136233000c625759e583
> Author: Abraham Elmahrek <ab...@elmahrek.com>
> Date: Fri Apr 26 15:10:24 2013 -0700
>
> SQOOP-927 Sqoop2: Integration: Mapreduce specific tests should be running on MiniCluster
>
> Handle MiniDFSCluster and MiniMRClientCluster on own.
>
> Set yarn.application.classpath to get over classpath errors.
> Set to use fair scheduler.
>
> :100644 100644 0abbb18... f09704b... M pom.xml
> :100644 100644 6eb3184... a4c2a5b... M test/src/test/java/org/apache/sqoop/integration/TomcatTestCase.java
> :100644 100644 0f48a8b... 758c885... M test/src/test/java/org/apache/sqoop/integration/server/VersionTest.java
>
>
> Diffs
> -----
>
> pom.xml 513b6d0570ee6f1ec32f8c35a403becc1cfa156a
> test/pom.xml 8001fcec29bf5a7d9c3119b9f6ef548f678e1ebe
> test/src/main/java/org/apache/sqoop/test/asserts/HdfsAsserts.java c17527285167104994d78f5919b7f11c0c31cd15
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopLocalRunner.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopMiniClusterRunner.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopRunner.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopRunnerFactory.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/testcases/ConnectorTestCase.java d10b94268babeb3dc7b1d2450776af5fd8449fd3
> test/src/main/java/org/apache/sqoop/test/testcases/TomcatTestCase.java efdfed4e2065c0c01c16da88cf55705ab13d7bcf
> test/src/main/java/org/apache/sqoop/test/utils/HdfsUtils.java 95dd177a11c5d822e35ca54e5d93785f0a40fbfc
>
> Diff: https://reviews.apache.org/r/10964/diff/
>
>
> Testing
> -------
>
> Ran integration tests without and with miniclusters.
> Currently need to use both miniclusters or neither for tests to work.
>
>
> Thanks,
>
> Abraham Elmahrek
>
>
Re: Review Request 10964: SQOOP-927 Sqoop2: Integration: Mapreduce specific
tests should be running on MiniCluster
Posted by Jarek Cecho <ja...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10964/#review27093
-----------------------------------------------------------
Hi Abe,
thank you very much for working on this patch! Please accept my apologies for late review! I'm having difficulties in Hadoop 100 profile - getting NullPointerExceptions in the cluster initialization. I was wondering if this patch version is not yet ready for Hadoop 1 or whether I should dig into my environment to see what is going on.
pom.xml
<https://reviews.apache.org/r/10964/#comment52748>
Can we wait until SQOOP-1218 will get committed? That will upgrade the version even further to 2.2.0 (first GA release of Hadoop 2).
test/pom.xml
<https://reviews.apache.org/r/10964/#comment52749>
One hour long timeout seems too much, what about 900 seconds (15 minutes)?
Jarcec
- Jarek Cecho
On Aug. 26, 2013, 5:24 p.m., Abraham Elmahrek wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/10964/
> -----------------------------------------------------------
>
> (Updated Aug. 26, 2013, 5:24 p.m.)
>
>
> Review request for Sqoop and Jarek Cecho.
>
>
> Bugs: SQOOP-927
> https://issues.apache.org/jira/browse/SQOOP-927
>
>
> Repository: sqoop-sqoop2
>
>
> Description
> -------
>
> commit ff7dba55a09dd7789a34136233000c625759e583
> Author: Abraham Elmahrek <ab...@elmahrek.com>
> Date: Fri Apr 26 15:10:24 2013 -0700
>
> SQOOP-927 Sqoop2: Integration: Mapreduce specific tests should be running on MiniCluster
>
> Handle MiniDFSCluster and MiniMRClientCluster on own.
>
> Set yarn.application.classpath to get over classpath errors.
> Set to use fair scheduler.
>
> :100644 100644 0abbb18... f09704b... M pom.xml
> :100644 100644 6eb3184... a4c2a5b... M test/src/test/java/org/apache/sqoop/integration/TomcatTestCase.java
> :100644 100644 0f48a8b... 758c885... M test/src/test/java/org/apache/sqoop/integration/server/VersionTest.java
>
>
> Diffs
> -----
>
> pom.xml 513b6d0570ee6f1ec32f8c35a403becc1cfa156a
> test/pom.xml 8001fcec29bf5a7d9c3119b9f6ef548f678e1ebe
> test/src/main/java/org/apache/sqoop/test/asserts/HdfsAsserts.java c17527285167104994d78f5919b7f11c0c31cd15
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopLocalRunner.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopMiniClusterRunner.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopRunner.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopRunnerFactory.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/testcases/ConnectorTestCase.java d10b94268babeb3dc7b1d2450776af5fd8449fd3
> test/src/main/java/org/apache/sqoop/test/testcases/TomcatTestCase.java efdfed4e2065c0c01c16da88cf55705ab13d7bcf
> test/src/main/java/org/apache/sqoop/test/utils/HdfsUtils.java 95dd177a11c5d822e35ca54e5d93785f0a40fbfc
>
> Diff: https://reviews.apache.org/r/10964/diff/
>
>
> Testing
> -------
>
> Ran integration tests without and with miniclusters.
> Currently need to use both miniclusters or neither for tests to work.
>
>
> Thanks,
>
> Abraham Elmahrek
>
>
Re: Review Request 10964: SQOOP-927 Sqoop2: Integration: Mapreduce specific
tests should be running on MiniCluster
Posted by Abraham Elmahrek <ab...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10964/
-----------------------------------------------------------
(Updated Aug. 26, 2013, 5:24 p.m.)
Review request for Sqoop and Jarek Cecho.
Bugs: SQOOP-927
https://issues.apache.org/jira/browse/SQOOP-927
Repository: sqoop-sqoop2
Description
-------
commit ff7dba55a09dd7789a34136233000c625759e583
Author: Abraham Elmahrek <ab...@elmahrek.com>
Date: Fri Apr 26 15:10:24 2013 -0700
SQOOP-927 Sqoop2: Integration: Mapreduce specific tests should be running on MiniCluster
Handle MiniDFSCluster and MiniMRClientCluster on own.
Set yarn.application.classpath to get over classpath errors.
Set to use fair scheduler.
:100644 100644 0abbb18... f09704b... M pom.xml
:100644 100644 6eb3184... a4c2a5b... M test/src/test/java/org/apache/sqoop/integration/TomcatTestCase.java
:100644 100644 0f48a8b... 758c885... M test/src/test/java/org/apache/sqoop/integration/server/VersionTest.java
Diffs (updated)
-----
pom.xml 513b6d0570ee6f1ec32f8c35a403becc1cfa156a
test/pom.xml 8001fcec29bf5a7d9c3119b9f6ef548f678e1ebe
test/src/main/java/org/apache/sqoop/test/asserts/HdfsAsserts.java c17527285167104994d78f5919b7f11c0c31cd15
test/src/main/java/org/apache/sqoop/test/hadoop/HadoopLocalRunner.java PRE-CREATION
test/src/main/java/org/apache/sqoop/test/hadoop/HadoopMiniClusterRunner.java PRE-CREATION
test/src/main/java/org/apache/sqoop/test/hadoop/HadoopRunner.java PRE-CREATION
test/src/main/java/org/apache/sqoop/test/hadoop/HadoopRunnerFactory.java PRE-CREATION
test/src/main/java/org/apache/sqoop/test/testcases/ConnectorTestCase.java d10b94268babeb3dc7b1d2450776af5fd8449fd3
test/src/main/java/org/apache/sqoop/test/testcases/TomcatTestCase.java efdfed4e2065c0c01c16da88cf55705ab13d7bcf
test/src/main/java/org/apache/sqoop/test/utils/HdfsUtils.java 95dd177a11c5d822e35ca54e5d93785f0a40fbfc
Diff: https://reviews.apache.org/r/10964/diff/
Testing
-------
Ran integration tests without and with miniclusters.
Currently need to use both miniclusters or neither for tests to work.
Thanks,
Abraham Elmahrek
Re: Review Request 10964: SQOOP-927 Sqoop2: Integration: Mapreduce specific
tests should be running on MiniCluster
Posted by Abraham Elmahrek <ab...@cloudera.com>.
> On Aug. 23, 2013, 11:24 p.m., Jarek Cecho wrote:
> > dist/src/main/server/conf/catalina.properties, line 47
> > <https://reviews.apache.org/r/10964/diff/7/?file=341604#file341604line47>
> >
> > Is this change relevant for the patch?
No. Sorry. I have to add that every time to test Sqoop. I'll remove it.
> On Aug. 23, 2013, 11:24 p.m., Jarek Cecho wrote:
> > test/pom.xml, line 120
> > <https://reviews.apache.org/r/10964/diff/7/?file=341606#file341606line120>
> >
> > Setting the timeout to 0 might be quite dangerous especially when running on Jenkins. Would it be possible to set here some high, but still finite number?
Sure, I'll set it to 1 hour.
> On Aug. 23, 2013, 11:24 p.m., Jarek Cecho wrote:
> > test/src/main/java/org/apache/sqoop/test/hadoop/HadoopMiniClusterRunner.java, lines 71-72
> > <https://reviews.apache.org/r/10964/diff/7/?file=341609#file341609line71>
> >
> > Nit: Wouldn't make sense to log "Stopping" and then start stopping the service?
I think you mean "Stopping..." then "Stopped"?
> On Aug. 23, 2013, 11:24 p.m., Jarek Cecho wrote:
> > test/src/main/java/org/apache/sqoop/test/hadoop/HadoopMiniClusterRunner.java, lines 74-75
> > <https://reviews.apache.org/r/10964/diff/7/?file=341609#file341609line74>
> >
> > Nit: Wouldn't make sense to log "Stopping" and then start stopping the service?
same as above
- Abraham
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10964/#review25479
-----------------------------------------------------------
On Aug. 16, 2013, 12:17 a.m., Abraham Elmahrek wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/10964/
> -----------------------------------------------------------
>
> (Updated Aug. 16, 2013, 12:17 a.m.)
>
>
> Review request for Sqoop and Jarek Cecho.
>
>
> Bugs: SQOOP-927
> https://issues.apache.org/jira/browse/SQOOP-927
>
>
> Repository: sqoop-sqoop2
>
>
> Description
> -------
>
> commit ff7dba55a09dd7789a34136233000c625759e583
> Author: Abraham Elmahrek <ab...@elmahrek.com>
> Date: Fri Apr 26 15:10:24 2013 -0700
>
> SQOOP-927 Sqoop2: Integration: Mapreduce specific tests should be running on MiniCluster
>
> Handle MiniDFSCluster and MiniMRClientCluster on own.
>
> Set yarn.application.classpath to get over classpath errors.
> Set to use fair scheduler.
>
> :100644 100644 0abbb18... f09704b... M pom.xml
> :100644 100644 6eb3184... a4c2a5b... M test/src/test/java/org/apache/sqoop/integration/TomcatTestCase.java
> :100644 100644 0f48a8b... 758c885... M test/src/test/java/org/apache/sqoop/integration/server/VersionTest.java
>
>
> Diffs
> -----
>
> dist/src/main/server/conf/catalina.properties 5a47295d6e1bfa2114f86d620bb1452bc576df71
> pom.xml 513b6d0570ee6f1ec32f8c35a403becc1cfa156a
> test/pom.xml 8001fcec29bf5a7d9c3119b9f6ef548f678e1ebe
> test/src/main/java/org/apache/sqoop/test/asserts/HdfsAsserts.java c17527285167104994d78f5919b7f11c0c31cd15
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopLocalRunner.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopMiniClusterRunner.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopRunner.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopRunnerFactory.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/testcases/ConnectorTestCase.java d10b94268babeb3dc7b1d2450776af5fd8449fd3
> test/src/main/java/org/apache/sqoop/test/testcases/TomcatTestCase.java efdfed4e2065c0c01c16da88cf55705ab13d7bcf
> test/src/main/java/org/apache/sqoop/test/utils/HdfsUtils.java 95dd177a11c5d822e35ca54e5d93785f0a40fbfc
>
> Diff: https://reviews.apache.org/r/10964/diff/
>
>
> Testing
> -------
>
> Ran integration tests without and with miniclusters.
> Currently need to use both miniclusters or neither for tests to work.
>
>
> Thanks,
>
> Abraham Elmahrek
>
>
Re: Review Request 10964: SQOOP-927 Sqoop2: Integration: Mapreduce specific
tests should be running on MiniCluster
Posted by Jarek Cecho <ja...@apache.org>.
> On Aug. 23, 2013, 11:24 p.m., Jarek Cecho wrote:
> > test/src/main/java/org/apache/sqoop/test/hadoop/HadoopMiniClusterRunner.java, lines 71-72
> > <https://reviews.apache.org/r/10964/diff/7/?file=341609#file341609line71>
> >
> > Nit: Wouldn't make sense to log "Stopping" and then start stopping the service?
>
> Abraham Elmahrek wrote:
> I think you mean "Stopping..." then "Stopped"?
My apologies I should have express myself better. I actually mean to exchange the order of the lines. Firstly log that we will do something and than do it, rather than do it and then say "we were doing it".
- Jarek
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10964/#review25479
-----------------------------------------------------------
On Aug. 26, 2013, 5:24 p.m., Abraham Elmahrek wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/10964/
> -----------------------------------------------------------
>
> (Updated Aug. 26, 2013, 5:24 p.m.)
>
>
> Review request for Sqoop and Jarek Cecho.
>
>
> Bugs: SQOOP-927
> https://issues.apache.org/jira/browse/SQOOP-927
>
>
> Repository: sqoop-sqoop2
>
>
> Description
> -------
>
> commit ff7dba55a09dd7789a34136233000c625759e583
> Author: Abraham Elmahrek <ab...@elmahrek.com>
> Date: Fri Apr 26 15:10:24 2013 -0700
>
> SQOOP-927 Sqoop2: Integration: Mapreduce specific tests should be running on MiniCluster
>
> Handle MiniDFSCluster and MiniMRClientCluster on own.
>
> Set yarn.application.classpath to get over classpath errors.
> Set to use fair scheduler.
>
> :100644 100644 0abbb18... f09704b... M pom.xml
> :100644 100644 6eb3184... a4c2a5b... M test/src/test/java/org/apache/sqoop/integration/TomcatTestCase.java
> :100644 100644 0f48a8b... 758c885... M test/src/test/java/org/apache/sqoop/integration/server/VersionTest.java
>
>
> Diffs
> -----
>
> pom.xml 513b6d0570ee6f1ec32f8c35a403becc1cfa156a
> test/pom.xml 8001fcec29bf5a7d9c3119b9f6ef548f678e1ebe
> test/src/main/java/org/apache/sqoop/test/asserts/HdfsAsserts.java c17527285167104994d78f5919b7f11c0c31cd15
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopLocalRunner.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopMiniClusterRunner.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopRunner.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopRunnerFactory.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/testcases/ConnectorTestCase.java d10b94268babeb3dc7b1d2450776af5fd8449fd3
> test/src/main/java/org/apache/sqoop/test/testcases/TomcatTestCase.java efdfed4e2065c0c01c16da88cf55705ab13d7bcf
> test/src/main/java/org/apache/sqoop/test/utils/HdfsUtils.java 95dd177a11c5d822e35ca54e5d93785f0a40fbfc
>
> Diff: https://reviews.apache.org/r/10964/diff/
>
>
> Testing
> -------
>
> Ran integration tests without and with miniclusters.
> Currently need to use both miniclusters or neither for tests to work.
>
>
> Thanks,
>
> Abraham Elmahrek
>
>
Re: Review Request 10964: SQOOP-927 Sqoop2: Integration: Mapreduce specific
tests should be running on MiniCluster
Posted by Jarek Cecho <ja...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10964/#review25479
-----------------------------------------------------------
Hi Abe,
thank you very much for working on this, greatly appreciated! I'm having some timeout issues/tests running forever on my machine. I do believe that it's more environmental, so I'll investigate that further.
dist/src/main/server/conf/catalina.properties
<https://reviews.apache.org/r/10964/#comment49911>
Is this change relevant for the patch?
test/pom.xml
<https://reviews.apache.org/r/10964/#comment49912>
Setting the timeout to 0 might be quite dangerous especially when running on Jenkins. Would it be possible to set here some high, but still finite number?
test/src/main/java/org/apache/sqoop/test/hadoop/HadoopMiniClusterRunner.java
<https://reviews.apache.org/r/10964/#comment49913>
Nit: tralling whitespace
test/src/main/java/org/apache/sqoop/test/hadoop/HadoopMiniClusterRunner.java
<https://reviews.apache.org/r/10964/#comment49914>
Nit: Wouldn't make sense to log "Stopping" and then start stopping the service?
test/src/main/java/org/apache/sqoop/test/hadoop/HadoopMiniClusterRunner.java
<https://reviews.apache.org/r/10964/#comment49915>
Nit: Wouldn't make sense to log "Stopping" and then start stopping the service?
test/src/main/java/org/apache/sqoop/test/hadoop/HadoopRunnerFactory.java
<https://reviews.apache.org/r/10964/#comment49916>
Nit: Can we rename the property to "sqoop.hadoop.runner.class" to match the Class names?
Jarcec
- Jarek Cecho
On Aug. 16, 2013, 12:17 a.m., Abraham Elmahrek wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/10964/
> -----------------------------------------------------------
>
> (Updated Aug. 16, 2013, 12:17 a.m.)
>
>
> Review request for Sqoop and Jarek Cecho.
>
>
> Bugs: SQOOP-927
> https://issues.apache.org/jira/browse/SQOOP-927
>
>
> Repository: sqoop-sqoop2
>
>
> Description
> -------
>
> commit ff7dba55a09dd7789a34136233000c625759e583
> Author: Abraham Elmahrek <ab...@elmahrek.com>
> Date: Fri Apr 26 15:10:24 2013 -0700
>
> SQOOP-927 Sqoop2: Integration: Mapreduce specific tests should be running on MiniCluster
>
> Handle MiniDFSCluster and MiniMRClientCluster on own.
>
> Set yarn.application.classpath to get over classpath errors.
> Set to use fair scheduler.
>
> :100644 100644 0abbb18... f09704b... M pom.xml
> :100644 100644 6eb3184... a4c2a5b... M test/src/test/java/org/apache/sqoop/integration/TomcatTestCase.java
> :100644 100644 0f48a8b... 758c885... M test/src/test/java/org/apache/sqoop/integration/server/VersionTest.java
>
>
> Diffs
> -----
>
> dist/src/main/server/conf/catalina.properties 5a47295d6e1bfa2114f86d620bb1452bc576df71
> pom.xml 513b6d0570ee6f1ec32f8c35a403becc1cfa156a
> test/pom.xml 8001fcec29bf5a7d9c3119b9f6ef548f678e1ebe
> test/src/main/java/org/apache/sqoop/test/asserts/HdfsAsserts.java c17527285167104994d78f5919b7f11c0c31cd15
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopLocalRunner.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopMiniClusterRunner.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopRunner.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopRunnerFactory.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/testcases/ConnectorTestCase.java d10b94268babeb3dc7b1d2450776af5fd8449fd3
> test/src/main/java/org/apache/sqoop/test/testcases/TomcatTestCase.java efdfed4e2065c0c01c16da88cf55705ab13d7bcf
> test/src/main/java/org/apache/sqoop/test/utils/HdfsUtils.java 95dd177a11c5d822e35ca54e5d93785f0a40fbfc
>
> Diff: https://reviews.apache.org/r/10964/diff/
>
>
> Testing
> -------
>
> Ran integration tests without and with miniclusters.
> Currently need to use both miniclusters or neither for tests to work.
>
>
> Thanks,
>
> Abraham Elmahrek
>
>