You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sqoop.apache.org by Abraham Elmahrek <ab...@cloudera.com> on 2013/07/04 02:15:51 UTC
Re: Review Request 10964: SQOOP-927 Sqoop2: Integration: Mapreduce specific
tests should be running on MiniCluster
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10964/
-----------------------------------------------------------
(Updated July 4, 2013, 12:15 a.m.)
Review request for Sqoop and Jarek Cecho.
Bugs: SQOOP-927
https://issues.apache.org/jira/browse/SQOOP-927
Repository: sqoop-sqoop2
Description
-------
commit ff7dba55a09dd7789a34136233000c625759e583
Author: Abraham Elmahrek <ab...@elmahrek.com>
Date: Fri Apr 26 15:10:24 2013 -0700
SQOOP-927 Sqoop2: Integration: Mapreduce specific tests should be running on MiniCluster
Handle MiniDFSCluster and MiniMRClientCluster on own.
Set yarn.application.classpath to get over classpath errors.
Set to use fair scheduler.
:100644 100644 0abbb18... f09704b... M pom.xml
:100644 100644 6eb3184... a4c2a5b... M test/src/test/java/org/apache/sqoop/integration/TomcatTestCase.java
:100644 100644 0f48a8b... 758c885... M test/src/test/java/org/apache/sqoop/integration/server/VersionTest.java
Diffs (updated)
-----
pom.xml 8785e01000cc6e7d5a74ffeb96b83d236a657df8
test/src/main/java/org/apache/sqoop/test/asserts/HdfsAsserts.java 056e6124b918ce5821c389e388a0cdfa68fd7fc0
test/src/main/java/org/apache/sqoop/test/testcases/HadoopTestCase.java PRE-CREATION
test/src/main/java/org/apache/sqoop/test/testcases/TomcatTestCase.java ca77e64486d80b38306b5b30e185e6278ad7eaf1
test/src/main/java/org/apache/sqoop/test/utils/HdfsUtils.java 95dd177a11c5d822e35ca54e5d93785f0a40fbfc
Diff: https://reviews.apache.org/r/10964/diff/
Testing
-------
Ran integration tests.
Thanks,
Abraham Elmahrek
Re: Review Request 10964: SQOOP-927 Sqoop2: Integration: Mapreduce specific
tests should be running on MiniCluster
Posted by Jarek Cecho <ja...@apache.org>.
> On July 11, 2013, 8:49 p.m., Jarek Cecho wrote:
> > test/src/main/java/org/apache/sqoop/test/hadoop/HadoopRunner.java, line 65
> > <https://reviews.apache.org/r/10964/diff/5/?file=320107#file320107line65>
> >
> > Shouldn't this be also based on the temporaryPath as the getDataDir() method?
>
> Abraham Elmahrek wrote:
> It should be based on the temporary path if it were the HadoopLocalRunner. Most Hadoop clusters will use some HDFS path though.
But still different tests needs to end up in different directory right? The temporary path is on local file system, but it's unique path should be usable on real cluster as well, right?
- Jarek
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10964/#review23015
-----------------------------------------------------------
On July 19, 2013, 11:32 p.m., Abraham Elmahrek wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/10964/
> -----------------------------------------------------------
>
> (Updated July 19, 2013, 11:32 p.m.)
>
>
> Review request for Sqoop and Jarek Cecho.
>
>
> Bugs: SQOOP-927
> https://issues.apache.org/jira/browse/SQOOP-927
>
>
> Repository: sqoop-sqoop2
>
>
> Description
> -------
>
> commit ff7dba55a09dd7789a34136233000c625759e583
> Author: Abraham Elmahrek <ab...@elmahrek.com>
> Date: Fri Apr 26 15:10:24 2013 -0700
>
> SQOOP-927 Sqoop2: Integration: Mapreduce specific tests should be running on MiniCluster
>
> Handle MiniDFSCluster and MiniMRClientCluster on own.
>
> Set yarn.application.classpath to get over classpath errors.
> Set to use fair scheduler.
>
> :100644 100644 0abbb18... f09704b... M pom.xml
> :100644 100644 6eb3184... a4c2a5b... M test/src/test/java/org/apache/sqoop/integration/TomcatTestCase.java
> :100644 100644 0f48a8b... 758c885... M test/src/test/java/org/apache/sqoop/integration/server/VersionTest.java
>
>
> Diffs
> -----
>
> pom.xml 8785e01000cc6e7d5a74ffeb96b83d236a657df8
> test/src/main/java/org/apache/sqoop/test/asserts/HdfsAsserts.java 056e6124b918ce5821c389e388a0cdfa68fd7fc0
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopLocalRunner.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopMiniClusterRunner.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopRunner.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopRunnerFactory.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/testcases/ConnectorTestCase.java 6aeadd4e54e47e5644270b15813b2d9c4cedc059
> test/src/main/java/org/apache/sqoop/test/testcases/TomcatTestCase.java ca77e64486d80b38306b5b30e185e6278ad7eaf1
> test/src/main/java/org/apache/sqoop/test/utils/HdfsUtils.java 95dd177a11c5d822e35ca54e5d93785f0a40fbfc
>
> Diff: https://reviews.apache.org/r/10964/diff/
>
>
> Testing
> -------
>
> Ran integration tests without and with miniclusters.
> Currently need to use both miniclusters or neither for tests to work.
>
>
> Thanks,
>
> Abraham Elmahrek
>
>
Re: Review Request 10964: SQOOP-927 Sqoop2: Integration: Mapreduce specific
tests should be running on MiniCluster
Posted by Abraham Elmahrek <ab...@cloudera.com>.
> On July 11, 2013, 8:49 p.m., Jarek Cecho wrote:
> > test/src/main/java/org/apache/sqoop/test/hadoop/HadoopRunner.java, line 65
> > <https://reviews.apache.org/r/10964/diff/5/?file=320107#file320107line65>
> >
> > Shouldn't this be also based on the temporaryPath as the getDataDir() method?
It should be based on the temporary path if it were the HadoopLocalRunner. Most Hadoop clusters will use some HDFS path though.
- Abraham
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10964/#review23015
-----------------------------------------------------------
On July 10, 2013, 11:27 p.m., Abraham Elmahrek wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/10964/
> -----------------------------------------------------------
>
> (Updated July 10, 2013, 11:27 p.m.)
>
>
> Review request for Sqoop and Jarek Cecho.
>
>
> Bugs: SQOOP-927
> https://issues.apache.org/jira/browse/SQOOP-927
>
>
> Repository: sqoop-sqoop2
>
>
> Description
> -------
>
> commit ff7dba55a09dd7789a34136233000c625759e583
> Author: Abraham Elmahrek <ab...@elmahrek.com>
> Date: Fri Apr 26 15:10:24 2013 -0700
>
> SQOOP-927 Sqoop2: Integration: Mapreduce specific tests should be running on MiniCluster
>
> Handle MiniDFSCluster and MiniMRClientCluster on own.
>
> Set yarn.application.classpath to get over classpath errors.
> Set to use fair scheduler.
>
> :100644 100644 0abbb18... f09704b... M pom.xml
> :100644 100644 6eb3184... a4c2a5b... M test/src/test/java/org/apache/sqoop/integration/TomcatTestCase.java
> :100644 100644 0f48a8b... 758c885... M test/src/test/java/org/apache/sqoop/integration/server/VersionTest.java
>
>
> Diffs
> -----
>
> pom.xml 8785e01000cc6e7d5a74ffeb96b83d236a657df8
> test/src/main/java/org/apache/sqoop/test/asserts/HdfsAsserts.java 056e6124b918ce5821c389e388a0cdfa68fd7fc0
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopLocalRunner.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopMiniClusterRunner.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopRunner.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopRunnerFactory.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/testcases/ConnectorTestCase.java 6aeadd4e54e47e5644270b15813b2d9c4cedc059
> test/src/main/java/org/apache/sqoop/test/testcases/TomcatTestCase.java ca77e64486d80b38306b5b30e185e6278ad7eaf1
> test/src/main/java/org/apache/sqoop/test/utils/HdfsUtils.java 95dd177a11c5d822e35ca54e5d93785f0a40fbfc
>
> Diff: https://reviews.apache.org/r/10964/diff/
>
>
> Testing
> -------
>
> Ran integration tests without and with miniclusters.
> Currently need to use both miniclusters or neither for tests to work.
>
>
> Thanks,
>
> Abraham Elmahrek
>
>
Re: Review Request 10964: SQOOP-927 Sqoop2: Integration: Mapreduce specific
tests should be running on MiniCluster
Posted by Jarek Cecho <ja...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10964/#review23015
-----------------------------------------------------------
Hi Abe,
thank you very much for incorporating all my suggestions! I do have couple more polishing suggestions:
test/src/main/java/org/apache/sqoop/test/hadoop/HadoopLocalRunner.java
<https://reviews.apache.org/r/10964/#comment46781>
Nit: Local cluster != "jobs locally"
When hadoop is running jobs locally, they are run with LocalJobRunner without any cluster. This is very important distinction because the code path is totally different. E.g. in local mode there is totally different MR implementation then in the cluster mode.
Suggested rewording:
HadoopRunner implementation that is using LocalJobRunner for executing mapreduce jobs and local filesystem instead of HDFS.
test/src/main/java/org/apache/sqoop/test/hadoop/HadoopLocalRunner.java
<https://reviews.apache.org/r/10964/#comment46797>
It seems to me that here is a type and the getDataDir() should be replaced with getTemporaryDirectory()?
test/src/main/java/org/apache/sqoop/test/hadoop/HadoopRunner.java
<https://reviews.apache.org/r/10964/#comment46783>
Proposed javadoc comment:
Hadoop cluster runner for testing purpose.
Runner provides methods needed to bootstrapping and using Hadoop cluster. This abstract implementation is agnostic about in what mode the Hadoop is running. Each mode will have it's own concrete implementation (for example LocalJobRunner, MiniCluster or Real existing cluster).
test/src/main/java/org/apache/sqoop/test/hadoop/HadoopRunner.java
<https://reviews.apache.org/r/10964/#comment46798>
Suggested rewording:
Temporary path that can be used as a root for other directories storing various data like logs or stored HDFS files.
test/src/main/java/org/apache/sqoop/test/hadoop/HadoopRunner.java
<https://reviews.apache.org/r/10964/#comment46784>
Nit: s/mini//
test/src/main/java/org/apache/sqoop/test/hadoop/HadoopRunner.java
<https://reviews.apache.org/r/10964/#comment46785>
Nit: s/mini//
test/src/main/java/org/apache/sqoop/test/hadoop/HadoopRunner.java
<https://reviews.apache.org/r/10964/#comment46794>
Suggested rewording:
Return working directory on HDFS instance that this HadoopRunner is using.
This directory might be on local filesystem in case of local mode.
test/src/main/java/org/apache/sqoop/test/hadoop/HadoopRunner.java
<https://reviews.apache.org/r/10964/#comment46799>
Shouldn't this be also based on the temporaryPath as the getDataDir() method?
test/src/main/java/org/apache/sqoop/test/hadoop/HadoopRunner.java
<https://reviews.apache.org/r/10964/#comment46793>
Suggested rewording:
Return directory on local filesystem where logs and other data generated by the Hadoop Cluster should be stored.
test/src/main/java/org/apache/sqoop/test/hadoop/HadoopRunner.java
<https://reviews.apache.org/r/10964/#comment46796>
Nit: It seems that on most places we are using the HdfsUtils.joinPathFragments instead of File() object. I would suggest to be consistent.
test/src/main/java/org/apache/sqoop/test/testcases/TomcatTestCase.java
<https://reviews.apache.org/r/10964/#comment46789>
Please do not remove the tmpPath variable. TomcatTestCase is the root test case that everything else is inheriting and therefore should be owner of this variable. Also in addition the path should be available to all children in case that there is need to store additional temporal information on the local file system.
test/src/main/java/org/apache/sqoop/test/testcases/TomcatTestCase.java
<https://reviews.apache.org/r/10964/#comment46790>
I believe that the timestamp is not necessary as the temporary path on local file system is guaranteed to be unique.
test/src/main/java/org/apache/sqoop/test/testcases/TomcatTestCase.java
<https://reviews.apache.org/r/10964/#comment46792>
The temporary path needs to be exposed for downstream test cases that will need to store additional temporary files for given test.
test/src/main/java/org/apache/sqoop/test/testcases/TomcatTestCase.java
<https://reviews.apache.org/r/10964/#comment46791>
Nit: Changing method order do not seem to be necessary here.
Jarcec
- Jarek Cecho
On July 10, 2013, 11:27 p.m., Abraham Elmahrek wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/10964/
> -----------------------------------------------------------
>
> (Updated July 10, 2013, 11:27 p.m.)
>
>
> Review request for Sqoop and Jarek Cecho.
>
>
> Bugs: SQOOP-927
> https://issues.apache.org/jira/browse/SQOOP-927
>
>
> Repository: sqoop-sqoop2
>
>
> Description
> -------
>
> commit ff7dba55a09dd7789a34136233000c625759e583
> Author: Abraham Elmahrek <ab...@elmahrek.com>
> Date: Fri Apr 26 15:10:24 2013 -0700
>
> SQOOP-927 Sqoop2: Integration: Mapreduce specific tests should be running on MiniCluster
>
> Handle MiniDFSCluster and MiniMRClientCluster on own.
>
> Set yarn.application.classpath to get over classpath errors.
> Set to use fair scheduler.
>
> :100644 100644 0abbb18... f09704b... M pom.xml
> :100644 100644 6eb3184... a4c2a5b... M test/src/test/java/org/apache/sqoop/integration/TomcatTestCase.java
> :100644 100644 0f48a8b... 758c885... M test/src/test/java/org/apache/sqoop/integration/server/VersionTest.java
>
>
> Diffs
> -----
>
> pom.xml 8785e01000cc6e7d5a74ffeb96b83d236a657df8
> test/src/main/java/org/apache/sqoop/test/asserts/HdfsAsserts.java 056e6124b918ce5821c389e388a0cdfa68fd7fc0
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopLocalRunner.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopMiniClusterRunner.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopRunner.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopRunnerFactory.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/testcases/ConnectorTestCase.java 6aeadd4e54e47e5644270b15813b2d9c4cedc059
> test/src/main/java/org/apache/sqoop/test/testcases/TomcatTestCase.java ca77e64486d80b38306b5b30e185e6278ad7eaf1
> test/src/main/java/org/apache/sqoop/test/utils/HdfsUtils.java 95dd177a11c5d822e35ca54e5d93785f0a40fbfc
>
> Diff: https://reviews.apache.org/r/10964/diff/
>
>
> Testing
> -------
>
> Ran integration tests without and with miniclusters.
> Currently need to use both miniclusters or neither for tests to work.
>
>
> Thanks,
>
> Abraham Elmahrek
>
>
Re: Review Request 10964: SQOOP-927 Sqoop2: Integration: Mapreduce specific
tests should be running on MiniCluster
Posted by Jarek Cecho <ja...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10964/#review23634
-----------------------------------------------------------
Hi Abe,
thank you very much for incorporating all my comments, greatly appreciated! It seems that I'm having timeout issues in TableExportTest, are you observing them as well?
test/src/main/java/org/apache/sqoop/test/hadoop/HadoopLocalRunner.java
<https://reviews.apache.org/r/10964/#comment47533>
Nit: I'm not sure that this sentence do make sense here.
test/src/main/java/org/apache/sqoop/test/hadoop/HadoopRunner.java
<https://reviews.apache.org/r/10964/#comment47534>
Nit: Trailing white space.
test/src/main/java/org/apache/sqoop/test/hadoop/HadoopRunner.java
<https://reviews.apache.org/r/10964/#comment47536>
Nit: Trailing white space.
Jarcec
- Jarek Cecho
On July 19, 2013, 11:32 p.m., Abraham Elmahrek wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/10964/
> -----------------------------------------------------------
>
> (Updated July 19, 2013, 11:32 p.m.)
>
>
> Review request for Sqoop and Jarek Cecho.
>
>
> Bugs: SQOOP-927
> https://issues.apache.org/jira/browse/SQOOP-927
>
>
> Repository: sqoop-sqoop2
>
>
> Description
> -------
>
> commit ff7dba55a09dd7789a34136233000c625759e583
> Author: Abraham Elmahrek <ab...@elmahrek.com>
> Date: Fri Apr 26 15:10:24 2013 -0700
>
> SQOOP-927 Sqoop2: Integration: Mapreduce specific tests should be running on MiniCluster
>
> Handle MiniDFSCluster and MiniMRClientCluster on own.
>
> Set yarn.application.classpath to get over classpath errors.
> Set to use fair scheduler.
>
> :100644 100644 0abbb18... f09704b... M pom.xml
> :100644 100644 6eb3184... a4c2a5b... M test/src/test/java/org/apache/sqoop/integration/TomcatTestCase.java
> :100644 100644 0f48a8b... 758c885... M test/src/test/java/org/apache/sqoop/integration/server/VersionTest.java
>
>
> Diffs
> -----
>
> pom.xml 8785e01000cc6e7d5a74ffeb96b83d236a657df8
> test/src/main/java/org/apache/sqoop/test/asserts/HdfsAsserts.java 056e6124b918ce5821c389e388a0cdfa68fd7fc0
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopLocalRunner.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopMiniClusterRunner.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopRunner.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopRunnerFactory.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/testcases/ConnectorTestCase.java 6aeadd4e54e47e5644270b15813b2d9c4cedc059
> test/src/main/java/org/apache/sqoop/test/testcases/TomcatTestCase.java ca77e64486d80b38306b5b30e185e6278ad7eaf1
> test/src/main/java/org/apache/sqoop/test/utils/HdfsUtils.java 95dd177a11c5d822e35ca54e5d93785f0a40fbfc
>
> Diff: https://reviews.apache.org/r/10964/diff/
>
>
> Testing
> -------
>
> Ran integration tests without and with miniclusters.
> Currently need to use both miniclusters or neither for tests to work.
>
>
> Thanks,
>
> Abraham Elmahrek
>
>
Re: Review Request 10964: SQOOP-927 Sqoop2: Integration: Mapreduce specific
tests should be running on MiniCluster
Posted by Jarek Cecho <ja...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10964/#review27347
-----------------------------------------------------------
Ship it!
Ship It!
- Jarek Cecho
On Oct. 18, 2013, 2 a.m., Abraham Elmahrek wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/10964/
> -----------------------------------------------------------
>
> (Updated Oct. 18, 2013, 2 a.m.)
>
>
> Review request for Sqoop and Jarek Cecho.
>
>
> Bugs: SQOOP-927
> https://issues.apache.org/jira/browse/SQOOP-927
>
>
> Repository: sqoop-sqoop2
>
>
> Description
> -------
>
> commit 2cc666f2aa1ac679a58099819a6985a19dfb5de9
> Author: Abraham Elmahrek <ab...@elmahrek.com>
> Date: Tue Jul 2 11:53:57 2013 -0700
>
> SQOOP-927 Sqoop2: Integration: Mapreduce specific tests should be running on MiniCluster
>
> Minicluster test output set to <temp dir>/minicluster.
>
> Set yarn.application.classpath to get over classpath errors.
> Set to use fair scheduler.
>
> :100644 100644 513b6d0... 524117e... M pom.xml
> :100644 100644 c175272... d8f2b8d... M test/src/main/java/org/apache/sqoop/test/asserts/HdfsAsserts.java
> :000000 100644 0000000... 44465b4... A test/src/main/java/org/apache/sqoop/test/hadoop/HadoopLocalRunner.java
> :000000 100644 0000000... b06dcab... A test/src/main/java/org/apache/sqoop/test/hadoop/HadoopMiniClusterRunner.java
> :000000 100644 0000000... 2516ff1... A test/src/main/java/org/apache/sqoop/test/hadoop/HadoopRunner.java
> :000000 100644 0000000... 020fa3f... A test/src/main/java/org/apache/sqoop/test/hadoop/HadoopRunnerFactory.java
> :100644 100644 d10b942... 5ec4fa4... M test/src/main/java/org/apache/sqoop/test/testcases/ConnectorTestCase.java
> :100644 100644 efdfed4... 7e2558f... M test/src/main/java/org/apache/sqoop/test/testcases/TomcatTestCase.java
> :100644 100644 95dd177... 59c5f15... M test/src/main/java/org/apache/sqoop/test/utils/HdfsUtils.java
>
>
> Diffs
> -----
>
> pom.xml 513b6d0570ee6f1ec32f8c35a403becc1cfa156a
> test/src/main/java/org/apache/sqoop/test/asserts/HdfsAsserts.java c17527285167104994d78f5919b7f11c0c31cd15
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopLocalRunner.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopMiniClusterRunner.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopRunner.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopRunnerFactory.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/testcases/ConnectorTestCase.java d10b94268babeb3dc7b1d2450776af5fd8449fd3
> test/src/main/java/org/apache/sqoop/test/testcases/TomcatTestCase.java efdfed4e2065c0c01c16da88cf55705ab13d7bcf
> test/src/main/java/org/apache/sqoop/test/utils/HdfsUtils.java 95dd177a11c5d822e35ca54e5d93785f0a40fbfc
>
> Diff: https://reviews.apache.org/r/10964/diff/
>
>
> Testing
> -------
>
> mvn clean integration-test -DfailIfNoTests=false -X -Phadoop100 -Dhadoop.profile=100 -Dsurefire.useFile=false -Dmaven.test.redirectTestOutputToFile=false
> mvn clean integration-test -DfailIfNoTests=false -X -Dsurefire.useFile=false -Dmaven.test.redirectTestOutputToFile=false
> mvn clean integration-test -DfailIfNoTests=false -X -Phadoop100 -Dhadoop.profile=100 -Dsurefire.useFile=false -Dmaven.test.redirectTestOutputToFile=false -Dsqoop.hadoop.runner.class=org.apache.sqoop.test.hadoop.HadoopLocalRunner
> mvn clean integration-test -DfailIfNoTests=false -X -Dsurefire.useFile=false -Dmaven.test.redirectTestOutputToFile=false -Dsqoop.hadoop.runner.class=org.apache.sqoop.test.hadoop.HadoopLocalRunner
>
>
> Thanks,
>
> Abraham Elmahrek
>
>
Re: Review Request 10964: SQOOP-927 Sqoop2: Integration: Mapreduce specific
tests should be running on MiniCluster
Posted by Abraham Elmahrek <ab...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10964/
-----------------------------------------------------------
(Updated Oct. 18, 2013, 2 a.m.)
Review request for Sqoop and Jarek Cecho.
Bugs: SQOOP-927
https://issues.apache.org/jira/browse/SQOOP-927
Repository: sqoop-sqoop2
Description (updated)
-------
commit 2cc666f2aa1ac679a58099819a6985a19dfb5de9
Author: Abraham Elmahrek <ab...@elmahrek.com>
Date: Tue Jul 2 11:53:57 2013 -0700
SQOOP-927 Sqoop2: Integration: Mapreduce specific tests should be running on MiniCluster
Minicluster test output set to <temp dir>/minicluster.
Set yarn.application.classpath to get over classpath errors.
Set to use fair scheduler.
:100644 100644 513b6d0... 524117e... M pom.xml
:100644 100644 c175272... d8f2b8d... M test/src/main/java/org/apache/sqoop/test/asserts/HdfsAsserts.java
:000000 100644 0000000... 44465b4... A test/src/main/java/org/apache/sqoop/test/hadoop/HadoopLocalRunner.java
:000000 100644 0000000... b06dcab... A test/src/main/java/org/apache/sqoop/test/hadoop/HadoopMiniClusterRunner.java
:000000 100644 0000000... 2516ff1... A test/src/main/java/org/apache/sqoop/test/hadoop/HadoopRunner.java
:000000 100644 0000000... 020fa3f... A test/src/main/java/org/apache/sqoop/test/hadoop/HadoopRunnerFactory.java
:100644 100644 d10b942... 5ec4fa4... M test/src/main/java/org/apache/sqoop/test/testcases/ConnectorTestCase.java
:100644 100644 efdfed4... 7e2558f... M test/src/main/java/org/apache/sqoop/test/testcases/TomcatTestCase.java
:100644 100644 95dd177... 59c5f15... M test/src/main/java/org/apache/sqoop/test/utils/HdfsUtils.java
Diffs (updated)
-----
pom.xml 513b6d0570ee6f1ec32f8c35a403becc1cfa156a
test/src/main/java/org/apache/sqoop/test/asserts/HdfsAsserts.java c17527285167104994d78f5919b7f11c0c31cd15
test/src/main/java/org/apache/sqoop/test/hadoop/HadoopLocalRunner.java PRE-CREATION
test/src/main/java/org/apache/sqoop/test/hadoop/HadoopMiniClusterRunner.java PRE-CREATION
test/src/main/java/org/apache/sqoop/test/hadoop/HadoopRunner.java PRE-CREATION
test/src/main/java/org/apache/sqoop/test/hadoop/HadoopRunnerFactory.java PRE-CREATION
test/src/main/java/org/apache/sqoop/test/testcases/ConnectorTestCase.java d10b94268babeb3dc7b1d2450776af5fd8449fd3
test/src/main/java/org/apache/sqoop/test/testcases/TomcatTestCase.java efdfed4e2065c0c01c16da88cf55705ab13d7bcf
test/src/main/java/org/apache/sqoop/test/utils/HdfsUtils.java 95dd177a11c5d822e35ca54e5d93785f0a40fbfc
Diff: https://reviews.apache.org/r/10964/diff/
Testing (updated)
-------
mvn clean integration-test -DfailIfNoTests=false -X -Phadoop100 -Dhadoop.profile=100 -Dsurefire.useFile=false -Dmaven.test.redirectTestOutputToFile=false
mvn clean integration-test -DfailIfNoTests=false -X -Dsurefire.useFile=false -Dmaven.test.redirectTestOutputToFile=false
mvn clean integration-test -DfailIfNoTests=false -X -Phadoop100 -Dhadoop.profile=100 -Dsurefire.useFile=false -Dmaven.test.redirectTestOutputToFile=false -Dsqoop.hadoop.runner.class=org.apache.sqoop.test.hadoop.HadoopLocalRunner
mvn clean integration-test -DfailIfNoTests=false -X -Dsurefire.useFile=false -Dmaven.test.redirectTestOutputToFile=false -Dsqoop.hadoop.runner.class=org.apache.sqoop.test.hadoop.HadoopLocalRunner
Thanks,
Abraham Elmahrek
Re: Review Request 10964: SQOOP-927 Sqoop2: Integration: Mapreduce specific
tests should be running on MiniCluster
Posted by Jarek Cecho <ja...@apache.org>.
On Oct. 16, 2013, 10:27 p.m., Abraham Elmahrek wrote:
> > Jarcec
>
> Abraham Elmahrek wrote:
> Unfortunately, there is a test case that can take up to 30 minutes I've found. The PartitionerTest uses the parameterized runner and can take a while to finish.
I see, I guess that I have too powerful machine as it needed only 6 minutes :-)
- Jarek
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10964/#review27093
-----------------------------------------------------------
On Aug. 26, 2013, 5:24 p.m., Abraham Elmahrek wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/10964/
> -----------------------------------------------------------
>
> (Updated Aug. 26, 2013, 5:24 p.m.)
>
>
> Review request for Sqoop and Jarek Cecho.
>
>
> Bugs: SQOOP-927
> https://issues.apache.org/jira/browse/SQOOP-927
>
>
> Repository: sqoop-sqoop2
>
>
> Description
> -------
>
> commit ff7dba55a09dd7789a34136233000c625759e583
> Author: Abraham Elmahrek <ab...@elmahrek.com>
> Date: Fri Apr 26 15:10:24 2013 -0700
>
> SQOOP-927 Sqoop2: Integration: Mapreduce specific tests should be running on MiniCluster
>
> Handle MiniDFSCluster and MiniMRClientCluster on own.
>
> Set yarn.application.classpath to get over classpath errors.
> Set to use fair scheduler.
>
> :100644 100644 0abbb18... f09704b... M pom.xml
> :100644 100644 6eb3184... a4c2a5b... M test/src/test/java/org/apache/sqoop/integration/TomcatTestCase.java
> :100644 100644 0f48a8b... 758c885... M test/src/test/java/org/apache/sqoop/integration/server/VersionTest.java
>
>
> Diffs
> -----
>
> pom.xml 513b6d0570ee6f1ec32f8c35a403becc1cfa156a
> test/pom.xml 8001fcec29bf5a7d9c3119b9f6ef548f678e1ebe
> test/src/main/java/org/apache/sqoop/test/asserts/HdfsAsserts.java c17527285167104994d78f5919b7f11c0c31cd15
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopLocalRunner.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopMiniClusterRunner.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopRunner.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopRunnerFactory.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/testcases/ConnectorTestCase.java d10b94268babeb3dc7b1d2450776af5fd8449fd3
> test/src/main/java/org/apache/sqoop/test/testcases/TomcatTestCase.java efdfed4e2065c0c01c16da88cf55705ab13d7bcf
> test/src/main/java/org/apache/sqoop/test/utils/HdfsUtils.java 95dd177a11c5d822e35ca54e5d93785f0a40fbfc
>
> Diff: https://reviews.apache.org/r/10964/diff/
>
>
> Testing
> -------
>
> Ran integration tests without and with miniclusters.
> Currently need to use both miniclusters or neither for tests to work.
>
>
> Thanks,
>
> Abraham Elmahrek
>
>
Re: Review Request 10964: SQOOP-927 Sqoop2: Integration: Mapreduce specific
tests should be running on MiniCluster
Posted by Abraham Elmahrek <ab...@cloudera.com>.
> On Oct. 16, 2013, 10:27 p.m., Jarek Cecho wrote:
> > Hi Abe,
> > thank you very much for working on this patch! Please accept my apologies for late review! I'm having difficulties in Hadoop 100 profile - getting NullPointerExceptions in the cluster initialization. I was wondering if this patch version is not yet ready for Hadoop 1 or whether I should dig into my environment to see what is going on.
> >
> >
> >
Jarcec, great points. I'm looking into what's going wrong with the Hadoop 100 profile right now. As always, thank you for the review!
> On Oct. 16, 2013, 10:27 p.m., Jarek Cecho wrote:
> > pom.xml, line 99
> > <https://reviews.apache.org/r/10964/diff/8/?file=344782#file344782line99>
> >
> > Can we wait until SQOOP-1218 will get committed? That will upgrade the version even further to 2.2.0 (first GA release of Hadoop 2).
Indeed.
> On Oct. 16, 2013, 10:27 p.m., Jarek Cecho wrote:
> > test/pom.xml, line 120
> > <https://reviews.apache.org/r/10964/diff/8/?file=344783#file344783line120>
> >
> > One hour long timeout seems too much, what about 900 seconds (15 minutes)?
Unfortunately, there is a test case that can take up to 30 minutes I've found. The PartitionerTest uses the parameterized runner and can take a while to finish.
On Oct. 16, 2013, 10:27 p.m., Abraham Elmahrek wrote:
> > Jarcec
Unfortunately, there is a test case that can take up to 30 minutes I've found. The PartitionerTest uses the parameterized runner and can take a while to finish.
- Abraham
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10964/#review27093
-----------------------------------------------------------
On Aug. 26, 2013, 5:24 p.m., Abraham Elmahrek wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/10964/
> -----------------------------------------------------------
>
> (Updated Aug. 26, 2013, 5:24 p.m.)
>
>
> Review request for Sqoop and Jarek Cecho.
>
>
> Bugs: SQOOP-927
> https://issues.apache.org/jira/browse/SQOOP-927
>
>
> Repository: sqoop-sqoop2
>
>
> Description
> -------
>
> commit ff7dba55a09dd7789a34136233000c625759e583
> Author: Abraham Elmahrek <ab...@elmahrek.com>
> Date: Fri Apr 26 15:10:24 2013 -0700
>
> SQOOP-927 Sqoop2: Integration: Mapreduce specific tests should be running on MiniCluster
>
> Handle MiniDFSCluster and MiniMRClientCluster on own.
>
> Set yarn.application.classpath to get over classpath errors.
> Set to use fair scheduler.
>
> :100644 100644 0abbb18... f09704b... M pom.xml
> :100644 100644 6eb3184... a4c2a5b... M test/src/test/java/org/apache/sqoop/integration/TomcatTestCase.java
> :100644 100644 0f48a8b... 758c885... M test/src/test/java/org/apache/sqoop/integration/server/VersionTest.java
>
>
> Diffs
> -----
>
> pom.xml 513b6d0570ee6f1ec32f8c35a403becc1cfa156a
> test/pom.xml 8001fcec29bf5a7d9c3119b9f6ef548f678e1ebe
> test/src/main/java/org/apache/sqoop/test/asserts/HdfsAsserts.java c17527285167104994d78f5919b7f11c0c31cd15
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopLocalRunner.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopMiniClusterRunner.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopRunner.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopRunnerFactory.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/testcases/ConnectorTestCase.java d10b94268babeb3dc7b1d2450776af5fd8449fd3
> test/src/main/java/org/apache/sqoop/test/testcases/TomcatTestCase.java efdfed4e2065c0c01c16da88cf55705ab13d7bcf
> test/src/main/java/org/apache/sqoop/test/utils/HdfsUtils.java 95dd177a11c5d822e35ca54e5d93785f0a40fbfc
>
> Diff: https://reviews.apache.org/r/10964/diff/
>
>
> Testing
> -------
>
> Ran integration tests without and with miniclusters.
> Currently need to use both miniclusters or neither for tests to work.
>
>
> Thanks,
>
> Abraham Elmahrek
>
>
Re: Review Request 10964: SQOOP-927 Sqoop2: Integration: Mapreduce specific
tests should be running on MiniCluster
Posted by Jarek Cecho <ja...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10964/#review27093
-----------------------------------------------------------
Hi Abe,
thank you very much for working on this patch! Please accept my apologies for late review! I'm having difficulties in Hadoop 100 profile - getting NullPointerExceptions in the cluster initialization. I was wondering if this patch version is not yet ready for Hadoop 1 or whether I should dig into my environment to see what is going on.
pom.xml
<https://reviews.apache.org/r/10964/#comment52748>
Can we wait until SQOOP-1218 will get committed? That will upgrade the version even further to 2.2.0 (first GA release of Hadoop 2).
test/pom.xml
<https://reviews.apache.org/r/10964/#comment52749>
One hour long timeout seems too much, what about 900 seconds (15 minutes)?
Jarcec
- Jarek Cecho
On Aug. 26, 2013, 5:24 p.m., Abraham Elmahrek wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/10964/
> -----------------------------------------------------------
>
> (Updated Aug. 26, 2013, 5:24 p.m.)
>
>
> Review request for Sqoop and Jarek Cecho.
>
>
> Bugs: SQOOP-927
> https://issues.apache.org/jira/browse/SQOOP-927
>
>
> Repository: sqoop-sqoop2
>
>
> Description
> -------
>
> commit ff7dba55a09dd7789a34136233000c625759e583
> Author: Abraham Elmahrek <ab...@elmahrek.com>
> Date: Fri Apr 26 15:10:24 2013 -0700
>
> SQOOP-927 Sqoop2: Integration: Mapreduce specific tests should be running on MiniCluster
>
> Handle MiniDFSCluster and MiniMRClientCluster on own.
>
> Set yarn.application.classpath to get over classpath errors.
> Set to use fair scheduler.
>
> :100644 100644 0abbb18... f09704b... M pom.xml
> :100644 100644 6eb3184... a4c2a5b... M test/src/test/java/org/apache/sqoop/integration/TomcatTestCase.java
> :100644 100644 0f48a8b... 758c885... M test/src/test/java/org/apache/sqoop/integration/server/VersionTest.java
>
>
> Diffs
> -----
>
> pom.xml 513b6d0570ee6f1ec32f8c35a403becc1cfa156a
> test/pom.xml 8001fcec29bf5a7d9c3119b9f6ef548f678e1ebe
> test/src/main/java/org/apache/sqoop/test/asserts/HdfsAsserts.java c17527285167104994d78f5919b7f11c0c31cd15
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopLocalRunner.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopMiniClusterRunner.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopRunner.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopRunnerFactory.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/testcases/ConnectorTestCase.java d10b94268babeb3dc7b1d2450776af5fd8449fd3
> test/src/main/java/org/apache/sqoop/test/testcases/TomcatTestCase.java efdfed4e2065c0c01c16da88cf55705ab13d7bcf
> test/src/main/java/org/apache/sqoop/test/utils/HdfsUtils.java 95dd177a11c5d822e35ca54e5d93785f0a40fbfc
>
> Diff: https://reviews.apache.org/r/10964/diff/
>
>
> Testing
> -------
>
> Ran integration tests without and with miniclusters.
> Currently need to use both miniclusters or neither for tests to work.
>
>
> Thanks,
>
> Abraham Elmahrek
>
>
Re: Review Request 10964: SQOOP-927 Sqoop2: Integration: Mapreduce specific
tests should be running on MiniCluster
Posted by Abraham Elmahrek <ab...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10964/
-----------------------------------------------------------
(Updated Aug. 26, 2013, 5:24 p.m.)
Review request for Sqoop and Jarek Cecho.
Bugs: SQOOP-927
https://issues.apache.org/jira/browse/SQOOP-927
Repository: sqoop-sqoop2
Description
-------
commit ff7dba55a09dd7789a34136233000c625759e583
Author: Abraham Elmahrek <ab...@elmahrek.com>
Date: Fri Apr 26 15:10:24 2013 -0700
SQOOP-927 Sqoop2: Integration: Mapreduce specific tests should be running on MiniCluster
Handle MiniDFSCluster and MiniMRClientCluster on own.
Set yarn.application.classpath to get over classpath errors.
Set to use fair scheduler.
:100644 100644 0abbb18... f09704b... M pom.xml
:100644 100644 6eb3184... a4c2a5b... M test/src/test/java/org/apache/sqoop/integration/TomcatTestCase.java
:100644 100644 0f48a8b... 758c885... M test/src/test/java/org/apache/sqoop/integration/server/VersionTest.java
Diffs (updated)
-----
pom.xml 513b6d0570ee6f1ec32f8c35a403becc1cfa156a
test/pom.xml 8001fcec29bf5a7d9c3119b9f6ef548f678e1ebe
test/src/main/java/org/apache/sqoop/test/asserts/HdfsAsserts.java c17527285167104994d78f5919b7f11c0c31cd15
test/src/main/java/org/apache/sqoop/test/hadoop/HadoopLocalRunner.java PRE-CREATION
test/src/main/java/org/apache/sqoop/test/hadoop/HadoopMiniClusterRunner.java PRE-CREATION
test/src/main/java/org/apache/sqoop/test/hadoop/HadoopRunner.java PRE-CREATION
test/src/main/java/org/apache/sqoop/test/hadoop/HadoopRunnerFactory.java PRE-CREATION
test/src/main/java/org/apache/sqoop/test/testcases/ConnectorTestCase.java d10b94268babeb3dc7b1d2450776af5fd8449fd3
test/src/main/java/org/apache/sqoop/test/testcases/TomcatTestCase.java efdfed4e2065c0c01c16da88cf55705ab13d7bcf
test/src/main/java/org/apache/sqoop/test/utils/HdfsUtils.java 95dd177a11c5d822e35ca54e5d93785f0a40fbfc
Diff: https://reviews.apache.org/r/10964/diff/
Testing
-------
Ran integration tests without and with miniclusters.
Currently need to use both miniclusters or neither for tests to work.
Thanks,
Abraham Elmahrek
Re: Review Request 10964: SQOOP-927 Sqoop2: Integration: Mapreduce specific
tests should be running on MiniCluster
Posted by Abraham Elmahrek <ab...@cloudera.com>.
> On Aug. 23, 2013, 11:24 p.m., Jarek Cecho wrote:
> > dist/src/main/server/conf/catalina.properties, line 47
> > <https://reviews.apache.org/r/10964/diff/7/?file=341604#file341604line47>
> >
> > Is this change relevant for the patch?
No. Sorry. I have to add that every time to test Sqoop. I'll remove it.
> On Aug. 23, 2013, 11:24 p.m., Jarek Cecho wrote:
> > test/pom.xml, line 120
> > <https://reviews.apache.org/r/10964/diff/7/?file=341606#file341606line120>
> >
> > Setting the timeout to 0 might be quite dangerous especially when running on Jenkins. Would it be possible to set here some high, but still finite number?
Sure, I'll set it to 1 hour.
> On Aug. 23, 2013, 11:24 p.m., Jarek Cecho wrote:
> > test/src/main/java/org/apache/sqoop/test/hadoop/HadoopMiniClusterRunner.java, lines 71-72
> > <https://reviews.apache.org/r/10964/diff/7/?file=341609#file341609line71>
> >
> > Nit: Wouldn't make sense to log "Stopping" and then start stopping the service?
I think you mean "Stopping..." then "Stopped"?
> On Aug. 23, 2013, 11:24 p.m., Jarek Cecho wrote:
> > test/src/main/java/org/apache/sqoop/test/hadoop/HadoopMiniClusterRunner.java, lines 74-75
> > <https://reviews.apache.org/r/10964/diff/7/?file=341609#file341609line74>
> >
> > Nit: Wouldn't make sense to log "Stopping" and then start stopping the service?
same as above
- Abraham
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10964/#review25479
-----------------------------------------------------------
On Aug. 16, 2013, 12:17 a.m., Abraham Elmahrek wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/10964/
> -----------------------------------------------------------
>
> (Updated Aug. 16, 2013, 12:17 a.m.)
>
>
> Review request for Sqoop and Jarek Cecho.
>
>
> Bugs: SQOOP-927
> https://issues.apache.org/jira/browse/SQOOP-927
>
>
> Repository: sqoop-sqoop2
>
>
> Description
> -------
>
> commit ff7dba55a09dd7789a34136233000c625759e583
> Author: Abraham Elmahrek <ab...@elmahrek.com>
> Date: Fri Apr 26 15:10:24 2013 -0700
>
> SQOOP-927 Sqoop2: Integration: Mapreduce specific tests should be running on MiniCluster
>
> Handle MiniDFSCluster and MiniMRClientCluster on own.
>
> Set yarn.application.classpath to get over classpath errors.
> Set to use fair scheduler.
>
> :100644 100644 0abbb18... f09704b... M pom.xml
> :100644 100644 6eb3184... a4c2a5b... M test/src/test/java/org/apache/sqoop/integration/TomcatTestCase.java
> :100644 100644 0f48a8b... 758c885... M test/src/test/java/org/apache/sqoop/integration/server/VersionTest.java
>
>
> Diffs
> -----
>
> dist/src/main/server/conf/catalina.properties 5a47295d6e1bfa2114f86d620bb1452bc576df71
> pom.xml 513b6d0570ee6f1ec32f8c35a403becc1cfa156a
> test/pom.xml 8001fcec29bf5a7d9c3119b9f6ef548f678e1ebe
> test/src/main/java/org/apache/sqoop/test/asserts/HdfsAsserts.java c17527285167104994d78f5919b7f11c0c31cd15
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopLocalRunner.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopMiniClusterRunner.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopRunner.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopRunnerFactory.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/testcases/ConnectorTestCase.java d10b94268babeb3dc7b1d2450776af5fd8449fd3
> test/src/main/java/org/apache/sqoop/test/testcases/TomcatTestCase.java efdfed4e2065c0c01c16da88cf55705ab13d7bcf
> test/src/main/java/org/apache/sqoop/test/utils/HdfsUtils.java 95dd177a11c5d822e35ca54e5d93785f0a40fbfc
>
> Diff: https://reviews.apache.org/r/10964/diff/
>
>
> Testing
> -------
>
> Ran integration tests without and with miniclusters.
> Currently need to use both miniclusters or neither for tests to work.
>
>
> Thanks,
>
> Abraham Elmahrek
>
>
Re: Review Request 10964: SQOOP-927 Sqoop2: Integration: Mapreduce specific
tests should be running on MiniCluster
Posted by Jarek Cecho <ja...@apache.org>.
> On Aug. 23, 2013, 11:24 p.m., Jarek Cecho wrote:
> > test/src/main/java/org/apache/sqoop/test/hadoop/HadoopMiniClusterRunner.java, lines 71-72
> > <https://reviews.apache.org/r/10964/diff/7/?file=341609#file341609line71>
> >
> > Nit: Wouldn't make sense to log "Stopping" and then start stopping the service?
>
> Abraham Elmahrek wrote:
> I think you mean "Stopping..." then "Stopped"?
My apologies I should have express myself better. I actually mean to exchange the order of the lines. Firstly log that we will do something and than do it, rather than do it and then say "we were doing it".
- Jarek
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10964/#review25479
-----------------------------------------------------------
On Aug. 26, 2013, 5:24 p.m., Abraham Elmahrek wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/10964/
> -----------------------------------------------------------
>
> (Updated Aug. 26, 2013, 5:24 p.m.)
>
>
> Review request for Sqoop and Jarek Cecho.
>
>
> Bugs: SQOOP-927
> https://issues.apache.org/jira/browse/SQOOP-927
>
>
> Repository: sqoop-sqoop2
>
>
> Description
> -------
>
> commit ff7dba55a09dd7789a34136233000c625759e583
> Author: Abraham Elmahrek <ab...@elmahrek.com>
> Date: Fri Apr 26 15:10:24 2013 -0700
>
> SQOOP-927 Sqoop2: Integration: Mapreduce specific tests should be running on MiniCluster
>
> Handle MiniDFSCluster and MiniMRClientCluster on own.
>
> Set yarn.application.classpath to get over classpath errors.
> Set to use fair scheduler.
>
> :100644 100644 0abbb18... f09704b... M pom.xml
> :100644 100644 6eb3184... a4c2a5b... M test/src/test/java/org/apache/sqoop/integration/TomcatTestCase.java
> :100644 100644 0f48a8b... 758c885... M test/src/test/java/org/apache/sqoop/integration/server/VersionTest.java
>
>
> Diffs
> -----
>
> pom.xml 513b6d0570ee6f1ec32f8c35a403becc1cfa156a
> test/pom.xml 8001fcec29bf5a7d9c3119b9f6ef548f678e1ebe
> test/src/main/java/org/apache/sqoop/test/asserts/HdfsAsserts.java c17527285167104994d78f5919b7f11c0c31cd15
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopLocalRunner.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopMiniClusterRunner.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopRunner.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopRunnerFactory.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/testcases/ConnectorTestCase.java d10b94268babeb3dc7b1d2450776af5fd8449fd3
> test/src/main/java/org/apache/sqoop/test/testcases/TomcatTestCase.java efdfed4e2065c0c01c16da88cf55705ab13d7bcf
> test/src/main/java/org/apache/sqoop/test/utils/HdfsUtils.java 95dd177a11c5d822e35ca54e5d93785f0a40fbfc
>
> Diff: https://reviews.apache.org/r/10964/diff/
>
>
> Testing
> -------
>
> Ran integration tests without and with miniclusters.
> Currently need to use both miniclusters or neither for tests to work.
>
>
> Thanks,
>
> Abraham Elmahrek
>
>
Re: Review Request 10964: SQOOP-927 Sqoop2: Integration: Mapreduce specific
tests should be running on MiniCluster
Posted by Jarek Cecho <ja...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10964/#review25479
-----------------------------------------------------------
Hi Abe,
thank you very much for working on this, greatly appreciated! I'm having some timeout issues/tests running forever on my machine. I do believe that it's more environmental, so I'll investigate that further.
dist/src/main/server/conf/catalina.properties
<https://reviews.apache.org/r/10964/#comment49911>
Is this change relevant for the patch?
test/pom.xml
<https://reviews.apache.org/r/10964/#comment49912>
Setting the timeout to 0 might be quite dangerous especially when running on Jenkins. Would it be possible to set here some high, but still finite number?
test/src/main/java/org/apache/sqoop/test/hadoop/HadoopMiniClusterRunner.java
<https://reviews.apache.org/r/10964/#comment49913>
Nit: tralling whitespace
test/src/main/java/org/apache/sqoop/test/hadoop/HadoopMiniClusterRunner.java
<https://reviews.apache.org/r/10964/#comment49914>
Nit: Wouldn't make sense to log "Stopping" and then start stopping the service?
test/src/main/java/org/apache/sqoop/test/hadoop/HadoopMiniClusterRunner.java
<https://reviews.apache.org/r/10964/#comment49915>
Nit: Wouldn't make sense to log "Stopping" and then start stopping the service?
test/src/main/java/org/apache/sqoop/test/hadoop/HadoopRunnerFactory.java
<https://reviews.apache.org/r/10964/#comment49916>
Nit: Can we rename the property to "sqoop.hadoop.runner.class" to match the Class names?
Jarcec
- Jarek Cecho
On Aug. 16, 2013, 12:17 a.m., Abraham Elmahrek wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/10964/
> -----------------------------------------------------------
>
> (Updated Aug. 16, 2013, 12:17 a.m.)
>
>
> Review request for Sqoop and Jarek Cecho.
>
>
> Bugs: SQOOP-927
> https://issues.apache.org/jira/browse/SQOOP-927
>
>
> Repository: sqoop-sqoop2
>
>
> Description
> -------
>
> commit ff7dba55a09dd7789a34136233000c625759e583
> Author: Abraham Elmahrek <ab...@elmahrek.com>
> Date: Fri Apr 26 15:10:24 2013 -0700
>
> SQOOP-927 Sqoop2: Integration: Mapreduce specific tests should be running on MiniCluster
>
> Handle MiniDFSCluster and MiniMRClientCluster on own.
>
> Set yarn.application.classpath to get over classpath errors.
> Set to use fair scheduler.
>
> :100644 100644 0abbb18... f09704b... M pom.xml
> :100644 100644 6eb3184... a4c2a5b... M test/src/test/java/org/apache/sqoop/integration/TomcatTestCase.java
> :100644 100644 0f48a8b... 758c885... M test/src/test/java/org/apache/sqoop/integration/server/VersionTest.java
>
>
> Diffs
> -----
>
> dist/src/main/server/conf/catalina.properties 5a47295d6e1bfa2114f86d620bb1452bc576df71
> pom.xml 513b6d0570ee6f1ec32f8c35a403becc1cfa156a
> test/pom.xml 8001fcec29bf5a7d9c3119b9f6ef548f678e1ebe
> test/src/main/java/org/apache/sqoop/test/asserts/HdfsAsserts.java c17527285167104994d78f5919b7f11c0c31cd15
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopLocalRunner.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopMiniClusterRunner.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopRunner.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopRunnerFactory.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/testcases/ConnectorTestCase.java d10b94268babeb3dc7b1d2450776af5fd8449fd3
> test/src/main/java/org/apache/sqoop/test/testcases/TomcatTestCase.java efdfed4e2065c0c01c16da88cf55705ab13d7bcf
> test/src/main/java/org/apache/sqoop/test/utils/HdfsUtils.java 95dd177a11c5d822e35ca54e5d93785f0a40fbfc
>
> Diff: https://reviews.apache.org/r/10964/diff/
>
>
> Testing
> -------
>
> Ran integration tests without and with miniclusters.
> Currently need to use both miniclusters or neither for tests to work.
>
>
> Thanks,
>
> Abraham Elmahrek
>
>
Re: Review Request 10964: SQOOP-927 Sqoop2: Integration: Mapreduce specific
tests should be running on MiniCluster
Posted by Abraham Elmahrek <ab...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10964/
-----------------------------------------------------------
(Updated Aug. 16, 2013, 12:17 a.m.)
Review request for Sqoop and Jarek Cecho.
Changes
-------
Rebased
Bugs: SQOOP-927
https://issues.apache.org/jira/browse/SQOOP-927
Repository: sqoop-sqoop2
Description
-------
commit ff7dba55a09dd7789a34136233000c625759e583
Author: Abraham Elmahrek <ab...@elmahrek.com>
Date: Fri Apr 26 15:10:24 2013 -0700
SQOOP-927 Sqoop2: Integration: Mapreduce specific tests should be running on MiniCluster
Handle MiniDFSCluster and MiniMRClientCluster on own.
Set yarn.application.classpath to get over classpath errors.
Set to use fair scheduler.
:100644 100644 0abbb18... f09704b... M pom.xml
:100644 100644 6eb3184... a4c2a5b... M test/src/test/java/org/apache/sqoop/integration/TomcatTestCase.java
:100644 100644 0f48a8b... 758c885... M test/src/test/java/org/apache/sqoop/integration/server/VersionTest.java
Diffs (updated)
-----
dist/src/main/server/conf/catalina.properties 5a47295d6e1bfa2114f86d620bb1452bc576df71
pom.xml 513b6d0570ee6f1ec32f8c35a403becc1cfa156a
test/pom.xml 8001fcec29bf5a7d9c3119b9f6ef548f678e1ebe
test/src/main/java/org/apache/sqoop/test/asserts/HdfsAsserts.java c17527285167104994d78f5919b7f11c0c31cd15
test/src/main/java/org/apache/sqoop/test/hadoop/HadoopLocalRunner.java PRE-CREATION
test/src/main/java/org/apache/sqoop/test/hadoop/HadoopMiniClusterRunner.java PRE-CREATION
test/src/main/java/org/apache/sqoop/test/hadoop/HadoopRunner.java PRE-CREATION
test/src/main/java/org/apache/sqoop/test/hadoop/HadoopRunnerFactory.java PRE-CREATION
test/src/main/java/org/apache/sqoop/test/testcases/ConnectorTestCase.java d10b94268babeb3dc7b1d2450776af5fd8449fd3
test/src/main/java/org/apache/sqoop/test/testcases/TomcatTestCase.java efdfed4e2065c0c01c16da88cf55705ab13d7bcf
test/src/main/java/org/apache/sqoop/test/utils/HdfsUtils.java 95dd177a11c5d822e35ca54e5d93785f0a40fbfc
Diff: https://reviews.apache.org/r/10964/diff/
Testing
-------
Ran integration tests without and with miniclusters.
Currently need to use both miniclusters or neither for tests to work.
Thanks,
Abraham Elmahrek
Re: Review Request 10964: SQOOP-927 Sqoop2: Integration: Mapreduce specific
tests should be running on MiniCluster
Posted by Abraham Elmahrek <ab...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10964/
-----------------------------------------------------------
(Updated July 19, 2013, 11:32 p.m.)
Review request for Sqoop and Jarek Cecho.
Bugs: SQOOP-927
https://issues.apache.org/jira/browse/SQOOP-927
Repository: sqoop-sqoop2
Description
-------
commit ff7dba55a09dd7789a34136233000c625759e583
Author: Abraham Elmahrek <ab...@elmahrek.com>
Date: Fri Apr 26 15:10:24 2013 -0700
SQOOP-927 Sqoop2: Integration: Mapreduce specific tests should be running on MiniCluster
Handle MiniDFSCluster and MiniMRClientCluster on own.
Set yarn.application.classpath to get over classpath errors.
Set to use fair scheduler.
:100644 100644 0abbb18... f09704b... M pom.xml
:100644 100644 6eb3184... a4c2a5b... M test/src/test/java/org/apache/sqoop/integration/TomcatTestCase.java
:100644 100644 0f48a8b... 758c885... M test/src/test/java/org/apache/sqoop/integration/server/VersionTest.java
Diffs (updated)
-----
pom.xml 8785e01000cc6e7d5a74ffeb96b83d236a657df8
test/src/main/java/org/apache/sqoop/test/asserts/HdfsAsserts.java 056e6124b918ce5821c389e388a0cdfa68fd7fc0
test/src/main/java/org/apache/sqoop/test/hadoop/HadoopLocalRunner.java PRE-CREATION
test/src/main/java/org/apache/sqoop/test/hadoop/HadoopMiniClusterRunner.java PRE-CREATION
test/src/main/java/org/apache/sqoop/test/hadoop/HadoopRunner.java PRE-CREATION
test/src/main/java/org/apache/sqoop/test/hadoop/HadoopRunnerFactory.java PRE-CREATION
test/src/main/java/org/apache/sqoop/test/testcases/ConnectorTestCase.java 6aeadd4e54e47e5644270b15813b2d9c4cedc059
test/src/main/java/org/apache/sqoop/test/testcases/TomcatTestCase.java ca77e64486d80b38306b5b30e185e6278ad7eaf1
test/src/main/java/org/apache/sqoop/test/utils/HdfsUtils.java 95dd177a11c5d822e35ca54e5d93785f0a40fbfc
Diff: https://reviews.apache.org/r/10964/diff/
Testing
-------
Ran integration tests without and with miniclusters.
Currently need to use both miniclusters or neither for tests to work.
Thanks,
Abraham Elmahrek
Re: Review Request 10964: SQOOP-927 Sqoop2: Integration: Mapreduce specific
tests should be running on MiniCluster
Posted by Abraham Elmahrek <ab...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10964/
-----------------------------------------------------------
(Updated July 10, 2013, 11:27 p.m.)
Review request for Sqoop and Jarek Cecho.
Bugs: SQOOP-927
https://issues.apache.org/jira/browse/SQOOP-927
Repository: sqoop-sqoop2
Description
-------
commit ff7dba55a09dd7789a34136233000c625759e583
Author: Abraham Elmahrek <ab...@elmahrek.com>
Date: Fri Apr 26 15:10:24 2013 -0700
SQOOP-927 Sqoop2: Integration: Mapreduce specific tests should be running on MiniCluster
Handle MiniDFSCluster and MiniMRClientCluster on own.
Set yarn.application.classpath to get over classpath errors.
Set to use fair scheduler.
:100644 100644 0abbb18... f09704b... M pom.xml
:100644 100644 6eb3184... a4c2a5b... M test/src/test/java/org/apache/sqoop/integration/TomcatTestCase.java
:100644 100644 0f48a8b... 758c885... M test/src/test/java/org/apache/sqoop/integration/server/VersionTest.java
Diffs (updated)
-----
pom.xml 8785e01000cc6e7d5a74ffeb96b83d236a657df8
test/src/main/java/org/apache/sqoop/test/asserts/HdfsAsserts.java 056e6124b918ce5821c389e388a0cdfa68fd7fc0
test/src/main/java/org/apache/sqoop/test/hadoop/HadoopLocalRunner.java PRE-CREATION
test/src/main/java/org/apache/sqoop/test/hadoop/HadoopMiniClusterRunner.java PRE-CREATION
test/src/main/java/org/apache/sqoop/test/hadoop/HadoopRunner.java PRE-CREATION
test/src/main/java/org/apache/sqoop/test/hadoop/HadoopRunnerFactory.java PRE-CREATION
test/src/main/java/org/apache/sqoop/test/testcases/ConnectorTestCase.java 6aeadd4e54e47e5644270b15813b2d9c4cedc059
test/src/main/java/org/apache/sqoop/test/testcases/TomcatTestCase.java ca77e64486d80b38306b5b30e185e6278ad7eaf1
test/src/main/java/org/apache/sqoop/test/utils/HdfsUtils.java 95dd177a11c5d822e35ca54e5d93785f0a40fbfc
Diff: https://reviews.apache.org/r/10964/diff/
Testing
-------
Ran integration tests without and with miniclusters.
Currently need to use both miniclusters or neither for tests to work.
Thanks,
Abraham Elmahrek
Re: Review Request 10964: SQOOP-927 Sqoop2: Integration: Mapreduce specific
tests should be running on MiniCluster
Posted by Abraham Elmahrek <ab...@cloudera.com>.
> On July 9, 2013, 6:14 p.m., Jarek Cecho wrote:
> > test/src/main/java/org/apache/sqoop/test/hadoop/HadoopStandaloneCluster.java, line 23
> > <https://reviews.apache.org/r/10964/diff/4/?file=318881#file318881line23>
> >
> > Nit: I'm afraid that the word "cluster" in the name can be quite confusing as this is not using any cluster at all. Would it be possible to rename it to something like LocalMode or LocalRunner?
I changed every thing to be suffixed with the word "Runner". I feel as though this overlaps with MapReduce nomenclature. Any thoughts?
> On July 9, 2013, 6:14 p.m., Jarek Cecho wrote:
> > test/src/main/java/org/apache/sqoop/test/hadoop/HadoopMiniCluster.java, lines 50-51
> > <https://reviews.apache.org/r/10964/diff/4/?file=318880#file318880line50>
> >
> > The getDataDir seems to be overloaded. Here it's used for local files that are generated by the MiniClusters, however we are also using it for the HDFS files. I would recommend to distinguish those two usages.
Added 'getTestDirectory' to symbolically discern between data directory and test directory.
> On July 9, 2013, 6:14 p.m., Jarek Cecho wrote:
> > test/src/main/java/org/apache/sqoop/test/testcases/TomcatTestCase.java, lines 62-75
> > <https://reviews.apache.org/r/10964/diff/4/?file=318883#file318883line62>
> >
> > I think that the main temporary path should be owned by this class because we are storing there a lot of information required for the Sqoop Server itself (logs, metastore, configuration, ....). Those files exists outside of the mapreduce/hdfs.
Doesn't it make more sense for the sqoop minicluster classes to manage their own temporary path? It seems like we should only provide a base directory.
> On July 9, 2013, 6:14 p.m., Jarek Cecho wrote:
> > test/src/main/java/org/apache/sqoop/test/testcases/ConnectorTestCase.java, lines 52-54
> > <https://reviews.apache.org/r/10964/diff/4/?file=318882#file318882line52>
> >
> > I think that we should be able to put the HadoopCluster initialization into method annotated with @BeforeClass and override this method in child if necessary. This way we should save some time bootstrapping the Minicluster for test cases that have more then one test method.
Works!
- Abraham
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10964/#review22919
-----------------------------------------------------------
On July 8, 2013, 8:47 p.m., Abraham Elmahrek wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/10964/
> -----------------------------------------------------------
>
> (Updated July 8, 2013, 8:47 p.m.)
>
>
> Review request for Sqoop and Jarek Cecho.
>
>
> Bugs: SQOOP-927
> https://issues.apache.org/jira/browse/SQOOP-927
>
>
> Repository: sqoop-sqoop2
>
>
> Description
> -------
>
> commit ff7dba55a09dd7789a34136233000c625759e583
> Author: Abraham Elmahrek <ab...@elmahrek.com>
> Date: Fri Apr 26 15:10:24 2013 -0700
>
> SQOOP-927 Sqoop2: Integration: Mapreduce specific tests should be running on MiniCluster
>
> Handle MiniDFSCluster and MiniMRClientCluster on own.
>
> Set yarn.application.classpath to get over classpath errors.
> Set to use fair scheduler.
>
> :100644 100644 0abbb18... f09704b... M pom.xml
> :100644 100644 6eb3184... a4c2a5b... M test/src/test/java/org/apache/sqoop/integration/TomcatTestCase.java
> :100644 100644 0f48a8b... 758c885... M test/src/test/java/org/apache/sqoop/integration/server/VersionTest.java
>
>
> Diffs
> -----
>
> pom.xml 8785e01000cc6e7d5a74ffeb96b83d236a657df8
> test/src/main/java/org/apache/sqoop/test/asserts/HdfsAsserts.java 056e6124b918ce5821c389e388a0cdfa68fd7fc0
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopCluster.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopClusterFactory.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopMiniCluster.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopStandaloneCluster.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/testcases/ConnectorTestCase.java 6aeadd4e54e47e5644270b15813b2d9c4cedc059
> test/src/main/java/org/apache/sqoop/test/testcases/TomcatTestCase.java ca77e64486d80b38306b5b30e185e6278ad7eaf1
> test/src/main/java/org/apache/sqoop/test/utils/HdfsUtils.java 95dd177a11c5d822e35ca54e5d93785f0a40fbfc
>
> Diff: https://reviews.apache.org/r/10964/diff/
>
>
> Testing
> -------
>
> Ran integration tests without and with miniclusters.
> Currently need to use both miniclusters or neither for tests to work.
>
>
> Thanks,
>
> Abraham Elmahrek
>
>
Re: Review Request 10964: SQOOP-927 Sqoop2: Integration: Mapreduce specific
tests should be running on MiniCluster
Posted by Jarek Cecho <ja...@apache.org>.
> On July 9, 2013, 6:14 p.m., Jarek Cecho wrote:
> > test/src/main/java/org/apache/sqoop/test/hadoop/HadoopMiniCluster.java, lines 50-51
> > <https://reviews.apache.org/r/10964/diff/4/?file=318880#file318880line50>
> >
> > The getDataDir seems to be overloaded. Here it's used for local files that are generated by the MiniClusters, however we are also using it for the HDFS files. I would recommend to distinguish those two usages.
>
> Abraham Elmahrek wrote:
> Added 'getTestDirectory' to symbolically discern between data directory and test directory.
Thank you!
> On July 9, 2013, 6:14 p.m., Jarek Cecho wrote:
> > test/src/main/java/org/apache/sqoop/test/hadoop/HadoopStandaloneCluster.java, line 23
> > <https://reviews.apache.org/r/10964/diff/4/?file=318881#file318881line23>
> >
> > Nit: I'm afraid that the word "cluster" in the name can be quite confusing as this is not using any cluster at all. Would it be possible to rename it to something like LocalMode or LocalRunner?
>
> Abraham Elmahrek wrote:
> I changed every thing to be suffixed with the word "Runner". I feel as though this overlaps with MapReduce nomenclature. Any thoughts?
I do not personally mind word Runner, but some other ideas:
* "Implementation" -> LocalImplementation, MiniClusterImplementation, RealClusterImplementation
* "Impl" -> LocalImpl, MiniClusterImpl, RealClusterImpl
> On July 9, 2013, 6:14 p.m., Jarek Cecho wrote:
> > test/src/main/java/org/apache/sqoop/test/testcases/TomcatTestCase.java, lines 62-75
> > <https://reviews.apache.org/r/10964/diff/4/?file=318883#file318883line62>
> >
> > I think that the main temporary path should be owned by this class because we are storing there a lot of information required for the Sqoop Server itself (logs, metastore, configuration, ....). Those files exists outside of the mapreduce/hdfs.
>
> Abraham Elmahrek wrote:
> Doesn't it make more sense for the sqoop minicluster classes to manage their own temporary path? It seems like we should only provide a base directory.
The tmpPath in this case is actually a base directory from this perspective. Child test cases are expected to use this, append additional sub directory and use it. It's not called "base" though because we already have one base path that is shared for all test cases, this one is specific for each particular testing method.
- Jarek
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10964/#review22919
-----------------------------------------------------------
On July 10, 2013, 11:27 p.m., Abraham Elmahrek wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/10964/
> -----------------------------------------------------------
>
> (Updated July 10, 2013, 11:27 p.m.)
>
>
> Review request for Sqoop and Jarek Cecho.
>
>
> Bugs: SQOOP-927
> https://issues.apache.org/jira/browse/SQOOP-927
>
>
> Repository: sqoop-sqoop2
>
>
> Description
> -------
>
> commit ff7dba55a09dd7789a34136233000c625759e583
> Author: Abraham Elmahrek <ab...@elmahrek.com>
> Date: Fri Apr 26 15:10:24 2013 -0700
>
> SQOOP-927 Sqoop2: Integration: Mapreduce specific tests should be running on MiniCluster
>
> Handle MiniDFSCluster and MiniMRClientCluster on own.
>
> Set yarn.application.classpath to get over classpath errors.
> Set to use fair scheduler.
>
> :100644 100644 0abbb18... f09704b... M pom.xml
> :100644 100644 6eb3184... a4c2a5b... M test/src/test/java/org/apache/sqoop/integration/TomcatTestCase.java
> :100644 100644 0f48a8b... 758c885... M test/src/test/java/org/apache/sqoop/integration/server/VersionTest.java
>
>
> Diffs
> -----
>
> pom.xml 8785e01000cc6e7d5a74ffeb96b83d236a657df8
> test/src/main/java/org/apache/sqoop/test/asserts/HdfsAsserts.java 056e6124b918ce5821c389e388a0cdfa68fd7fc0
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopLocalRunner.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopMiniClusterRunner.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopRunner.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopRunnerFactory.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/testcases/ConnectorTestCase.java 6aeadd4e54e47e5644270b15813b2d9c4cedc059
> test/src/main/java/org/apache/sqoop/test/testcases/TomcatTestCase.java ca77e64486d80b38306b5b30e185e6278ad7eaf1
> test/src/main/java/org/apache/sqoop/test/utils/HdfsUtils.java 95dd177a11c5d822e35ca54e5d93785f0a40fbfc
>
> Diff: https://reviews.apache.org/r/10964/diff/
>
>
> Testing
> -------
>
> Ran integration tests without and with miniclusters.
> Currently need to use both miniclusters or neither for tests to work.
>
>
> Thanks,
>
> Abraham Elmahrek
>
>
Re: Review Request 10964: SQOOP-927 Sqoop2: Integration: Mapreduce specific
tests should be running on MiniCluster
Posted by Jarek Cecho <ja...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10964/#review22919
-----------------------------------------------------------
Hi Abe,
thank you very much for incorporating all my suggestions. I think that we are on track to get it committed very soon!
pom.xml
<https://reviews.apache.org/r/10964/#comment46678>
Do you think that it would make sense to bump the version up to the latest 2.0.5-alpha release?
test/src/main/java/org/apache/sqoop/test/hadoop/HadoopCluster.java
<https://reviews.apache.org/r/10964/#comment46668>
Please provide JavaDoc describing purpose of this abstract class.
test/src/main/java/org/apache/sqoop/test/hadoop/HadoopCluster.java
<https://reviews.apache.org/r/10964/#comment46667>
Nit: This comment seems to be copied from the TomcatTestCase, but I'm not sure that it's applicable here. The method setTemporaryPath() can set any arbitrary directory so the pattern might not be accurate (the class itself can be used outside TomcatTestCase).
test/src/main/java/org/apache/sqoop/test/hadoop/HadoopCluster.java
<https://reviews.apache.org/r/10964/#comment46692>
Nit: Please improve the java doc a bit by for example stating when this method will be called or what are the allowed/disallowed operations. It seems that the expected usage to call this method only once prior running start() method.
test/src/main/java/org/apache/sqoop/test/hadoop/HadoopClusterFactory.java
<https://reviews.apache.org/r/10964/#comment46666>
I would suggest to start the property name with prefix "sqoop." to avoid any possible conflict with Hadoop property of the same name.
test/src/main/java/org/apache/sqoop/test/hadoop/HadoopMiniCluster.java
<https://reviews.apache.org/r/10964/#comment46691>
The getDataDir seems to be overloaded. Here it's used for local files that are generated by the MiniClusters, however we are also using it for the HDFS files. I would recommend to distinguish those two usages.
test/src/main/java/org/apache/sqoop/test/hadoop/HadoopStandaloneCluster.java
<https://reviews.apache.org/r/10964/#comment46670>
Nit: I'm afraid that the word "cluster" in the name can be quite confusing as this is not using any cluster at all. Would it be possible to rename it to something like LocalMode or LocalRunner?
test/src/main/java/org/apache/sqoop/test/testcases/ConnectorTestCase.java
<https://reviews.apache.org/r/10964/#comment46693>
I think that we should be able to put the HadoopCluster initialization into method annotated with @BeforeClass and override this method in child if necessary. This way we should save some time bootstrapping the Minicluster for test cases that have more then one test method.
test/src/main/java/org/apache/sqoop/test/testcases/TomcatTestCase.java
<https://reviews.apache.org/r/10964/#comment46671>
Nit: This change do not seem to be necessary.
test/src/main/java/org/apache/sqoop/test/testcases/TomcatTestCase.java
<https://reviews.apache.org/r/10964/#comment46681>
I think that the main temporary path should be owned by this class because we are storing there a lot of information required for the Sqoop Server itself (logs, metastore, configuration, ....). Those files exists outside of the mapreduce/hdfs.
test/src/main/java/org/apache/sqoop/test/testcases/TomcatTestCase.java
<https://reviews.apache.org/r/10964/#comment46684>
I don't think that we should be cleaning up all the directories after the test. The directory created on local file system will be cleaned up automatically by either next test or maven itself. The question is what to do with the directories on HDFS in case of minicluster or future real cluster. I would personally vote to not remove them as such action will complicate investigation of non deterministic issues and will most likely lead to a disabling the removal anyway.
test/src/main/java/org/apache/sqoop/test/testcases/TomcatTestCase.java
<https://reviews.apache.org/r/10964/#comment46683>
I would suggest to distinguish between main temporary directory for local files (configuration, logs, ...) and temporary directory for HDFS files (on minicluster). The local directory should remain inside the "target" directory of maven structures, so that it will be cleaned up on "mvn clean". The HDFS directory on Local Mode should remain inside the "target" dir as well. I think that in case of a mini cluster the path can be arbitrary as entire file system is recreated every time. However we will have to remain the directory separation for each test method in case that we will be reusing the MiniCluster for multiple tests.
test/src/main/java/org/apache/sqoop/test/testcases/TomcatTestCase.java
<https://reviews.apache.org/r/10964/#comment46673>
Nit: The method reorder do not seem to be necessary.
test/src/main/java/org/apache/sqoop/test/utils/HdfsUtils.java
<https://reviews.apache.org/r/10964/#comment46674>
Nit: Please remove the comment as it's no longer applicable.
test/src/main/java/org/apache/sqoop/test/utils/HdfsUtils.java
<https://reviews.apache.org/r/10964/#comment46675>
Nit: Is the logger removal intentional?
test/src/main/java/org/apache/sqoop/test/utils/HdfsUtils.java
<https://reviews.apache.org/r/10964/#comment46680>
Nit: This method seems to be joining multiple path fragments into one single path. Maybe it would be worth to rename the method and javadoc to reflect that?
Jarcec
- Jarek Cecho
On July 8, 2013, 8:47 p.m., Abraham Elmahrek wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/10964/
> -----------------------------------------------------------
>
> (Updated July 8, 2013, 8:47 p.m.)
>
>
> Review request for Sqoop and Jarek Cecho.
>
>
> Bugs: SQOOP-927
> https://issues.apache.org/jira/browse/SQOOP-927
>
>
> Repository: sqoop-sqoop2
>
>
> Description
> -------
>
> commit ff7dba55a09dd7789a34136233000c625759e583
> Author: Abraham Elmahrek <ab...@elmahrek.com>
> Date: Fri Apr 26 15:10:24 2013 -0700
>
> SQOOP-927 Sqoop2: Integration: Mapreduce specific tests should be running on MiniCluster
>
> Handle MiniDFSCluster and MiniMRClientCluster on own.
>
> Set yarn.application.classpath to get over classpath errors.
> Set to use fair scheduler.
>
> :100644 100644 0abbb18... f09704b... M pom.xml
> :100644 100644 6eb3184... a4c2a5b... M test/src/test/java/org/apache/sqoop/integration/TomcatTestCase.java
> :100644 100644 0f48a8b... 758c885... M test/src/test/java/org/apache/sqoop/integration/server/VersionTest.java
>
>
> Diffs
> -----
>
> pom.xml 8785e01000cc6e7d5a74ffeb96b83d236a657df8
> test/src/main/java/org/apache/sqoop/test/asserts/HdfsAsserts.java 056e6124b918ce5821c389e388a0cdfa68fd7fc0
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopCluster.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopClusterFactory.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopMiniCluster.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/hadoop/HadoopStandaloneCluster.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/testcases/ConnectorTestCase.java 6aeadd4e54e47e5644270b15813b2d9c4cedc059
> test/src/main/java/org/apache/sqoop/test/testcases/TomcatTestCase.java ca77e64486d80b38306b5b30e185e6278ad7eaf1
> test/src/main/java/org/apache/sqoop/test/utils/HdfsUtils.java 95dd177a11c5d822e35ca54e5d93785f0a40fbfc
>
> Diff: https://reviews.apache.org/r/10964/diff/
>
>
> Testing
> -------
>
> Ran integration tests without and with miniclusters.
> Currently need to use both miniclusters or neither for tests to work.
>
>
> Thanks,
>
> Abraham Elmahrek
>
>
Re: Review Request 10964: SQOOP-927 Sqoop2: Integration: Mapreduce specific
tests should be running on MiniCluster
Posted by Abraham Elmahrek <ab...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10964/
-----------------------------------------------------------
(Updated July 8, 2013, 8:47 p.m.)
Review request for Sqoop and Jarek Cecho.
Changes
-------
Miniclusters are loaded on a per test basis so that the default class can be overridden.
Bugs: SQOOP-927
https://issues.apache.org/jira/browse/SQOOP-927
Repository: sqoop-sqoop2
Description
-------
commit ff7dba55a09dd7789a34136233000c625759e583
Author: Abraham Elmahrek <ab...@elmahrek.com>
Date: Fri Apr 26 15:10:24 2013 -0700
SQOOP-927 Sqoop2: Integration: Mapreduce specific tests should be running on MiniCluster
Handle MiniDFSCluster and MiniMRClientCluster on own.
Set yarn.application.classpath to get over classpath errors.
Set to use fair scheduler.
:100644 100644 0abbb18... f09704b... M pom.xml
:100644 100644 6eb3184... a4c2a5b... M test/src/test/java/org/apache/sqoop/integration/TomcatTestCase.java
:100644 100644 0f48a8b... 758c885... M test/src/test/java/org/apache/sqoop/integration/server/VersionTest.java
Diffs (updated)
-----
pom.xml 8785e01000cc6e7d5a74ffeb96b83d236a657df8
test/src/main/java/org/apache/sqoop/test/asserts/HdfsAsserts.java 056e6124b918ce5821c389e388a0cdfa68fd7fc0
test/src/main/java/org/apache/sqoop/test/hadoop/HadoopCluster.java PRE-CREATION
test/src/main/java/org/apache/sqoop/test/hadoop/HadoopClusterFactory.java PRE-CREATION
test/src/main/java/org/apache/sqoop/test/hadoop/HadoopMiniCluster.java PRE-CREATION
test/src/main/java/org/apache/sqoop/test/hadoop/HadoopStandaloneCluster.java PRE-CREATION
test/src/main/java/org/apache/sqoop/test/testcases/ConnectorTestCase.java 6aeadd4e54e47e5644270b15813b2d9c4cedc059
test/src/main/java/org/apache/sqoop/test/testcases/TomcatTestCase.java ca77e64486d80b38306b5b30e185e6278ad7eaf1
test/src/main/java/org/apache/sqoop/test/utils/HdfsUtils.java 95dd177a11c5d822e35ca54e5d93785f0a40fbfc
Diff: https://reviews.apache.org/r/10964/diff/
Testing
-------
Ran integration tests without and with miniclusters.
Currently need to use both miniclusters or neither for tests to work.
Thanks,
Abraham Elmahrek
Re: Review Request 10964: SQOOP-927 Sqoop2: Integration: Mapreduce specific
tests should be running on MiniCluster
Posted by Abraham Elmahrek <ab...@cloudera.com>.
> On July 8, 2013, 3:38 p.m., Jarek Cecho wrote:
> > Hi Abe,
> > thank you very much for rebasing this patch after all the changes that happened on the sqoop2 branch, greatly appreciated!
> >
> > It seems that current implementation is forcing the minicluster to be run either for all test cases or to none. We have multiple types of test cases, some of them like VersionTest are not executing any mapreduce jobs and thus using the minicluster for them by default seems to be unnecessary waste of resources. What about creating the "Hadoop Mini Cluster" as a first level citizen in the test module similarly as is the SqoopMiniCluster implementation that the tests cases are simply reusing? We can create abstract HadoopMiniCluster class with two concrete implementations for LocalJobRunner and MiniCluster. Each test case class could then choose default implementation - for example TomcatTestCase would use "LocalJobRunner" by default (and transitively also VersionTest) whereas the ConnectorTestCase which is mostly running mapreduce jobs would by default use the "MiniCluster". We could also do it overridable by system property, so that user can force the implementation as he seems fi
t (similarly as DatabaseProviderFactory is doing for database providers). This solution would be very easily extensible with running on real cluster by creating different concrete implementation.
Jarcec,
Makes sense. I've taken your feedback into consideration and have made the suggested modifications.
> On July 8, 2013, 3:38 p.m., Jarek Cecho wrote:
> > test/src/main/java/org/apache/sqoop/test/asserts/HdfsAsserts.java, lines 71-78
> > <https://reviews.apache.org/r/10964/diff/3/?file=318133#file318133line71>
> >
> > Is this removal intentional?
Nope... added it back.
- Abraham
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10964/#review22752
-----------------------------------------------------------
On July 4, 2013, 7:59 p.m., Abraham Elmahrek wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/10964/
> -----------------------------------------------------------
>
> (Updated July 4, 2013, 7:59 p.m.)
>
>
> Review request for Sqoop and Jarek Cecho.
>
>
> Bugs: SQOOP-927
> https://issues.apache.org/jira/browse/SQOOP-927
>
>
> Repository: sqoop-sqoop2
>
>
> Description
> -------
>
> commit ff7dba55a09dd7789a34136233000c625759e583
> Author: Abraham Elmahrek <ab...@elmahrek.com>
> Date: Fri Apr 26 15:10:24 2013 -0700
>
> SQOOP-927 Sqoop2: Integration: Mapreduce specific tests should be running on MiniCluster
>
> Handle MiniDFSCluster and MiniMRClientCluster on own.
>
> Set yarn.application.classpath to get over classpath errors.
> Set to use fair scheduler.
>
> :100644 100644 0abbb18... f09704b... M pom.xml
> :100644 100644 6eb3184... a4c2a5b... M test/src/test/java/org/apache/sqoop/integration/TomcatTestCase.java
> :100644 100644 0f48a8b... 758c885... M test/src/test/java/org/apache/sqoop/integration/server/VersionTest.java
>
>
> Diffs
> -----
>
> pom.xml 8785e01000cc6e7d5a74ffeb96b83d236a657df8
> test/src/main/java/org/apache/sqoop/test/asserts/HdfsAsserts.java 056e6124b918ce5821c389e388a0cdfa68fd7fc0
> test/src/main/java/org/apache/sqoop/test/testcases/HadoopTestCase.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/testcases/TomcatTestCase.java ca77e64486d80b38306b5b30e185e6278ad7eaf1
> test/src/main/java/org/apache/sqoop/test/utils/HdfsUtils.java 95dd177a11c5d822e35ca54e5d93785f0a40fbfc
>
> Diff: https://reviews.apache.org/r/10964/diff/
>
>
> Testing
> -------
>
> Ran integration tests without and with miniclusters.
> Currently need to use both miniclusters or neither for tests to work.
>
>
> Thanks,
>
> Abraham Elmahrek
>
>
Re: Review Request 10964: SQOOP-927 Sqoop2: Integration: Mapreduce specific
tests should be running on MiniCluster
Posted by Jarek Cecho <ja...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10964/#review22752
-----------------------------------------------------------
Hi Abe,
thank you very much for rebasing this patch after all the changes that happened on the sqoop2 branch, greatly appreciated!
It seems that current implementation is forcing the minicluster to be run either for all test cases or to none. We have multiple types of test cases, some of them like VersionTest are not executing any mapreduce jobs and thus using the minicluster for them by default seems to be unnecessary waste of resources. What about creating the "Hadoop Mini Cluster" as a first level citizen in the test module similarly as is the SqoopMiniCluster implementation that the tests cases are simply reusing? We can create abstract HadoopMiniCluster class with two concrete implementations for LocalJobRunner and MiniCluster. Each test case class could then choose default implementation - for example TomcatTestCase would use "LocalJobRunner" by default (and transitively also VersionTest) whereas the ConnectorTestCase which is mostly running mapreduce jobs would by default use the "MiniCluster". We could also do it overridable by system property, so that user can force the implementation as he seems fit (s
imilarly as DatabaseProviderFactory is doing for database providers). This solution would be very easily extensible with running on real cluster by creating different concrete implementation.
test/src/main/java/org/apache/sqoop/test/asserts/HdfsAsserts.java
<https://reviews.apache.org/r/10964/#comment46451>
Nit: Can we remove this deprecated comment?
test/src/main/java/org/apache/sqoop/test/asserts/HdfsAsserts.java
<https://reviews.apache.org/r/10964/#comment46452>
Nit: Trailing whitespace characters.
test/src/main/java/org/apache/sqoop/test/asserts/HdfsAsserts.java
<https://reviews.apache.org/r/10964/#comment46531>
Is this removal intentional?
test/src/main/java/org/apache/sqoop/test/testcases/HadoopTestCase.java
<https://reviews.apache.org/r/10964/#comment46534>
This file is missing license header.
test/src/main/java/org/apache/sqoop/test/testcases/HadoopTestCase.java
<https://reviews.apache.org/r/10964/#comment46532>
Nit: Trailing white space characters.
test/src/main/java/org/apache/sqoop/test/testcases/HadoopTestCase.java
<https://reviews.apache.org/r/10964/#comment46535>
Do we need to restart the minicluster for each individual test method? Would @BeforeClass be sufficient here?
test/src/main/java/org/apache/sqoop/test/utils/HdfsUtils.java
<https://reviews.apache.org/r/10964/#comment46533>
Nit: Trailing white space characters.
Jarcec
- Jarek Cecho
On July 4, 2013, 7:59 p.m., Abraham Elmahrek wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/10964/
> -----------------------------------------------------------
>
> (Updated July 4, 2013, 7:59 p.m.)
>
>
> Review request for Sqoop and Jarek Cecho.
>
>
> Bugs: SQOOP-927
> https://issues.apache.org/jira/browse/SQOOP-927
>
>
> Repository: sqoop-sqoop2
>
>
> Description
> -------
>
> commit ff7dba55a09dd7789a34136233000c625759e583
> Author: Abraham Elmahrek <ab...@elmahrek.com>
> Date: Fri Apr 26 15:10:24 2013 -0700
>
> SQOOP-927 Sqoop2: Integration: Mapreduce specific tests should be running on MiniCluster
>
> Handle MiniDFSCluster and MiniMRClientCluster on own.
>
> Set yarn.application.classpath to get over classpath errors.
> Set to use fair scheduler.
>
> :100644 100644 0abbb18... f09704b... M pom.xml
> :100644 100644 6eb3184... a4c2a5b... M test/src/test/java/org/apache/sqoop/integration/TomcatTestCase.java
> :100644 100644 0f48a8b... 758c885... M test/src/test/java/org/apache/sqoop/integration/server/VersionTest.java
>
>
> Diffs
> -----
>
> pom.xml 8785e01000cc6e7d5a74ffeb96b83d236a657df8
> test/src/main/java/org/apache/sqoop/test/asserts/HdfsAsserts.java 056e6124b918ce5821c389e388a0cdfa68fd7fc0
> test/src/main/java/org/apache/sqoop/test/testcases/HadoopTestCase.java PRE-CREATION
> test/src/main/java/org/apache/sqoop/test/testcases/TomcatTestCase.java ca77e64486d80b38306b5b30e185e6278ad7eaf1
> test/src/main/java/org/apache/sqoop/test/utils/HdfsUtils.java 95dd177a11c5d822e35ca54e5d93785f0a40fbfc
>
> Diff: https://reviews.apache.org/r/10964/diff/
>
>
> Testing
> -------
>
> Ran integration tests without and with miniclusters.
> Currently need to use both miniclusters or neither for tests to work.
>
>
> Thanks,
>
> Abraham Elmahrek
>
>
Re: Review Request 10964: SQOOP-927 Sqoop2: Integration: Mapreduce specific
tests should be running on MiniCluster
Posted by Abraham Elmahrek <ab...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10964/
-----------------------------------------------------------
(Updated July 4, 2013, 7:59 p.m.)
Review request for Sqoop and Jarek Cecho.
Bugs: SQOOP-927
https://issues.apache.org/jira/browse/SQOOP-927
Repository: sqoop-sqoop2
Description
-------
commit ff7dba55a09dd7789a34136233000c625759e583
Author: Abraham Elmahrek <ab...@elmahrek.com>
Date: Fri Apr 26 15:10:24 2013 -0700
SQOOP-927 Sqoop2: Integration: Mapreduce specific tests should be running on MiniCluster
Handle MiniDFSCluster and MiniMRClientCluster on own.
Set yarn.application.classpath to get over classpath errors.
Set to use fair scheduler.
:100644 100644 0abbb18... f09704b... M pom.xml
:100644 100644 6eb3184... a4c2a5b... M test/src/test/java/org/apache/sqoop/integration/TomcatTestCase.java
:100644 100644 0f48a8b... 758c885... M test/src/test/java/org/apache/sqoop/integration/server/VersionTest.java
Diffs
-----
pom.xml 8785e01000cc6e7d5a74ffeb96b83d236a657df8
test/src/main/java/org/apache/sqoop/test/asserts/HdfsAsserts.java 056e6124b918ce5821c389e388a0cdfa68fd7fc0
test/src/main/java/org/apache/sqoop/test/testcases/HadoopTestCase.java PRE-CREATION
test/src/main/java/org/apache/sqoop/test/testcases/TomcatTestCase.java ca77e64486d80b38306b5b30e185e6278ad7eaf1
test/src/main/java/org/apache/sqoop/test/utils/HdfsUtils.java 95dd177a11c5d822e35ca54e5d93785f0a40fbfc
Diff: https://reviews.apache.org/r/10964/diff/
Testing (updated)
-------
Ran integration tests without and with miniclusters.
Currently need to use both miniclusters or neither for tests to work.
Thanks,
Abraham Elmahrek