You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@crunch.apache.org by Josh Wills <jw...@apache.org> on 2012/06/27 06:26:15 UTC
getting crunch running on hadoop 2.0.0
Hey all,
That last commit from me "fixes" (in some loose sense of the word) the
WordCountHBaseTest, which was the last test that we needed to get
working in order to have all of the Crunch unit tests passing under
Hadoop-2.0.0.
The complexity stems from the fact that the WordCountHBaseTest starts
up a mini hbase/mapreduce cluster in order to run, and I had to do
some work to make sure that all of the proper dependencies and class
definitions that were needed for the test case to pass were available
to the mini YARN instance that runs w/HBaseTestingUtility under hadoop
2.0.0.
Vinod, if you get a chance to take a look at the test case and see if
there is anything I could have done to make it cleaner, I would be
much obliged.
J
Re: getting crunch running on hadoop 2.0.0
Posted by Josh Wills <jw...@cloudera.com>.
None, but I'm not particularly knowledgable w/Eclipse's handling of
unit tests and how they get cleaned up. I rarely run more than one
unit test at a time in Eclipse. I'd be surprised about the conflicts,
since I think that all of the other unit tests run in local mode, and
don't make use of a mini cluster.
There was awhile there when the sbt test cycle of Scrunch was giving
me fits b/c I didn't realize that it was running all of the tests
simultaneously and they were conflicting with each other, but I doubt
that is the case here.
On Tue, Jun 26, 2012 at 9:39 PM, Gabriel Reid <ga...@gmail.com> wrote:
> Something somewhat related: yesterday I had an issue when running the unit tests (or a subset of the tests) from within Eclipse. I noticed that if the WordCountHbaseTest was run, a lot of tests after it would fail. Not running the WordCountHbaseTest (ie setting @Ignore on it) seemed to resolve the problem, an all tests would run through fine.
>
> This was only an issue when running in Eclipse; running the tests from maven seemed to work fine.
>
> Any ideas on this?
>
> On 27 Jun 2012, at 06:26, Josh Wills <jw...@apache.org> wrote:
>
>> Hey all,
>>
>> That last commit from me "fixes" (in some loose sense of the word) the
>> WordCountHBaseTest, which was the last test that we needed to get
>> working in order to have all of the Crunch unit tests passing under
>> Hadoop-2.0.0.
>>
>> The complexity stems from the fact that the WordCountHBaseTest starts
>> up a mini hbase/mapreduce cluster in order to run, and I had to do
>> some work to make sure that all of the proper dependencies and class
>> definitions that were needed for the test case to pass were available
>> to the mini YARN instance that runs w/HBaseTestingUtility under hadoop
>> 2.0.0.
>>
>> Vinod, if you get a chance to take a look at the test case and see if
>> there is anything I could have done to make it cleaner, I would be
>> much obliged.
>>
>> J
--
Director of Data Science
Cloudera
Twitter: @josh_wills
Re: getting crunch running on hadoop 2.0.0
Posted by Gabriel Reid <ga...@gmail.com>.
Something somewhat related: yesterday I had an issue when running the unit tests (or a subset of the tests) from within Eclipse. I noticed that if the WordCountHbaseTest was run, a lot of tests after it would fail. Not running the WordCountHbaseTest (ie setting @Ignore on it) seemed to resolve the problem, an all tests would run through fine.
This was only an issue when running in Eclipse; running the tests from maven seemed to work fine.
Any ideas on this?
On 27 Jun 2012, at 06:26, Josh Wills <jw...@apache.org> wrote:
> Hey all,
>
> That last commit from me "fixes" (in some loose sense of the word) the
> WordCountHBaseTest, which was the last test that we needed to get
> working in order to have all of the Crunch unit tests passing under
> Hadoop-2.0.0.
>
> The complexity stems from the fact that the WordCountHBaseTest starts
> up a mini hbase/mapreduce cluster in order to run, and I had to do
> some work to make sure that all of the proper dependencies and class
> definitions that were needed for the test case to pass were available
> to the mini YARN instance that runs w/HBaseTestingUtility under hadoop
> 2.0.0.
>
> Vinod, if you get a chance to take a look at the test case and see if
> there is anything I could have done to make it cleaner, I would be
> much obliged.
>
> J
Re: getting crunch running on hadoop 2.0.0
Posted by Josh Wills <jw...@cloudera.com>.
Oh, and I wanted to mention: getting all of the unit tests to pass
under hadoop 2.0.0 was one of the blockers for getting Crunch added to
bigtop: https://issues.apache.org/jira/browse/BIGTOP-612
Now it's out of the way.
On Tue, Jun 26, 2012 at 9:26 PM, Josh Wills <jw...@apache.org> wrote:
> Hey all,
>
> That last commit from me "fixes" (in some loose sense of the word) the
> WordCountHBaseTest, which was the last test that we needed to get
> working in order to have all of the Crunch unit tests passing under
> Hadoop-2.0.0.
>
> The complexity stems from the fact that the WordCountHBaseTest starts
> up a mini hbase/mapreduce cluster in order to run, and I had to do
> some work to make sure that all of the proper dependencies and class
> definitions that were needed for the test case to pass were available
> to the mini YARN instance that runs w/HBaseTestingUtility under hadoop
> 2.0.0.
>
> Vinod, if you get a chance to take a look at the test case and see if
> there is anything I could have done to make it cleaner, I would be
> much obliged.
>
> J
--
Director of Data Science
Cloudera
Twitter: @josh_wills