You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Russell Jurney <ru...@gmail.com> on 2012/02/21 01:59:27 UTC
Pig unit tests minus Java
I need to write unit tests that start with raw data on HDFS and plumb all the way through to a web browser. Writing Java isn't desirable, so PigUnit isn't right.
Anyone have any ideas? I'd like it to be as simple as possible.
Russell Jurney http://datasyndrome.com
Re: Pig unit tests minus Java
Posted by John Meagher <jo...@gmail.com>.
You can use JUnit for system tests like that, but it ends up being a
mess. You would need a JUnit test that ran hadoop, ran any other
server pieces you needed, then you can use Selenium
http://seleniumhq.org/ for the browser side of the test.
On Tue, Feb 21, 2012 at 05:23, Dmitriy Ryaboy <dv...@gmail.com> wrote:
> That doesn't sound like a unit test. Unit tests test a unit of work -- a class, a function. You want to involve everything from hdfs to the browser... Sounds like some decomposition of the problem is in order.
>
> On Feb 20, 2012, at 4:59 PM, Russell Jurney <ru...@gmail.com> wrote:
>
>> I need to write unit tests that start with raw data on HDFS and plumb all the way through to a web browser. Writing Java isn't desirable, so PigUnit isn't right.
>>
>> Anyone have any ideas? I'd like it to be as simple as possible.
>>
>> Russell Jurney http://datasyndrome.com
Re: Pig unit tests minus Java
Posted by Dmitriy Ryaboy <dv...@gmail.com>.
That doesn't sound like a unit test. Unit tests test a unit of work -- a class, a function. You want to involve everything from hdfs to the browser... Sounds like some decomposition of the problem is in order.
On Feb 20, 2012, at 4:59 PM, Russell Jurney <ru...@gmail.com> wrote:
> I need to write unit tests that start with raw data on HDFS and plumb all the way through to a web browser. Writing Java isn't desirable, so PigUnit isn't right.
>
> Anyone have any ideas? I'd like it to be as simple as possible.
>
> Russell Jurney http://datasyndrome.com
Re: Pig unit tests minus Java
Posted by Russell Jurney <ru...@gmail.com>.
Thanks, this looks good.
Russell Jurney http://datasyndrome.com
On Feb 21, 2012, at 8:07 AM, Alan Gates <ga...@hortonworks.com> wrote:
> Have you looked at our e2e test harness? This might be more appropriate for what you're trying to do, though I don't know about the web browser part. Check out HowToTest on the wiki for info on the e2e harness.
>
> Alan.
>
> On Feb 20, 2012, at 4:59 PM, Russell Jurney wrote:
>
>> I need to write unit tests that start with raw data on HDFS and plumb all the way through to a web browser. Writing Java isn't desirable, so PigUnit isn't right.
>>
>> Anyone have any ideas? I'd like it to be as simple as possible.
>>
>> Russell Jurney http://datasyndrome.com
>
Re: Pig unit tests minus Java
Posted by Russell Jurney <ru...@gmail.com>.
Thanks, I'll check e2e out.
What I really want to do is:
Run a pig script that stores to MongoDB. wget it as json, verify it's sanity. I think I can run PigUnit in JRuby or Jython. I also want to do this continuously with Jenkins, any tips there?
Also, a question about PigUnit. The docs show the filenames loaded and store as 'input' and 'output.'. These match the variable names for string input/output for tests. Coincidence, or can I use any path I want, in unmodified scripts, and run PigUnit on them? Sorry if this is obvious from the examples, I'm on a plane without wifi and it's bugging me.
Aside: Who woulda thought that a cross-country Delta Boeing 767-300ER(G) wouldn't have wifi in lay flat seating up front?
Russell Jurney http://datasyndrome.com
On Feb 21, 2012, at 8:07 AM, Alan Gates <ga...@hortonworks.com> wrote:
> Have you looked at our e2e test harness? This might be more appropriate for what you're trying to do, though I don't know about the web browser part. Check out HowToTest on the wiki for info on the e2e harness.
>
> Alan.
>
> On Feb 20, 2012, at 4:59 PM, Russell Jurney wrote:
>
>> I need to write unit tests that start with raw data on HDFS and plumb all the way through to a web browser. Writing Java isn't desirable, so PigUnit isn't right.
>>
>> Anyone have any ideas? I'd like it to be as simple as possible.
>>
>> Russell Jurney http://datasyndrome.com
>
Re: Pig unit tests minus Java
Posted by Alan Gates <ga...@hortonworks.com>.
Have you looked at our e2e test harness? This might be more appropriate for what you're trying to do, though I don't know about the web browser part. Check out HowToTest on the wiki for info on the e2e harness.
Alan.
On Feb 20, 2012, at 4:59 PM, Russell Jurney wrote:
> I need to write unit tests that start with raw data on HDFS and plumb all the way through to a web browser. Writing Java isn't desirable, so PigUnit isn't right.
>
> Anyone have any ideas? I'd like it to be as simple as possible.
>
> Russell Jurney http://datasyndrome.com