You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Johnny Zhang <xi...@cloudera.com> on 2012/11/14 03:37:55 UTC
Re: Pig UT last nearly 8 hours and TestEvalPipeline2 lasts for 37 minutes
Hi, lulynn:
Yes, whole Pig unit tests run about 8 hours.
TestEvalPipline runs about 26 mins and TestEvalPiplineLocal runs about 3
mins.
Hope it is helpful,
Johnny
On Tue, Nov 13, 2012 at 6:28 PM, lulynn_2008 <lu...@163.com> wrote:
> Hi all,
>
> The whole pig UT last for nearly 8 hours, and TestEvalPipeline2 last for
> 37 minutes.
>
> My questions are:
> how long pig UT will last in normal?
> Do we have jenkins for pig UT? If yes, please attach the link. Thanks
>
> Thanks
>
>
Re: Re: Pig UT last nearly 8 hours and TestEvalPipeline2 lasts for 37 minutes
Posted by Jonathan Coveney <jc...@gmail.com>.
There are a couple of ways to shorten the time... one (super helpful one)
would be to look at tests using the MiniCluster, and convert them to use
local mode. A lot of tests are run using a full MR job when they aren't
testing a piece of Pig relevant to that interop.
Another way is to split up the tests and run them on a different CI worker.
Obviously this doesn't help if you're running it locally.
2012/11/14 Johnny Zhang <xi...@cloudera.com>
> Hi, lulynn_2008:
> I am not aware of how to shorten the time.
>
> Johnny
>
> On Tue, Nov 13, 2012 at 7:27 PM, lulynn_2008 <lu...@163.com> wrote:
>
> > Thanks.
> > Then my environment is normal.
> > Is there any way to shorten the time? I think maybe we can find a way to
> > shorten the time.
> >
> >
> >
> >
> >
> >
> >
> >
> > At 2012-11-14 10:37:55,"Johnny Zhang" <xi...@cloudera.com> wrote:
> > >Hi, lulynn:
> > >Yes, whole Pig unit tests run about 8 hours.
> > >TestEvalPipline runs about 26 mins and TestEvalPiplineLocal runs about 3
> > >mins.
> > >
> > >Hope it is helpful,
> > >Johnny
> > >
> > >
> > >
> > >On Tue, Nov 13, 2012 at 6:28 PM, lulynn_2008 <lu...@163.com>
> wrote:
> > >
> > >> Hi all,
> > >>
> > >> The whole pig UT last for nearly 8 hours, and TestEvalPipeline2 last
> for
> > >> 37 minutes.
> > >>
> > >> My questions are:
> > >> how long pig UT will last in normal?
> > >> Do we have jenkins for pig UT? If yes, please attach the link. Thanks
> > >>
> > >> Thanks
> > >>
> > >>
> >
>
Re: Re: Re: Pig UT last nearly 8 hours and TestEvalPipeline2 lasts
for 37 minutes
Posted by Vitalii Tymchyshyn <ti...@gmail.com>.
Well, you could try multi JVM parallelization. AFAIR maven has the option.
Re: Re: Re: Pig UT last nearly 8 hours and TestEvalPipeline2 lasts
for 37 minutes
Posted by Jonathan Coveney <jc...@gmail.com>.
Pig is very much not thread safe. It uses static methods to add stuff to
contexts all over the place. It would be a ton of work to fix this.
2012/11/20 Cheolsoo Park <ch...@cloudera.com>
> Hi,
>
> I actually tried to run entire unit test suite in multiple threads, and I
> used this junit extension:
> http://tempusfugitlibrary.org/documentation/junit/parallel/
>
> This library is nice because all I had to do is to add @RunWith to each
> test suite, and it seems to work.
>
> However, I ran into several problems as follows:
>
> 1) Pig is not entirely thread-safe as far as I can tell. Yes, some work
> done was done in the past to make Pig thread safe, but that's limited to
> the front-end. For example,
> https://issues.apache.org/jira/browse/PIG-1874
>
> But to run unit tests in multiple threads, we need to ensure that both
> front-end and back-end are thread safe. But they're apparently not. For
> example, here a reported race condition in local mode:
>
> http://search-hadoop.com/m/2OdLNRMwXa2/Intermittent+NullPointerException&subj=Intermittent+NullPointerException
>
> 2) Many unit test cases are not written to run in parallel. For example,
> each test case changes the cwd of MiniCluster in TestLoad. There are also
> many other test cases where read/write files to the same location.
>
> 3) Parallelizing test cases doesn't always reduce the execution time. In
> our test suites, the distribution of test cases is not uniform. Some test
> suites contain many test cases while some has only one. What I found is
> that test suites with many test cases actually run slower than running
> sequentially due to context switching overhead. Unfortunately, tempus-fugit
> doesn't provide fine-grained control over the number of threads. It blindly
> runs every test case in separate threads.
>
> Hope this is useful.
>
> Thanks,
> Cheolsoo
>
> On Mon, Nov 19, 2012 at 10:57 PM, lulynn_2008 <lu...@163.com> wrote:
>
> > Maybe we can run some UT paralleled.
> >
> >
> >
> >
> >
> >
> >
> >
> > At 2012-11-15 03:12:27,"Johnny Zhang" <xi...@cloudera.com> wrote:
> > >Hi, lulynn_2008:
> > >I am not aware of how to shorten the time.
> > >
> > >Johnny
> > >
> > >On Tue, Nov 13, 2012 at 7:27 PM, lulynn_2008 <lu...@163.com>
> wrote:
> > >
> > >> Thanks.
> > >> Then my environment is normal.
> > >> Is there any way to shorten the time? I think maybe we can find a way
> to
> > >> shorten the time.
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >> At 2012-11-14 10:37:55,"Johnny Zhang" <xi...@cloudera.com> wrote:
> > >> >Hi, lulynn:
> > >> >Yes, whole Pig unit tests run about 8 hours.
> > >> >TestEvalPipline runs about 26 mins and TestEvalPiplineLocal runs
> about
> > 3
> > >> >mins.
> > >> >
> > >> >Hope it is helpful,
> > >> >Johnny
> > >> >
> > >> >
> > >> >
> > >> >On Tue, Nov 13, 2012 at 6:28 PM, lulynn_2008 <lu...@163.com>
> > wrote:
> > >> >
> > >> >> Hi all,
> > >> >>
> > >> >> The whole pig UT last for nearly 8 hours, and TestEvalPipeline2
> last
> > for
> > >> >> 37 minutes.
> > >> >>
> > >> >> My questions are:
> > >> >> how long pig UT will last in normal?
> > >> >> Do we have jenkins for pig UT? If yes, please attach the link.
> Thanks
> > >> >>
> > >> >> Thanks
> > >> >>
> > >> >>
> > >>
> >
>
Re: Re: Re: Pig UT last nearly 8 hours and TestEvalPipeline2 lasts
for 37 minutes
Posted by Cheolsoo Park <ch...@cloudera.com>.
Hi,
I actually tried to run entire unit test suite in multiple threads, and I
used this junit extension:
http://tempusfugitlibrary.org/documentation/junit/parallel/
This library is nice because all I had to do is to add @RunWith to each
test suite, and it seems to work.
However, I ran into several problems as follows:
1) Pig is not entirely thread-safe as far as I can tell. Yes, some work
done was done in the past to make Pig thread safe, but that's limited to
the front-end. For example,
https://issues.apache.org/jira/browse/PIG-1874
But to run unit tests in multiple threads, we need to ensure that both
front-end and back-end are thread safe. But they're apparently not. For
example, here a reported race condition in local mode:
http://search-hadoop.com/m/2OdLNRMwXa2/Intermittent+NullPointerException&subj=Intermittent+NullPointerException
2) Many unit test cases are not written to run in parallel. For example,
each test case changes the cwd of MiniCluster in TestLoad. There are also
many other test cases where read/write files to the same location.
3) Parallelizing test cases doesn't always reduce the execution time. In
our test suites, the distribution of test cases is not uniform. Some test
suites contain many test cases while some has only one. What I found is
that test suites with many test cases actually run slower than running
sequentially due to context switching overhead. Unfortunately, tempus-fugit
doesn't provide fine-grained control over the number of threads. It blindly
runs every test case in separate threads.
Hope this is useful.
Thanks,
Cheolsoo
On Mon, Nov 19, 2012 at 10:57 PM, lulynn_2008 <lu...@163.com> wrote:
> Maybe we can run some UT paralleled.
>
>
>
>
>
>
>
>
> At 2012-11-15 03:12:27,"Johnny Zhang" <xi...@cloudera.com> wrote:
> >Hi, lulynn_2008:
> >I am not aware of how to shorten the time.
> >
> >Johnny
> >
> >On Tue, Nov 13, 2012 at 7:27 PM, lulynn_2008 <lu...@163.com> wrote:
> >
> >> Thanks.
> >> Then my environment is normal.
> >> Is there any way to shorten the time? I think maybe we can find a way to
> >> shorten the time.
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> At 2012-11-14 10:37:55,"Johnny Zhang" <xi...@cloudera.com> wrote:
> >> >Hi, lulynn:
> >> >Yes, whole Pig unit tests run about 8 hours.
> >> >TestEvalPipline runs about 26 mins and TestEvalPiplineLocal runs about
> 3
> >> >mins.
> >> >
> >> >Hope it is helpful,
> >> >Johnny
> >> >
> >> >
> >> >
> >> >On Tue, Nov 13, 2012 at 6:28 PM, lulynn_2008 <lu...@163.com>
> wrote:
> >> >
> >> >> Hi all,
> >> >>
> >> >> The whole pig UT last for nearly 8 hours, and TestEvalPipeline2 last
> for
> >> >> 37 minutes.
> >> >>
> >> >> My questions are:
> >> >> how long pig UT will last in normal?
> >> >> Do we have jenkins for pig UT? If yes, please attach the link. Thanks
> >> >>
> >> >> Thanks
> >> >>
> >> >>
> >>
>
Re:Re: Re: Pig UT last nearly 8 hours and TestEvalPipeline2 lasts
for 37 minutes
Posted by lulynn_2008 <lu...@163.com>.
Maybe we can run some UT paralleled.
At 2012-11-15 03:12:27,"Johnny Zhang" <xi...@cloudera.com> wrote:
>Hi, lulynn_2008:
>I am not aware of how to shorten the time.
>
>Johnny
>
>On Tue, Nov 13, 2012 at 7:27 PM, lulynn_2008 <lu...@163.com> wrote:
>
>> Thanks.
>> Then my environment is normal.
>> Is there any way to shorten the time? I think maybe we can find a way to
>> shorten the time.
>>
>>
>>
>>
>>
>>
>>
>>
>> At 2012-11-14 10:37:55,"Johnny Zhang" <xi...@cloudera.com> wrote:
>> >Hi, lulynn:
>> >Yes, whole Pig unit tests run about 8 hours.
>> >TestEvalPipline runs about 26 mins and TestEvalPiplineLocal runs about 3
>> >mins.
>> >
>> >Hope it is helpful,
>> >Johnny
>> >
>> >
>> >
>> >On Tue, Nov 13, 2012 at 6:28 PM, lulynn_2008 <lu...@163.com> wrote:
>> >
>> >> Hi all,
>> >>
>> >> The whole pig UT last for nearly 8 hours, and TestEvalPipeline2 last for
>> >> 37 minutes.
>> >>
>> >> My questions are:
>> >> how long pig UT will last in normal?
>> >> Do we have jenkins for pig UT? If yes, please attach the link. Thanks
>> >>
>> >> Thanks
>> >>
>> >>
>>
Re: Re: Pig UT last nearly 8 hours and TestEvalPipeline2 lasts for 37 minutes
Posted by Johnny Zhang <xi...@cloudera.com>.
Hi, lulynn_2008:
I am not aware of how to shorten the time.
Johnny
On Tue, Nov 13, 2012 at 7:27 PM, lulynn_2008 <lu...@163.com> wrote:
> Thanks.
> Then my environment is normal.
> Is there any way to shorten the time? I think maybe we can find a way to
> shorten the time.
>
>
>
>
>
>
>
>
> At 2012-11-14 10:37:55,"Johnny Zhang" <xi...@cloudera.com> wrote:
> >Hi, lulynn:
> >Yes, whole Pig unit tests run about 8 hours.
> >TestEvalPipline runs about 26 mins and TestEvalPiplineLocal runs about 3
> >mins.
> >
> >Hope it is helpful,
> >Johnny
> >
> >
> >
> >On Tue, Nov 13, 2012 at 6:28 PM, lulynn_2008 <lu...@163.com> wrote:
> >
> >> Hi all,
> >>
> >> The whole pig UT last for nearly 8 hours, and TestEvalPipeline2 last for
> >> 37 minutes.
> >>
> >> My questions are:
> >> how long pig UT will last in normal?
> >> Do we have jenkins for pig UT? If yes, please attach the link. Thanks
> >>
> >> Thanks
> >>
> >>
>
Re:Re: Pig UT last nearly 8 hours and TestEvalPipeline2 lasts for
37 minutes
Posted by lulynn_2008 <lu...@163.com>.
Thanks.
Then my environment is normal.
Is there any way to shorten the time? I think maybe we can find a way to shorten the time.
At 2012-11-14 10:37:55,"Johnny Zhang" <xi...@cloudera.com> wrote:
>Hi, lulynn:
>Yes, whole Pig unit tests run about 8 hours.
>TestEvalPipline runs about 26 mins and TestEvalPiplineLocal runs about 3
>mins.
>
>Hope it is helpful,
>Johnny
>
>
>
>On Tue, Nov 13, 2012 at 6:28 PM, lulynn_2008 <lu...@163.com> wrote:
>
>> Hi all,
>>
>> The whole pig UT last for nearly 8 hours, and TestEvalPipeline2 last for
>> 37 minutes.
>>
>> My questions are:
>> how long pig UT will last in normal?
>> Do we have jenkins for pig UT? If yes, please attach the link. Thanks
>>
>> Thanks
>>
>>