You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Prasanth Jayachandran <pr...@apache.org> on 2015/09/29 01:23:58 UTC
Splitting tests in PTest2
Hello everyone
Regarding HIVE-11642, the tests for MiniLlapCliDriver are not split across multiple nodes. I can see from the logs that MiniTezCliDriver are broken down into many tests and ran on different machines. http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5447/test-results/ <http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5447/test-results/>
How can we do the same for MiniLlapCliDriver tests? Do we need to do something in PTest2 to enable splitting of tests?
MiniLlapCliDriver runs the same tests are MiniTez but in llap mode.
Thanks
Prasanth
Re: Splitting tests in PTest2
Posted by Prasanth Jayachandran <pj...@hortonworks.com>.
Szehon,
I committed a different patch now to llap branch, HIVE-11923. This will let llap and tez test to run much faster hopefully before the 2h timeout. Once all tests passes, committed then we can split up the tests to run in parallel. Even after HIVE-11923 if the patch didn’t complete before timeout then I will commit the patch and provide you an update to split up the tests.
Thanks
Prasanth
> On Sep 28, 2015, at 7:15 PM, Szehon Ho <sz...@cloudera.com> wrote:
>
> OK so there is a bit of a chicken-egg problem, I cannot change it until
> 11642 is in, otherwise it will break the other builds.
>
> Once its in I can update the properties file to reference the new
> LlapCliDriver. Maybe you can check it in and I will try enabling it
> tonight.
>
> On Mon, Sep 28, 2015 at 5:05 PM, Szehon Ho <sz...@cloudera.com> wrote:
>
>> OK it may be possible, I will take a look!
>>
>> Thanks
>> Szehon
>>
>> On Mon, Sep 28, 2015 at 4:56 PM, Prasanth Jayachandran <
>> pjayachandran@hortonworks.com> wrote:
>>
>>> @Szehon.. MiniLlapCliDriver uses the same set of files from minitez. Can
>>> that be reused as such without duplicating?
>>>
>>>
>>>> On Sep 28, 2015, at 6:49 PM, Prasanth Jayachandran <
>>> pjayachandran@hortonworks.com> wrote:
>>>>
>>>> Thanks Szehon! I will look at it and update you once done.
>>>>
>>>>> On Sep 28, 2015, at 6:37 PM, Szehon Ho <sz...@cloudera.com> wrote:
>>>>>
>>>>> Yes thanks for looking into it. Can you edit
>>> testconfiguration.properties
>>>>> to add a section for LlapCliDriver tests, see HIVE-7254 for details.
>>> That
>>>>> way the framework can identify which tests it needs to run and then
>>> split
>>>>> it into parallel batches across test nodes. Otherwise it will just
>>> run all
>>>>> of the LlapCliDriver tests using -Dtest=TestMiniLlapCliDriver in one
>>> huge
>>>>> batch.
>>>>>
>>>>> Once you finish, let me know and I'll tweak the test properties to set
>>> a
>>>>> batch size for that one.
>>>>>
>>>>> Thanks
>>>>> Szehon
>>>>>
>>>>> On Mon, Sep 28, 2015 at 4:23 PM, Prasanth Jayachandran <
>>> prasanthj@apache.org
>>>>>> wrote:
>>>>>
>>>>>> Hello everyone
>>>>>>
>>>>>> Regarding HIVE-11642, the tests for MiniLlapCliDriver are not split
>>> across
>>>>>> multiple nodes. I can see from the logs that MiniTezCliDriver are
>>> broken
>>>>>> down into many tests and ran on different machines.
>>>>>>
>>> http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5447/test-results/
>>>>>> <
>>>>>>
>>> http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5447/test-results/
>>>>>>>
>>>>>>
>>>>>> How can we do the same for MiniLlapCliDriver tests? Do we need to do
>>>>>> something in PTest2 to enable splitting of tests?
>>>>>> MiniLlapCliDriver runs the same tests are MiniTez but in llap mode.
>>>>>>
>>>>>> Thanks
>>>>>> Prasanth
>>>>
>>>
>>>
>>
Re: Splitting tests in PTest2
Posted by Szehon Ho <sz...@cloudera.com>.
OK so there is a bit of a chicken-egg problem, I cannot change it until
11642 is in, otherwise it will break the other builds.
Once its in I can update the properties file to reference the new
LlapCliDriver. Maybe you can check it in and I will try enabling it
tonight.
On Mon, Sep 28, 2015 at 5:05 PM, Szehon Ho <sz...@cloudera.com> wrote:
> OK it may be possible, I will take a look!
>
> Thanks
> Szehon
>
> On Mon, Sep 28, 2015 at 4:56 PM, Prasanth Jayachandran <
> pjayachandran@hortonworks.com> wrote:
>
>> @Szehon.. MiniLlapCliDriver uses the same set of files from minitez. Can
>> that be reused as such without duplicating?
>>
>>
>> > On Sep 28, 2015, at 6:49 PM, Prasanth Jayachandran <
>> pjayachandran@hortonworks.com> wrote:
>> >
>> > Thanks Szehon! I will look at it and update you once done.
>> >
>> >> On Sep 28, 2015, at 6:37 PM, Szehon Ho <sz...@cloudera.com> wrote:
>> >>
>> >> Yes thanks for looking into it. Can you edit
>> testconfiguration.properties
>> >> to add a section for LlapCliDriver tests, see HIVE-7254 for details.
>> That
>> >> way the framework can identify which tests it needs to run and then
>> split
>> >> it into parallel batches across test nodes. Otherwise it will just
>> run all
>> >> of the LlapCliDriver tests using -Dtest=TestMiniLlapCliDriver in one
>> huge
>> >> batch.
>> >>
>> >> Once you finish, let me know and I'll tweak the test properties to set
>> a
>> >> batch size for that one.
>> >>
>> >> Thanks
>> >> Szehon
>> >>
>> >> On Mon, Sep 28, 2015 at 4:23 PM, Prasanth Jayachandran <
>> prasanthj@apache.org
>> >>> wrote:
>> >>
>> >>> Hello everyone
>> >>>
>> >>> Regarding HIVE-11642, the tests for MiniLlapCliDriver are not split
>> across
>> >>> multiple nodes. I can see from the logs that MiniTezCliDriver are
>> broken
>> >>> down into many tests and ran on different machines.
>> >>>
>> http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5447/test-results/
>> >>> <
>> >>>
>> http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5447/test-results/
>> >>>>
>> >>>
>> >>> How can we do the same for MiniLlapCliDriver tests? Do we need to do
>> >>> something in PTest2 to enable splitting of tests?
>> >>> MiniLlapCliDriver runs the same tests are MiniTez but in llap mode.
>> >>>
>> >>> Thanks
>> >>> Prasanth
>> >
>>
>>
>
Re: Splitting tests in PTest2
Posted by Szehon Ho <sz...@cloudera.com>.
OK it may be possible, I will take a look!
Thanks
Szehon
On Mon, Sep 28, 2015 at 4:56 PM, Prasanth Jayachandran <
pjayachandran@hortonworks.com> wrote:
> @Szehon.. MiniLlapCliDriver uses the same set of files from minitez. Can
> that be reused as such without duplicating?
>
>
> > On Sep 28, 2015, at 6:49 PM, Prasanth Jayachandran <
> pjayachandran@hortonworks.com> wrote:
> >
> > Thanks Szehon! I will look at it and update you once done.
> >
> >> On Sep 28, 2015, at 6:37 PM, Szehon Ho <sz...@cloudera.com> wrote:
> >>
> >> Yes thanks for looking into it. Can you edit
> testconfiguration.properties
> >> to add a section for LlapCliDriver tests, see HIVE-7254 for details.
> That
> >> way the framework can identify which tests it needs to run and then
> split
> >> it into parallel batches across test nodes. Otherwise it will just run
> all
> >> of the LlapCliDriver tests using -Dtest=TestMiniLlapCliDriver in one
> huge
> >> batch.
> >>
> >> Once you finish, let me know and I'll tweak the test properties to set a
> >> batch size for that one.
> >>
> >> Thanks
> >> Szehon
> >>
> >> On Mon, Sep 28, 2015 at 4:23 PM, Prasanth Jayachandran <
> prasanthj@apache.org
> >>> wrote:
> >>
> >>> Hello everyone
> >>>
> >>> Regarding HIVE-11642, the tests for MiniLlapCliDriver are not split
> across
> >>> multiple nodes. I can see from the logs that MiniTezCliDriver are
> broken
> >>> down into many tests and ran on different machines.
> >>>
> http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5447/test-results/
> >>> <
> >>>
> http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5447/test-results/
> >>>>
> >>>
> >>> How can we do the same for MiniLlapCliDriver tests? Do we need to do
> >>> something in PTest2 to enable splitting of tests?
> >>> MiniLlapCliDriver runs the same tests are MiniTez but in llap mode.
> >>>
> >>> Thanks
> >>> Prasanth
> >
>
>
Re: Splitting tests in PTest2
Posted by Prasanth Jayachandran <pj...@hortonworks.com>.
@Szehon.. MiniLlapCliDriver uses the same set of files from minitez. Can that be reused as such without duplicating?
> On Sep 28, 2015, at 6:49 PM, Prasanth Jayachandran <pj...@hortonworks.com> wrote:
>
> Thanks Szehon! I will look at it and update you once done.
>
>> On Sep 28, 2015, at 6:37 PM, Szehon Ho <sz...@cloudera.com> wrote:
>>
>> Yes thanks for looking into it. Can you edit testconfiguration.properties
>> to add a section for LlapCliDriver tests, see HIVE-7254 for details. That
>> way the framework can identify which tests it needs to run and then split
>> it into parallel batches across test nodes. Otherwise it will just run all
>> of the LlapCliDriver tests using -Dtest=TestMiniLlapCliDriver in one huge
>> batch.
>>
>> Once you finish, let me know and I'll tweak the test properties to set a
>> batch size for that one.
>>
>> Thanks
>> Szehon
>>
>> On Mon, Sep 28, 2015 at 4:23 PM, Prasanth Jayachandran <prasanthj@apache.org
>>> wrote:
>>
>>> Hello everyone
>>>
>>> Regarding HIVE-11642, the tests for MiniLlapCliDriver are not split across
>>> multiple nodes. I can see from the logs that MiniTezCliDriver are broken
>>> down into many tests and ran on different machines.
>>> http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5447/test-results/
>>> <
>>> http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5447/test-results/
>>>>
>>>
>>> How can we do the same for MiniLlapCliDriver tests? Do we need to do
>>> something in PTest2 to enable splitting of tests?
>>> MiniLlapCliDriver runs the same tests are MiniTez but in llap mode.
>>>
>>> Thanks
>>> Prasanth
>
Re: Splitting tests in PTest2
Posted by Prasanth Jayachandran <pj...@hortonworks.com>.
Thanks Szehon! I will look at it and update you once done.
> On Sep 28, 2015, at 6:37 PM, Szehon Ho <sz...@cloudera.com> wrote:
>
> Yes thanks for looking into it. Can you edit testconfiguration.properties
> to add a section for LlapCliDriver tests, see HIVE-7254 for details. That
> way the framework can identify which tests it needs to run and then split
> it into parallel batches across test nodes. Otherwise it will just run all
> of the LlapCliDriver tests using -Dtest=TestMiniLlapCliDriver in one huge
> batch.
>
> Once you finish, let me know and I'll tweak the test properties to set a
> batch size for that one.
>
> Thanks
> Szehon
>
> On Mon, Sep 28, 2015 at 4:23 PM, Prasanth Jayachandran <prasanthj@apache.org
>> wrote:
>
>> Hello everyone
>>
>> Regarding HIVE-11642, the tests for MiniLlapCliDriver are not split across
>> multiple nodes. I can see from the logs that MiniTezCliDriver are broken
>> down into many tests and ran on different machines.
>> http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5447/test-results/
>> <
>> http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5447/test-results/
>>>
>>
>> How can we do the same for MiniLlapCliDriver tests? Do we need to do
>> something in PTest2 to enable splitting of tests?
>> MiniLlapCliDriver runs the same tests are MiniTez but in llap mode.
>>
>> Thanks
>> Prasanth
Re: Splitting tests in PTest2
Posted by Szehon Ho <sz...@cloudera.com>.
Yes thanks for looking into it. Can you edit testconfiguration.properties
to add a section for LlapCliDriver tests, see HIVE-7254 for details. That
way the framework can identify which tests it needs to run and then split
it into parallel batches across test nodes. Otherwise it will just run all
of the LlapCliDriver tests using -Dtest=TestMiniLlapCliDriver in one huge
batch.
Once you finish, let me know and I'll tweak the test properties to set a
batch size for that one.
Thanks
Szehon
On Mon, Sep 28, 2015 at 4:23 PM, Prasanth Jayachandran <prasanthj@apache.org
> wrote:
> Hello everyone
>
> Regarding HIVE-11642, the tests for MiniLlapCliDriver are not split across
> multiple nodes. I can see from the logs that MiniTezCliDriver are broken
> down into many tests and ran on different machines.
> http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5447/test-results/
> <
> http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5447/test-results/
> >
>
> How can we do the same for MiniLlapCliDriver tests? Do we need to do
> something in PTest2 to enable splitting of tests?
> MiniLlapCliDriver runs the same tests are MiniTez but in llap mode.
>
> Thanks
> Prasanth