You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@impala.apache.org by Tim Armstrong <ta...@cloudera.com> on 2019/08/05 06:27:27 UTC
Precommit test runtime
Hi All,
Just letting you know I tweaked the precommit testing job -
https://jenkins.impala.io/job/parallel-all-tests - so that it runs the
backend and frontend tests separately from the other tests.
If you have ideas for reducing build+test runtime, please share or
implement them. The frontend authorisation tests are running way too long,
but the long pole is still the other job with the Python tests. I looked at
a recent job (https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/6942/
<https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/6942/consoleFull>) and
here's the time taken per step:
0:00:20 Checked out repos
0:01:04 System setup done
0:00:27 Hadoop-lzo built
0:00:33 python bootstrap
0:00:47 download toolchain and cdh components
0:29:36 build impala
0:34:42 load data
0:45:08 serial tests
0:03:45 mini stress tests (all skipped or xfailed)
0:56:16 parallel tests
1:37:49 custom cluster tests
0:15:55 auth tests
0:01:48 archiving
- Tim
Re: Precommit test runtime
Posted by Tim Armstrong <ta...@cloudera.com>.
I have a fix for the second one and I move the permanent one to exhaustive
here: https://gerrit.cloudera.org/#/c/13967/.
I agree we don't need to test all those UDFs in a serial fashion - it's
still useful to make sure we can load a variety of hive UDFs but I think we
could just do that in test_udfs.py. Actually I think a lot of the permanent
UDFs don't need to be custom cluster tests in principle - if we always set
--local_library_dir for the test cluster and created the UDFs in a unique
database, I think we could do all of the tests in the parallel suite,
except for the ones that stop and start the cluster.
On Tue, Aug 6, 2019 at 6:31 AM Tamas Mate <tm...@cloudera.com> wrote:
> From the longest tests these two looks like an easy ~5-8 min win. What do
> you think?
>
> 1) test_permanent_udfs.py::test_java_udfs_hive_integration - runs for ~5
> minutes consistently, I think it is not necessary to execute it with this
> amount of UDFS
>
> https://github.com/apache/impala/blob/master/tests/custom_cluster/test_permanent_udfs.py#L187
>
> 2) test_breakpad.py::test_abort_writes_minidump - runs for 4 minutes most
> probably due to the CLUSTER_WAIT_TIMEOUT_IN_SECONDS, this could be set
> lower for this test
>
> https://github.com/apache/impala/blob/master/tests/custom_cluster/test_breakpad.py#L155
>
> https://github.com/apache/impala/blob/master/tests/common/impala_cluster.py#L67
>
> On Tue, Aug 6, 2019 at 2:10 PM Tamas Mate <tm...@cloudera.com> wrote:
>
> > The source file was corrupted, that is why the sentry curiosity came up.
> > Updated the Gist
> > <https://gist.github.com/tmater/df43cb35a25e7dda65aa24ed50f3fcbe> with
> > the new report.
> >
> > On Mon, Aug 5, 2019 at 8:35 PM Tamas Mate <tm...@cloudera.com> wrote:
> >
> >> No, possibly the mail server rejected it without notification. Uploaded
> >> as a secret Gist
> >> <https://gist.github.com/tmater/df43cb35a25e7dda65aa24ed50f3fcbe>.
> >>
> >> On Mon, Aug 5, 2019 at 7:44 PM Tim Armstrong <ta...@cloudera.com>
> >> wrote:
> >>
> >>> Did you forget to attach it?
> >>>
> >>> On Mon, Aug 5, 2019 at 10:32 AM Tamas Mate <tm...@cloudera.com> wrote:
> >>>
> >>> > Hi Tim, All,
> >>> >
> >>> > I made a quick log analysis on the linked job. Just blindly checked
> the
> >>> > differences between the consecutive lines to look for hotspots, so
> slow
> >>> > operations that are logging continuously remain hidden. There are
> some
> >>> > tests that running for minutes, will start checking them later. Also,
> >>> there
> >>> > is this curiosity, I haven't looked into it yet either:
> >>> > 21:15:09 --> Starting the Sentry Policy Server
> >>> > 21:21:45 Stopping Sentry
> >>> > 21:27:56 19/08/04 21:06:23 INFO testutil.SentryServicePinger: Sentry
> >>> > Service ping succeeded.
> >>> > 21:27:56 --> Starting Ranger Server
> >>> > 21:30:39 Restarting Apache Ranger Admin
> >>> > 21:30:39 Apache Ranger Admin Service is not running
> >>> > 21:34:16 Starting Apache Ranger Admin Service
> >>> > 21:34:16 Apache Ranger Admin Service with pid 41746 has started.
> >>> >
> >>> > Attached the report to this mail.
> >>> >
> >>> > Best regards,
> >>> > Tamas
> >>> >
> >>> > On Mon, Aug 5, 2019 at 8:28 AM Tim Armstrong <
> tarmstrong@cloudera.com>
> >>> > wrote:
> >>> >
> >>> >> Hi All,
> >>> >> Just letting you know I tweaked the precommit testing job -
> >>> >> https://jenkins.impala.io/job/parallel-all-tests - so that it runs
> >>> the
> >>> >> backend and frontend tests separately from the other tests.
> >>> >>
> >>> >> If you have ideas for reducing build+test runtime, please share or
> >>> >> implement them. The frontend authorisation tests are running way too
> >>> long,
> >>> >> but the long pole is still the other job with the Python tests. I
> >>> looked
> >>> >> at
> >>> >> a recent job (
> >>> >> https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/6942/
> >>> >> <
> >>>
> https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/6942/consoleFull
> >>> >)
> >>> >> and
> >>> >> here's the time taken per step:
> >>> >>
> >>> >> 0:00:20 Checked out repos
> >>> >> 0:01:04 System setup done
> >>> >> 0:00:27 Hadoop-lzo built
> >>> >> 0:00:33 python bootstrap
> >>> >> 0:00:47 download toolchain and cdh components
> >>> >> 0:29:36 build impala
> >>> >> 0:34:42 load data
> >>> >> 0:45:08 serial tests
> >>> >> 0:03:45 mini stress tests (all skipped or xfailed)
> >>> >> 0:56:16 parallel tests
> >>> >> 1:37:49 custom cluster tests
> >>> >> 0:15:55 auth tests
> >>> >> 0:01:48 archiving
> >>> >>
> >>> >> - Tim
> >>> >>
> >>> >
> >>> >
> >>>
> >>
> >>
> >> --
> >> Tamas Mate
> >> Senior Customer Operations Engineer
> >> Cloudera
> >>
> >
> >
> > --
> > Tamas Mate
> > Senior Customer Operations Engineer
> > Cloudera
> >
>
>
> --
> Tamas Mate
> Senior Customer Operations Engineer
> Cloudera
>
Re: Precommit test runtime
Posted by Tamas Mate <tm...@cloudera.com>.
From the longest tests these two looks like an easy ~5-8 min win. What do
you think?
1) test_permanent_udfs.py::test_java_udfs_hive_integration - runs for ~5
minutes consistently, I think it is not necessary to execute it with this
amount of UDFS
https://github.com/apache/impala/blob/master/tests/custom_cluster/test_permanent_udfs.py#L187
2) test_breakpad.py::test_abort_writes_minidump - runs for 4 minutes most
probably due to the CLUSTER_WAIT_TIMEOUT_IN_SECONDS, this could be set
lower for this test
https://github.com/apache/impala/blob/master/tests/custom_cluster/test_breakpad.py#L155
https://github.com/apache/impala/blob/master/tests/common/impala_cluster.py#L67
On Tue, Aug 6, 2019 at 2:10 PM Tamas Mate <tm...@cloudera.com> wrote:
> The source file was corrupted, that is why the sentry curiosity came up.
> Updated the Gist
> <https://gist.github.com/tmater/df43cb35a25e7dda65aa24ed50f3fcbe> with
> the new report.
>
> On Mon, Aug 5, 2019 at 8:35 PM Tamas Mate <tm...@cloudera.com> wrote:
>
>> No, possibly the mail server rejected it without notification. Uploaded
>> as a secret Gist
>> <https://gist.github.com/tmater/df43cb35a25e7dda65aa24ed50f3fcbe>.
>>
>> On Mon, Aug 5, 2019 at 7:44 PM Tim Armstrong <ta...@cloudera.com>
>> wrote:
>>
>>> Did you forget to attach it?
>>>
>>> On Mon, Aug 5, 2019 at 10:32 AM Tamas Mate <tm...@cloudera.com> wrote:
>>>
>>> > Hi Tim, All,
>>> >
>>> > I made a quick log analysis on the linked job. Just blindly checked the
>>> > differences between the consecutive lines to look for hotspots, so slow
>>> > operations that are logging continuously remain hidden. There are some
>>> > tests that running for minutes, will start checking them later. Also,
>>> there
>>> > is this curiosity, I haven't looked into it yet either:
>>> > 21:15:09 --> Starting the Sentry Policy Server
>>> > 21:21:45 Stopping Sentry
>>> > 21:27:56 19/08/04 21:06:23 INFO testutil.SentryServicePinger: Sentry
>>> > Service ping succeeded.
>>> > 21:27:56 --> Starting Ranger Server
>>> > 21:30:39 Restarting Apache Ranger Admin
>>> > 21:30:39 Apache Ranger Admin Service is not running
>>> > 21:34:16 Starting Apache Ranger Admin Service
>>> > 21:34:16 Apache Ranger Admin Service with pid 41746 has started.
>>> >
>>> > Attached the report to this mail.
>>> >
>>> > Best regards,
>>> > Tamas
>>> >
>>> > On Mon, Aug 5, 2019 at 8:28 AM Tim Armstrong <ta...@cloudera.com>
>>> > wrote:
>>> >
>>> >> Hi All,
>>> >> Just letting you know I tweaked the precommit testing job -
>>> >> https://jenkins.impala.io/job/parallel-all-tests - so that it runs
>>> the
>>> >> backend and frontend tests separately from the other tests.
>>> >>
>>> >> If you have ideas for reducing build+test runtime, please share or
>>> >> implement them. The frontend authorisation tests are running way too
>>> long,
>>> >> but the long pole is still the other job with the Python tests. I
>>> looked
>>> >> at
>>> >> a recent job (
>>> >> https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/6942/
>>> >> <
>>> https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/6942/consoleFull
>>> >)
>>> >> and
>>> >> here's the time taken per step:
>>> >>
>>> >> 0:00:20 Checked out repos
>>> >> 0:01:04 System setup done
>>> >> 0:00:27 Hadoop-lzo built
>>> >> 0:00:33 python bootstrap
>>> >> 0:00:47 download toolchain and cdh components
>>> >> 0:29:36 build impala
>>> >> 0:34:42 load data
>>> >> 0:45:08 serial tests
>>> >> 0:03:45 mini stress tests (all skipped or xfailed)
>>> >> 0:56:16 parallel tests
>>> >> 1:37:49 custom cluster tests
>>> >> 0:15:55 auth tests
>>> >> 0:01:48 archiving
>>> >>
>>> >> - Tim
>>> >>
>>> >
>>> >
>>>
>>
>>
>> --
>> Tamas Mate
>> Senior Customer Operations Engineer
>> Cloudera
>>
>
>
> --
> Tamas Mate
> Senior Customer Operations Engineer
> Cloudera
>
--
Tamas Mate
Senior Customer Operations Engineer
Cloudera
Re: Precommit test runtime
Posted by Tamas Mate <tm...@cloudera.com>.
The source file was corrupted, that is why the sentry curiosity came up.
Updated the Gist
<https://gist.github.com/tmater/df43cb35a25e7dda65aa24ed50f3fcbe> with the
new report.
On Mon, Aug 5, 2019 at 8:35 PM Tamas Mate <tm...@cloudera.com> wrote:
> No, possibly the mail server rejected it without notification. Uploaded as
> a secret Gist
> <https://gist.github.com/tmater/df43cb35a25e7dda65aa24ed50f3fcbe>.
>
> On Mon, Aug 5, 2019 at 7:44 PM Tim Armstrong <ta...@cloudera.com>
> wrote:
>
>> Did you forget to attach it?
>>
>> On Mon, Aug 5, 2019 at 10:32 AM Tamas Mate <tm...@cloudera.com> wrote:
>>
>> > Hi Tim, All,
>> >
>> > I made a quick log analysis on the linked job. Just blindly checked the
>> > differences between the consecutive lines to look for hotspots, so slow
>> > operations that are logging continuously remain hidden. There are some
>> > tests that running for minutes, will start checking them later. Also,
>> there
>> > is this curiosity, I haven't looked into it yet either:
>> > 21:15:09 --> Starting the Sentry Policy Server
>> > 21:21:45 Stopping Sentry
>> > 21:27:56 19/08/04 21:06:23 INFO testutil.SentryServicePinger: Sentry
>> > Service ping succeeded.
>> > 21:27:56 --> Starting Ranger Server
>> > 21:30:39 Restarting Apache Ranger Admin
>> > 21:30:39 Apache Ranger Admin Service is not running
>> > 21:34:16 Starting Apache Ranger Admin Service
>> > 21:34:16 Apache Ranger Admin Service with pid 41746 has started.
>> >
>> > Attached the report to this mail.
>> >
>> > Best regards,
>> > Tamas
>> >
>> > On Mon, Aug 5, 2019 at 8:28 AM Tim Armstrong <ta...@cloudera.com>
>> > wrote:
>> >
>> >> Hi All,
>> >> Just letting you know I tweaked the precommit testing job -
>> >> https://jenkins.impala.io/job/parallel-all-tests - so that it runs the
>> >> backend and frontend tests separately from the other tests.
>> >>
>> >> If you have ideas for reducing build+test runtime, please share or
>> >> implement them. The frontend authorisation tests are running way too
>> long,
>> >> but the long pole is still the other job with the Python tests. I
>> looked
>> >> at
>> >> a recent job (
>> >> https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/6942/
>> >> <
>> https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/6942/consoleFull
>> >)
>> >> and
>> >> here's the time taken per step:
>> >>
>> >> 0:00:20 Checked out repos
>> >> 0:01:04 System setup done
>> >> 0:00:27 Hadoop-lzo built
>> >> 0:00:33 python bootstrap
>> >> 0:00:47 download toolchain and cdh components
>> >> 0:29:36 build impala
>> >> 0:34:42 load data
>> >> 0:45:08 serial tests
>> >> 0:03:45 mini stress tests (all skipped or xfailed)
>> >> 0:56:16 parallel tests
>> >> 1:37:49 custom cluster tests
>> >> 0:15:55 auth tests
>> >> 0:01:48 archiving
>> >>
>> >> - Tim
>> >>
>> >
>> >
>>
>
>
> --
> Tamas Mate
> Senior Customer Operations Engineer
> Cloudera
>
--
Tamas Mate
Senior Customer Operations Engineer
Cloudera
Re: Precommit test runtime
Posted by Tamas Mate <tm...@cloudera.com>.
No, possibly the mail server rejected it without notification. Uploaded as
a secret Gist
<https://gist.github.com/tmater/df43cb35a25e7dda65aa24ed50f3fcbe>.
On Mon, Aug 5, 2019 at 7:44 PM Tim Armstrong <ta...@cloudera.com>
wrote:
> Did you forget to attach it?
>
> On Mon, Aug 5, 2019 at 10:32 AM Tamas Mate <tm...@cloudera.com> wrote:
>
> > Hi Tim, All,
> >
> > I made a quick log analysis on the linked job. Just blindly checked the
> > differences between the consecutive lines to look for hotspots, so slow
> > operations that are logging continuously remain hidden. There are some
> > tests that running for minutes, will start checking them later. Also,
> there
> > is this curiosity, I haven't looked into it yet either:
> > 21:15:09 --> Starting the Sentry Policy Server
> > 21:21:45 Stopping Sentry
> > 21:27:56 19/08/04 21:06:23 INFO testutil.SentryServicePinger: Sentry
> > Service ping succeeded.
> > 21:27:56 --> Starting Ranger Server
> > 21:30:39 Restarting Apache Ranger Admin
> > 21:30:39 Apache Ranger Admin Service is not running
> > 21:34:16 Starting Apache Ranger Admin Service
> > 21:34:16 Apache Ranger Admin Service with pid 41746 has started.
> >
> > Attached the report to this mail.
> >
> > Best regards,
> > Tamas
> >
> > On Mon, Aug 5, 2019 at 8:28 AM Tim Armstrong <ta...@cloudera.com>
> > wrote:
> >
> >> Hi All,
> >> Just letting you know I tweaked the precommit testing job -
> >> https://jenkins.impala.io/job/parallel-all-tests - so that it runs the
> >> backend and frontend tests separately from the other tests.
> >>
> >> If you have ideas for reducing build+test runtime, please share or
> >> implement them. The frontend authorisation tests are running way too
> long,
> >> but the long pole is still the other job with the Python tests. I looked
> >> at
> >> a recent job (
> >> https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/6942/
> >> <
> https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/6942/consoleFull>)
> >> and
> >> here's the time taken per step:
> >>
> >> 0:00:20 Checked out repos
> >> 0:01:04 System setup done
> >> 0:00:27 Hadoop-lzo built
> >> 0:00:33 python bootstrap
> >> 0:00:47 download toolchain and cdh components
> >> 0:29:36 build impala
> >> 0:34:42 load data
> >> 0:45:08 serial tests
> >> 0:03:45 mini stress tests (all skipped or xfailed)
> >> 0:56:16 parallel tests
> >> 1:37:49 custom cluster tests
> >> 0:15:55 auth tests
> >> 0:01:48 archiving
> >>
> >> - Tim
> >>
> >
> >
>
--
Tamas Mate
Senior Customer Operations Engineer
Cloudera
Re: Precommit test runtime
Posted by Tim Armstrong <ta...@cloudera.com>.
Did you forget to attach it?
On Mon, Aug 5, 2019 at 10:32 AM Tamas Mate <tm...@cloudera.com> wrote:
> Hi Tim, All,
>
> I made a quick log analysis on the linked job. Just blindly checked the
> differences between the consecutive lines to look for hotspots, so slow
> operations that are logging continuously remain hidden. There are some
> tests that running for minutes, will start checking them later. Also, there
> is this curiosity, I haven't looked into it yet either:
> 21:15:09 --> Starting the Sentry Policy Server
> 21:21:45 Stopping Sentry
> 21:27:56 19/08/04 21:06:23 INFO testutil.SentryServicePinger: Sentry
> Service ping succeeded.
> 21:27:56 --> Starting Ranger Server
> 21:30:39 Restarting Apache Ranger Admin
> 21:30:39 Apache Ranger Admin Service is not running
> 21:34:16 Starting Apache Ranger Admin Service
> 21:34:16 Apache Ranger Admin Service with pid 41746 has started.
>
> Attached the report to this mail.
>
> Best regards,
> Tamas
>
> On Mon, Aug 5, 2019 at 8:28 AM Tim Armstrong <ta...@cloudera.com>
> wrote:
>
>> Hi All,
>> Just letting you know I tweaked the precommit testing job -
>> https://jenkins.impala.io/job/parallel-all-tests - so that it runs the
>> backend and frontend tests separately from the other tests.
>>
>> If you have ideas for reducing build+test runtime, please share or
>> implement them. The frontend authorisation tests are running way too long,
>> but the long pole is still the other job with the Python tests. I looked
>> at
>> a recent job (
>> https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/6942/
>> <https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/6942/consoleFull>)
>> and
>> here's the time taken per step:
>>
>> 0:00:20 Checked out repos
>> 0:01:04 System setup done
>> 0:00:27 Hadoop-lzo built
>> 0:00:33 python bootstrap
>> 0:00:47 download toolchain and cdh components
>> 0:29:36 build impala
>> 0:34:42 load data
>> 0:45:08 serial tests
>> 0:03:45 mini stress tests (all skipped or xfailed)
>> 0:56:16 parallel tests
>> 1:37:49 custom cluster tests
>> 0:15:55 auth tests
>> 0:01:48 archiving
>>
>> - Tim
>>
>
>
Re: Precommit test runtime
Posted by Tamas Mate <tm...@cloudera.com>.
Hi Tim, All,
I made a quick log analysis on the linked job. Just blindly checked the
differences between the consecutive lines to look for hotspots, so slow
operations that are logging continuously remain hidden. There are some
tests that running for minutes, will start checking them later. Also, there
is this curiosity, I haven't looked into it yet either:
21:15:09 --> Starting the Sentry Policy Server
21:21:45 Stopping Sentry
21:27:56 19/08/04 21:06:23 INFO testutil.SentryServicePinger: Sentry
Service ping succeeded.
21:27:56 --> Starting Ranger Server
21:30:39 Restarting Apache Ranger Admin
21:30:39 Apache Ranger Admin Service is not running
21:34:16 Starting Apache Ranger Admin Service
21:34:16 Apache Ranger Admin Service with pid 41746 has started.
Attached the report to this mail.
Best regards,
Tamas
On Mon, Aug 5, 2019 at 8:28 AM Tim Armstrong <ta...@cloudera.com>
wrote:
> Hi All,
> Just letting you know I tweaked the precommit testing job -
> https://jenkins.impala.io/job/parallel-all-tests - so that it runs the
> backend and frontend tests separately from the other tests.
>
> If you have ideas for reducing build+test runtime, please share or
> implement them. The frontend authorisation tests are running way too long,
> but the long pole is still the other job with the Python tests. I looked at
> a recent job (
> https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/6942/
> <https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/6942/consoleFull>)
> and
> here's the time taken per step:
>
> 0:00:20 Checked out repos
> 0:01:04 System setup done
> 0:00:27 Hadoop-lzo built
> 0:00:33 python bootstrap
> 0:00:47 download toolchain and cdh components
> 0:29:36 build impala
> 0:34:42 load data
> 0:45:08 serial tests
> 0:03:45 mini stress tests (all skipped or xfailed)
> 0:56:16 parallel tests
> 1:37:49 custom cluster tests
> 0:15:55 auth tests
> 0:01:48 archiving
>
> - Tim
>