You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@impala.apache.org by Pooja Nilangekar <po...@cloudera.com> on 2018/10/11 17:15:16 UTC
Re: parallel-all-tests update
I agree with the approach of running the tests on supported platforms. In
the past couple of days, there have been multiple issues (IMPALA-7678
<https://issues.apache.org/jira/browse/IMPALA-7678> & IMPALA-7690
<https://issues.apache.org/jira/browse/IMPALA-7690>) due to python2.6
compatibility issues. Additionally, python2.6 might be one of the place
where multiple linux distributions differ. So it probably makes sense to
add another centos-from-scratch or similar, rather than adding ad-hoc tests
to address issues as they come up.
Thanks,
Pooja
On Thu, Sep 27, 2018 at 2:43 PM Philip Zeyliger <ph...@cloudera.com> wrote:
> I think the approach to catch that is to just run the tests on the relevant
> platform. (E.g., with test-with-docker or something to make the Jenkins
> setup less painful.) Looking through some recent commits, I found:
>
> 304e02cf6238a3c1e9537337f802bcf7a92df5db "{}".format(1)
> 9cfa228c2e itertools.count(start=1)
> cf4f314922f13fbf54c6d7300ceaa2229bf5916a shutil.make_archive(); Multiple
> with statements in the same context
> d91bc44021f28a5d113285be9c920bd1a646a9b9 {dict_comprehension}
> ??? a set comprehension; remember fixing it; couldn't find it.
>
> So, it's a mix of both classes, but I don't see recurring things that would
> be easy to catch.
>
> -- Philip
>
>
> On Thu, Sep 27, 2018 at 2:14 PM Tim Armstrong <ta...@cloudera.com>
> wrote:
>
> > Is it worth adding some regexes or similar to the gerrit bot to catch
> > itertools.count() usage? Or do we not expect repeated bugs of this form?
> >
> > On Thu, Sep 27, 2018 at 12:59 PM Tim Armstrong <ta...@cloudera.com>
> > wrote:
> >
> > > I also added it to the pre-review tests. Let me know if you see any
> > issues.
> > >
> > > On Thu, Sep 27, 2018 at 12:33 PM Philip Zeyliger <ph...@cloudera.com>
> > > wrote:
> > >
> > >> Hi folks,
> > >>
> > >> To address IMPALA-6543, there's a new test in parallel-all-tests that
> > >> makes
> > >> sure that any Python scripts use Python2.6-compatible syntax. Note
> that
> > >> this will catch the "try/catch/finally" style of bug, but not the
> > >> "itertools.count(start=1)" kind of bug (python2.7 changed the
> signature
> > of
> > >> itertools.count).
> > >>
> > >> The relevant diff was:
> > >>
> > >> $diff -u /tmp/b /tmp/a
> > >> --- /tmp/b 2018-09-27 12:29:15.000000000 -0700
> > >> +++ /tmp/a 2018-09-27 12:28:58.000000000 -0700
> > >> @@ -23,6 +23,14 @@
> > >> if (restring != null && !restring.equals("SUCCESS")) {
> > >> failed_job_urls.add(result.getAbsoluteUrl())
> > >> }
> > >> +}, Python26Compatibility: {
> > >> + result = build job: 'python26-incompatibility-check', propagate:
> > >> false, parameters:
> > >> + [string(name: 'IMPALA_REPO_URL', value: IMPALA_REPO_URL),
> > >> + string(name: 'IMPALA_REPO_BRANCH', value: IMPALA_REPO_BRANCH)]
> > >> + restring = result.getResult()
> > >> + if (restring != null && !restring.equals("SUCCESS")) {
> > >> + failed_job_urls.add(result.getAbsoluteUrl())
> > >> + }
> > >> }, TidyAndBuildOnlyAndRat: {
> > >> result = build job: 'clang-tidy-ub1604', propagate: false,
> > >> parameters:
> > >> [string(name: 'IMPALA_REPO_URL', value: IMPALA_REPO_URL),
> > >>
> > >> This will show up in your GVO builds. I've tested it, but of course
> let
> > me
> > >> know if you run into any trouble.
> > >>
> > >> Thanks!
> > >>
> > >
> >
>
Re: parallel-all-tests update
Posted by Jim Apple <jb...@cloudera.com>.
dev@impala sets the return address to itself, not the author of the post.
You are nowhere near the first person to be bitten by this, which is why I
am posting to dev@ and which I hope is some consolation.
On Thu, Oct 11, 2018 at 9:21 PM Philip Zeyliger <ph...@cloudera.com> wrote:
> -everyone
>
> I took advantage of your work on the packer template and wrote
>
> http://github.mtv.cloudera.com/philip/Impala/commit/4c9786eb6455a13f442b771e9296f204a692c2c2
> . I'll keep poking at it until it works. Right now it's getting as far as
> most of the C++ build. Annoyingly, it's now at the "slow" part.
>
>
> https://master-02.jenkins.cloudera.com/job/impala-test-with-docker-parameterized/123
> is building it; we'll see whether it worked when I wake up :) I'm using
> centos:6 as the base.
>
> -- Philip
>
>
>
> On Thu, Oct 11, 2018 at 1:15 PM Laszlo Gaal <la...@cloudera.com>
> wrote:
>
> > I'd be happy to pitch in; this is aligned with other test infra stuff I'm
> > working on.
> >
> > On Thu, Oct 11, 2018 at 7:34 PM Philip Zeyliger <ph...@cloudera.com>
> > wrote:
> >
> > > I'm working a little bit (in the background) on getting
> > > bin/bootstrap_system.sh to work. I'm somewhat optimistic that I can get
> > it
> > > to work :)
> > >
> > > I filed https://issues.apache.org/jira/browse/IMPALA-7698. Let me know
> > if
> > > you want to help dig in!
> > >
> > > -- Philip
> > >
> > > On Thu, Oct 11, 2018 at 10:16 AM Pooja Nilangekar <
> > > pooja.nilangekar@cloudera.com> wrote:
> > >
> > > > I agree with the approach of running the tests on supported
> platforms.
> > In
> > > > the past couple of days, there have been multiple issues (IMPALA-7678
> > > > <https://issues.apache.org/jira/browse/IMPALA-7678> & IMPALA-7690
> > > > <https://issues.apache.org/jira/browse/IMPALA-7690>) due to
> python2.6
> > > > compatibility issues. Additionally, python2.6 might be one of the
> place
> > > > where multiple linux distributions differ. So it probably makes sense
> > to
> > > > add another centos-from-scratch or similar, rather than adding ad-hoc
> > > tests
> > > > to address issues as they come up.
> > > >
> > > > Thanks,
> > > > Pooja
> > > >
> > > >
> > > > On Thu, Sep 27, 2018 at 2:43 PM Philip Zeyliger <philip@cloudera.com
> >
> > > > wrote:
> > > >
> > > > > I think the approach to catch that is to just run the tests on the
> > > > relevant
> > > > > platform. (E.g., with test-with-docker or something to make the
> > Jenkins
> > > > > setup less painful.) Looking through some recent commits, I found:
> > > > >
> > > > > 304e02cf6238a3c1e9537337f802bcf7a92df5db "{}".format(1)
> > > > > 9cfa228c2e itertools.count(start=1)
> > > > > cf4f314922f13fbf54c6d7300ceaa2229bf5916a shutil.make_archive();
> > > Multiple
> > > > > with statements in the same context
> > > > > d91bc44021f28a5d113285be9c920bd1a646a9b9 {dict_comprehension}
> > > > > ??? a set comprehension; remember fixing it; couldn't find it.
> > > > >
> > > > > So, it's a mix of both classes, but I don't see recurring things
> that
> > > > would
> > > > > be easy to catch.
> > > > >
> > > > > -- Philip
> > > > >
> > > > >
> > > > > On Thu, Sep 27, 2018 at 2:14 PM Tim Armstrong <
> > tarmstrong@cloudera.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > Is it worth adding some regexes or similar to the gerrit bot to
> > catch
> > > > > > itertools.count() usage? Or do we not expect repeated bugs of
> this
> > > > form?
> > > > > >
> > > > > > On Thu, Sep 27, 2018 at 12:59 PM Tim Armstrong <
> > > > tarmstrong@cloudera.com>
> > > > > > wrote:
> > > > > >
> > > > > > > I also added it to the pre-review tests. Let me know if you see
> > any
> > > > > > issues.
> > > > > > >
> > > > > > > On Thu, Sep 27, 2018 at 12:33 PM Philip Zeyliger <
> > > > philip@cloudera.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > >> Hi folks,
> > > > > > >>
> > > > > > >> To address IMPALA-6543, there's a new test in
> parallel-all-tests
> > > > that
> > > > > > >> makes
> > > > > > >> sure that any Python scripts use Python2.6-compatible syntax.
> > Note
> > > > > that
> > > > > > >> this will catch the "try/catch/finally" style of bug, but not
> > the
> > > > > > >> "itertools.count(start=1)" kind of bug (python2.7 changed the
> > > > > signature
> > > > > > of
> > > > > > >> itertools.count).
> > > > > > >>
> > > > > > >> The relevant diff was:
> > > > > > >>
> > > > > > >> $diff -u /tmp/b /tmp/a
> > > > > > >> --- /tmp/b 2018-09-27 12:29:15.000000000 -0700
> > > > > > >> +++ /tmp/a 2018-09-27 12:28:58.000000000 -0700
> > > > > > >> @@ -23,6 +23,14 @@
> > > > > > >> if (restring != null && !restring.equals("SUCCESS")) {
> > > > > > >> failed_job_urls.add(result.getAbsoluteUrl())
> > > > > > >> }
> > > > > > >> +}, Python26Compatibility: {
> > > > > > >> + result = build job: 'python26-incompatibility-check',
> > > > propagate:
> > > > > > >> false, parameters:
> > > > > > >> + [string(name: 'IMPALA_REPO_URL', value: IMPALA_REPO_URL),
> > > > > > >> + string(name: 'IMPALA_REPO_BRANCH', value:
> > > IMPALA_REPO_BRANCH)]
> > > > > > >> + restring = result.getResult()
> > > > > > >> + if (restring != null && !restring.equals("SUCCESS")) {
> > > > > > >> + failed_job_urls.add(result.getAbsoluteUrl())
> > > > > > >> + }
> > > > > > >> }, TidyAndBuildOnlyAndRat: {
> > > > > > >> result = build job: 'clang-tidy-ub1604', propagate:
> false,
> > > > > > >> parameters:
> > > > > > >> [string(name: 'IMPALA_REPO_URL', value: IMPALA_REPO_URL),
> > > > > > >>
> > > > > > >> This will show up in your GVO builds. I've tested it, but of
> > > course
> > > > > let
> > > > > > me
> > > > > > >> know if you run into any trouble.
> > > > > > >>
> > > > > > >> Thanks!
> > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: parallel-all-tests update
Posted by Laszlo Gaal <la...@cloudera.com>.
update: pg_hba.conf contains 'ident' only on CentOS (both 6 and 7); on
Ubuntu it's 'peer'.
On Fri, Oct 12, 2018 at 3:47 PM Laszlo Gaal <la...@cloudera.com>
wrote:
> It failed; apparently the setup script failed to change the pgsql access
> mode to "trust".
> I spun up a centos:6 container and looked around inside:
> - CentOS is 6.10
> - PostgreSQL is 8.4,
> - pg_hba.conf contains 'ident' for the default authN mode, so this
> location:
>
> http://github.mtv.cloudera.com/philip/Impala/commit/4c9786eb6455a13f442b771e9296f204a692c2c2#diff-26d1538cd6054d485c261fb103625e79R189
> should say
> sudo sed -ri 's/local +all +all +*ident*/local all all trust/g'
> $PG_AUTH_FILE
>
> As the packer template targeted CentOS 7, you may well find other
> locations where the tweaks will prove to be specific to that OS.
>
> On Fri, Oct 12, 2018 at 6:21 AM Philip Zeyliger <ph...@cloudera.com>
> wrote:
>
>> -everyone
>>
>> I took advantage of your work on the packer template and wrote
>>
>> http://github.mtv.cloudera.com/philip/Impala/commit/4c9786eb6455a13f442b771e9296f204a692c2c2
>> . I'll keep poking at it until it works. Right now it's getting as far as
>> most of the C++ build. Annoyingly, it's now at the "slow" part.
>>
>>
>> https://master-02.jenkins.cloudera.com/job/impala-test-with-docker-parameterized/123
>> is building it; we'll see whether it worked when I wake up :) I'm using
>> centos:6 as the base.
>>
>> -- Philip
>>
>>
>>
>> On Thu, Oct 11, 2018 at 1:15 PM Laszlo Gaal <la...@cloudera.com>
>> wrote:
>>
>> > I'd be happy to pitch in; this is aligned with other test infra stuff
>> I'm
>> > working on.
>> >
>> > On Thu, Oct 11, 2018 at 7:34 PM Philip Zeyliger <ph...@cloudera.com>
>> > wrote:
>> >
>> > > I'm working a little bit (in the background) on getting
>> > > bin/bootstrap_system.sh to work. I'm somewhat optimistic that I can
>> get
>> > it
>> > > to work :)
>> > >
>> > > I filed https://issues.apache.org/jira/browse/IMPALA-7698. Let me
>> know
>> > if
>> > > you want to help dig in!
>> > >
>> > > -- Philip
>> > >
>> > > On Thu, Oct 11, 2018 at 10:16 AM Pooja Nilangekar <
>> > > pooja.nilangekar@cloudera.com> wrote:
>> > >
>> > > > I agree with the approach of running the tests on supported
>> platforms.
>> > In
>> > > > the past couple of days, there have been multiple issues
>> (IMPALA-7678
>> > > > <https://issues.apache.org/jira/browse/IMPALA-7678> & IMPALA-7690
>> > > > <https://issues.apache.org/jira/browse/IMPALA-7690>) due to
>> python2.6
>> > > > compatibility issues. Additionally, python2.6 might be one of the
>> place
>> > > > where multiple linux distributions differ. So it probably makes
>> sense
>> > to
>> > > > add another centos-from-scratch or similar, rather than adding
>> ad-hoc
>> > > tests
>> > > > to address issues as they come up.
>> > > >
>> > > > Thanks,
>> > > > Pooja
>> > > >
>> > > >
>> > > > On Thu, Sep 27, 2018 at 2:43 PM Philip Zeyliger <
>> philip@cloudera.com>
>> > > > wrote:
>> > > >
>> > > > > I think the approach to catch that is to just run the tests on the
>> > > > relevant
>> > > > > platform. (E.g., with test-with-docker or something to make the
>> > Jenkins
>> > > > > setup less painful.) Looking through some recent commits, I found:
>> > > > >
>> > > > > 304e02cf6238a3c1e9537337f802bcf7a92df5db "{}".format(1)
>> > > > > 9cfa228c2e itertools.count(start=1)
>> > > > > cf4f314922f13fbf54c6d7300ceaa2229bf5916a shutil.make_archive();
>> > > Multiple
>> > > > > with statements in the same context
>> > > > > d91bc44021f28a5d113285be9c920bd1a646a9b9 {dict_comprehension}
>> > > > > ??? a set comprehension; remember fixing it; couldn't find it.
>> > > > >
>> > > > > So, it's a mix of both classes, but I don't see recurring things
>> that
>> > > > would
>> > > > > be easy to catch.
>> > > > >
>> > > > > -- Philip
>> > > > >
>> > > > >
>> > > > > On Thu, Sep 27, 2018 at 2:14 PM Tim Armstrong <
>> > tarmstrong@cloudera.com
>> > > >
>> > > > > wrote:
>> > > > >
>> > > > > > Is it worth adding some regexes or similar to the gerrit bot to
>> > catch
>> > > > > > itertools.count() usage? Or do we not expect repeated bugs of
>> this
>> > > > form?
>> > > > > >
>> > > > > > On Thu, Sep 27, 2018 at 12:59 PM Tim Armstrong <
>> > > > tarmstrong@cloudera.com>
>> > > > > > wrote:
>> > > > > >
>> > > > > > > I also added it to the pre-review tests. Let me know if you
>> see
>> > any
>> > > > > > issues.
>> > > > > > >
>> > > > > > > On Thu, Sep 27, 2018 at 12:33 PM Philip Zeyliger <
>> > > > philip@cloudera.com>
>> > > > > > > wrote:
>> > > > > > >
>> > > > > > >> Hi folks,
>> > > > > > >>
>> > > > > > >> To address IMPALA-6543, there's a new test in
>> parallel-all-tests
>> > > > that
>> > > > > > >> makes
>> > > > > > >> sure that any Python scripts use Python2.6-compatible syntax.
>> > Note
>> > > > > that
>> > > > > > >> this will catch the "try/catch/finally" style of bug, but not
>> > the
>> > > > > > >> "itertools.count(start=1)" kind of bug (python2.7 changed the
>> > > > > signature
>> > > > > > of
>> > > > > > >> itertools.count).
>> > > > > > >>
>> > > > > > >> The relevant diff was:
>> > > > > > >>
>> > > > > > >> $diff -u /tmp/b /tmp/a
>> > > > > > >> --- /tmp/b 2018-09-27 12:29:15.000000000 -0700
>> > > > > > >> +++ /tmp/a 2018-09-27 12:28:58.000000000 -0700
>> > > > > > >> @@ -23,6 +23,14 @@
>> > > > > > >> if (restring != null && !restring.equals("SUCCESS")) {
>> > > > > > >> failed_job_urls.add(result.getAbsoluteUrl())
>> > > > > > >> }
>> > > > > > >> +}, Python26Compatibility: {
>> > > > > > >> + result = build job: 'python26-incompatibility-check',
>> > > > propagate:
>> > > > > > >> false, parameters:
>> > > > > > >> + [string(name: 'IMPALA_REPO_URL', value:
>> IMPALA_REPO_URL),
>> > > > > > >> + string(name: 'IMPALA_REPO_BRANCH', value:
>> > > IMPALA_REPO_BRANCH)]
>> > > > > > >> + restring = result.getResult()
>> > > > > > >> + if (restring != null && !restring.equals("SUCCESS")) {
>> > > > > > >> + failed_job_urls.add(result.getAbsoluteUrl())
>> > > > > > >> + }
>> > > > > > >> }, TidyAndBuildOnlyAndRat: {
>> > > > > > >> result = build job: 'clang-tidy-ub1604', propagate:
>> false,
>> > > > > > >> parameters:
>> > > > > > >> [string(name: 'IMPALA_REPO_URL', value:
>> IMPALA_REPO_URL),
>> > > > > > >>
>> > > > > > >> This will show up in your GVO builds. I've tested it, but of
>> > > course
>> > > > > let
>> > > > > > me
>> > > > > > >> know if you run into any trouble.
>> > > > > > >>
>> > > > > > >> Thanks!
>> > > > > > >>
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
Re: parallel-all-tests update
Posted by Laszlo Gaal <la...@cloudera.com>.
It failed; apparently the setup script failed to change the pgsql access
mode to "trust".
I spun up a centos:6 container and looked around inside:
- CentOS is 6.10
- PostgreSQL is 8.4,
- pg_hba.conf contains 'ident' for the default authN mode, so this location:
http://github.mtv.cloudera.com/philip/Impala/commit/4c9786eb6455a13f442b771e9296f204a692c2c2#diff-26d1538cd6054d485c261fb103625e79R189
should say
sudo sed -ri 's/local +all +all +*ident*/local all all trust/g'
$PG_AUTH_FILE
As the packer template targeted CentOS 7, you may well find other locations
where the tweaks will prove to be specific to that OS.
On Fri, Oct 12, 2018 at 6:21 AM Philip Zeyliger <ph...@cloudera.com> wrote:
> -everyone
>
> I took advantage of your work on the packer template and wrote
>
> http://github.mtv.cloudera.com/philip/Impala/commit/4c9786eb6455a13f442b771e9296f204a692c2c2
> . I'll keep poking at it until it works. Right now it's getting as far as
> most of the C++ build. Annoyingly, it's now at the "slow" part.
>
>
> https://master-02.jenkins.cloudera.com/job/impala-test-with-docker-parameterized/123
> is building it; we'll see whether it worked when I wake up :) I'm using
> centos:6 as the base.
>
> -- Philip
>
>
>
> On Thu, Oct 11, 2018 at 1:15 PM Laszlo Gaal <la...@cloudera.com>
> wrote:
>
> > I'd be happy to pitch in; this is aligned with other test infra stuff I'm
> > working on.
> >
> > On Thu, Oct 11, 2018 at 7:34 PM Philip Zeyliger <ph...@cloudera.com>
> > wrote:
> >
> > > I'm working a little bit (in the background) on getting
> > > bin/bootstrap_system.sh to work. I'm somewhat optimistic that I can get
> > it
> > > to work :)
> > >
> > > I filed https://issues.apache.org/jira/browse/IMPALA-7698. Let me know
> > if
> > > you want to help dig in!
> > >
> > > -- Philip
> > >
> > > On Thu, Oct 11, 2018 at 10:16 AM Pooja Nilangekar <
> > > pooja.nilangekar@cloudera.com> wrote:
> > >
> > > > I agree with the approach of running the tests on supported
> platforms.
> > In
> > > > the past couple of days, there have been multiple issues (IMPALA-7678
> > > > <https://issues.apache.org/jira/browse/IMPALA-7678> & IMPALA-7690
> > > > <https://issues.apache.org/jira/browse/IMPALA-7690>) due to
> python2.6
> > > > compatibility issues. Additionally, python2.6 might be one of the
> place
> > > > where multiple linux distributions differ. So it probably makes sense
> > to
> > > > add another centos-from-scratch or similar, rather than adding ad-hoc
> > > tests
> > > > to address issues as they come up.
> > > >
> > > > Thanks,
> > > > Pooja
> > > >
> > > >
> > > > On Thu, Sep 27, 2018 at 2:43 PM Philip Zeyliger <philip@cloudera.com
> >
> > > > wrote:
> > > >
> > > > > I think the approach to catch that is to just run the tests on the
> > > > relevant
> > > > > platform. (E.g., with test-with-docker or something to make the
> > Jenkins
> > > > > setup less painful.) Looking through some recent commits, I found:
> > > > >
> > > > > 304e02cf6238a3c1e9537337f802bcf7a92df5db "{}".format(1)
> > > > > 9cfa228c2e itertools.count(start=1)
> > > > > cf4f314922f13fbf54c6d7300ceaa2229bf5916a shutil.make_archive();
> > > Multiple
> > > > > with statements in the same context
> > > > > d91bc44021f28a5d113285be9c920bd1a646a9b9 {dict_comprehension}
> > > > > ??? a set comprehension; remember fixing it; couldn't find it.
> > > > >
> > > > > So, it's a mix of both classes, but I don't see recurring things
> that
> > > > would
> > > > > be easy to catch.
> > > > >
> > > > > -- Philip
> > > > >
> > > > >
> > > > > On Thu, Sep 27, 2018 at 2:14 PM Tim Armstrong <
> > tarmstrong@cloudera.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > Is it worth adding some regexes or similar to the gerrit bot to
> > catch
> > > > > > itertools.count() usage? Or do we not expect repeated bugs of
> this
> > > > form?
> > > > > >
> > > > > > On Thu, Sep 27, 2018 at 12:59 PM Tim Armstrong <
> > > > tarmstrong@cloudera.com>
> > > > > > wrote:
> > > > > >
> > > > > > > I also added it to the pre-review tests. Let me know if you see
> > any
> > > > > > issues.
> > > > > > >
> > > > > > > On Thu, Sep 27, 2018 at 12:33 PM Philip Zeyliger <
> > > > philip@cloudera.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > >> Hi folks,
> > > > > > >>
> > > > > > >> To address IMPALA-6543, there's a new test in
> parallel-all-tests
> > > > that
> > > > > > >> makes
> > > > > > >> sure that any Python scripts use Python2.6-compatible syntax.
> > Note
> > > > > that
> > > > > > >> this will catch the "try/catch/finally" style of bug, but not
> > the
> > > > > > >> "itertools.count(start=1)" kind of bug (python2.7 changed the
> > > > > signature
> > > > > > of
> > > > > > >> itertools.count).
> > > > > > >>
> > > > > > >> The relevant diff was:
> > > > > > >>
> > > > > > >> $diff -u /tmp/b /tmp/a
> > > > > > >> --- /tmp/b 2018-09-27 12:29:15.000000000 -0700
> > > > > > >> +++ /tmp/a 2018-09-27 12:28:58.000000000 -0700
> > > > > > >> @@ -23,6 +23,14 @@
> > > > > > >> if (restring != null && !restring.equals("SUCCESS")) {
> > > > > > >> failed_job_urls.add(result.getAbsoluteUrl())
> > > > > > >> }
> > > > > > >> +}, Python26Compatibility: {
> > > > > > >> + result = build job: 'python26-incompatibility-check',
> > > > propagate:
> > > > > > >> false, parameters:
> > > > > > >> + [string(name: 'IMPALA_REPO_URL', value: IMPALA_REPO_URL),
> > > > > > >> + string(name: 'IMPALA_REPO_BRANCH', value:
> > > IMPALA_REPO_BRANCH)]
> > > > > > >> + restring = result.getResult()
> > > > > > >> + if (restring != null && !restring.equals("SUCCESS")) {
> > > > > > >> + failed_job_urls.add(result.getAbsoluteUrl())
> > > > > > >> + }
> > > > > > >> }, TidyAndBuildOnlyAndRat: {
> > > > > > >> result = build job: 'clang-tidy-ub1604', propagate:
> false,
> > > > > > >> parameters:
> > > > > > >> [string(name: 'IMPALA_REPO_URL', value: IMPALA_REPO_URL),
> > > > > > >>
> > > > > > >> This will show up in your GVO builds. I've tested it, but of
> > > course
> > > > > let
> > > > > > me
> > > > > > >> know if you run into any trouble.
> > > > > > >>
> > > > > > >> Thanks!
> > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: parallel-all-tests update
Posted by Philip Zeyliger <ph...@cloudera.com>.
-everyone
I took advantage of your work on the packer template and wrote
http://github.mtv.cloudera.com/philip/Impala/commit/4c9786eb6455a13f442b771e9296f204a692c2c2
. I'll keep poking at it until it works. Right now it's getting as far as
most of the C++ build. Annoyingly, it's now at the "slow" part.
https://master-02.jenkins.cloudera.com/job/impala-test-with-docker-parameterized/123
is building it; we'll see whether it worked when I wake up :) I'm using
centos:6 as the base.
-- Philip
On Thu, Oct 11, 2018 at 1:15 PM Laszlo Gaal <la...@cloudera.com>
wrote:
> I'd be happy to pitch in; this is aligned with other test infra stuff I'm
> working on.
>
> On Thu, Oct 11, 2018 at 7:34 PM Philip Zeyliger <ph...@cloudera.com>
> wrote:
>
> > I'm working a little bit (in the background) on getting
> > bin/bootstrap_system.sh to work. I'm somewhat optimistic that I can get
> it
> > to work :)
> >
> > I filed https://issues.apache.org/jira/browse/IMPALA-7698. Let me know
> if
> > you want to help dig in!
> >
> > -- Philip
> >
> > On Thu, Oct 11, 2018 at 10:16 AM Pooja Nilangekar <
> > pooja.nilangekar@cloudera.com> wrote:
> >
> > > I agree with the approach of running the tests on supported platforms.
> In
> > > the past couple of days, there have been multiple issues (IMPALA-7678
> > > <https://issues.apache.org/jira/browse/IMPALA-7678> & IMPALA-7690
> > > <https://issues.apache.org/jira/browse/IMPALA-7690>) due to python2.6
> > > compatibility issues. Additionally, python2.6 might be one of the place
> > > where multiple linux distributions differ. So it probably makes sense
> to
> > > add another centos-from-scratch or similar, rather than adding ad-hoc
> > tests
> > > to address issues as they come up.
> > >
> > > Thanks,
> > > Pooja
> > >
> > >
> > > On Thu, Sep 27, 2018 at 2:43 PM Philip Zeyliger <ph...@cloudera.com>
> > > wrote:
> > >
> > > > I think the approach to catch that is to just run the tests on the
> > > relevant
> > > > platform. (E.g., with test-with-docker or something to make the
> Jenkins
> > > > setup less painful.) Looking through some recent commits, I found:
> > > >
> > > > 304e02cf6238a3c1e9537337f802bcf7a92df5db "{}".format(1)
> > > > 9cfa228c2e itertools.count(start=1)
> > > > cf4f314922f13fbf54c6d7300ceaa2229bf5916a shutil.make_archive();
> > Multiple
> > > > with statements in the same context
> > > > d91bc44021f28a5d113285be9c920bd1a646a9b9 {dict_comprehension}
> > > > ??? a set comprehension; remember fixing it; couldn't find it.
> > > >
> > > > So, it's a mix of both classes, but I don't see recurring things that
> > > would
> > > > be easy to catch.
> > > >
> > > > -- Philip
> > > >
> > > >
> > > > On Thu, Sep 27, 2018 at 2:14 PM Tim Armstrong <
> tarmstrong@cloudera.com
> > >
> > > > wrote:
> > > >
> > > > > Is it worth adding some regexes or similar to the gerrit bot to
> catch
> > > > > itertools.count() usage? Or do we not expect repeated bugs of this
> > > form?
> > > > >
> > > > > On Thu, Sep 27, 2018 at 12:59 PM Tim Armstrong <
> > > tarmstrong@cloudera.com>
> > > > > wrote:
> > > > >
> > > > > > I also added it to the pre-review tests. Let me know if you see
> any
> > > > > issues.
> > > > > >
> > > > > > On Thu, Sep 27, 2018 at 12:33 PM Philip Zeyliger <
> > > philip@cloudera.com>
> > > > > > wrote:
> > > > > >
> > > > > >> Hi folks,
> > > > > >>
> > > > > >> To address IMPALA-6543, there's a new test in parallel-all-tests
> > > that
> > > > > >> makes
> > > > > >> sure that any Python scripts use Python2.6-compatible syntax.
> Note
> > > > that
> > > > > >> this will catch the "try/catch/finally" style of bug, but not
> the
> > > > > >> "itertools.count(start=1)" kind of bug (python2.7 changed the
> > > > signature
> > > > > of
> > > > > >> itertools.count).
> > > > > >>
> > > > > >> The relevant diff was:
> > > > > >>
> > > > > >> $diff -u /tmp/b /tmp/a
> > > > > >> --- /tmp/b 2018-09-27 12:29:15.000000000 -0700
> > > > > >> +++ /tmp/a 2018-09-27 12:28:58.000000000 -0700
> > > > > >> @@ -23,6 +23,14 @@
> > > > > >> if (restring != null && !restring.equals("SUCCESS")) {
> > > > > >> failed_job_urls.add(result.getAbsoluteUrl())
> > > > > >> }
> > > > > >> +}, Python26Compatibility: {
> > > > > >> + result = build job: 'python26-incompatibility-check',
> > > propagate:
> > > > > >> false, parameters:
> > > > > >> + [string(name: 'IMPALA_REPO_URL', value: IMPALA_REPO_URL),
> > > > > >> + string(name: 'IMPALA_REPO_BRANCH', value:
> > IMPALA_REPO_BRANCH)]
> > > > > >> + restring = result.getResult()
> > > > > >> + if (restring != null && !restring.equals("SUCCESS")) {
> > > > > >> + failed_job_urls.add(result.getAbsoluteUrl())
> > > > > >> + }
> > > > > >> }, TidyAndBuildOnlyAndRat: {
> > > > > >> result = build job: 'clang-tidy-ub1604', propagate: false,
> > > > > >> parameters:
> > > > > >> [string(name: 'IMPALA_REPO_URL', value: IMPALA_REPO_URL),
> > > > > >>
> > > > > >> This will show up in your GVO builds. I've tested it, but of
> > course
> > > > let
> > > > > me
> > > > > >> know if you run into any trouble.
> > > > > >>
> > > > > >> Thanks!
> > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: parallel-all-tests update
Posted by Laszlo Gaal <la...@cloudera.com>.
I'd be happy to pitch in; this is aligned with other test infra stuff I'm
working on.
On Thu, Oct 11, 2018 at 7:34 PM Philip Zeyliger <ph...@cloudera.com> wrote:
> I'm working a little bit (in the background) on getting
> bin/bootstrap_system.sh to work. I'm somewhat optimistic that I can get it
> to work :)
>
> I filed https://issues.apache.org/jira/browse/IMPALA-7698. Let me know if
> you want to help dig in!
>
> -- Philip
>
> On Thu, Oct 11, 2018 at 10:16 AM Pooja Nilangekar <
> pooja.nilangekar@cloudera.com> wrote:
>
> > I agree with the approach of running the tests on supported platforms. In
> > the past couple of days, there have been multiple issues (IMPALA-7678
> > <https://issues.apache.org/jira/browse/IMPALA-7678> & IMPALA-7690
> > <https://issues.apache.org/jira/browse/IMPALA-7690>) due to python2.6
> > compatibility issues. Additionally, python2.6 might be one of the place
> > where multiple linux distributions differ. So it probably makes sense to
> > add another centos-from-scratch or similar, rather than adding ad-hoc
> tests
> > to address issues as they come up.
> >
> > Thanks,
> > Pooja
> >
> >
> > On Thu, Sep 27, 2018 at 2:43 PM Philip Zeyliger <ph...@cloudera.com>
> > wrote:
> >
> > > I think the approach to catch that is to just run the tests on the
> > relevant
> > > platform. (E.g., with test-with-docker or something to make the Jenkins
> > > setup less painful.) Looking through some recent commits, I found:
> > >
> > > 304e02cf6238a3c1e9537337f802bcf7a92df5db "{}".format(1)
> > > 9cfa228c2e itertools.count(start=1)
> > > cf4f314922f13fbf54c6d7300ceaa2229bf5916a shutil.make_archive();
> Multiple
> > > with statements in the same context
> > > d91bc44021f28a5d113285be9c920bd1a646a9b9 {dict_comprehension}
> > > ??? a set comprehension; remember fixing it; couldn't find it.
> > >
> > > So, it's a mix of both classes, but I don't see recurring things that
> > would
> > > be easy to catch.
> > >
> > > -- Philip
> > >
> > >
> > > On Thu, Sep 27, 2018 at 2:14 PM Tim Armstrong <tarmstrong@cloudera.com
> >
> > > wrote:
> > >
> > > > Is it worth adding some regexes or similar to the gerrit bot to catch
> > > > itertools.count() usage? Or do we not expect repeated bugs of this
> > form?
> > > >
> > > > On Thu, Sep 27, 2018 at 12:59 PM Tim Armstrong <
> > tarmstrong@cloudera.com>
> > > > wrote:
> > > >
> > > > > I also added it to the pre-review tests. Let me know if you see any
> > > > issues.
> > > > >
> > > > > On Thu, Sep 27, 2018 at 12:33 PM Philip Zeyliger <
> > philip@cloudera.com>
> > > > > wrote:
> > > > >
> > > > >> Hi folks,
> > > > >>
> > > > >> To address IMPALA-6543, there's a new test in parallel-all-tests
> > that
> > > > >> makes
> > > > >> sure that any Python scripts use Python2.6-compatible syntax. Note
> > > that
> > > > >> this will catch the "try/catch/finally" style of bug, but not the
> > > > >> "itertools.count(start=1)" kind of bug (python2.7 changed the
> > > signature
> > > > of
> > > > >> itertools.count).
> > > > >>
> > > > >> The relevant diff was:
> > > > >>
> > > > >> $diff -u /tmp/b /tmp/a
> > > > >> --- /tmp/b 2018-09-27 12:29:15.000000000 -0700
> > > > >> +++ /tmp/a 2018-09-27 12:28:58.000000000 -0700
> > > > >> @@ -23,6 +23,14 @@
> > > > >> if (restring != null && !restring.equals("SUCCESS")) {
> > > > >> failed_job_urls.add(result.getAbsoluteUrl())
> > > > >> }
> > > > >> +}, Python26Compatibility: {
> > > > >> + result = build job: 'python26-incompatibility-check',
> > propagate:
> > > > >> false, parameters:
> > > > >> + [string(name: 'IMPALA_REPO_URL', value: IMPALA_REPO_URL),
> > > > >> + string(name: 'IMPALA_REPO_BRANCH', value:
> IMPALA_REPO_BRANCH)]
> > > > >> + restring = result.getResult()
> > > > >> + if (restring != null && !restring.equals("SUCCESS")) {
> > > > >> + failed_job_urls.add(result.getAbsoluteUrl())
> > > > >> + }
> > > > >> }, TidyAndBuildOnlyAndRat: {
> > > > >> result = build job: 'clang-tidy-ub1604', propagate: false,
> > > > >> parameters:
> > > > >> [string(name: 'IMPALA_REPO_URL', value: IMPALA_REPO_URL),
> > > > >>
> > > > >> This will show up in your GVO builds. I've tested it, but of
> course
> > > let
> > > > me
> > > > >> know if you run into any trouble.
> > > > >>
> > > > >> Thanks!
> > > > >>
> > > > >
> > > >
> > >
> >
>
Re: parallel-all-tests update
Posted by Philip Zeyliger <ph...@cloudera.com>.
I'm working a little bit (in the background) on getting
bin/bootstrap_system.sh to work. I'm somewhat optimistic that I can get it
to work :)
I filed https://issues.apache.org/jira/browse/IMPALA-7698. Let me know if
you want to help dig in!
-- Philip
On Thu, Oct 11, 2018 at 10:16 AM Pooja Nilangekar <
pooja.nilangekar@cloudera.com> wrote:
> I agree with the approach of running the tests on supported platforms. In
> the past couple of days, there have been multiple issues (IMPALA-7678
> <https://issues.apache.org/jira/browse/IMPALA-7678> & IMPALA-7690
> <https://issues.apache.org/jira/browse/IMPALA-7690>) due to python2.6
> compatibility issues. Additionally, python2.6 might be one of the place
> where multiple linux distributions differ. So it probably makes sense to
> add another centos-from-scratch or similar, rather than adding ad-hoc tests
> to address issues as they come up.
>
> Thanks,
> Pooja
>
>
> On Thu, Sep 27, 2018 at 2:43 PM Philip Zeyliger <ph...@cloudera.com>
> wrote:
>
> > I think the approach to catch that is to just run the tests on the
> relevant
> > platform. (E.g., with test-with-docker or something to make the Jenkins
> > setup less painful.) Looking through some recent commits, I found:
> >
> > 304e02cf6238a3c1e9537337f802bcf7a92df5db "{}".format(1)
> > 9cfa228c2e itertools.count(start=1)
> > cf4f314922f13fbf54c6d7300ceaa2229bf5916a shutil.make_archive(); Multiple
> > with statements in the same context
> > d91bc44021f28a5d113285be9c920bd1a646a9b9 {dict_comprehension}
> > ??? a set comprehension; remember fixing it; couldn't find it.
> >
> > So, it's a mix of both classes, but I don't see recurring things that
> would
> > be easy to catch.
> >
> > -- Philip
> >
> >
> > On Thu, Sep 27, 2018 at 2:14 PM Tim Armstrong <ta...@cloudera.com>
> > wrote:
> >
> > > Is it worth adding some regexes or similar to the gerrit bot to catch
> > > itertools.count() usage? Or do we not expect repeated bugs of this
> form?
> > >
> > > On Thu, Sep 27, 2018 at 12:59 PM Tim Armstrong <
> tarmstrong@cloudera.com>
> > > wrote:
> > >
> > > > I also added it to the pre-review tests. Let me know if you see any
> > > issues.
> > > >
> > > > On Thu, Sep 27, 2018 at 12:33 PM Philip Zeyliger <
> philip@cloudera.com>
> > > > wrote:
> > > >
> > > >> Hi folks,
> > > >>
> > > >> To address IMPALA-6543, there's a new test in parallel-all-tests
> that
> > > >> makes
> > > >> sure that any Python scripts use Python2.6-compatible syntax. Note
> > that
> > > >> this will catch the "try/catch/finally" style of bug, but not the
> > > >> "itertools.count(start=1)" kind of bug (python2.7 changed the
> > signature
> > > of
> > > >> itertools.count).
> > > >>
> > > >> The relevant diff was:
> > > >>
> > > >> $diff -u /tmp/b /tmp/a
> > > >> --- /tmp/b 2018-09-27 12:29:15.000000000 -0700
> > > >> +++ /tmp/a 2018-09-27 12:28:58.000000000 -0700
> > > >> @@ -23,6 +23,14 @@
> > > >> if (restring != null && !restring.equals("SUCCESS")) {
> > > >> failed_job_urls.add(result.getAbsoluteUrl())
> > > >> }
> > > >> +}, Python26Compatibility: {
> > > >> + result = build job: 'python26-incompatibility-check',
> propagate:
> > > >> false, parameters:
> > > >> + [string(name: 'IMPALA_REPO_URL', value: IMPALA_REPO_URL),
> > > >> + string(name: 'IMPALA_REPO_BRANCH', value: IMPALA_REPO_BRANCH)]
> > > >> + restring = result.getResult()
> > > >> + if (restring != null && !restring.equals("SUCCESS")) {
> > > >> + failed_job_urls.add(result.getAbsoluteUrl())
> > > >> + }
> > > >> }, TidyAndBuildOnlyAndRat: {
> > > >> result = build job: 'clang-tidy-ub1604', propagate: false,
> > > >> parameters:
> > > >> [string(name: 'IMPALA_REPO_URL', value: IMPALA_REPO_URL),
> > > >>
> > > >> This will show up in your GVO builds. I've tested it, but of course
> > let
> > > me
> > > >> know if you run into any trouble.
> > > >>
> > > >> Thanks!
> > > >>
> > > >
> > >
> >
>