You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@impala.apache.org by Tim Armstrong <ta...@cloudera.com> on 2020/10/01 23:22:06 UTC
Re: Discuss Impala ARM CI integration
I think we got stalled here. I never got anywhere with configuring commits@
to accept emails from openlab and not sure that it's possible.
In order to get something going, my suggestion is to copy what Kudu did and
have this go to a custom mailing list. If you follow the thread here it
describes what they did
https://lists.apache.org/thread.html/rfcd230882ec34171462a0c8c53ef6e8efcad50c9d67f8da18a508d07%40%3Cdev.kudu.apache.org%3E
Jim, it looks like the PMC chair is needed to create a new mailing list via
https://selfserve.apache.org/ (see
https://issues.apache.org/jira/browse/INFRA-20604), so maybe we can go
ahead with that route?
- Tim
I think it would be good to run the tests on impala.jenkins.io but I don't
want to hold up things further. Another interesting technology is the ARM
emulation support in Docker.
On Sun, Sep 20, 2020 at 8:53 PM Jim Apple <jb...@apache.org> wrote:
> I'm very excited about this development! I don't see any jobs at
> status.openlabtesting.org, though -- is
> http://status.openlabtesting.org/builds?job_name=impala-build-test-arm64
> the right link? Maybe only the owner of a job can see it?
>
> As far as the price, based on https://ec2instances.info/, you might be
> able
> to eyeball the cost. I'm no longer a Jenkins admin, so I'll let the other
> community members who are admins speak more to this, if it seems
> feasible to them.
>
> On Sat, Sep 19, 2020 at 2:31 AM Tianhua huang <hu...@gmail.com>
> wrote:
>
> > I am sorry there is no API with Openlab. But we use ZUUL in our system,
> you
> > can call ZUUL api to get the job information,
> > simple like: curl
> >
> >
> http://status.openlabtesting.org/api/builds?job_name=impala-build-test-arm64
> > | python -m json.tool
> > The result we will get is jobs info of 'impala-build-test-arm64',
> including
> > job status, job log url and so on, I am not sure if we can integrate with
> > the job info into impala.jenkins.io? Then the arm job still runs
> triggered
> > by OpenLab, and jenkins.impala.io only to get the result and show
> > the result in it.
> >
> > Not sure it's worth doing this, maybe let commits@impala.apache.org
> allow
> > receiving the result email from Openlab is more easy:)
> >
> >
> >
> > On Sat, Sep 19, 2020 at 11:03 AM Shant Hovsepian <
> > shant@superdupershant.com>
> > wrote:
> >
> > > Would it be possible to setup an API Key or webhook with
> openlabstesting?
> > >
> > > Then we'd have an Impala Jenkins job that uses the API to call
> > > openlabstesting to do the build and report the results back.
> > >
> > > -Shant
> > >
> > > On Fri, Sep 18, 2020 at 10:24 PM Tianhua huang <
> > huangtianhua223@gmail.com>
> > > wrote:
> > >
> > > > Hi Shant,
> > > >
> > > > Thanks for your reply, maybe it's a good idea to trigger arm testing
> by
> > > > jenkins.impala.io, we can donate an arm instance, but I don't know
> how
> > > to
> > > > do it. Maybe the Jenkins owner of Impala can help me?
> > > >
> > > >
> > > > On Sat, Sep 19, 2020 at 2:46 AM Shant Hovsepian <
> > > shant@superdupershant.com
> > > > >
> > > > wrote:
> > > >
> > > > > Great work Tianhua!
> > > > >
> > > > > This is a very exciting enhancement.
> > > > >
> > > > > dev@ isn't usually meant for automated test run output. Would it
> be
> > > > > possible to get the openlabtesting runs triggered by
> > jenkins.impala.io
> > > > > then
> > > > > have the results of the test run come from that jenkins instance?
> > This
> > > > > should solve the email problem as the notification can come from
> the
> > > > > jenkins account and still give us one centralized place to look at
> > > > various
> > > > > test statuses.
> > > > >
> > > > > -Shant
> > > > >
> > > > > On Thu, Sep 17, 2020 at 8:08 AM Tianhua huang <
> > > huangtianhua223@gmail.com
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > Impala can run/test on arm64 platform after our efforts, the
> issues
> > > we
> > > > > > fixed you can find in
> > > > https://issues.apache.org/jira/browse/IMPALA-9376
> > > > > >
> > > > > > We are particularly grateful to Tim and Quanlong for their help,
> > > thanks
> > > > > > very much!
> > > > > >
> > > > > > And I have integrated Impala arm64 CI with OpenLab(
> > > > > > https://openlabtesting.org/ , a community to do open source
> > project
> > > > > > testing),
> > > > > > see
> > > > > >
> > > >
> > http://status.openlabtesting.org/builds?job_name=impala-build-test-arm64
> > > > > > the
> > > > > > periodic test job runs for a few days already and runs once per
> > day.
> > > > > >
> > > > > > Tim suggested us to send the arm64 test result to
> > > > > > commits@impala.apache.org,
> > > > > > but seems it can receive the emails only ncome from @apache.org.
> > > > > > Now I wonder if we can send the test result to Impala community
> > > > maillist
> > > > > > here dev@impala.apache.org, to tell developers that Impala can
> > runs
> > > on
> > > > > > arm64, to find the compatibility in time.
> > > > > >
> > > > > > What do you think? Or maybe there is any other good suggestion?
> > > > > >
> > > > > > Waiting for your reply.
> > > > > >
> > > > > > Thank you very much!
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Discuss Impala ARM CI integration
Posted by Jim Apple <ap...@jbapple.com>.
OK, we're all set on the builds@ list. According to the confirmation email,
"it will appear on https://lists.apache.org/ within a few minutes of the
first post to it."
On Fri, Oct 2, 2020 at 5:38 PM Tim Armstrong <ta...@cloudera.com>
wrote:
> Thanks Jim!
>
> On Fri, Oct 2, 2020 at 11:06 AM Jim Apple <ap...@jbapple.com> wrote:
>
> > SG; submitted request for builds@
> >
> > Tim, I made you and I moderators. You're welcome to decline. :-)
> Volunteers
> > are welcome, but I expect zero moderation work - I set the list to
> > "subscribers can post; all others are rejected", so no moderation
> messages
> > should show up.
> >
> > The form said it will be acted on in 24 hours and the PMC will be
> informed
> > when the mailing list is ready.
> >
> > Also, FWIW, aside from Docker, AWS supports some ARM EC2 types.
> >
> > On Thu, Oct 1, 2020 at 4:22 PM Tim Armstrong <ta...@cloudera.com>
> > wrote:
> >
> > > I think we got stalled here. I never got anywhere with configuring
> > commits@
> > > to accept emails from openlab and not sure that it's possible.
> > >
> > > In order to get something going, my suggestion is to copy what Kudu did
> > and
> > > have this go to a custom mailing list. If you follow the thread here it
> > > describes what they did
> > >
> > >
> >
> https://lists.apache.org/thread.html/rfcd230882ec34171462a0c8c53ef6e8efcad50c9d67f8da18a508d07%40%3Cdev.kudu.apache.org%3E
> > >
> > > Jim, it looks like the PMC chair is needed to create a new mailing list
> > via
> > > https://selfserve.apache.org/ (see
> > > https://issues.apache.org/jira/browse/INFRA-20604), so maybe we can go
> > > ahead with that route?
> > >
> > > - Tim
> > >
> > > I think it would be good to run the tests on impala.jenkins.io but I
> > don't
> > > want to hold up things further. Another interesting technology is the
> ARM
> > > emulation support in Docker.
> > >
> > > On Sun, Sep 20, 2020 at 8:53 PM Jim Apple <jb...@apache.org> wrote:
> > >
> > > > I'm very excited about this development! I don't see any jobs at
> > > > status.openlabtesting.org, though -- is
> > > >
> > http://status.openlabtesting.org/builds?job_name=impala-build-test-arm64
> > > > the right link? Maybe only the owner of a job can see it?
> > > >
> > > > As far as the price, based on https://ec2instances.info/, you might
> be
> > > > able
> > > > to eyeball the cost. I'm no longer a Jenkins admin, so I'll let the
> > other
> > > > community members who are admins speak more to this, if it seems
> > > > feasible to them.
> > > >
> > > > On Sat, Sep 19, 2020 at 2:31 AM Tianhua huang <
> > huangtianhua223@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > I am sorry there is no API with Openlab. But we use ZUUL in our
> > system,
> > > > you
> > > > > can call ZUUL api to get the job information,
> > > > > simple like: curl
> > > > >
> > > > >
> > > >
> > >
> >
> http://status.openlabtesting.org/api/builds?job_name=impala-build-test-arm64
> > > > > | python -m json.tool
> > > > > The result we will get is jobs info of 'impala-build-test-arm64',
> > > > including
> > > > > job status, job log url and so on, I am not sure if we can
> integrate
> > > with
> > > > > the job info into impala.jenkins.io? Then the arm job still runs
> > > > triggered
> > > > > by OpenLab, and jenkins.impala.io only to get the result and show
> > > > > the result in it.
> > > > >
> > > > > Not sure it's worth doing this, maybe let
> commits@impala.apache.org
> > > > allow
> > > > > receiving the result email from Openlab is more easy:)
> > > > >
> > > > >
> > > > >
> > > > > On Sat, Sep 19, 2020 at 11:03 AM Shant Hovsepian <
> > > > > shant@superdupershant.com>
> > > > > wrote:
> > > > >
> > > > > > Would it be possible to setup an API Key or webhook with
> > > > openlabstesting?
> > > > > >
> > > > > > Then we'd have an Impala Jenkins job that uses the API to call
> > > > > > openlabstesting to do the build and report the results back.
> > > > > >
> > > > > > -Shant
> > > > > >
> > > > > > On Fri, Sep 18, 2020 at 10:24 PM Tianhua huang <
> > > > > huangtianhua223@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Shant,
> > > > > > >
> > > > > > > Thanks for your reply, maybe it's a good idea to trigger arm
> > > testing
> > > > by
> > > > > > > jenkins.impala.io, we can donate an arm instance, but I don't
> > know
> > > > how
> > > > > > to
> > > > > > > do it. Maybe the Jenkins owner of Impala can help me?
> > > > > > >
> > > > > > >
> > > > > > > On Sat, Sep 19, 2020 at 2:46 AM Shant Hovsepian <
> > > > > > shant@superdupershant.com
> > > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Great work Tianhua!
> > > > > > > >
> > > > > > > > This is a very exciting enhancement.
> > > > > > > >
> > > > > > > > dev@ isn't usually meant for automated test run output.
> Would
> > it
> > > > be
> > > > > > > > possible to get the openlabtesting runs triggered by
> > > > > jenkins.impala.io
> > > > > > > > then
> > > > > > > > have the results of the test run come from that jenkins
> > instance?
> > > > > This
> > > > > > > > should solve the email problem as the notification can come
> > from
> > > > the
> > > > > > > > jenkins account and still give us one centralized place to
> look
> > > at
> > > > > > > various
> > > > > > > > test statuses.
> > > > > > > >
> > > > > > > > -Shant
> > > > > > > >
> > > > > > > > On Thu, Sep 17, 2020 at 8:08 AM Tianhua huang <
> > > > > > huangtianhua223@gmail.com
> > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi all,
> > > > > > > > >
> > > > > > > > > Impala can run/test on arm64 platform after our efforts,
> the
> > > > issues
> > > > > > we
> > > > > > > > > fixed you can find in
> > > > > > > https://issues.apache.org/jira/browse/IMPALA-9376
> > > > > > > > >
> > > > > > > > > We are particularly grateful to Tim and Quanlong for their
> > > help,
> > > > > > thanks
> > > > > > > > > very much!
> > > > > > > > >
> > > > > > > > > And I have integrated Impala arm64 CI with OpenLab(
> > > > > > > > > https://openlabtesting.org/ , a community to do open
> source
> > > > > project
> > > > > > > > > testing),
> > > > > > > > > see
> > > > > > > > >
> > > > > > >
> > > > >
> > >
> http://status.openlabtesting.org/builds?job_name=impala-build-test-arm64
> > > > > > > > > the
> > > > > > > > > periodic test job runs for a few days already and runs once
> > per
> > > > > day.
> > > > > > > > >
> > > > > > > > > Tim suggested us to send the arm64 test result to
> > > > > > > > > commits@impala.apache.org,
> > > > > > > > > but seems it can receive the emails only ncome from @
> > > apache.org.
> > > > > > > > > Now I wonder if we can send the test result to Impala
> > community
> > > > > > > maillist
> > > > > > > > > here dev@impala.apache.org, to tell developers that Impala
> > can
> > > > > runs
> > > > > > on
> > > > > > > > > arm64, to find the compatibility in time.
> > > > > > > > >
> > > > > > > > > What do you think? Or maybe there is any other good
> > suggestion?
> > > > > > > > >
> > > > > > > > > Waiting for your reply.
> > > > > > > > >
> > > > > > > > > Thank you very much!
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Discuss Impala ARM CI integration
Posted by Tim Armstrong <ta...@cloudera.com>.
Thanks Jim!
On Fri, Oct 2, 2020 at 11:06 AM Jim Apple <ap...@jbapple.com> wrote:
> SG; submitted request for builds@
>
> Tim, I made you and I moderators. You're welcome to decline. :-) Volunteers
> are welcome, but I expect zero moderation work - I set the list to
> "subscribers can post; all others are rejected", so no moderation messages
> should show up.
>
> The form said it will be acted on in 24 hours and the PMC will be informed
> when the mailing list is ready.
>
> Also, FWIW, aside from Docker, AWS supports some ARM EC2 types.
>
> On Thu, Oct 1, 2020 at 4:22 PM Tim Armstrong <ta...@cloudera.com>
> wrote:
>
> > I think we got stalled here. I never got anywhere with configuring
> commits@
> > to accept emails from openlab and not sure that it's possible.
> >
> > In order to get something going, my suggestion is to copy what Kudu did
> and
> > have this go to a custom mailing list. If you follow the thread here it
> > describes what they did
> >
> >
> https://lists.apache.org/thread.html/rfcd230882ec34171462a0c8c53ef6e8efcad50c9d67f8da18a508d07%40%3Cdev.kudu.apache.org%3E
> >
> > Jim, it looks like the PMC chair is needed to create a new mailing list
> via
> > https://selfserve.apache.org/ (see
> > https://issues.apache.org/jira/browse/INFRA-20604), so maybe we can go
> > ahead with that route?
> >
> > - Tim
> >
> > I think it would be good to run the tests on impala.jenkins.io but I
> don't
> > want to hold up things further. Another interesting technology is the ARM
> > emulation support in Docker.
> >
> > On Sun, Sep 20, 2020 at 8:53 PM Jim Apple <jb...@apache.org> wrote:
> >
> > > I'm very excited about this development! I don't see any jobs at
> > > status.openlabtesting.org, though -- is
> > >
> http://status.openlabtesting.org/builds?job_name=impala-build-test-arm64
> > > the right link? Maybe only the owner of a job can see it?
> > >
> > > As far as the price, based on https://ec2instances.info/, you might be
> > > able
> > > to eyeball the cost. I'm no longer a Jenkins admin, so I'll let the
> other
> > > community members who are admins speak more to this, if it seems
> > > feasible to them.
> > >
> > > On Sat, Sep 19, 2020 at 2:31 AM Tianhua huang <
> huangtianhua223@gmail.com
> > >
> > > wrote:
> > >
> > > > I am sorry there is no API with Openlab. But we use ZUUL in our
> system,
> > > you
> > > > can call ZUUL api to get the job information,
> > > > simple like: curl
> > > >
> > > >
> > >
> >
> http://status.openlabtesting.org/api/builds?job_name=impala-build-test-arm64
> > > > | python -m json.tool
> > > > The result we will get is jobs info of 'impala-build-test-arm64',
> > > including
> > > > job status, job log url and so on, I am not sure if we can integrate
> > with
> > > > the job info into impala.jenkins.io? Then the arm job still runs
> > > triggered
> > > > by OpenLab, and jenkins.impala.io only to get the result and show
> > > > the result in it.
> > > >
> > > > Not sure it's worth doing this, maybe let commits@impala.apache.org
> > > allow
> > > > receiving the result email from Openlab is more easy:)
> > > >
> > > >
> > > >
> > > > On Sat, Sep 19, 2020 at 11:03 AM Shant Hovsepian <
> > > > shant@superdupershant.com>
> > > > wrote:
> > > >
> > > > > Would it be possible to setup an API Key or webhook with
> > > openlabstesting?
> > > > >
> > > > > Then we'd have an Impala Jenkins job that uses the API to call
> > > > > openlabstesting to do the build and report the results back.
> > > > >
> > > > > -Shant
> > > > >
> > > > > On Fri, Sep 18, 2020 at 10:24 PM Tianhua huang <
> > > > huangtianhua223@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi Shant,
> > > > > >
> > > > > > Thanks for your reply, maybe it's a good idea to trigger arm
> > testing
> > > by
> > > > > > jenkins.impala.io, we can donate an arm instance, but I don't
> know
> > > how
> > > > > to
> > > > > > do it. Maybe the Jenkins owner of Impala can help me?
> > > > > >
> > > > > >
> > > > > > On Sat, Sep 19, 2020 at 2:46 AM Shant Hovsepian <
> > > > > shant@superdupershant.com
> > > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Great work Tianhua!
> > > > > > >
> > > > > > > This is a very exciting enhancement.
> > > > > > >
> > > > > > > dev@ isn't usually meant for automated test run output. Would
> it
> > > be
> > > > > > > possible to get the openlabtesting runs triggered by
> > > > jenkins.impala.io
> > > > > > > then
> > > > > > > have the results of the test run come from that jenkins
> instance?
> > > > This
> > > > > > > should solve the email problem as the notification can come
> from
> > > the
> > > > > > > jenkins account and still give us one centralized place to look
> > at
> > > > > > various
> > > > > > > test statuses.
> > > > > > >
> > > > > > > -Shant
> > > > > > >
> > > > > > > On Thu, Sep 17, 2020 at 8:08 AM Tianhua huang <
> > > > > huangtianhua223@gmail.com
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > Impala can run/test on arm64 platform after our efforts, the
> > > issues
> > > > > we
> > > > > > > > fixed you can find in
> > > > > > https://issues.apache.org/jira/browse/IMPALA-9376
> > > > > > > >
> > > > > > > > We are particularly grateful to Tim and Quanlong for their
> > help,
> > > > > thanks
> > > > > > > > very much!
> > > > > > > >
> > > > > > > > And I have integrated Impala arm64 CI with OpenLab(
> > > > > > > > https://openlabtesting.org/ , a community to do open source
> > > > project
> > > > > > > > testing),
> > > > > > > > see
> > > > > > > >
> > > > > >
> > > >
> > http://status.openlabtesting.org/builds?job_name=impala-build-test-arm64
> > > > > > > > the
> > > > > > > > periodic test job runs for a few days already and runs once
> per
> > > > day.
> > > > > > > >
> > > > > > > > Tim suggested us to send the arm64 test result to
> > > > > > > > commits@impala.apache.org,
> > > > > > > > but seems it can receive the emails only ncome from @
> > apache.org.
> > > > > > > > Now I wonder if we can send the test result to Impala
> community
> > > > > > maillist
> > > > > > > > here dev@impala.apache.org, to tell developers that Impala
> can
> > > > runs
> > > > > on
> > > > > > > > arm64, to find the compatibility in time.
> > > > > > > >
> > > > > > > > What do you think? Or maybe there is any other good
> suggestion?
> > > > > > > >
> > > > > > > > Waiting for your reply.
> > > > > > > >
> > > > > > > > Thank you very much!
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Discuss Impala ARM CI integration
Posted by Tianhua huang <hu...@gmail.com>.
Thank you all:)
The details of the arm64 test job as below:
1. The job runs on instance which flavor is 16vCPUs | 32GB, using Ubuntu
18.04 64bit with ARM, with a 500GB root volume
2. The job runs once every day on utc 12:30
3. The job script general is ./bin/bootstrap_development.sh &
bin/run-all-tests.sh
you can find the script
https://github.com/theopenlab/openlab-zuul-jobs/blob/master/playbooks/impala-build-test-arm64/run.yaml
4. we send the job results to builds@impala.apache.org
On Sat, Oct 10, 2020 at 5:33 AM Tim Armstrong <ta...@cloudera.com>
wrote:
> It came through ok -
> https://lists.apache.org/list.html?builds@impala.apache.org
>
> On Thu, Oct 8, 2020 at 6:42 PM Tianhua huang <hu...@gmail.com>
> wrote:
>
> > Hi Jim and Tim,
> >
> > Thank you very much.
> > I am back from vacation, sorry to reply so late, I will configure the arm
> > job to send the test result to builds@impala.apache.org today.
> >
> > On Tue, Oct 6, 2020 at 7:50 AM Jim Apple <ap...@jbapple.com> wrote:
> >
> > > I did a test on EC2 Graviton 2, and I was able to start from a blank
> > Ubuntu
> > > 18.04 image and get all the way to all tests passing. This *included
> > > building the native-toolchain from scratch, with no manual input
> > required*
> > > - it just bootstrapped with no fuss! I used a r6g.2xlarge, which has a
> > list
> > > price of 40 cents per hour, though you have to pay for EBS, too. If we
> > get
> > > to a place where we want this to be in the gerrit merge step, I expect
> it
> > > to be smooth sailing.
> > >
> > > I'm really impressed with those of you who have worked on the pieces of
> > > this to put it together! Thank you for your contributions to the
> > community!
> > >
> > > Here, roughly, are how one could duplicate my test:
> > >
> > > 1. Launch an r6g.2xlarge using Ubuntu 18.04, Arm64, with a 500GB
> root
> > > volume. Leave the following ports open:
> > > - $YOUR_IP, udp 60000-61000 for mosh, in both IPV4 and IPv6
> > > - $YOUR_IP, tcp 22 for ssh, in both IPV4 and IPv6
> > > - All TCP and UDP from itself, for loopback
> > > 2. ssh in and
> > > - apt-get update
> > > - apt-get install tmux git mosh
> > > 3. Terminate your ssh session and connect again, this time with mosh
> > > 4. Start a tmux session
> > > 5. git clone https://github.com/apache/impala
> > > 6. cd impala
> > > 7. export IMPALA_HOME=$(pwd)
> > > 8. ./bin/bootstrap_development.sh && ./bin/run-all-tests.sh
> > >
> > > I'd love to see a performance comparison between two different
> equal-cost
> > > clusters on AWS: one Arm64, one x86-64, but I'll leave that to the
> > testers
> > > with larger budgets of both time and money! :-D
> > >
> > > On Fri, Oct 2, 2020 at 11:05 AM Jim Apple <ap...@jbapple.com> wrote:
> > >
> > > > SG; submitted request for builds@
> > > >
> > > > Tim, I made you and I moderators. You're welcome to decline. :-)
> > > > Volunteers are welcome, but I expect zero moderation work - I set the
> > > list
> > > > to "subscribers can post; all others are rejected", so no moderation
> > > > messages should show up.
> > > >
> > > > The form said it will be acted on in 24 hours and the PMC will be
> > > informed
> > > > when the mailing list is ready.
> > > >
> > > > Also, FWIW, aside from Docker, AWS supports some ARM EC2 types.
> > > >
> > > > On Thu, Oct 1, 2020 at 4:22 PM Tim Armstrong <
> tarmstrong@cloudera.com>
> > > > wrote:
> > > >
> > > >> I think we got stalled here. I never got anywhere with configuring
> > > >> commits@
> > > >> to accept emails from openlab and not sure that it's possible.
> > > >>
> > > >> In order to get something going, my suggestion is to copy what Kudu
> > did
> > > >> and
> > > >> have this go to a custom mailing list. If you follow the thread here
> > it
> > > >> describes what they did
> > > >>
> > > >>
> > >
> >
> https://lists.apache.org/thread.html/rfcd230882ec34171462a0c8c53ef6e8efcad50c9d67f8da18a508d07%40%3Cdev.kudu.apache.org%3E
> > > >>
> > > >> Jim, it looks like the PMC chair is needed to create a new mailing
> > list
> > > >> via
> > > >> https://selfserve.apache.org/ (see
> > > >> https://issues.apache.org/jira/browse/INFRA-20604), so maybe we can
> > go
> > > >> ahead with that route?
> > > >>
> > > >> - Tim
> > > >>
> > > >> I think it would be good to run the tests on impala.jenkins.io but
> I
> > > >> don't
> > > >> want to hold up things further. Another interesting technology is
> the
> > > ARM
> > > >> emulation support in Docker.
> > > >>
> > > >> On Sun, Sep 20, 2020 at 8:53 PM Jim Apple <jb...@apache.org>
> wrote:
> > > >>
> > > >> > I'm very excited about this development! I don't see any jobs at
> > > >> > status.openlabtesting.org, though -- is
> > > >> >
> > > >>
> > >
> http://status.openlabtesting.org/builds?job_name=impala-build-test-arm64
> > > >> > the right link? Maybe only the owner of a job can see it?
> > > >> >
> > > >> > As far as the price, based on https://ec2instances.info/, you
> might
> > > be
> > > >> > able
> > > >> > to eyeball the cost. I'm no longer a Jenkins admin, so I'll let
> the
> > > >> other
> > > >> > community members who are admins speak more to this, if it seems
> > > >> > feasible to them.
> > > >> >
> > > >> > On Sat, Sep 19, 2020 at 2:31 AM Tianhua huang <
> > > >> huangtianhua223@gmail.com>
> > > >> > wrote:
> > > >> >
> > > >> > > I am sorry there is no API with Openlab. But we use ZUUL in our
> > > >> system,
> > > >> > you
> > > >> > > can call ZUUL api to get the job information,
> > > >> > > simple like: curl
> > > >> > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> http://status.openlabtesting.org/api/builds?job_name=impala-build-test-arm64
> > > >> > > | python -m json.tool
> > > >> > > The result we will get is jobs info of
> 'impala-build-test-arm64',
> > > >> > including
> > > >> > > job status, job log url and so on, I am not sure if we can
> > integrate
> > > >> with
> > > >> > > the job info into impala.jenkins.io? Then the arm job still
> runs
> > > >> > triggered
> > > >> > > by OpenLab, and jenkins.impala.io only to get the result and
> show
> > > >> > > the result in it.
> > > >> > >
> > > >> > > Not sure it's worth doing this, maybe let
> > commits@impala.apache.org
> > > >> > allow
> > > >> > > receiving the result email from Openlab is more easy:)
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> > > On Sat, Sep 19, 2020 at 11:03 AM Shant Hovsepian <
> > > >> > > shant@superdupershant.com>
> > > >> > > wrote:
> > > >> > >
> > > >> > > > Would it be possible to setup an API Key or webhook with
> > > >> > openlabstesting?
> > > >> > > >
> > > >> > > > Then we'd have an Impala Jenkins job that uses the API to call
> > > >> > > > openlabstesting to do the build and report the results back.
> > > >> > > >
> > > >> > > > -Shant
> > > >> > > >
> > > >> > > > On Fri, Sep 18, 2020 at 10:24 PM Tianhua huang <
> > > >> > > huangtianhua223@gmail.com>
> > > >> > > > wrote:
> > > >> > > >
> > > >> > > > > Hi Shant,
> > > >> > > > >
> > > >> > > > > Thanks for your reply, maybe it's a good idea to trigger arm
> > > >> testing
> > > >> > by
> > > >> > > > > jenkins.impala.io, we can donate an arm instance, but I
> don't
> > > >> know
> > > >> > how
> > > >> > > > to
> > > >> > > > > do it. Maybe the Jenkins owner of Impala can help me?
> > > >> > > > >
> > > >> > > > >
> > > >> > > > > On Sat, Sep 19, 2020 at 2:46 AM Shant Hovsepian <
> > > >> > > > shant@superdupershant.com
> > > >> > > > > >
> > > >> > > > > wrote:
> > > >> > > > >
> > > >> > > > > > Great work Tianhua!
> > > >> > > > > >
> > > >> > > > > > This is a very exciting enhancement.
> > > >> > > > > >
> > > >> > > > > > dev@ isn't usually meant for automated test run output.
> > Would
> > > >> it
> > > >> > be
> > > >> > > > > > possible to get the openlabtesting runs triggered by
> > > >> > > jenkins.impala.io
> > > >> > > > > > then
> > > >> > > > > > have the results of the test run come from that jenkins
> > > >> instance?
> > > >> > > This
> > > >> > > > > > should solve the email problem as the notification can
> come
> > > from
> > > >> > the
> > > >> > > > > > jenkins account and still give us one centralized place to
> > > look
> > > >> at
> > > >> > > > > various
> > > >> > > > > > test statuses.
> > > >> > > > > >
> > > >> > > > > > -Shant
> > > >> > > > > >
> > > >> > > > > > On Thu, Sep 17, 2020 at 8:08 AM Tianhua huang <
> > > >> > > > huangtianhua223@gmail.com
> > > >> > > > > >
> > > >> > > > > > wrote:
> > > >> > > > > >
> > > >> > > > > > > Hi all,
> > > >> > > > > > >
> > > >> > > > > > > Impala can run/test on arm64 platform after our efforts,
> > the
> > > >> > issues
> > > >> > > > we
> > > >> > > > > > > fixed you can find in
> > > >> > > > > https://issues.apache.org/jira/browse/IMPALA-9376
> > > >> > > > > > >
> > > >> > > > > > > We are particularly grateful to Tim and Quanlong for
> their
> > > >> help,
> > > >> > > > thanks
> > > >> > > > > > > very much!
> > > >> > > > > > >
> > > >> > > > > > > And I have integrated Impala arm64 CI with OpenLab(
> > > >> > > > > > > https://openlabtesting.org/ , a community to do open
> > source
> > > >> > > project
> > > >> > > > > > > testing),
> > > >> > > > > > > see
> > > >> > > > > > >
> > > >> > > > >
> > > >> > >
> > > >>
> > >
> http://status.openlabtesting.org/builds?job_name=impala-build-test-arm64
> > > >> > > > > > > the
> > > >> > > > > > > periodic test job runs for a few days already and runs
> > once
> > > >> per
> > > >> > > day.
> > > >> > > > > > >
> > > >> > > > > > > Tim suggested us to send the arm64 test result to
> > > >> > > > > > > commits@impala.apache.org,
> > > >> > > > > > > but seems it can receive the emails only ncome from @
> > > >> apache.org.
> > > >> > > > > > > Now I wonder if we can send the test result to Impala
> > > >> community
> > > >> > > > > maillist
> > > >> > > > > > > here dev@impala.apache.org, to tell developers that
> > Impala
> > > >> can
> > > >> > > runs
> > > >> > > > on
> > > >> > > > > > > arm64, to find the compatibility in time.
> > > >> > > > > > >
> > > >> > > > > > > What do you think? Or maybe there is any other good
> > > >> suggestion?
> > > >> > > > > > >
> > > >> > > > > > > Waiting for your reply.
> > > >> > > > > > >
> > > >> > > > > > > Thank you very much!
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > > >
> > >
> >
>
Re: Discuss Impala ARM CI integration
Posted by Tim Armstrong <ta...@cloudera.com>.
It came through ok -
https://lists.apache.org/list.html?builds@impala.apache.org
On Thu, Oct 8, 2020 at 6:42 PM Tianhua huang <hu...@gmail.com>
wrote:
> Hi Jim and Tim,
>
> Thank you very much.
> I am back from vacation, sorry to reply so late, I will configure the arm
> job to send the test result to builds@impala.apache.org today.
>
> On Tue, Oct 6, 2020 at 7:50 AM Jim Apple <ap...@jbapple.com> wrote:
>
> > I did a test on EC2 Graviton 2, and I was able to start from a blank
> Ubuntu
> > 18.04 image and get all the way to all tests passing. This *included
> > building the native-toolchain from scratch, with no manual input
> required*
> > - it just bootstrapped with no fuss! I used a r6g.2xlarge, which has a
> list
> > price of 40 cents per hour, though you have to pay for EBS, too. If we
> get
> > to a place where we want this to be in the gerrit merge step, I expect it
> > to be smooth sailing.
> >
> > I'm really impressed with those of you who have worked on the pieces of
> > this to put it together! Thank you for your contributions to the
> community!
> >
> > Here, roughly, are how one could duplicate my test:
> >
> > 1. Launch an r6g.2xlarge using Ubuntu 18.04, Arm64, with a 500GB root
> > volume. Leave the following ports open:
> > - $YOUR_IP, udp 60000-61000 for mosh, in both IPV4 and IPv6
> > - $YOUR_IP, tcp 22 for ssh, in both IPV4 and IPv6
> > - All TCP and UDP from itself, for loopback
> > 2. ssh in and
> > - apt-get update
> > - apt-get install tmux git mosh
> > 3. Terminate your ssh session and connect again, this time with mosh
> > 4. Start a tmux session
> > 5. git clone https://github.com/apache/impala
> > 6. cd impala
> > 7. export IMPALA_HOME=$(pwd)
> > 8. ./bin/bootstrap_development.sh && ./bin/run-all-tests.sh
> >
> > I'd love to see a performance comparison between two different equal-cost
> > clusters on AWS: one Arm64, one x86-64, but I'll leave that to the
> testers
> > with larger budgets of both time and money! :-D
> >
> > On Fri, Oct 2, 2020 at 11:05 AM Jim Apple <ap...@jbapple.com> wrote:
> >
> > > SG; submitted request for builds@
> > >
> > > Tim, I made you and I moderators. You're welcome to decline. :-)
> > > Volunteers are welcome, but I expect zero moderation work - I set the
> > list
> > > to "subscribers can post; all others are rejected", so no moderation
> > > messages should show up.
> > >
> > > The form said it will be acted on in 24 hours and the PMC will be
> > informed
> > > when the mailing list is ready.
> > >
> > > Also, FWIW, aside from Docker, AWS supports some ARM EC2 types.
> > >
> > > On Thu, Oct 1, 2020 at 4:22 PM Tim Armstrong <ta...@cloudera.com>
> > > wrote:
> > >
> > >> I think we got stalled here. I never got anywhere with configuring
> > >> commits@
> > >> to accept emails from openlab and not sure that it's possible.
> > >>
> > >> In order to get something going, my suggestion is to copy what Kudu
> did
> > >> and
> > >> have this go to a custom mailing list. If you follow the thread here
> it
> > >> describes what they did
> > >>
> > >>
> >
> https://lists.apache.org/thread.html/rfcd230882ec34171462a0c8c53ef6e8efcad50c9d67f8da18a508d07%40%3Cdev.kudu.apache.org%3E
> > >>
> > >> Jim, it looks like the PMC chair is needed to create a new mailing
> list
> > >> via
> > >> https://selfserve.apache.org/ (see
> > >> https://issues.apache.org/jira/browse/INFRA-20604), so maybe we can
> go
> > >> ahead with that route?
> > >>
> > >> - Tim
> > >>
> > >> I think it would be good to run the tests on impala.jenkins.io but I
> > >> don't
> > >> want to hold up things further. Another interesting technology is the
> > ARM
> > >> emulation support in Docker.
> > >>
> > >> On Sun, Sep 20, 2020 at 8:53 PM Jim Apple <jb...@apache.org> wrote:
> > >>
> > >> > I'm very excited about this development! I don't see any jobs at
> > >> > status.openlabtesting.org, though -- is
> > >> >
> > >>
> > http://status.openlabtesting.org/builds?job_name=impala-build-test-arm64
> > >> > the right link? Maybe only the owner of a job can see it?
> > >> >
> > >> > As far as the price, based on https://ec2instances.info/, you might
> > be
> > >> > able
> > >> > to eyeball the cost. I'm no longer a Jenkins admin, so I'll let the
> > >> other
> > >> > community members who are admins speak more to this, if it seems
> > >> > feasible to them.
> > >> >
> > >> > On Sat, Sep 19, 2020 at 2:31 AM Tianhua huang <
> > >> huangtianhua223@gmail.com>
> > >> > wrote:
> > >> >
> > >> > > I am sorry there is no API with Openlab. But we use ZUUL in our
> > >> system,
> > >> > you
> > >> > > can call ZUUL api to get the job information,
> > >> > > simple like: curl
> > >> > >
> > >> > >
> > >> >
> > >>
> >
> http://status.openlabtesting.org/api/builds?job_name=impala-build-test-arm64
> > >> > > | python -m json.tool
> > >> > > The result we will get is jobs info of 'impala-build-test-arm64',
> > >> > including
> > >> > > job status, job log url and so on, I am not sure if we can
> integrate
> > >> with
> > >> > > the job info into impala.jenkins.io? Then the arm job still runs
> > >> > triggered
> > >> > > by OpenLab, and jenkins.impala.io only to get the result and show
> > >> > > the result in it.
> > >> > >
> > >> > > Not sure it's worth doing this, maybe let
> commits@impala.apache.org
> > >> > allow
> > >> > > receiving the result email from Openlab is more easy:)
> > >> > >
> > >> > >
> > >> > >
> > >> > > On Sat, Sep 19, 2020 at 11:03 AM Shant Hovsepian <
> > >> > > shant@superdupershant.com>
> > >> > > wrote:
> > >> > >
> > >> > > > Would it be possible to setup an API Key or webhook with
> > >> > openlabstesting?
> > >> > > >
> > >> > > > Then we'd have an Impala Jenkins job that uses the API to call
> > >> > > > openlabstesting to do the build and report the results back.
> > >> > > >
> > >> > > > -Shant
> > >> > > >
> > >> > > > On Fri, Sep 18, 2020 at 10:24 PM Tianhua huang <
> > >> > > huangtianhua223@gmail.com>
> > >> > > > wrote:
> > >> > > >
> > >> > > > > Hi Shant,
> > >> > > > >
> > >> > > > > Thanks for your reply, maybe it's a good idea to trigger arm
> > >> testing
> > >> > by
> > >> > > > > jenkins.impala.io, we can donate an arm instance, but I don't
> > >> know
> > >> > how
> > >> > > > to
> > >> > > > > do it. Maybe the Jenkins owner of Impala can help me?
> > >> > > > >
> > >> > > > >
> > >> > > > > On Sat, Sep 19, 2020 at 2:46 AM Shant Hovsepian <
> > >> > > > shant@superdupershant.com
> > >> > > > > >
> > >> > > > > wrote:
> > >> > > > >
> > >> > > > > > Great work Tianhua!
> > >> > > > > >
> > >> > > > > > This is a very exciting enhancement.
> > >> > > > > >
> > >> > > > > > dev@ isn't usually meant for automated test run output.
> Would
> > >> it
> > >> > be
> > >> > > > > > possible to get the openlabtesting runs triggered by
> > >> > > jenkins.impala.io
> > >> > > > > > then
> > >> > > > > > have the results of the test run come from that jenkins
> > >> instance?
> > >> > > This
> > >> > > > > > should solve the email problem as the notification can come
> > from
> > >> > the
> > >> > > > > > jenkins account and still give us one centralized place to
> > look
> > >> at
> > >> > > > > various
> > >> > > > > > test statuses.
> > >> > > > > >
> > >> > > > > > -Shant
> > >> > > > > >
> > >> > > > > > On Thu, Sep 17, 2020 at 8:08 AM Tianhua huang <
> > >> > > > huangtianhua223@gmail.com
> > >> > > > > >
> > >> > > > > > wrote:
> > >> > > > > >
> > >> > > > > > > Hi all,
> > >> > > > > > >
> > >> > > > > > > Impala can run/test on arm64 platform after our efforts,
> the
> > >> > issues
> > >> > > > we
> > >> > > > > > > fixed you can find in
> > >> > > > > https://issues.apache.org/jira/browse/IMPALA-9376
> > >> > > > > > >
> > >> > > > > > > We are particularly grateful to Tim and Quanlong for their
> > >> help,
> > >> > > > thanks
> > >> > > > > > > very much!
> > >> > > > > > >
> > >> > > > > > > And I have integrated Impala arm64 CI with OpenLab(
> > >> > > > > > > https://openlabtesting.org/ , a community to do open
> source
> > >> > > project
> > >> > > > > > > testing),
> > >> > > > > > > see
> > >> > > > > > >
> > >> > > > >
> > >> > >
> > >>
> > http://status.openlabtesting.org/builds?job_name=impala-build-test-arm64
> > >> > > > > > > the
> > >> > > > > > > periodic test job runs for a few days already and runs
> once
> > >> per
> > >> > > day.
> > >> > > > > > >
> > >> > > > > > > Tim suggested us to send the arm64 test result to
> > >> > > > > > > commits@impala.apache.org,
> > >> > > > > > > but seems it can receive the emails only ncome from @
> > >> apache.org.
> > >> > > > > > > Now I wonder if we can send the test result to Impala
> > >> community
> > >> > > > > maillist
> > >> > > > > > > here dev@impala.apache.org, to tell developers that
> Impala
> > >> can
> > >> > > runs
> > >> > > > on
> > >> > > > > > > arm64, to find the compatibility in time.
> > >> > > > > > >
> > >> > > > > > > What do you think? Or maybe there is any other good
> > >> suggestion?
> > >> > > > > > >
> > >> > > > > > > Waiting for your reply.
> > >> > > > > > >
> > >> > > > > > > Thank you very much!
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> >
>
Re: Discuss Impala ARM CI integration
Posted by Tianhua huang <hu...@gmail.com>.
Hi Jim and Tim,
Thank you very much.
I am back from vacation, sorry to reply so late, I will configure the arm
job to send the test result to builds@impala.apache.org today.
On Tue, Oct 6, 2020 at 7:50 AM Jim Apple <ap...@jbapple.com> wrote:
> I did a test on EC2 Graviton 2, and I was able to start from a blank Ubuntu
> 18.04 image and get all the way to all tests passing. This *included
> building the native-toolchain from scratch, with no manual input required*
> - it just bootstrapped with no fuss! I used a r6g.2xlarge, which has a list
> price of 40 cents per hour, though you have to pay for EBS, too. If we get
> to a place where we want this to be in the gerrit merge step, I expect it
> to be smooth sailing.
>
> I'm really impressed with those of you who have worked on the pieces of
> this to put it together! Thank you for your contributions to the community!
>
> Here, roughly, are how one could duplicate my test:
>
> 1. Launch an r6g.2xlarge using Ubuntu 18.04, Arm64, with a 500GB root
> volume. Leave the following ports open:
> - $YOUR_IP, udp 60000-61000 for mosh, in both IPV4 and IPv6
> - $YOUR_IP, tcp 22 for ssh, in both IPV4 and IPv6
> - All TCP and UDP from itself, for loopback
> 2. ssh in and
> - apt-get update
> - apt-get install tmux git mosh
> 3. Terminate your ssh session and connect again, this time with mosh
> 4. Start a tmux session
> 5. git clone https://github.com/apache/impala
> 6. cd impala
> 7. export IMPALA_HOME=$(pwd)
> 8. ./bin/bootstrap_development.sh && ./bin/run-all-tests.sh
>
> I'd love to see a performance comparison between two different equal-cost
> clusters on AWS: one Arm64, one x86-64, but I'll leave that to the testers
> with larger budgets of both time and money! :-D
>
> On Fri, Oct 2, 2020 at 11:05 AM Jim Apple <ap...@jbapple.com> wrote:
>
> > SG; submitted request for builds@
> >
> > Tim, I made you and I moderators. You're welcome to decline. :-)
> > Volunteers are welcome, but I expect zero moderation work - I set the
> list
> > to "subscribers can post; all others are rejected", so no moderation
> > messages should show up.
> >
> > The form said it will be acted on in 24 hours and the PMC will be
> informed
> > when the mailing list is ready.
> >
> > Also, FWIW, aside from Docker, AWS supports some ARM EC2 types.
> >
> > On Thu, Oct 1, 2020 at 4:22 PM Tim Armstrong <ta...@cloudera.com>
> > wrote:
> >
> >> I think we got stalled here. I never got anywhere with configuring
> >> commits@
> >> to accept emails from openlab and not sure that it's possible.
> >>
> >> In order to get something going, my suggestion is to copy what Kudu did
> >> and
> >> have this go to a custom mailing list. If you follow the thread here it
> >> describes what they did
> >>
> >>
> https://lists.apache.org/thread.html/rfcd230882ec34171462a0c8c53ef6e8efcad50c9d67f8da18a508d07%40%3Cdev.kudu.apache.org%3E
> >>
> >> Jim, it looks like the PMC chair is needed to create a new mailing list
> >> via
> >> https://selfserve.apache.org/ (see
> >> https://issues.apache.org/jira/browse/INFRA-20604), so maybe we can go
> >> ahead with that route?
> >>
> >> - Tim
> >>
> >> I think it would be good to run the tests on impala.jenkins.io but I
> >> don't
> >> want to hold up things further. Another interesting technology is the
> ARM
> >> emulation support in Docker.
> >>
> >> On Sun, Sep 20, 2020 at 8:53 PM Jim Apple <jb...@apache.org> wrote:
> >>
> >> > I'm very excited about this development! I don't see any jobs at
> >> > status.openlabtesting.org, though -- is
> >> >
> >>
> http://status.openlabtesting.org/builds?job_name=impala-build-test-arm64
> >> > the right link? Maybe only the owner of a job can see it?
> >> >
> >> > As far as the price, based on https://ec2instances.info/, you might
> be
> >> > able
> >> > to eyeball the cost. I'm no longer a Jenkins admin, so I'll let the
> >> other
> >> > community members who are admins speak more to this, if it seems
> >> > feasible to them.
> >> >
> >> > On Sat, Sep 19, 2020 at 2:31 AM Tianhua huang <
> >> huangtianhua223@gmail.com>
> >> > wrote:
> >> >
> >> > > I am sorry there is no API with Openlab. But we use ZUUL in our
> >> system,
> >> > you
> >> > > can call ZUUL api to get the job information,
> >> > > simple like: curl
> >> > >
> >> > >
> >> >
> >>
> http://status.openlabtesting.org/api/builds?job_name=impala-build-test-arm64
> >> > > | python -m json.tool
> >> > > The result we will get is jobs info of 'impala-build-test-arm64',
> >> > including
> >> > > job status, job log url and so on, I am not sure if we can integrate
> >> with
> >> > > the job info into impala.jenkins.io? Then the arm job still runs
> >> > triggered
> >> > > by OpenLab, and jenkins.impala.io only to get the result and show
> >> > > the result in it.
> >> > >
> >> > > Not sure it's worth doing this, maybe let commits@impala.apache.org
> >> > allow
> >> > > receiving the result email from Openlab is more easy:)
> >> > >
> >> > >
> >> > >
> >> > > On Sat, Sep 19, 2020 at 11:03 AM Shant Hovsepian <
> >> > > shant@superdupershant.com>
> >> > > wrote:
> >> > >
> >> > > > Would it be possible to setup an API Key or webhook with
> >> > openlabstesting?
> >> > > >
> >> > > > Then we'd have an Impala Jenkins job that uses the API to call
> >> > > > openlabstesting to do the build and report the results back.
> >> > > >
> >> > > > -Shant
> >> > > >
> >> > > > On Fri, Sep 18, 2020 at 10:24 PM Tianhua huang <
> >> > > huangtianhua223@gmail.com>
> >> > > > wrote:
> >> > > >
> >> > > > > Hi Shant,
> >> > > > >
> >> > > > > Thanks for your reply, maybe it's a good idea to trigger arm
> >> testing
> >> > by
> >> > > > > jenkins.impala.io, we can donate an arm instance, but I don't
> >> know
> >> > how
> >> > > > to
> >> > > > > do it. Maybe the Jenkins owner of Impala can help me?
> >> > > > >
> >> > > > >
> >> > > > > On Sat, Sep 19, 2020 at 2:46 AM Shant Hovsepian <
> >> > > > shant@superdupershant.com
> >> > > > > >
> >> > > > > wrote:
> >> > > > >
> >> > > > > > Great work Tianhua!
> >> > > > > >
> >> > > > > > This is a very exciting enhancement.
> >> > > > > >
> >> > > > > > dev@ isn't usually meant for automated test run output. Would
> >> it
> >> > be
> >> > > > > > possible to get the openlabtesting runs triggered by
> >> > > jenkins.impala.io
> >> > > > > > then
> >> > > > > > have the results of the test run come from that jenkins
> >> instance?
> >> > > This
> >> > > > > > should solve the email problem as the notification can come
> from
> >> > the
> >> > > > > > jenkins account and still give us one centralized place to
> look
> >> at
> >> > > > > various
> >> > > > > > test statuses.
> >> > > > > >
> >> > > > > > -Shant
> >> > > > > >
> >> > > > > > On Thu, Sep 17, 2020 at 8:08 AM Tianhua huang <
> >> > > > huangtianhua223@gmail.com
> >> > > > > >
> >> > > > > > wrote:
> >> > > > > >
> >> > > > > > > Hi all,
> >> > > > > > >
> >> > > > > > > Impala can run/test on arm64 platform after our efforts, the
> >> > issues
> >> > > > we
> >> > > > > > > fixed you can find in
> >> > > > > https://issues.apache.org/jira/browse/IMPALA-9376
> >> > > > > > >
> >> > > > > > > We are particularly grateful to Tim and Quanlong for their
> >> help,
> >> > > > thanks
> >> > > > > > > very much!
> >> > > > > > >
> >> > > > > > > And I have integrated Impala arm64 CI with OpenLab(
> >> > > > > > > https://openlabtesting.org/ , a community to do open source
> >> > > project
> >> > > > > > > testing),
> >> > > > > > > see
> >> > > > > > >
> >> > > > >
> >> > >
> >>
> http://status.openlabtesting.org/builds?job_name=impala-build-test-arm64
> >> > > > > > > the
> >> > > > > > > periodic test job runs for a few days already and runs once
> >> per
> >> > > day.
> >> > > > > > >
> >> > > > > > > Tim suggested us to send the arm64 test result to
> >> > > > > > > commits@impala.apache.org,
> >> > > > > > > but seems it can receive the emails only ncome from @
> >> apache.org.
> >> > > > > > > Now I wonder if we can send the test result to Impala
> >> community
> >> > > > > maillist
> >> > > > > > > here dev@impala.apache.org, to tell developers that Impala
> >> can
> >> > > runs
> >> > > > on
> >> > > > > > > arm64, to find the compatibility in time.
> >> > > > > > >
> >> > > > > > > What do you think? Or maybe there is any other good
> >> suggestion?
> >> > > > > > >
> >> > > > > > > Waiting for your reply.
> >> > > > > > >
> >> > > > > > > Thank you very much!
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
>
Re: Discuss Impala ARM CI integration
Posted by Jim Apple <ap...@jbapple.com>.
I did a test on EC2 Graviton 2, and I was able to start from a blank Ubuntu
18.04 image and get all the way to all tests passing. This *included
building the native-toolchain from scratch, with no manual input required*
- it just bootstrapped with no fuss! I used a r6g.2xlarge, which has a list
price of 40 cents per hour, though you have to pay for EBS, too. If we get
to a place where we want this to be in the gerrit merge step, I expect it
to be smooth sailing.
I'm really impressed with those of you who have worked on the pieces of
this to put it together! Thank you for your contributions to the community!
Here, roughly, are how one could duplicate my test:
1. Launch an r6g.2xlarge using Ubuntu 18.04, Arm64, with a 500GB root
volume. Leave the following ports open:
- $YOUR_IP, udp 60000-61000 for mosh, in both IPV4 and IPv6
- $YOUR_IP, tcp 22 for ssh, in both IPV4 and IPv6
- All TCP and UDP from itself, for loopback
2. ssh in and
- apt-get update
- apt-get install tmux git mosh
3. Terminate your ssh session and connect again, this time with mosh
4. Start a tmux session
5. git clone https://github.com/apache/impala
6. cd impala
7. export IMPALA_HOME=$(pwd)
8. ./bin/bootstrap_development.sh && ./bin/run-all-tests.sh
I'd love to see a performance comparison between two different equal-cost
clusters on AWS: one Arm64, one x86-64, but I'll leave that to the testers
with larger budgets of both time and money! :-D
On Fri, Oct 2, 2020 at 11:05 AM Jim Apple <ap...@jbapple.com> wrote:
> SG; submitted request for builds@
>
> Tim, I made you and I moderators. You're welcome to decline. :-)
> Volunteers are welcome, but I expect zero moderation work - I set the list
> to "subscribers can post; all others are rejected", so no moderation
> messages should show up.
>
> The form said it will be acted on in 24 hours and the PMC will be informed
> when the mailing list is ready.
>
> Also, FWIW, aside from Docker, AWS supports some ARM EC2 types.
>
> On Thu, Oct 1, 2020 at 4:22 PM Tim Armstrong <ta...@cloudera.com>
> wrote:
>
>> I think we got stalled here. I never got anywhere with configuring
>> commits@
>> to accept emails from openlab and not sure that it's possible.
>>
>> In order to get something going, my suggestion is to copy what Kudu did
>> and
>> have this go to a custom mailing list. If you follow the thread here it
>> describes what they did
>>
>> https://lists.apache.org/thread.html/rfcd230882ec34171462a0c8c53ef6e8efcad50c9d67f8da18a508d07%40%3Cdev.kudu.apache.org%3E
>>
>> Jim, it looks like the PMC chair is needed to create a new mailing list
>> via
>> https://selfserve.apache.org/ (see
>> https://issues.apache.org/jira/browse/INFRA-20604), so maybe we can go
>> ahead with that route?
>>
>> - Tim
>>
>> I think it would be good to run the tests on impala.jenkins.io but I
>> don't
>> want to hold up things further. Another interesting technology is the ARM
>> emulation support in Docker.
>>
>> On Sun, Sep 20, 2020 at 8:53 PM Jim Apple <jb...@apache.org> wrote:
>>
>> > I'm very excited about this development! I don't see any jobs at
>> > status.openlabtesting.org, though -- is
>> >
>> http://status.openlabtesting.org/builds?job_name=impala-build-test-arm64
>> > the right link? Maybe only the owner of a job can see it?
>> >
>> > As far as the price, based on https://ec2instances.info/, you might be
>> > able
>> > to eyeball the cost. I'm no longer a Jenkins admin, so I'll let the
>> other
>> > community members who are admins speak more to this, if it seems
>> > feasible to them.
>> >
>> > On Sat, Sep 19, 2020 at 2:31 AM Tianhua huang <
>> huangtianhua223@gmail.com>
>> > wrote:
>> >
>> > > I am sorry there is no API with Openlab. But we use ZUUL in our
>> system,
>> > you
>> > > can call ZUUL api to get the job information,
>> > > simple like: curl
>> > >
>> > >
>> >
>> http://status.openlabtesting.org/api/builds?job_name=impala-build-test-arm64
>> > > | python -m json.tool
>> > > The result we will get is jobs info of 'impala-build-test-arm64',
>> > including
>> > > job status, job log url and so on, I am not sure if we can integrate
>> with
>> > > the job info into impala.jenkins.io? Then the arm job still runs
>> > triggered
>> > > by OpenLab, and jenkins.impala.io only to get the result and show
>> > > the result in it.
>> > >
>> > > Not sure it's worth doing this, maybe let commits@impala.apache.org
>> > allow
>> > > receiving the result email from Openlab is more easy:)
>> > >
>> > >
>> > >
>> > > On Sat, Sep 19, 2020 at 11:03 AM Shant Hovsepian <
>> > > shant@superdupershant.com>
>> > > wrote:
>> > >
>> > > > Would it be possible to setup an API Key or webhook with
>> > openlabstesting?
>> > > >
>> > > > Then we'd have an Impala Jenkins job that uses the API to call
>> > > > openlabstesting to do the build and report the results back.
>> > > >
>> > > > -Shant
>> > > >
>> > > > On Fri, Sep 18, 2020 at 10:24 PM Tianhua huang <
>> > > huangtianhua223@gmail.com>
>> > > > wrote:
>> > > >
>> > > > > Hi Shant,
>> > > > >
>> > > > > Thanks for your reply, maybe it's a good idea to trigger arm
>> testing
>> > by
>> > > > > jenkins.impala.io, we can donate an arm instance, but I don't
>> know
>> > how
>> > > > to
>> > > > > do it. Maybe the Jenkins owner of Impala can help me?
>> > > > >
>> > > > >
>> > > > > On Sat, Sep 19, 2020 at 2:46 AM Shant Hovsepian <
>> > > > shant@superdupershant.com
>> > > > > >
>> > > > > wrote:
>> > > > >
>> > > > > > Great work Tianhua!
>> > > > > >
>> > > > > > This is a very exciting enhancement.
>> > > > > >
>> > > > > > dev@ isn't usually meant for automated test run output. Would
>> it
>> > be
>> > > > > > possible to get the openlabtesting runs triggered by
>> > > jenkins.impala.io
>> > > > > > then
>> > > > > > have the results of the test run come from that jenkins
>> instance?
>> > > This
>> > > > > > should solve the email problem as the notification can come from
>> > the
>> > > > > > jenkins account and still give us one centralized place to look
>> at
>> > > > > various
>> > > > > > test statuses.
>> > > > > >
>> > > > > > -Shant
>> > > > > >
>> > > > > > On Thu, Sep 17, 2020 at 8:08 AM Tianhua huang <
>> > > > huangtianhua223@gmail.com
>> > > > > >
>> > > > > > wrote:
>> > > > > >
>> > > > > > > Hi all,
>> > > > > > >
>> > > > > > > Impala can run/test on arm64 platform after our efforts, the
>> > issues
>> > > > we
>> > > > > > > fixed you can find in
>> > > > > https://issues.apache.org/jira/browse/IMPALA-9376
>> > > > > > >
>> > > > > > > We are particularly grateful to Tim and Quanlong for their
>> help,
>> > > > thanks
>> > > > > > > very much!
>> > > > > > >
>> > > > > > > And I have integrated Impala arm64 CI with OpenLab(
>> > > > > > > https://openlabtesting.org/ , a community to do open source
>> > > project
>> > > > > > > testing),
>> > > > > > > see
>> > > > > > >
>> > > > >
>> > >
>> http://status.openlabtesting.org/builds?job_name=impala-build-test-arm64
>> > > > > > > the
>> > > > > > > periodic test job runs for a few days already and runs once
>> per
>> > > day.
>> > > > > > >
>> > > > > > > Tim suggested us to send the arm64 test result to
>> > > > > > > commits@impala.apache.org,
>> > > > > > > but seems it can receive the emails only ncome from @
>> apache.org.
>> > > > > > > Now I wonder if we can send the test result to Impala
>> community
>> > > > > maillist
>> > > > > > > here dev@impala.apache.org, to tell developers that Impala
>> can
>> > > runs
>> > > > on
>> > > > > > > arm64, to find the compatibility in time.
>> > > > > > >
>> > > > > > > What do you think? Or maybe there is any other good
>> suggestion?
>> > > > > > >
>> > > > > > > Waiting for your reply.
>> > > > > > >
>> > > > > > > Thank you very much!
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
Re: Discuss Impala ARM CI integration
Posted by Jim Apple <ap...@jbapple.com>.
SG; submitted request for builds@
Tim, I made you and I moderators. You're welcome to decline. :-) Volunteers
are welcome, but I expect zero moderation work - I set the list to
"subscribers can post; all others are rejected", so no moderation messages
should show up.
The form said it will be acted on in 24 hours and the PMC will be informed
when the mailing list is ready.
Also, FWIW, aside from Docker, AWS supports some ARM EC2 types.
On Thu, Oct 1, 2020 at 4:22 PM Tim Armstrong <ta...@cloudera.com>
wrote:
> I think we got stalled here. I never got anywhere with configuring commits@
> to accept emails from openlab and not sure that it's possible.
>
> In order to get something going, my suggestion is to copy what Kudu did and
> have this go to a custom mailing list. If you follow the thread here it
> describes what they did
>
> https://lists.apache.org/thread.html/rfcd230882ec34171462a0c8c53ef6e8efcad50c9d67f8da18a508d07%40%3Cdev.kudu.apache.org%3E
>
> Jim, it looks like the PMC chair is needed to create a new mailing list via
> https://selfserve.apache.org/ (see
> https://issues.apache.org/jira/browse/INFRA-20604), so maybe we can go
> ahead with that route?
>
> - Tim
>
> I think it would be good to run the tests on impala.jenkins.io but I don't
> want to hold up things further. Another interesting technology is the ARM
> emulation support in Docker.
>
> On Sun, Sep 20, 2020 at 8:53 PM Jim Apple <jb...@apache.org> wrote:
>
> > I'm very excited about this development! I don't see any jobs at
> > status.openlabtesting.org, though -- is
> > http://status.openlabtesting.org/builds?job_name=impala-build-test-arm64
> > the right link? Maybe only the owner of a job can see it?
> >
> > As far as the price, based on https://ec2instances.info/, you might be
> > able
> > to eyeball the cost. I'm no longer a Jenkins admin, so I'll let the other
> > community members who are admins speak more to this, if it seems
> > feasible to them.
> >
> > On Sat, Sep 19, 2020 at 2:31 AM Tianhua huang <huangtianhua223@gmail.com
> >
> > wrote:
> >
> > > I am sorry there is no API with Openlab. But we use ZUUL in our system,
> > you
> > > can call ZUUL api to get the job information,
> > > simple like: curl
> > >
> > >
> >
> http://status.openlabtesting.org/api/builds?job_name=impala-build-test-arm64
> > > | python -m json.tool
> > > The result we will get is jobs info of 'impala-build-test-arm64',
> > including
> > > job status, job log url and so on, I am not sure if we can integrate
> with
> > > the job info into impala.jenkins.io? Then the arm job still runs
> > triggered
> > > by OpenLab, and jenkins.impala.io only to get the result and show
> > > the result in it.
> > >
> > > Not sure it's worth doing this, maybe let commits@impala.apache.org
> > allow
> > > receiving the result email from Openlab is more easy:)
> > >
> > >
> > >
> > > On Sat, Sep 19, 2020 at 11:03 AM Shant Hovsepian <
> > > shant@superdupershant.com>
> > > wrote:
> > >
> > > > Would it be possible to setup an API Key or webhook with
> > openlabstesting?
> > > >
> > > > Then we'd have an Impala Jenkins job that uses the API to call
> > > > openlabstesting to do the build and report the results back.
> > > >
> > > > -Shant
> > > >
> > > > On Fri, Sep 18, 2020 at 10:24 PM Tianhua huang <
> > > huangtianhua223@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi Shant,
> > > > >
> > > > > Thanks for your reply, maybe it's a good idea to trigger arm
> testing
> > by
> > > > > jenkins.impala.io, we can donate an arm instance, but I don't know
> > how
> > > > to
> > > > > do it. Maybe the Jenkins owner of Impala can help me?
> > > > >
> > > > >
> > > > > On Sat, Sep 19, 2020 at 2:46 AM Shant Hovsepian <
> > > > shant@superdupershant.com
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > Great work Tianhua!
> > > > > >
> > > > > > This is a very exciting enhancement.
> > > > > >
> > > > > > dev@ isn't usually meant for automated test run output. Would it
> > be
> > > > > > possible to get the openlabtesting runs triggered by
> > > jenkins.impala.io
> > > > > > then
> > > > > > have the results of the test run come from that jenkins instance?
> > > This
> > > > > > should solve the email problem as the notification can come from
> > the
> > > > > > jenkins account and still give us one centralized place to look
> at
> > > > > various
> > > > > > test statuses.
> > > > > >
> > > > > > -Shant
> > > > > >
> > > > > > On Thu, Sep 17, 2020 at 8:08 AM Tianhua huang <
> > > > huangtianhua223@gmail.com
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > Impala can run/test on arm64 platform after our efforts, the
> > issues
> > > > we
> > > > > > > fixed you can find in
> > > > > https://issues.apache.org/jira/browse/IMPALA-9376
> > > > > > >
> > > > > > > We are particularly grateful to Tim and Quanlong for their
> help,
> > > > thanks
> > > > > > > very much!
> > > > > > >
> > > > > > > And I have integrated Impala arm64 CI with OpenLab(
> > > > > > > https://openlabtesting.org/ , a community to do open source
> > > project
> > > > > > > testing),
> > > > > > > see
> > > > > > >
> > > > >
> > >
> http://status.openlabtesting.org/builds?job_name=impala-build-test-arm64
> > > > > > > the
> > > > > > > periodic test job runs for a few days already and runs once per
> > > day.
> > > > > > >
> > > > > > > Tim suggested us to send the arm64 test result to
> > > > > > > commits@impala.apache.org,
> > > > > > > but seems it can receive the emails only ncome from @
> apache.org.
> > > > > > > Now I wonder if we can send the test result to Impala community
> > > > > maillist
> > > > > > > here dev@impala.apache.org, to tell developers that Impala can
> > > runs
> > > > on
> > > > > > > arm64, to find the compatibility in time.
> > > > > > >
> > > > > > > What do you think? Or maybe there is any other good suggestion?
> > > > > > >
> > > > > > > Waiting for your reply.
> > > > > > >
> > > > > > > Thank you very much!
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>