You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Dongjoon Hyun <do...@gmail.com> on 2019/11/19 21:35:39 UTC
Migration `Spark QA Compile` Jenkins jobs to GitHub Action
Hi, All.
Apache Spark community used the following dashboard as post-hook
verifications.
https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Compile/
There are six registered jobs.
1. spark-branch-2.4-compile-maven-hadoop-2.6
2. spark-branch-2.4-compile-maven-hadoop-2.7
3. spark-branch-2.4-lint
4. spark-master-compile-maven-hadoop-2.7
5. spark-master-compile-maven-hadoop-3.2
6. spark-master-lint
Now, we added `GitHub Action` jobs. You can see the green check at every
commit.
https://github.com/apache/spark/commits/master
https://github.com/apache/spark/commits/branch-2.4
If you click the green check, you can see the detail.
The followings are the example runs at the last commits on both branches.
https://github.com/apache/spark/runs/310411948 (master)
https://github.com/apache/spark/runs/309522646 (branch-2.4)
New `GitHub Action` have more combination than the old Jenkins jobs.
- branch-2.4-scala-2.11-hadoop-2.6 (compile/package/install)
- branch-2.4-scala-2.12-hadoop-2.6 (compile/package/install)
- branch-2.4-scala-2.11-hadoop-2.7 (compile/package/install)
- branch-2.4-scala-2.12-hadoop-2.7 (compile/package/install)
- branch-2.4-linters (Scala/Java/Python/R)
- master-scala-2.12-hadoop-2.7 (compile/package/install)
- master-scala-2.12-hadoop-3.2 (compile/package/install)
- master-scala-2.12-hadoop-3.2-jdk11 (compile/package/install)
- master-linters (Scala/Java/Python/R)
In addition, this is a part of Apache Spark code base and everyone can make
contributions on this.
Finally, as the last piece of this work, we are going to remove the legacy
Jenkins jobs via the following JIRA issue.
https://issues.apache.org/jira/browse/SPARK-29935
Please let me know if you have any concerns on this.
(We can keep the legacy jobs, but two of them are already broken.)
Bests,
Dongjoon.
Re: Migration `Spark QA Compile` Jenkins jobs to GitHub Action
Posted by Dongjoon Hyun <do...@gmail.com>.
Thank you, Sean, Shane, and Xiao!
Bests,
Dongjoon.
On Tue, Nov 19, 2019 at 2:15 PM Shane Knapp <sk...@berkeley.edu> wrote:
> i had a few minutes aaaaand everything has been deleted!
>
> On Tue, Nov 19, 2019 at 2:02 PM Shane Knapp <sk...@berkeley.edu> wrote:
> >
> > thank sean!
> >
> > i am all for moving these jobs to github actions, and will be doing
> > this 'soon' as i'm @ kubecon this week.
> >
> > btw the R ecosystem definitely needs some attention, however, but
> > that's an issue for another time. :)
> >
> > On Tue, Nov 19, 2019 at 1:49 PM Sean Owen <sr...@gmail.com> wrote:
> > >
> > > I would favor moving whatever we can to Github. It's difficult to
> > > modify the Jenkins instances without Shane's valiant help, and over
> > > time makes more sense to modernize and integrate it into the project.
> > >
> > > On Tue, Nov 19, 2019 at 3:35 PM Dongjoon Hyun <do...@gmail.com>
> wrote:
> > > >
> > > > Hi, All.
> > > >
> > > > Apache Spark community used the following dashboard as post-hook
> verifications.
> > > >
> > > >
> https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Compile/
> > > >
> > > > There are six registered jobs.
> > > >
> > > > 1. spark-branch-2.4-compile-maven-hadoop-2.6
> > > > 2. spark-branch-2.4-compile-maven-hadoop-2.7
> > > > 3. spark-branch-2.4-lint
> > > > 4. spark-master-compile-maven-hadoop-2.7
> > > > 5. spark-master-compile-maven-hadoop-3.2
> > > > 6. spark-master-lint
> > > >
> > > > Now, we added `GitHub Action` jobs. You can see the green check at
> every commit.
> > > >
> > > > https://github.com/apache/spark/commits/master
> > > > https://github.com/apache/spark/commits/branch-2.4
> > > >
> > > > If you click the green check, you can see the detail.
> > > > The followings are the example runs at the last commits on both
> branches.
> > > >
> > > > https://github.com/apache/spark/runs/310411948 (master)
> > > > https://github.com/apache/spark/runs/309522646 (branch-2.4)
> > > >
> > > > New `GitHub Action` have more combination than the old Jenkins jobs.
> > > >
> > > > - branch-2.4-scala-2.11-hadoop-2.6 (compile/package/install)
> > > > - branch-2.4-scala-2.12-hadoop-2.6 (compile/package/install)
> > > > - branch-2.4-scala-2.11-hadoop-2.7 (compile/package/install)
> > > > - branch-2.4-scala-2.12-hadoop-2.7 (compile/package/install)
> > > > - branch-2.4-linters (Scala/Java/Python/R)
> > > > - master-scala-2.12-hadoop-2.7 (compile/package/install)
> > > > - master-scala-2.12-hadoop-3.2 (compile/package/install)
> > > > - master-scala-2.12-hadoop-3.2-jdk11 (compile/package/install)
> > > > - master-linters (Scala/Java/Python/R)
> > > >
> > > > In addition, this is a part of Apache Spark code base and everyone
> can make contributions on this.
> > > >
> > > > Finally, as the last piece of this work, we are going to remove the
> legacy Jenkins jobs via the following JIRA issue.
> > > >
> > > > https://issues.apache.org/jira/browse/SPARK-29935
> > > >
> > > > Please let me know if you have any concerns on this.
> > > > (We can keep the legacy jobs, but two of them are already broken.)
> > > >
> > > > Bests,
> > > > Dongjoon.
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
> > >
> >
> >
> > --
> > Shane Knapp
> > UC Berkeley EECS Research / RISELab Staff Technical Lead
> > https://rise.cs.berkeley.edu
>
>
>
> --
> Shane Knapp
> UC Berkeley EECS Research / RISELab Staff Technical Lead
> https://rise.cs.berkeley.edu
>
Re: Migration `Spark QA Compile` Jenkins jobs to GitHub Action
Posted by Shane Knapp <sk...@berkeley.edu>.
i had a few minutes aaaaand everything has been deleted!
On Tue, Nov 19, 2019 at 2:02 PM Shane Knapp <sk...@berkeley.edu> wrote:
>
> thank sean!
>
> i am all for moving these jobs to github actions, and will be doing
> this 'soon' as i'm @ kubecon this week.
>
> btw the R ecosystem definitely needs some attention, however, but
> that's an issue for another time. :)
>
> On Tue, Nov 19, 2019 at 1:49 PM Sean Owen <sr...@gmail.com> wrote:
> >
> > I would favor moving whatever we can to Github. It's difficult to
> > modify the Jenkins instances without Shane's valiant help, and over
> > time makes more sense to modernize and integrate it into the project.
> >
> > On Tue, Nov 19, 2019 at 3:35 PM Dongjoon Hyun <do...@gmail.com> wrote:
> > >
> > > Hi, All.
> > >
> > > Apache Spark community used the following dashboard as post-hook verifications.
> > >
> > > https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Compile/
> > >
> > > There are six registered jobs.
> > >
> > > 1. spark-branch-2.4-compile-maven-hadoop-2.6
> > > 2. spark-branch-2.4-compile-maven-hadoop-2.7
> > > 3. spark-branch-2.4-lint
> > > 4. spark-master-compile-maven-hadoop-2.7
> > > 5. spark-master-compile-maven-hadoop-3.2
> > > 6. spark-master-lint
> > >
> > > Now, we added `GitHub Action` jobs. You can see the green check at every commit.
> > >
> > > https://github.com/apache/spark/commits/master
> > > https://github.com/apache/spark/commits/branch-2.4
> > >
> > > If you click the green check, you can see the detail.
> > > The followings are the example runs at the last commits on both branches.
> > >
> > > https://github.com/apache/spark/runs/310411948 (master)
> > > https://github.com/apache/spark/runs/309522646 (branch-2.4)
> > >
> > > New `GitHub Action` have more combination than the old Jenkins jobs.
> > >
> > > - branch-2.4-scala-2.11-hadoop-2.6 (compile/package/install)
> > > - branch-2.4-scala-2.12-hadoop-2.6 (compile/package/install)
> > > - branch-2.4-scala-2.11-hadoop-2.7 (compile/package/install)
> > > - branch-2.4-scala-2.12-hadoop-2.7 (compile/package/install)
> > > - branch-2.4-linters (Scala/Java/Python/R)
> > > - master-scala-2.12-hadoop-2.7 (compile/package/install)
> > > - master-scala-2.12-hadoop-3.2 (compile/package/install)
> > > - master-scala-2.12-hadoop-3.2-jdk11 (compile/package/install)
> > > - master-linters (Scala/Java/Python/R)
> > >
> > > In addition, this is a part of Apache Spark code base and everyone can make contributions on this.
> > >
> > > Finally, as the last piece of this work, we are going to remove the legacy Jenkins jobs via the following JIRA issue.
> > >
> > > https://issues.apache.org/jira/browse/SPARK-29935
> > >
> > > Please let me know if you have any concerns on this.
> > > (We can keep the legacy jobs, but two of them are already broken.)
> > >
> > > Bests,
> > > Dongjoon.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
> >
>
>
> --
> Shane Knapp
> UC Berkeley EECS Research / RISELab Staff Technical Lead
> https://rise.cs.berkeley.edu
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: Migration `Spark QA Compile` Jenkins jobs to GitHub Action
Posted by Shane Knapp <sk...@berkeley.edu>.
thank sean!
i am all for moving these jobs to github actions, and will be doing
this 'soon' as i'm @ kubecon this week.
btw the R ecosystem definitely needs some attention, however, but
that's an issue for another time. :)
On Tue, Nov 19, 2019 at 1:49 PM Sean Owen <sr...@gmail.com> wrote:
>
> I would favor moving whatever we can to Github. It's difficult to
> modify the Jenkins instances without Shane's valiant help, and over
> time makes more sense to modernize and integrate it into the project.
>
> On Tue, Nov 19, 2019 at 3:35 PM Dongjoon Hyun <do...@gmail.com> wrote:
> >
> > Hi, All.
> >
> > Apache Spark community used the following dashboard as post-hook verifications.
> >
> > https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Compile/
> >
> > There are six registered jobs.
> >
> > 1. spark-branch-2.4-compile-maven-hadoop-2.6
> > 2. spark-branch-2.4-compile-maven-hadoop-2.7
> > 3. spark-branch-2.4-lint
> > 4. spark-master-compile-maven-hadoop-2.7
> > 5. spark-master-compile-maven-hadoop-3.2
> > 6. spark-master-lint
> >
> > Now, we added `GitHub Action` jobs. You can see the green check at every commit.
> >
> > https://github.com/apache/spark/commits/master
> > https://github.com/apache/spark/commits/branch-2.4
> >
> > If you click the green check, you can see the detail.
> > The followings are the example runs at the last commits on both branches.
> >
> > https://github.com/apache/spark/runs/310411948 (master)
> > https://github.com/apache/spark/runs/309522646 (branch-2.4)
> >
> > New `GitHub Action` have more combination than the old Jenkins jobs.
> >
> > - branch-2.4-scala-2.11-hadoop-2.6 (compile/package/install)
> > - branch-2.4-scala-2.12-hadoop-2.6 (compile/package/install)
> > - branch-2.4-scala-2.11-hadoop-2.7 (compile/package/install)
> > - branch-2.4-scala-2.12-hadoop-2.7 (compile/package/install)
> > - branch-2.4-linters (Scala/Java/Python/R)
> > - master-scala-2.12-hadoop-2.7 (compile/package/install)
> > - master-scala-2.12-hadoop-3.2 (compile/package/install)
> > - master-scala-2.12-hadoop-3.2-jdk11 (compile/package/install)
> > - master-linters (Scala/Java/Python/R)
> >
> > In addition, this is a part of Apache Spark code base and everyone can make contributions on this.
> >
> > Finally, as the last piece of this work, we are going to remove the legacy Jenkins jobs via the following JIRA issue.
> >
> > https://issues.apache.org/jira/browse/SPARK-29935
> >
> > Please let me know if you have any concerns on this.
> > (We can keep the legacy jobs, but two of them are already broken.)
> >
> > Bests,
> > Dongjoon.
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
--
Shane Knapp
UC Berkeley EECS Research / RISELab Staff Technical Lead
https://rise.cs.berkeley.edu
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: Migration `Spark QA Compile` Jenkins jobs to GitHub Action
Posted by Sean Owen <sr...@gmail.com>.
I would favor moving whatever we can to Github. It's difficult to
modify the Jenkins instances without Shane's valiant help, and over
time makes more sense to modernize and integrate it into the project.
On Tue, Nov 19, 2019 at 3:35 PM Dongjoon Hyun <do...@gmail.com> wrote:
>
> Hi, All.
>
> Apache Spark community used the following dashboard as post-hook verifications.
>
> https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Compile/
>
> There are six registered jobs.
>
> 1. spark-branch-2.4-compile-maven-hadoop-2.6
> 2. spark-branch-2.4-compile-maven-hadoop-2.7
> 3. spark-branch-2.4-lint
> 4. spark-master-compile-maven-hadoop-2.7
> 5. spark-master-compile-maven-hadoop-3.2
> 6. spark-master-lint
>
> Now, we added `GitHub Action` jobs. You can see the green check at every commit.
>
> https://github.com/apache/spark/commits/master
> https://github.com/apache/spark/commits/branch-2.4
>
> If you click the green check, you can see the detail.
> The followings are the example runs at the last commits on both branches.
>
> https://github.com/apache/spark/runs/310411948 (master)
> https://github.com/apache/spark/runs/309522646 (branch-2.4)
>
> New `GitHub Action` have more combination than the old Jenkins jobs.
>
> - branch-2.4-scala-2.11-hadoop-2.6 (compile/package/install)
> - branch-2.4-scala-2.12-hadoop-2.6 (compile/package/install)
> - branch-2.4-scala-2.11-hadoop-2.7 (compile/package/install)
> - branch-2.4-scala-2.12-hadoop-2.7 (compile/package/install)
> - branch-2.4-linters (Scala/Java/Python/R)
> - master-scala-2.12-hadoop-2.7 (compile/package/install)
> - master-scala-2.12-hadoop-3.2 (compile/package/install)
> - master-scala-2.12-hadoop-3.2-jdk11 (compile/package/install)
> - master-linters (Scala/Java/Python/R)
>
> In addition, this is a part of Apache Spark code base and everyone can make contributions on this.
>
> Finally, as the last piece of this work, we are going to remove the legacy Jenkins jobs via the following JIRA issue.
>
> https://issues.apache.org/jira/browse/SPARK-29935
>
> Please let me know if you have any concerns on this.
> (We can keep the legacy jobs, but two of them are already broken.)
>
> Bests,
> Dongjoon.
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org