You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Marcelo Vanzin <va...@cloudera.com> on 2016/11/15 21:21:10 UTC
Running lint-java during PR builds?
Hey all,
Is there a reason why lint-java is not run during PR builds? I see it
seems to be maven-only, is it really expensive to run after an sbt
build?
I see a lot of PRs coming in to fix Java style issues, and those all
seem a little unnecessary. Either we're enforcing style checks or
we're not, and right now it seems we aren't.
--
Marcelo
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: Running lint-java during PR builds?
Posted by Hyukjin Kwon <gu...@gmail.com>.
Ah, now it's merged so that it can be ran via SBT and it runs when there
are some changes in .java file only.
On Mon, 28 May 2018, 6:23 pm Steve Loughran, <st...@hortonworks.com> wrote:
>
>
> > On 21 May 2018, at 17:20, Marcelo Vanzin <va...@cloudera.com> wrote:
> >
> > Is there a way to trigger it conditionally? e.g. only if the diff
> > touches java files.
> >
>
> what about adding it as another command which could be added alongside
> "jenkins test this please", something like "lint this please"
>
> > On Mon, May 21, 2018 at 9:17 AM, Felix Cheung <fe...@hotmail.com>
> wrote:
> >> One concern is with the volume of test runs on Travis.
> >>
> >> In ASF projects Travis could get significantly
> >> backed up since - if I recall - all of ASF shares one queue.
> >>
> >> At the number of PRs Spark has this could be a big issue.
> >>
> >>
> >> ________________________________
> >> From: Marcelo Vanzin <va...@cloudera.com>
> >> Sent: Monday, May 21, 2018 9:08:28 AM
> >> To: Hyukjin Kwon
> >> Cc: Dongjoon Hyun; dev
> >> Subject: Re: Running lint-java during PR builds?
> >>
> >> I'm fine with it. I tried to use the existing checkstyle sbt plugin
> >> (trying to fix SPARK-22269), but it depends on an ancient version of
> >> checkstyle, and I don't know sbt enough to figure out how to hack
> >> classpaths and class loaders when applying rules, so gave up.
> >>
> >> On Mon, May 21, 2018 at 1:47 AM, Hyukjin Kwon <gu...@gmail.com>
> wrote:
> >>> I am going to open an INFRA JIRA if there's no explicit objection in
> few
> >>> days.
> >>>
> >>> 2018-05-21 13:09 GMT+08:00 Hyukjin Kwon <gu...@gmail.com>:
> >>>>
> >>>> I would like to revive this proposal. Travis CI. Shall we give this
> try?
> >>>> I
> >>>> think it's worth trying it.
> >>>>
> >>>> 2016-11-17 3:50 GMT+08:00 Dongjoon Hyun <do...@apache.org>:
> >>>>>
> >>>>> Hi, Marcelo and Ryan.
> >>>>>
> >>>>> That was the main purpose of my proposal about Travis.CI.
> >>>>> IMO, that is the only way to achieve that without any harmful
> >>>>> side-effect
> >>>>> on Jenkins infra.
> >>>>>
> >>>>> Spark is already ready for that. Like AppVoyer, if one of you files
> an
> >>>>> INFRA jira issue to enable that, they will turn on that. Then, we can
> >>>>> try it
> >>>>> and see the result. Also, you can turn off easily again if you don't
> >>>>> want.
> >>>>>
> >>>>> Without this, we will consume more community efforts. For example, we
> >>>>> merged lint-java error fix PR seven hours ago, but the master branch
> >>>>> still
> >>>>> has one lint-java error.
> >>>>>
> >>>>> https://travis-ci.org/dongjoon-hyun/spark/jobs/176351319
> >>>>>
> >>>>> Actually, I've been monitoring the history here. (It's synced every
> 30
> >>>>> minutes.)
> >>>>>
> >>>>> https://travis-ci.org/dongjoon-hyun/spark/builds
> >>>>>
> >>>>> Could we give a change to this?
> >>>>>
> >>>>> Bests,
> >>>>> Dongjoon.
> >>>>>
> >>>>> On 2016-11-15 13:40 (-0800), "Shixiong(Ryan) Zhu"
> >>>>> <sh...@databricks.com> wrote:
> >>>>>> I remember it's because you need to run `mvn install` before running
> >>>>>> lint-java if the maven cache is empty, and `mvn install` is pretty
> >>>>>> heavy.
> >>>>>>
> >>>>>> On Tue, Nov 15, 2016 at 1:21 PM, Marcelo Vanzin <
> vanzin@cloudera.com>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Hey all,
> >>>>>>>
> >>>>>>> Is there a reason why lint-java is not run during PR builds? I see
> >>>>>>> it
> >>>>>>> seems to be maven-only, is it really expensive to run after an sbt
> >>>>>>> build?
> >>>>>>>
> >>>>>>> I see a lot of PRs coming in to fix Java style issues, and those
> all
> >>>>>>> seem a little unnecessary. Either we're enforcing style checks or
> >>>>>>> we're not, and right now it seems we aren't.
> >>>>>>>
> >>>>>>> --
> >>>>>>> Marcelo
> >>>>>>>
> >>>>>>>
> >>>>>>>
> ---------------------------------------------------------------------
> >>>>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>> ---------------------------------------------------------------------
> >>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
> >>>>>
> >>>>
> >>>
> >>
> >>
> >>
> >> --
> >> Marcelo
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
> >>
> >
> >
> >
> > --
> > Marcelo
> >
> > ---------------------------------------------------------------------
> > To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
> >
>
>
Re: Running lint-java during PR builds?
Posted by Steve Loughran <st...@hortonworks.com>.
> On 21 May 2018, at 17:20, Marcelo Vanzin <va...@cloudera.com> wrote:
>
> Is there a way to trigger it conditionally? e.g. only if the diff
> touches java files.
>
what about adding it as another command which could be added alongside "jenkins test this please", something like "lint this please"
> On Mon, May 21, 2018 at 9:17 AM, Felix Cheung <fe...@hotmail.com> wrote:
>> One concern is with the volume of test runs on Travis.
>>
>> In ASF projects Travis could get significantly
>> backed up since - if I recall - all of ASF shares one queue.
>>
>> At the number of PRs Spark has this could be a big issue.
>>
>>
>> ________________________________
>> From: Marcelo Vanzin <va...@cloudera.com>
>> Sent: Monday, May 21, 2018 9:08:28 AM
>> To: Hyukjin Kwon
>> Cc: Dongjoon Hyun; dev
>> Subject: Re: Running lint-java during PR builds?
>>
>> I'm fine with it. I tried to use the existing checkstyle sbt plugin
>> (trying to fix SPARK-22269), but it depends on an ancient version of
>> checkstyle, and I don't know sbt enough to figure out how to hack
>> classpaths and class loaders when applying rules, so gave up.
>>
>> On Mon, May 21, 2018 at 1:47 AM, Hyukjin Kwon <gu...@gmail.com> wrote:
>>> I am going to open an INFRA JIRA if there's no explicit objection in few
>>> days.
>>>
>>> 2018-05-21 13:09 GMT+08:00 Hyukjin Kwon <gu...@gmail.com>:
>>>>
>>>> I would like to revive this proposal. Travis CI. Shall we give this try?
>>>> I
>>>> think it's worth trying it.
>>>>
>>>> 2016-11-17 3:50 GMT+08:00 Dongjoon Hyun <do...@apache.org>:
>>>>>
>>>>> Hi, Marcelo and Ryan.
>>>>>
>>>>> That was the main purpose of my proposal about Travis.CI.
>>>>> IMO, that is the only way to achieve that without any harmful
>>>>> side-effect
>>>>> on Jenkins infra.
>>>>>
>>>>> Spark is already ready for that. Like AppVoyer, if one of you files an
>>>>> INFRA jira issue to enable that, they will turn on that. Then, we can
>>>>> try it
>>>>> and see the result. Also, you can turn off easily again if you don't
>>>>> want.
>>>>>
>>>>> Without this, we will consume more community efforts. For example, we
>>>>> merged lint-java error fix PR seven hours ago, but the master branch
>>>>> still
>>>>> has one lint-java error.
>>>>>
>>>>> https://travis-ci.org/dongjoon-hyun/spark/jobs/176351319
>>>>>
>>>>> Actually, I've been monitoring the history here. (It's synced every 30
>>>>> minutes.)
>>>>>
>>>>> https://travis-ci.org/dongjoon-hyun/spark/builds
>>>>>
>>>>> Could we give a change to this?
>>>>>
>>>>> Bests,
>>>>> Dongjoon.
>>>>>
>>>>> On 2016-11-15 13:40 (-0800), "Shixiong(Ryan) Zhu"
>>>>> <sh...@databricks.com> wrote:
>>>>>> I remember it's because you need to run `mvn install` before running
>>>>>> lint-java if the maven cache is empty, and `mvn install` is pretty
>>>>>> heavy.
>>>>>>
>>>>>> On Tue, Nov 15, 2016 at 1:21 PM, Marcelo Vanzin <va...@cloudera.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hey all,
>>>>>>>
>>>>>>> Is there a reason why lint-java is not run during PR builds? I see
>>>>>>> it
>>>>>>> seems to be maven-only, is it really expensive to run after an sbt
>>>>>>> build?
>>>>>>>
>>>>>>> I see a lot of PRs coming in to fix Java style issues, and those all
>>>>>>> seem a little unnecessary. Either we're enforcing style checks or
>>>>>>> we're not, and right now it seems we aren't.
>>>>>>>
>>>>>>> --
>>>>>>> Marcelo
>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>>
>>>>
>>>
>>
>>
>>
>> --
>> Marcelo
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>
>
>
>
> --
> Marcelo
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: Running lint-java during PR builds?
Posted by Hyukjin Kwon <gu...@gmail.com>.
I opened a PR - https://github.com/apache/spark/pull/21399 to run it with
SBT.
2018-05-22 2:18 GMT+08:00 Reynold Xin <rx...@databricks.com>:
> Can we look into if there is a plugin for sbt that works and then we can
> put everything into one single builder?
>
> On Mon, May 21, 2018 at 11:17 AM Dongjoon Hyun <do...@gmail.com>
> wrote:
>
>> Thank you for reconsidering this, Hyukjin. :)
>>
>> Bests,
>> Dongjoon.
>>
>>
>> On Mon, May 21, 2018 at 9:20 AM, Marcelo Vanzin <va...@cloudera.com>
>> wrote:
>>
>>> Is there a way to trigger it conditionally? e.g. only if the diff
>>> touches java files.
>>>
>>> On Mon, May 21, 2018 at 9:17 AM, Felix Cheung <fe...@hotmail.com>
>>> wrote:
>>> > One concern is with the volume of test runs on Travis.
>>> >
>>> > In ASF projects Travis could get significantly
>>> > backed up since - if I recall - all of ASF shares one queue.
>>> >
>>> > At the number of PRs Spark has this could be a big issue.
>>> >
>>> >
>>> > ________________________________
>>> > From: Marcelo Vanzin <va...@cloudera.com>
>>> > Sent: Monday, May 21, 2018 9:08:28 AM
>>> > To: Hyukjin Kwon
>>> > Cc: Dongjoon Hyun; dev
>>> > Subject: Re: Running lint-java during PR builds?
>>> >
>>> > I'm fine with it. I tried to use the existing checkstyle sbt plugin
>>> > (trying to fix SPARK-22269), but it depends on an ancient version of
>>> > checkstyle, and I don't know sbt enough to figure out how to hack
>>> > classpaths and class loaders when applying rules, so gave up.
>>> >
>>> > On Mon, May 21, 2018 at 1:47 AM, Hyukjin Kwon <gu...@gmail.com>
>>> wrote:
>>> >> I am going to open an INFRA JIRA if there's no explicit objection in
>>> few
>>> >> days.
>>> >>
>>> >> 2018-05-21 13:09 GMT+08:00 Hyukjin Kwon <gu...@gmail.com>:
>>> >>>
>>> >>> I would like to revive this proposal. Travis CI. Shall we give this
>>> try?
>>> >>> I
>>> >>> think it's worth trying it.
>>> >>>
>>> >>> 2016-11-17 3:50 GMT+08:00 Dongjoon Hyun <do...@apache.org>:
>>> >>>>
>>> >>>> Hi, Marcelo and Ryan.
>>> >>>>
>>> >>>> That was the main purpose of my proposal about Travis.CI.
>>> >>>> IMO, that is the only way to achieve that without any harmful
>>> >>>> side-effect
>>> >>>> on Jenkins infra.
>>> >>>>
>>> >>>> Spark is already ready for that. Like AppVoyer, if one of you files
>>> an
>>> >>>> INFRA jira issue to enable that, they will turn on that. Then, we
>>> can
>>> >>>> try it
>>> >>>> and see the result. Also, you can turn off easily again if you don't
>>> >>>> want.
>>> >>>>
>>> >>>> Without this, we will consume more community efforts. For example,
>>> we
>>> >>>> merged lint-java error fix PR seven hours ago, but the master branch
>>> >>>> still
>>> >>>> has one lint-java error.
>>> >>>>
>>> >>>> https://travis-ci.org/dongjoon-hyun/spark/jobs/176351319
>>> >>>>
>>> >>>> Actually, I've been monitoring the history here. (It's synced every
>>> 30
>>> >>>> minutes.)
>>> >>>>
>>> >>>> https://travis-ci.org/dongjoon-hyun/spark/builds
>>> >>>>
>>> >>>> Could we give a change to this?
>>> >>>>
>>> >>>> Bests,
>>> >>>> Dongjoon.
>>> >>>>
>>> >>>> On 2016-11-15 13:40 (-0800), "Shixiong(Ryan) Zhu"
>>> >>>> <sh...@databricks.com> wrote:
>>> >>>> > I remember it's because you need to run `mvn install` before
>>> running
>>> >>>> > lint-java if the maven cache is empty, and `mvn install` is pretty
>>> >>>> > heavy.
>>> >>>> >
>>> >>>> > On Tue, Nov 15, 2016 at 1:21 PM, Marcelo Vanzin <
>>> vanzin@cloudera.com>
>>> >>>> > wrote:
>>> >>>> >
>>> >>>> > > Hey all,
>>> >>>> > >
>>> >>>> > > Is there a reason why lint-java is not run during PR builds? I
>>> see
>>> >>>> > > it
>>> >>>> > > seems to be maven-only, is it really expensive to run after an
>>> sbt
>>> >>>> > > build?
>>> >>>> > >
>>> >>>> > > I see a lot of PRs coming in to fix Java style issues, and
>>> those all
>>> >>>> > > seem a little unnecessary. Either we're enforcing style checks
>>> or
>>> >>>> > > we're not, and right now it seems we aren't.
>>> >>>> > >
>>> >>>> > > --
>>> >>>> > > Marcelo
>>> >>>> > >
>>> >>>> > >
>>> >>>> > > ------------------------------------------------------------
>>> ---------
>>> >>>> > > To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>> >>>> > >
>>> >>>> > >
>>> >>>> >
>>> >>>>
>>> >>>> ------------------------------------------------------------
>>> ---------
>>> >>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>> >>>>
>>> >>>
>>> >>
>>> >
>>> >
>>> >
>>> > --
>>> > Marcelo
>>> >
>>> > ---------------------------------------------------------------------
>>> > To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>> >
>>>
>>>
>>>
>>> --
>>> Marcelo
>>>
>>
>>
Re: Running lint-java during PR builds?
Posted by Reynold Xin <rx...@databricks.com>.
Can we look into if there is a plugin for sbt that works and then we can
put everything into one single builder?
On Mon, May 21, 2018 at 11:17 AM Dongjoon Hyun <do...@gmail.com>
wrote:
> Thank you for reconsidering this, Hyukjin. :)
>
> Bests,
> Dongjoon.
>
>
> On Mon, May 21, 2018 at 9:20 AM, Marcelo Vanzin <va...@cloudera.com>
> wrote:
>
>> Is there a way to trigger it conditionally? e.g. only if the diff
>> touches java files.
>>
>> On Mon, May 21, 2018 at 9:17 AM, Felix Cheung <fe...@hotmail.com>
>> wrote:
>> > One concern is with the volume of test runs on Travis.
>> >
>> > In ASF projects Travis could get significantly
>> > backed up since - if I recall - all of ASF shares one queue.
>> >
>> > At the number of PRs Spark has this could be a big issue.
>> >
>> >
>> > ________________________________
>> > From: Marcelo Vanzin <va...@cloudera.com>
>> > Sent: Monday, May 21, 2018 9:08:28 AM
>> > To: Hyukjin Kwon
>> > Cc: Dongjoon Hyun; dev
>> > Subject: Re: Running lint-java during PR builds?
>> >
>> > I'm fine with it. I tried to use the existing checkstyle sbt plugin
>> > (trying to fix SPARK-22269), but it depends on an ancient version of
>> > checkstyle, and I don't know sbt enough to figure out how to hack
>> > classpaths and class loaders when applying rules, so gave up.
>> >
>> > On Mon, May 21, 2018 at 1:47 AM, Hyukjin Kwon <gu...@gmail.com>
>> wrote:
>> >> I am going to open an INFRA JIRA if there's no explicit objection in
>> few
>> >> days.
>> >>
>> >> 2018-05-21 13:09 GMT+08:00 Hyukjin Kwon <gu...@gmail.com>:
>> >>>
>> >>> I would like to revive this proposal. Travis CI. Shall we give this
>> try?
>> >>> I
>> >>> think it's worth trying it.
>> >>>
>> >>> 2016-11-17 3:50 GMT+08:00 Dongjoon Hyun <do...@apache.org>:
>> >>>>
>> >>>> Hi, Marcelo and Ryan.
>> >>>>
>> >>>> That was the main purpose of my proposal about Travis.CI.
>> >>>> IMO, that is the only way to achieve that without any harmful
>> >>>> side-effect
>> >>>> on Jenkins infra.
>> >>>>
>> >>>> Spark is already ready for that. Like AppVoyer, if one of you files
>> an
>> >>>> INFRA jira issue to enable that, they will turn on that. Then, we can
>> >>>> try it
>> >>>> and see the result. Also, you can turn off easily again if you don't
>> >>>> want.
>> >>>>
>> >>>> Without this, we will consume more community efforts. For example, we
>> >>>> merged lint-java error fix PR seven hours ago, but the master branch
>> >>>> still
>> >>>> has one lint-java error.
>> >>>>
>> >>>> https://travis-ci.org/dongjoon-hyun/spark/jobs/176351319
>> >>>>
>> >>>> Actually, I've been monitoring the history here. (It's synced every
>> 30
>> >>>> minutes.)
>> >>>>
>> >>>> https://travis-ci.org/dongjoon-hyun/spark/builds
>> >>>>
>> >>>> Could we give a change to this?
>> >>>>
>> >>>> Bests,
>> >>>> Dongjoon.
>> >>>>
>> >>>> On 2016-11-15 13:40 (-0800), "Shixiong(Ryan) Zhu"
>> >>>> <sh...@databricks.com> wrote:
>> >>>> > I remember it's because you need to run `mvn install` before
>> running
>> >>>> > lint-java if the maven cache is empty, and `mvn install` is pretty
>> >>>> > heavy.
>> >>>> >
>> >>>> > On Tue, Nov 15, 2016 at 1:21 PM, Marcelo Vanzin <
>> vanzin@cloudera.com>
>> >>>> > wrote:
>> >>>> >
>> >>>> > > Hey all,
>> >>>> > >
>> >>>> > > Is there a reason why lint-java is not run during PR builds? I
>> see
>> >>>> > > it
>> >>>> > > seems to be maven-only, is it really expensive to run after an
>> sbt
>> >>>> > > build?
>> >>>> > >
>> >>>> > > I see a lot of PRs coming in to fix Java style issues, and those
>> all
>> >>>> > > seem a little unnecessary. Either we're enforcing style checks or
>> >>>> > > we're not, and right now it seems we aren't.
>> >>>> > >
>> >>>> > > --
>> >>>> > > Marcelo
>> >>>> > >
>> >>>> > >
>> >>>> > >
>> ---------------------------------------------------------------------
>> >>>> > > To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>> >>>> > >
>> >>>> > >
>> >>>> >
>> >>>>
>> >>>> ---------------------------------------------------------------------
>> >>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>> >>>>
>> >>>
>> >>
>> >
>> >
>> >
>> > --
>> > Marcelo
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>> >
>>
>>
>>
>> --
>> Marcelo
>>
>
>
Re: Running lint-java during PR builds?
Posted by Dongjoon Hyun <do...@gmail.com>.
Thank you for reconsidering this, Hyukjin. :)
Bests,
Dongjoon.
On Mon, May 21, 2018 at 9:20 AM, Marcelo Vanzin <va...@cloudera.com> wrote:
> Is there a way to trigger it conditionally? e.g. only if the diff
> touches java files.
>
> On Mon, May 21, 2018 at 9:17 AM, Felix Cheung <fe...@hotmail.com>
> wrote:
> > One concern is with the volume of test runs on Travis.
> >
> > In ASF projects Travis could get significantly
> > backed up since - if I recall - all of ASF shares one queue.
> >
> > At the number of PRs Spark has this could be a big issue.
> >
> >
> > ________________________________
> > From: Marcelo Vanzin <va...@cloudera.com>
> > Sent: Monday, May 21, 2018 9:08:28 AM
> > To: Hyukjin Kwon
> > Cc: Dongjoon Hyun; dev
> > Subject: Re: Running lint-java during PR builds?
> >
> > I'm fine with it. I tried to use the existing checkstyle sbt plugin
> > (trying to fix SPARK-22269), but it depends on an ancient version of
> > checkstyle, and I don't know sbt enough to figure out how to hack
> > classpaths and class loaders when applying rules, so gave up.
> >
> > On Mon, May 21, 2018 at 1:47 AM, Hyukjin Kwon <gu...@gmail.com>
> wrote:
> >> I am going to open an INFRA JIRA if there's no explicit objection in few
> >> days.
> >>
> >> 2018-05-21 13:09 GMT+08:00 Hyukjin Kwon <gu...@gmail.com>:
> >>>
> >>> I would like to revive this proposal. Travis CI. Shall we give this
> try?
> >>> I
> >>> think it's worth trying it.
> >>>
> >>> 2016-11-17 3:50 GMT+08:00 Dongjoon Hyun <do...@apache.org>:
> >>>>
> >>>> Hi, Marcelo and Ryan.
> >>>>
> >>>> That was the main purpose of my proposal about Travis.CI.
> >>>> IMO, that is the only way to achieve that without any harmful
> >>>> side-effect
> >>>> on Jenkins infra.
> >>>>
> >>>> Spark is already ready for that. Like AppVoyer, if one of you files an
> >>>> INFRA jira issue to enable that, they will turn on that. Then, we can
> >>>> try it
> >>>> and see the result. Also, you can turn off easily again if you don't
> >>>> want.
> >>>>
> >>>> Without this, we will consume more community efforts. For example, we
> >>>> merged lint-java error fix PR seven hours ago, but the master branch
> >>>> still
> >>>> has one lint-java error.
> >>>>
> >>>> https://travis-ci.org/dongjoon-hyun/spark/jobs/176351319
> >>>>
> >>>> Actually, I've been monitoring the history here. (It's synced every 30
> >>>> minutes.)
> >>>>
> >>>> https://travis-ci.org/dongjoon-hyun/spark/builds
> >>>>
> >>>> Could we give a change to this?
> >>>>
> >>>> Bests,
> >>>> Dongjoon.
> >>>>
> >>>> On 2016-11-15 13:40 (-0800), "Shixiong(Ryan) Zhu"
> >>>> <sh...@databricks.com> wrote:
> >>>> > I remember it's because you need to run `mvn install` before running
> >>>> > lint-java if the maven cache is empty, and `mvn install` is pretty
> >>>> > heavy.
> >>>> >
> >>>> > On Tue, Nov 15, 2016 at 1:21 PM, Marcelo Vanzin <
> vanzin@cloudera.com>
> >>>> > wrote:
> >>>> >
> >>>> > > Hey all,
> >>>> > >
> >>>> > > Is there a reason why lint-java is not run during PR builds? I see
> >>>> > > it
> >>>> > > seems to be maven-only, is it really expensive to run after an sbt
> >>>> > > build?
> >>>> > >
> >>>> > > I see a lot of PRs coming in to fix Java style issues, and those
> all
> >>>> > > seem a little unnecessary. Either we're enforcing style checks or
> >>>> > > we're not, and right now it seems we aren't.
> >>>> > >
> >>>> > > --
> >>>> > > Marcelo
> >>>> > >
> >>>> > >
> >>>> > > ------------------------------------------------------------
> ---------
> >>>> > > To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
> >>>> > >
> >>>> > >
> >>>> >
> >>>>
> >>>> ---------------------------------------------------------------------
> >>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
> >>>>
> >>>
> >>
> >
> >
> >
> > --
> > Marcelo
> >
> > ---------------------------------------------------------------------
> > To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
> >
>
>
>
> --
> Marcelo
>
Re: Running lint-java during PR builds?
Posted by Marcelo Vanzin <va...@cloudera.com>.
Is there a way to trigger it conditionally? e.g. only if the diff
touches java files.
On Mon, May 21, 2018 at 9:17 AM, Felix Cheung <fe...@hotmail.com> wrote:
> One concern is with the volume of test runs on Travis.
>
> In ASF projects Travis could get significantly
> backed up since - if I recall - all of ASF shares one queue.
>
> At the number of PRs Spark has this could be a big issue.
>
>
> ________________________________
> From: Marcelo Vanzin <va...@cloudera.com>
> Sent: Monday, May 21, 2018 9:08:28 AM
> To: Hyukjin Kwon
> Cc: Dongjoon Hyun; dev
> Subject: Re: Running lint-java during PR builds?
>
> I'm fine with it. I tried to use the existing checkstyle sbt plugin
> (trying to fix SPARK-22269), but it depends on an ancient version of
> checkstyle, and I don't know sbt enough to figure out how to hack
> classpaths and class loaders when applying rules, so gave up.
>
> On Mon, May 21, 2018 at 1:47 AM, Hyukjin Kwon <gu...@gmail.com> wrote:
>> I am going to open an INFRA JIRA if there's no explicit objection in few
>> days.
>>
>> 2018-05-21 13:09 GMT+08:00 Hyukjin Kwon <gu...@gmail.com>:
>>>
>>> I would like to revive this proposal. Travis CI. Shall we give this try?
>>> I
>>> think it's worth trying it.
>>>
>>> 2016-11-17 3:50 GMT+08:00 Dongjoon Hyun <do...@apache.org>:
>>>>
>>>> Hi, Marcelo and Ryan.
>>>>
>>>> That was the main purpose of my proposal about Travis.CI.
>>>> IMO, that is the only way to achieve that without any harmful
>>>> side-effect
>>>> on Jenkins infra.
>>>>
>>>> Spark is already ready for that. Like AppVoyer, if one of you files an
>>>> INFRA jira issue to enable that, they will turn on that. Then, we can
>>>> try it
>>>> and see the result. Also, you can turn off easily again if you don't
>>>> want.
>>>>
>>>> Without this, we will consume more community efforts. For example, we
>>>> merged lint-java error fix PR seven hours ago, but the master branch
>>>> still
>>>> has one lint-java error.
>>>>
>>>> https://travis-ci.org/dongjoon-hyun/spark/jobs/176351319
>>>>
>>>> Actually, I've been monitoring the history here. (It's synced every 30
>>>> minutes.)
>>>>
>>>> https://travis-ci.org/dongjoon-hyun/spark/builds
>>>>
>>>> Could we give a change to this?
>>>>
>>>> Bests,
>>>> Dongjoon.
>>>>
>>>> On 2016-11-15 13:40 (-0800), "Shixiong(Ryan) Zhu"
>>>> <sh...@databricks.com> wrote:
>>>> > I remember it's because you need to run `mvn install` before running
>>>> > lint-java if the maven cache is empty, and `mvn install` is pretty
>>>> > heavy.
>>>> >
>>>> > On Tue, Nov 15, 2016 at 1:21 PM, Marcelo Vanzin <va...@cloudera.com>
>>>> > wrote:
>>>> >
>>>> > > Hey all,
>>>> > >
>>>> > > Is there a reason why lint-java is not run during PR builds? I see
>>>> > > it
>>>> > > seems to be maven-only, is it really expensive to run after an sbt
>>>> > > build?
>>>> > >
>>>> > > I see a lot of PRs coming in to fix Java style issues, and those all
>>>> > > seem a little unnecessary. Either we're enforcing style checks or
>>>> > > we're not, and right now it seems we aren't.
>>>> > >
>>>> > > --
>>>> > > Marcelo
>>>> > >
>>>> > >
>>>> > > ---------------------------------------------------------------------
>>>> > > To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>> > >
>>>> > >
>>>> >
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>>
>>>
>>
>
>
>
> --
> Marcelo
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
--
Marcelo
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: Running lint-java during PR builds?
Posted by Felix Cheung <fe...@hotmail.com>.
One concern is with the volume of test runs on Travis.
In ASF projects Travis could get significantly
backed up since - if I recall - all of ASF shares one queue.
At the number of PRs Spark has this could be a big issue.
________________________________
From: Marcelo Vanzin <va...@cloudera.com>
Sent: Monday, May 21, 2018 9:08:28 AM
To: Hyukjin Kwon
Cc: Dongjoon Hyun; dev
Subject: Re: Running lint-java during PR builds?
I'm fine with it. I tried to use the existing checkstyle sbt plugin
(trying to fix SPARK-22269), but it depends on an ancient version of
checkstyle, and I don't know sbt enough to figure out how to hack
classpaths and class loaders when applying rules, so gave up.
On Mon, May 21, 2018 at 1:47 AM, Hyukjin Kwon <gu...@gmail.com> wrote:
> I am going to open an INFRA JIRA if there's no explicit objection in few
> days.
>
> 2018-05-21 13:09 GMT+08:00 Hyukjin Kwon <gu...@gmail.com>:
>>
>> I would like to revive this proposal. Travis CI. Shall we give this try? I
>> think it's worth trying it.
>>
>> 2016-11-17 3:50 GMT+08:00 Dongjoon Hyun <do...@apache.org>:
>>>
>>> Hi, Marcelo and Ryan.
>>>
>>> That was the main purpose of my proposal about Travis.CI.
>>> IMO, that is the only way to achieve that without any harmful side-effect
>>> on Jenkins infra.
>>>
>>> Spark is already ready for that. Like AppVoyer, if one of you files an
>>> INFRA jira issue to enable that, they will turn on that. Then, we can try it
>>> and see the result. Also, you can turn off easily again if you don't want.
>>>
>>> Without this, we will consume more community efforts. For example, we
>>> merged lint-java error fix PR seven hours ago, but the master branch still
>>> has one lint-java error.
>>>
>>> https://travis-ci.org/dongjoon-hyun/spark/jobs/176351319
>>>
>>> Actually, I've been monitoring the history here. (It's synced every 30
>>> minutes.)
>>>
>>> https://travis-ci.org/dongjoon-hyun/spark/builds
>>>
>>> Could we give a change to this?
>>>
>>> Bests,
>>> Dongjoon.
>>>
>>> On 2016-11-15 13:40 (-0800), "Shixiong(Ryan) Zhu"
>>> <sh...@databricks.com> wrote:
>>> > I remember it's because you need to run `mvn install` before running
>>> > lint-java if the maven cache is empty, and `mvn install` is pretty
>>> > heavy.
>>> >
>>> > On Tue, Nov 15, 2016 at 1:21 PM, Marcelo Vanzin <va...@cloudera.com>
>>> > wrote:
>>> >
>>> > > Hey all,
>>> > >
>>> > > Is there a reason why lint-java is not run during PR builds? I see it
>>> > > seems to be maven-only, is it really expensive to run after an sbt
>>> > > build?
>>> > >
>>> > > I see a lot of PRs coming in to fix Java style issues, and those all
>>> > > seem a little unnecessary. Either we're enforcing style checks or
>>> > > we're not, and right now it seems we aren't.
>>> > >
>>> > > --
>>> > > Marcelo
>>> > >
>>> > > ---------------------------------------------------------------------
>>> > > To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>> > >
>>> > >
>>> >
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>
>>
>
--
Marcelo
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: Running lint-java during PR builds?
Posted by Marcelo Vanzin <va...@cloudera.com>.
I'm fine with it. I tried to use the existing checkstyle sbt plugin
(trying to fix SPARK-22269), but it depends on an ancient version of
checkstyle, and I don't know sbt enough to figure out how to hack
classpaths and class loaders when applying rules, so gave up.
On Mon, May 21, 2018 at 1:47 AM, Hyukjin Kwon <gu...@gmail.com> wrote:
> I am going to open an INFRA JIRA if there's no explicit objection in few
> days.
>
> 2018-05-21 13:09 GMT+08:00 Hyukjin Kwon <gu...@gmail.com>:
>>
>> I would like to revive this proposal. Travis CI. Shall we give this try? I
>> think it's worth trying it.
>>
>> 2016-11-17 3:50 GMT+08:00 Dongjoon Hyun <do...@apache.org>:
>>>
>>> Hi, Marcelo and Ryan.
>>>
>>> That was the main purpose of my proposal about Travis.CI.
>>> IMO, that is the only way to achieve that without any harmful side-effect
>>> on Jenkins infra.
>>>
>>> Spark is already ready for that. Like AppVoyer, if one of you files an
>>> INFRA jira issue to enable that, they will turn on that. Then, we can try it
>>> and see the result. Also, you can turn off easily again if you don't want.
>>>
>>> Without this, we will consume more community efforts. For example, we
>>> merged lint-java error fix PR seven hours ago, but the master branch still
>>> has one lint-java error.
>>>
>>> https://travis-ci.org/dongjoon-hyun/spark/jobs/176351319
>>>
>>> Actually, I've been monitoring the history here. (It's synced every 30
>>> minutes.)
>>>
>>> https://travis-ci.org/dongjoon-hyun/spark/builds
>>>
>>> Could we give a change to this?
>>>
>>> Bests,
>>> Dongjoon.
>>>
>>> On 2016-11-15 13:40 (-0800), "Shixiong(Ryan) Zhu"
>>> <sh...@databricks.com> wrote:
>>> > I remember it's because you need to run `mvn install` before running
>>> > lint-java if the maven cache is empty, and `mvn install` is pretty
>>> > heavy.
>>> >
>>> > On Tue, Nov 15, 2016 at 1:21 PM, Marcelo Vanzin <va...@cloudera.com>
>>> > wrote:
>>> >
>>> > > Hey all,
>>> > >
>>> > > Is there a reason why lint-java is not run during PR builds? I see it
>>> > > seems to be maven-only, is it really expensive to run after an sbt
>>> > > build?
>>> > >
>>> > > I see a lot of PRs coming in to fix Java style issues, and those all
>>> > > seem a little unnecessary. Either we're enforcing style checks or
>>> > > we're not, and right now it seems we aren't.
>>> > >
>>> > > --
>>> > > Marcelo
>>> > >
>>> > > ---------------------------------------------------------------------
>>> > > To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>> > >
>>> > >
>>> >
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>
>>
>
--
Marcelo
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: Running lint-java during PR builds?
Posted by Hyukjin Kwon <gu...@gmail.com>.
I am going to open an INFRA JIRA if there's no explicit objection in few
days.
2018-05-21 13:09 GMT+08:00 Hyukjin Kwon <gu...@gmail.com>:
> I would like to revive this proposal. Travis CI. Shall we give this try? I
> think it's worth trying it.
>
> 2016-11-17 3:50 GMT+08:00 Dongjoon Hyun <do...@apache.org>:
>
>> Hi, Marcelo and Ryan.
>>
>> That was the main purpose of my proposal about Travis.CI.
>> IMO, that is the only way to achieve that without any harmful side-effect
>> on Jenkins infra.
>>
>> Spark is already ready for that. Like AppVoyer, if one of you files an
>> INFRA jira issue to enable that, they will turn on that. Then, we can try
>> it and see the result. Also, you can turn off easily again if you don't
>> want.
>>
>> Without this, we will consume more community efforts. For example, we
>> merged lint-java error fix PR seven hours ago, but the master branch still
>> has one lint-java error.
>>
>> https://travis-ci.org/dongjoon-hyun/spark/jobs/176351319
>>
>> Actually, I've been monitoring the history here. (It's synced every 30
>> minutes.)
>>
>> https://travis-ci.org/dongjoon-hyun/spark/builds
>>
>> Could we give a change to this?
>>
>> Bests,
>> Dongjoon.
>>
>> On 2016-11-15 13:40 (-0800), "Shixiong(Ryan) Zhu" <
>> shixiong@databricks.com> wrote:
>> > I remember it's because you need to run `mvn install` before running
>> > lint-java if the maven cache is empty, and `mvn install` is pretty
>> heavy.
>> >
>> > On Tue, Nov 15, 2016 at 1:21 PM, Marcelo Vanzin <va...@cloudera.com>
>> wrote:
>> >
>> > > Hey all,
>> > >
>> > > Is there a reason why lint-java is not run during PR builds? I see it
>> > > seems to be maven-only, is it really expensive to run after an sbt
>> > > build?
>> > >
>> > > I see a lot of PRs coming in to fix Java style issues, and those all
>> > > seem a little unnecessary. Either we're enforcing style checks or
>> > > we're not, and right now it seems we aren't.
>> > >
>> > > --
>> > > Marcelo
>> > >
>> > > ---------------------------------------------------------------------
>> > > To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>> > >
>> > >
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>
>>
>
Re: Running lint-java during PR builds?
Posted by Hyukjin Kwon <gu...@gmail.com>.
I would like to revive this proposal. Travis CI. Shall we give this try? I
think it's worth trying it.
2016-11-17 3:50 GMT+08:00 Dongjoon Hyun <do...@apache.org>:
> Hi, Marcelo and Ryan.
>
> That was the main purpose of my proposal about Travis.CI.
> IMO, that is the only way to achieve that without any harmful side-effect
> on Jenkins infra.
>
> Spark is already ready for that. Like AppVoyer, if one of you files an
> INFRA jira issue to enable that, they will turn on that. Then, we can try
> it and see the result. Also, you can turn off easily again if you don't
> want.
>
> Without this, we will consume more community efforts. For example, we
> merged lint-java error fix PR seven hours ago, but the master branch still
> has one lint-java error.
>
> https://travis-ci.org/dongjoon-hyun/spark/jobs/176351319
>
> Actually, I've been monitoring the history here. (It's synced every 30
> minutes.)
>
> https://travis-ci.org/dongjoon-hyun/spark/builds
>
> Could we give a change to this?
>
> Bests,
> Dongjoon.
>
> On 2016-11-15 13:40 (-0800), "Shixiong(Ryan) Zhu" <sh...@databricks.com>
> wrote:
> > I remember it's because you need to run `mvn install` before running
> > lint-java if the maven cache is empty, and `mvn install` is pretty heavy.
> >
> > On Tue, Nov 15, 2016 at 1:21 PM, Marcelo Vanzin <va...@cloudera.com>
> wrote:
> >
> > > Hey all,
> > >
> > > Is there a reason why lint-java is not run during PR builds? I see it
> > > seems to be maven-only, is it really expensive to run after an sbt
> > > build?
> > >
> > > I see a lot of PRs coming in to fix Java style issues, and those all
> > > seem a little unnecessary. Either we're enforcing style checks or
> > > we're not, and right now it seems we aren't.
> > >
> > > --
> > > Marcelo
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
> > >
> > >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>
Re: Running lint-java during PR builds?
Posted by Dongjoon Hyun <do...@apache.org>.
Hi, Marcelo and Ryan.
That was the main purpose of my proposal about Travis.CI.
IMO, that is the only way to achieve that without any harmful side-effect on Jenkins infra.
Spark is already ready for that. Like AppVoyer, if one of you files an INFRA jira issue to enable that, they will turn on that. Then, we can try it and see the result. Also, you can turn off easily again if you don't want.
Without this, we will consume more community efforts. For example, we merged lint-java error fix PR seven hours ago, but the master branch still has one lint-java error.
https://travis-ci.org/dongjoon-hyun/spark/jobs/176351319
Actually, I've been monitoring the history here. (It's synced every 30 minutes.)
https://travis-ci.org/dongjoon-hyun/spark/builds
Could we give a change to this?
Bests,
Dongjoon.
On 2016-11-15 13:40 (-0800), "Shixiong(Ryan) Zhu" <sh...@databricks.com> wrote:
> I remember it's because you need to run `mvn install` before running
> lint-java if the maven cache is empty, and `mvn install` is pretty heavy.
>
> On Tue, Nov 15, 2016 at 1:21 PM, Marcelo Vanzin <va...@cloudera.com> wrote:
>
> > Hey all,
> >
> > Is there a reason why lint-java is not run during PR builds? I see it
> > seems to be maven-only, is it really expensive to run after an sbt
> > build?
> >
> > I see a lot of PRs coming in to fix Java style issues, and those all
> > seem a little unnecessary. Either we're enforcing style checks or
> > we're not, and right now it seems we aren't.
> >
> > --
> > Marcelo
> >
> > ---------------------------------------------------------------------
> > To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
> >
> >
>
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: Running lint-java during PR builds?
Posted by "Shixiong(Ryan) Zhu" <sh...@databricks.com>.
I remember it's because you need to run `mvn install` before running
lint-java if the maven cache is empty, and `mvn install` is pretty heavy.
On Tue, Nov 15, 2016 at 1:21 PM, Marcelo Vanzin <va...@cloudera.com> wrote:
> Hey all,
>
> Is there a reason why lint-java is not run during PR builds? I see it
> seems to be maven-only, is it really expensive to run after an sbt
> build?
>
> I see a lot of PRs coming in to fix Java style issues, and those all
> seem a little unnecessary. Either we're enforcing style checks or
> we're not, and right now it seems we aren't.
>
> --
> Marcelo
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>