You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Dongjoon Hyun <do...@apache.org> on 2016/05/22 20:25:04 UTC
Using Travis for JDK7/8 compilation and lint-java.
Hi, All.
I want to propose the followings.
- Turn on Travis CI for Apache Spark PR queue.
- Recommend this for contributors, too
Currently, Spark provides Travis CI configuration file to help contributors
check Scala/Java style conformance and JDK7/8 compilation easily during
their preparing pull requests. Please note that it's only about static
analysis.
- For Oracle JDK7, mvn -DskipTests install and run `dev/lint-java`.
- For Oracle JDK7, mvn -DskipTests install and run `dev/lint-java`.
Scalastyle is included in the step 'mvn install', too.
Yep, if you turn on your Travis CI configuration, you can already see the
results on your branches before making PR. I wrote this email to prevent
more failures proactively and community-widely.
For stability, I have been monitoring that for two weeks. It detects the
failures or recovery on JDK7 builds or Java linter on Spark master branch
correctly. The only exceptional case I observed rarely is `timeout` failure
due to hangs of maven. But, as we know, it's happen in our Jenkins
SparkPullRequestBuilder, too. I think we can ignore that.
I'm sure that this will save much more community's efforts on the static
errors by preventing them at the very early stage. But, there might be
another reason not to do this. I'm wondering about your thoughts.
I can make a Apache INFRA Jira issue for this if there is some consensus.
Warmly,
Dongjoon.
Re: Using Travis for JDK7/8 compilation and lint-java.
Posted by Hyukjin Kwon <gu...@gmail.com>.
+1 - I wouldn't be bothered if a build becomes longer if I can write
cleaner codes without manually running it.
I have just looked though the related PRs and JIRAs and it looks generally
okay and reasonable to me.
2016-05-23 18:54 GMT+09:00 Steve Loughran <st...@hortonworks.com>:
>
> On 23 May 2016, at 05:21, Dongjoon Hyun <do...@apache.org> wrote:
>
> Thank you for feedback. Sure, correctly, that's the reason why the current
> SparkPullRequestBuilder do not run `lint-java`. :-)
>
> In addition, that's the same reason why contributors are reluctant to run
> `lint-java` and causes breaking on JDK7 builds.
>
> Such a tedious and time-consuming job should be done by CI without human
> interventions.
>
> By the way, why do you think we need to wait for that? We should not wait
> for any CIs, we should continue our own work.
>
>
>
> +1
>
> Any time you spend waiting for tests to complete is time you could be
> doing useful things. I've had VMs running jenkins watching git branches do
> this in the past: every time I push
>
>
>
> My proposal isn't for making you wait to watch the result. There are two
> use cases I want for us to focus here.
>
> Case 1: When you make a PR to Spark PR queue.
>
> Travis CI will finish before SparkPullRequestBuilder.
> We will run the followings in parallel mode.
> 1. Current SparkPullRequestBuilder: JDK8 + sbt build + (no Java
> Linter)
> 2. Travis: JDK7 + mvn build + Java Linter
> 3. Travis: JDK8 + mvn build + Java Linter
> As we know, 1 is the longest time-consuming one which have lots of
> works (except maven building or lint- java). You don't need to wait more
> in many cases. Yes, in many cases, not all the cases.
>
>
> Case 2: When you prepare a PR on your branch.
>
> If you are at the final commit (maybe already-squashed), just go to
> case 1.
>
> However, usually, we makes lots of commits locally while making
> preparing our PR.
> And, finally we squashed them into one and send a PR to Spark.
> I mean you can use Travis CI during preparing your PRs.
> Again, don't wait for Travis CI. Just push it sometime or at every
> commit, and continue your work.
>
> At the final stage when you finish your coding, squash your commits
> into one,
> and amend your commit title or messages, see the Travis CI.
> Or, you can monitor Travis CI result on status menu bar.
> If it shows green icon, you have nothing to do.
>
> https://docs.travis-ci.com/user/apps/
>
> To sum up, I think we don't need to wait for any CIs. It's like an email.
> `Send and back to work.`
>
>
>
> I'd add another, which is "do a build and test of this patch while I get
> on with something else, that is, things which aren't ready for review, just
> the work you've done in the past hour or two which you'd like tested out
>
Re: Using Travis for JDK7/8 compilation and lint-java.
Posted by Steve Loughran <st...@hortonworks.com>.
On 23 May 2016, at 05:21, Dongjoon Hyun <do...@apache.org>> wrote:
Thank you for feedback. Sure, correctly, that's the reason why the current SparkPullRequestBuilder do not run `lint-java`. :-)
In addition, that's the same reason why contributors are reluctant to run `lint-java` and causes breaking on JDK7 builds.
Such a tedious and time-consuming job should be done by CI without human interventions.
By the way, why do you think we need to wait for that? We should not wait for any CIs, we should continue our own work.
+1
Any time you spend waiting for tests to complete is time you could be doing useful things. I've had VMs running jenkins watching git branches do this in the past: every time I push
My proposal isn't for making you wait to watch the result. There are two use cases I want for us to focus here.
Case 1: When you make a PR to Spark PR queue.
Travis CI will finish before SparkPullRequestBuilder.
We will run the followings in parallel mode.
1. Current SparkPullRequestBuilder: JDK8 + sbt build + (no Java Linter)
2. Travis: JDK7 + mvn build + Java Linter
3. Travis: JDK8 + mvn build + Java Linter
As we know, 1 is the longest time-consuming one which have lots of works (except maven building or lint- java). You don't need to wait more in many cases. Yes, in many cases, not all the cases.
Case 2: When you prepare a PR on your branch.
If you are at the final commit (maybe already-squashed), just go to case 1.
However, usually, we makes lots of commits locally while making preparing our PR.
And, finally we squashed them into one and send a PR to Spark.
I mean you can use Travis CI during preparing your PRs.
Again, don't wait for Travis CI. Just push it sometime or at every commit, and continue your work.
At the final stage when you finish your coding, squash your commits into one,
and amend your commit title or messages, see the Travis CI.
Or, you can monitor Travis CI result on status menu bar.
If it shows green icon, you have nothing to do.
https://docs.travis-ci.com/user/apps/
To sum up, I think we don't need to wait for any CIs. It's like an email. `Send and back to work.`
I'd add another, which is "do a build and test of this patch while I get on with something else, that is, things which aren't ready for review, just the work you've done in the past hour or two which you'd like tested out
Re: Using Travis for JDK7/8 compilation and lint-java.
Posted by shane knapp <sk...@berkeley.edu>.
chiming in, as i'm the one who currently maintains the CI infrastructure... :)
+1 on not having more than one CI system... there's no way i can
commit to keeping an eye on anything else other than jenkins.
and i agree wholeheartedly w/michael: if it's this important, let's
add it to the jenkins builds.
On Mon, May 23, 2016 at 12:20 PM, Michael Armbrust
<mi...@databricks.com> wrote:
> We did turn on travis a few years ago, but ended up turning it off because
> it was failing (I believe because of insufficient resources) which was
> confusing for developers. I wouldn't be opposed to turning it on if it
> provides more/faster signal, but its not obvious to me that it would. In
> particular, do we know that given the rate PRs are created if we will hit
> rate limits?
>
> Really my main feedback is, if the java linter is important we should
> probably have it as part of the canonical build process. I worry about
> having more than one set of CI infrastructure to maintain.
>
> On Mon, May 23, 2016 at 9:43 AM, Dongjoon Hyun <do...@apache.org> wrote:
>>
>> Thank you, Steve and Hyukjin.
>>
>> And, don't worry, Ted.
>>
>> Travis launches new VMs for every PR.
>>
>> Apache Spark repository uses the following setting.
>>
>> VM: Google Compute Engine
>> OS: Ubuntu 14.04.3 LTS Server Edition 64bit
>> CPU: ~2 CORE
>> RAM: 7.5GB
>>
>> FYI, you can find more information about this here.
>>
>>
>> https://docs.travis-ci.com/user/ci-environment/#Virtualization-environments
>>
>> Dongjoon.
>>
>>
>>
>> On Mon, May 23, 2016 at 6:32 AM, Ted Yu <yu...@gmail.com> wrote:
>>>
>>> Do you know if more than one PR would be verified on the same machine ?
>>>
>>> I wonder whether the 'mvn install' from two simultaneous PR builds may
>>> have conflict.
>>>
>>> On Sun, May 22, 2016 at 9:21 PM, Dongjoon Hyun <do...@apache.org>
>>> wrote:
>>>>
>>>> Thank you for feedback. Sure, correctly, that's the reason why the
>>>> current SparkPullRequestBuilder do not run `lint-java`. :-)
>>>>
>>>> In addition, that's the same reason why contributors are reluctant to
>>>> run `lint-java` and causes breaking on JDK7 builds.
>>>>
>>>> Such a tedious and time-consuming job should be done by CI without human
>>>> interventions.
>>>>
>>>> By the way, why do you think we need to wait for that? We should not
>>>> wait for any CIs, we should continue our own work.
>>>>
>>>> My proposal isn't for making you wait to watch the result. There are two
>>>> use cases I want for us to focus here.
>>>>
>>>> Case 1: When you make a PR to Spark PR queue.
>>>>
>>>> Travis CI will finish before SparkPullRequestBuilder.
>>>> We will run the followings in parallel mode.
>>>> 1. Current SparkPullRequestBuilder: JDK8 + sbt build + (no Java
>>>> Linter)
>>>> 2. Travis: JDK7 + mvn build + Java Linter
>>>> 3. Travis: JDK8 + mvn build + Java Linter
>>>> As we know, 1 is the longest time-consuming one which have lots of
>>>> works (except maven building or lint- java). You don't need to wait more in
>>>> many cases. Yes, in many cases, not all the cases.
>>>>
>>>>
>>>> Case 2: When you prepare a PR on your branch.
>>>>
>>>> If you are at the final commit (maybe already-squashed), just go to
>>>> case 1.
>>>>
>>>> However, usually, we makes lots of commits locally while making
>>>> preparing our PR.
>>>> And, finally we squashed them into one and send a PR to Spark.
>>>> I mean you can use Travis CI during preparing your PRs.
>>>> Again, don't wait for Travis CI. Just push it sometime or at every
>>>> commit, and continue your work.
>>>>
>>>> At the final stage when you finish your coding, squash your commits
>>>> into one,
>>>> and amend your commit title or messages, see the Travis CI.
>>>> Or, you can monitor Travis CI result on status menu bar.
>>>> If it shows green icon, you have nothing to do.
>>>>
>>>> https://docs.travis-ci.com/user/apps/
>>>>
>>>> To sum up, I think we don't need to wait for any CIs. It's like an
>>>> email. `Send and back to work.`
>>>>
>>>> Dongjoon.
>>>>
>>>>
>>>> On Sun, May 22, 2016 at 8:32 PM, Ted Yu <yu...@gmail.com> wrote:
>>>>>
>>>>> Without Zinc, 'mvn -DskipTests clean install' takes ~30 minutes.
>>>>>
>>>>> Maybe not everyone is willing to wait that long.
>>>>>
>>>>> On Sun, May 22, 2016 at 1:30 PM, Dongjoon Hyun <do...@apache.org>
>>>>> wrote:
>>>>>>
>>>>>> Oh, Sure. My bad!
>>>>>>
>>>>>> - For Oracle JDK7, mvn -DskipTests install and run `dev/lint-java`.
>>>>>> - For Oracle JDK8, mvn -DskipTests install and run `dev/lint-java`.
>>>>>>
>>>>>> Thank you, Ted.
>>>>>>
>>>>>> Dongjoon.
>>>>>>
>>>>>> On Sun, May 22, 2016 at 1:29 PM, Ted Yu <yu...@gmail.com> wrote:
>>>>>>>
>>>>>>> The following line was repeated twice:
>>>>>>>
>>>>>>> - For Oracle JDK7, mvn -DskipTests install and run `dev/lint-java`.
>>>>>>>
>>>>>>> Did you intend to cover JDK 8 ?
>>>>>>>
>>>>>>> Cheers
>>>>>>>
>>>>>>> On Sun, May 22, 2016 at 1:25 PM, Dongjoon Hyun <do...@apache.org>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Hi, All.
>>>>>>>>
>>>>>>>> I want to propose the followings.
>>>>>>>>
>>>>>>>> - Turn on Travis CI for Apache Spark PR queue.
>>>>>>>> - Recommend this for contributors, too
>>>>>>>>
>>>>>>>> Currently, Spark provides Travis CI configuration file to help
>>>>>>>> contributors check Scala/Java style conformance and JDK7/8 compilation
>>>>>>>> easily during their preparing pull requests. Please note that it's only
>>>>>>>> about static analysis.
>>>>>>>>
>>>>>>>> - For Oracle JDK7, mvn -DskipTests install and run `dev/lint-java`.
>>>>>>>> - For Oracle JDK7, mvn -DskipTests install and run `dev/lint-java`.
>>>>>>>> Scalastyle is included in the step 'mvn install', too.
>>>>>>>>
>>>>>>>> Yep, if you turn on your Travis CI configuration, you can already
>>>>>>>> see the results on your branches before making PR. I wrote this email to
>>>>>>>> prevent more failures proactively and community-widely.
>>>>>>>>
>>>>>>>> For stability, I have been monitoring that for two weeks. It detects
>>>>>>>> the failures or recovery on JDK7 builds or Java linter on Spark master
>>>>>>>> branch correctly. The only exceptional case I observed rarely is `timeout`
>>>>>>>> failure due to hangs of maven. But, as we know, it's happen in our Jenkins
>>>>>>>> SparkPullRequestBuilder, too. I think we can ignore that.
>>>>>>>>
>>>>>>>> I'm sure that this will save much more community's efforts on the
>>>>>>>> static errors by preventing them at the very early stage. But, there might
>>>>>>>> be another reason not to do this. I'm wondering about your thoughts.
>>>>>>>>
>>>>>>>> I can make a Apache INFRA Jira issue for this if there is some
>>>>>>>> consensus.
>>>>>>>>
>>>>>>>> Warmly,
>>>>>>>> Dongjoon.
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: Using Travis for JDK7/8 compilation and lint-java.
Posted by Dongjoon Hyun <do...@apache.org>.
Yep. Let's hold on. :)
On Tue, May 24, 2016 at 3:45 PM, shane knapp <sk...@berkeley.edu> wrote:
> > Sure, could you give me the permission for Spark Jira?
> >
> > Although we haven't decided yet, I can add Travis related section
> > (summarizing current configurations and expected VM HW, etc).
> >
> i can't give you permissions -- that has to be (most likely) through
> someone @ databricks, like michael.
>
> let's hold off on adding a section until we actually decide that this
> is critical and something that cannot be done currently w/jenkins.
>
> > It's just a Wiki, you can delete the Travis Section anytime.
> >
> ayup.
>
Re: Using Travis for JDK7/8 compilation and lint-java.
Posted by shane knapp <sk...@berkeley.edu>.
> Another clarification: not databricks, but the Apache Spark PMC grants
> access to the JIRA / wiki. That said... I'm not actually sure how its done.
word. i'll make the changes if we need to.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: Using Travis for JDK7/8 compilation and lint-java.
Posted by Michael Armbrust <mi...@databricks.com>.
>
> i can't give you permissions -- that has to be (most likely) through
> someone @ databricks, like michael.
>
Another clarification: not databricks, but the Apache Spark PMC grants
access to the JIRA / wiki. That said... I'm not actually sure how its done.
Re: Using Travis for JDK7/8 compilation and lint-java.
Posted by shane knapp <sk...@berkeley.edu>.
> Sure, could you give me the permission for Spark Jira?
>
> Although we haven't decided yet, I can add Travis related section
> (summarizing current configurations and expected VM HW, etc).
>
i can't give you permissions -- that has to be (most likely) through
someone @ databricks, like michael.
let's hold off on adding a section until we actually decide that this
is critical and something that cannot be done currently w/jenkins.
> It's just a Wiki, you can delete the Travis Section anytime.
>
ayup.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: Using Travis for JDK7/8 compilation and lint-java.
Posted by Dongjoon Hyun <do...@apache.org>.
Thank you, Shane.
Sure, could you give me the permission for Spark Jira?
Although we haven't decided yet, I can add Travis related section
(summarizing current configurations and expected VM HW, etc).
That will be helpful for further discussions.
It's just a Wiki, you can delete the Travis Section anytime.
Dongjoon.
On Tue, May 24, 2016 at 12:15 PM, shane knapp <sk...@berkeley.edu> wrote:
> > As Sean said, Vanzin made a PR for JDK7 compilation. We can ignore the
> issue
> > of JDK7 compilation.
> >
> vanzin and i are working together on this right now... we currently
> have java 7u79 installed on all of the workers. if some random test
> failures keep happening during his tests, i will roll out 7u80 (which
> is known to be 'good') and set that as the default java on all of the
> workers (which is currently 7-79).
>
> > To: Shane
> > For Travis CI, I volunteer for turning on/off and taking care about
> > complains during that test period.
> >
> > I hope these two helps you make a positive decision on Travis CI test
> drive.
> > Any other potential issues we should consider?
> >
> if this is the route we decide to go, could you please update the qa
> infra wiki entry and add a relevant section for the travis setup?
>
> https://cwiki.apache.org/confluence/display/SPARK/Spark+QA+Infrastructure
>
> thanks,
>
> shane
>
Re: Using Travis for JDK7/8 compilation and lint-java.
Posted by shane knapp <sk...@berkeley.edu>.
> As Sean said, Vanzin made a PR for JDK7 compilation. We can ignore the issue
> of JDK7 compilation.
>
vanzin and i are working together on this right now... we currently
have java 7u79 installed on all of the workers. if some random test
failures keep happening during his tests, i will roll out 7u80 (which
is known to be 'good') and set that as the default java on all of the
workers (which is currently 7-79).
> To: Shane
> For Travis CI, I volunteer for turning on/off and taking care about
> complains during that test period.
>
> I hope these two helps you make a positive decision on Travis CI test drive.
> Any other potential issues we should consider?
>
if this is the route we decide to go, could you please update the qa
infra wiki entry and add a relevant section for the travis setup?
https://cwiki.apache.org/confluence/display/SPARK/Spark+QA+Infrastructure
thanks,
shane
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: Using Travis for JDK7/8 compilation and lint-java.
Posted by Dongjoon Hyun <do...@apache.org>.
Hi, All.
As Sean said, Vanzin made a PR for JDK7 compilation. We can ignore the
issue of JDK7 compilation.
The remaining issues are the java-linter and maven installation test.
To: Michael
For the rate limit, Apache Foundation seems to use 30 concurrent according
to the INFRA blog.
https://blogs.apache.org/infra/
However, this does not include the Travis CI queue in personal users.
We have over 800 github registered contributors.
They can run Travis CI on their branch without this limitation like me.
To: Shane
For Travis CI, I volunteer for turning on/off and taking care about
complains during that test period.
I hope these two helps you make a positive decision on Travis CI test drive.
Any other potential issues we should consider?
Dongjoon.
Re: Using Travis for JDK7/8 compilation and lint-java.
Posted by Dongjoon Hyun <do...@apache.org>.
Thank you, Sean!
On Mon, May 23, 2016 at 2:09 PM, Sean Owen <so...@cloudera.com> wrote:
> No, because then none of the Java 8 support can build. Marcelo has a JIRA
> for handling that the right way with bootstrap class path config.
>
> Ideally it can be rolled into Jenkins though there are possibly historical
> reasons it was not enabled before. Best to fix those if possible but if not
> I'd rather have some automated checking than none. Checking lint is
> reasonably important.
>
> On Mon, May 23, 2016, 15:58 Ted Yu <yu...@gmail.com> wrote:
>
>> For #1 below, currently Jenkins uses Java 8:
>>
>> JAVA_HOME=/usr/java/jdk1.8.0_60
>>
>>
>> How about switching to Java 7 ?
>>
>>
>> My two cents.
>>
>>
>>
Re: Using Travis for JDK7/8 compilation and lint-java.
Posted by Sean Owen <so...@cloudera.com>.
No, because then none of the Java 8 support can build. Marcelo has a JIRA
for handling that the right way with bootstrap class path config.
Ideally it can be rolled into Jenkins though there are possibly historical
reasons it was not enabled before. Best to fix those if possible but if not
I'd rather have some automated checking than none. Checking lint is
reasonably important.
On Mon, May 23, 2016, 15:58 Ted Yu <yu...@gmail.com> wrote:
> For #1 below, currently Jenkins uses Java 8:
>
> JAVA_HOME=/usr/java/jdk1.8.0_60
>
>
> How about switching to Java 7 ?
>
>
> My two cents.
>
>
>
Re: Using Travis for JDK7/8 compilation and lint-java.
Posted by Ted Yu <yu...@gmail.com>.
For #1 below, currently Jenkins uses Java 8:
JAVA_HOME=/usr/java/jdk1.8.0_60
How about switching to Java 7 ?
My two cents.
On Mon, May 23, 2016 at 1:24 PM, Dongjoon Hyun <do...@apache.org> wrote:
> Thank you for your opinion!
>
> Sure. I know that history and totally agree with all your concerns.
> I indeed has hesitated about sending this kind of suggestion for a while.
>
> If Travis CI cannot handle those simple jobs at this time again,
> we must turn off from Spark PR queue.
> We can see the result quickly in one or two days.
> To turn on/off, Spark have nothing to do. INFRA team will do that.
>
> In fact, the goal is not about using another CI (like Travis), it is about
> preventing the followings.
>
> 1. JDK7 compilation errors. (Recently, 2 days ago and 5 days ago)
> 2. Java static errors. (Not critical but more frequently.)
> 3. Maven installation errors. (A month ago, it's reported in this mailing
> list.)
>
> Scala 2.10 compilation errors are fixed nearly instantly. But, 1~3 were
> not.
> If SparkPullRequestBuilder can do the above 1~3, that's the best for us.
> Do you think it is possible in some ways?
>
> By the way, as of today, Spark has 724 Java files and 96762 lines (without
> comment/blank).
> It's about 1/3 of Scala code. It's not small.
> --------------------------------------------------------------------------
> Language files blank comment code
> --------------------------------------------------------------------------
> Scala 2368 63578 124904 322518
> Java 724 18569 23445
> 96762
>
> Dongjoon.
>
>
>
> On Mon, May 23, 2016 at 12:20 PM, Michael Armbrust <michael@databricks.com
> > wrote:
>
>> We did turn on travis a few years ago, but ended up turning it off
>> because it was failing (I believe because of insufficient resources) which
>> was confusing for developers. I wouldn't be opposed to turning it on if it
>> provides more/faster signal, but its not obvious to me that it would. In
>> particular, do we know that given the rate PRs are created if we will hit
>> rate limits?
>>
>> Really my main feedback is, if the java linter is important we should
>> probably have it as part of the canonical build process. I worry about
>> having more than one set of CI infrastructure to maintain.
>>
>> On Mon, May 23, 2016 at 9:43 AM, Dongjoon Hyun <do...@apache.org>
>> wrote:
>>
>>> Thank you, Steve and Hyukjin.
>>>
>>> And, don't worry, Ted.
>>>
>>> Travis launches new VMs for every PR.
>>>
>>> Apache Spark repository uses the following setting.
>>>
>>> VM: Google Compute Engine
>>> OS: Ubuntu 14.04.3 LTS Server Edition 64bit
>>> CPU: ~2 CORE
>>> RAM: 7.5GB
>>>
>>> FYI, you can find more information about this here.
>>>
>>>
>>> https://docs.travis-ci.com/user/ci-environment/#Virtualization-environments
>>>
>>> Dongjoon.
>>>
>>>
>>>
>>> On Mon, May 23, 2016 at 6:32 AM, Ted Yu <yu...@gmail.com> wrote:
>>>
>>>> Do you know if more than one PR would be verified on the same machine ?
>>>>
>>>> I wonder whether the 'mvn install' from two simultaneous PR builds may
>>>> have conflict.
>>>>
>>>> On Sun, May 22, 2016 at 9:21 PM, Dongjoon Hyun <do...@apache.org>
>>>> wrote:
>>>>
>>>>> Thank you for feedback. Sure, correctly, that's the reason why the
>>>>> current SparkPullRequestBuilder do not run `lint-java`. :-)
>>>>>
>>>>> In addition, that's the same reason why contributors are reluctant to
>>>>> run `lint-java` and causes breaking on JDK7 builds.
>>>>>
>>>>> Such a tedious and time-consuming job should be done by CI without
>>>>> human interventions.
>>>>>
>>>>> By the way, why do you think we need to wait for that? We should not
>>>>> wait for any CIs, we should continue our own work.
>>>>>
>>>>> My proposal isn't for making you wait to watch the result. There are
>>>>> two use cases I want for us to focus here.
>>>>>
>>>>> Case 1: When you make a PR to Spark PR queue.
>>>>>
>>>>> Travis CI will finish before SparkPullRequestBuilder.
>>>>> We will run the followings in parallel mode.
>>>>> 1. Current SparkPullRequestBuilder: JDK8 + sbt build + (no
>>>>> Java Linter)
>>>>> 2. Travis: JDK7 + mvn build + Java Linter
>>>>> 3. Travis: JDK8 + mvn build + Java Linter
>>>>> As we know, 1 is the longest time-consuming one which have lots
>>>>> of works (except maven building or lint- java). You don't need to wait
>>>>> more in many cases. Yes, in many cases, not all the cases.
>>>>>
>>>>>
>>>>> Case 2: When you prepare a PR on your branch.
>>>>>
>>>>> If you are at the final commit (maybe already-squashed), just go
>>>>> to case 1.
>>>>>
>>>>> However, usually, we makes lots of commits locally while making
>>>>> preparing our PR.
>>>>> And, finally we squashed them into one and send a PR to Spark.
>>>>> I mean you can use Travis CI during preparing your PRs.
>>>>> Again, don't wait for Travis CI. Just push it sometime or at every
>>>>> commit, and continue your work.
>>>>>
>>>>> At the final stage when you finish your coding, squash your
>>>>> commits into one,
>>>>> and amend your commit title or messages, see the Travis CI.
>>>>> Or, you can monitor Travis CI result on status menu bar.
>>>>> If it shows green icon, you have nothing to do.
>>>>>
>>>>> https://docs.travis-ci.com/user/apps/
>>>>>
>>>>> To sum up, I think we don't need to wait for any CIs. It's like an
>>>>> email. `Send and back to work.`
>>>>>
>>>>> Dongjoon.
>>>>>
>>>>>
>>>>> On Sun, May 22, 2016 at 8:32 PM, Ted Yu <yu...@gmail.com> wrote:
>>>>>
>>>>>> Without Zinc, 'mvn -DskipTests clean install' takes ~30 minutes.
>>>>>>
>>>>>> Maybe not everyone is willing to wait that long.
>>>>>>
>>>>>> On Sun, May 22, 2016 at 1:30 PM, Dongjoon Hyun <do...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>>> Oh, Sure. My bad!
>>>>>>>
>>>>>>> - For Oracle JDK7, mvn -DskipTests install and run `dev/lint-java`.
>>>>>>> - For Oracle JDK8, mvn -DskipTests install and run `dev/lint-java`.
>>>>>>>
>>>>>>> Thank you, Ted.
>>>>>>>
>>>>>>> Dongjoon.
>>>>>>>
>>>>>>> On Sun, May 22, 2016 at 1:29 PM, Ted Yu <yu...@gmail.com> wrote:
>>>>>>>
>>>>>>>> The following line was repeated twice:
>>>>>>>>
>>>>>>>> - For Oracle JDK7, mvn -DskipTests install and run `dev/lint-java`.
>>>>>>>>
>>>>>>>> Did you intend to cover JDK 8 ?
>>>>>>>>
>>>>>>>> Cheers
>>>>>>>>
>>>>>>>> On Sun, May 22, 2016 at 1:25 PM, Dongjoon Hyun <dongjoon@apache.org
>>>>>>>> > wrote:
>>>>>>>>
>>>>>>>>> Hi, All.
>>>>>>>>>
>>>>>>>>> I want to propose the followings.
>>>>>>>>>
>>>>>>>>> - Turn on Travis CI for Apache Spark PR queue.
>>>>>>>>> - Recommend this for contributors, too
>>>>>>>>>
>>>>>>>>> Currently, Spark provides Travis CI configuration file to help
>>>>>>>>> contributors check Scala/Java style conformance and JDK7/8 compilation
>>>>>>>>> easily during their preparing pull requests. Please note that it's only
>>>>>>>>> about static analysis.
>>>>>>>>>
>>>>>>>>> - For Oracle JDK7, mvn -DskipTests install and run `dev/lint-java`.
>>>>>>>>> - For Oracle JDK7, mvn -DskipTests install and run `dev/lint-java`.
>>>>>>>>> Scalastyle is included in the step 'mvn install', too.
>>>>>>>>>
>>>>>>>>> Yep, if you turn on your Travis CI configuration, you can already
>>>>>>>>> see the results on your branches before making PR. I wrote this email to
>>>>>>>>> prevent more failures proactively and community-widely.
>>>>>>>>>
>>>>>>>>> For stability, I have been monitoring that for two weeks. It
>>>>>>>>> detects the failures or recovery on JDK7 builds or Java linter on Spark
>>>>>>>>> master branch correctly. The only exceptional case I observed rarely is
>>>>>>>>> `timeout` failure due to hangs of maven. But, as we know, it's happen in
>>>>>>>>> our Jenkins SparkPullRequestBuilder, too. I think we can ignore that.
>>>>>>>>>
>>>>>>>>> I'm sure that this will save much more community's efforts on the
>>>>>>>>> static errors by preventing them at the very early stage. But, there might
>>>>>>>>> be another reason not to do this. I'm wondering about your thoughts.
>>>>>>>>>
>>>>>>>>> I can make a Apache INFRA Jira issue for this if there is some
>>>>>>>>> consensus.
>>>>>>>>>
>>>>>>>>> Warmly,
>>>>>>>>> Dongjoon.
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Using Travis for JDK7/8 compilation and lint-java.
Posted by Dongjoon Hyun <do...@apache.org>.
I want to clarify something here.
For Travis CI, it's free for open source projects and there is only one
management point, `.travis.xml`, for Spark community.
It's not some like physical Jenkins cluster farm. It's just a cloud service
like Github.
PS.
I'm also not an employee of Travis(or Github). :-)
If Spark uses Travis CI freely, they might dislike me for the heavy traffic.
On Mon, May 23, 2016 at 1:26 PM, Dongjoon Hyun <do...@apache.org> wrote:
> Thank you, Shane!
>
> I really hope that SparkPullRequestBuilder handle them if possible.
>
> Dongjoon.
>
> On Mon, May 23, 2016 at 1:24 PM, Dongjoon Hyun <do...@apache.org>
> wrote:
>
>> Thank you for your opinion!
>>
>> Sure. I know that history and totally agree with all your concerns.
>> I indeed has hesitated about sending this kind of suggestion for a while.
>>
>> If Travis CI cannot handle those simple jobs at this time again,
>> we must turn off from Spark PR queue.
>> We can see the result quickly in one or two days.
>> To turn on/off, Spark have nothing to do. INFRA team will do that.
>>
>> In fact, the goal is not about using another CI (like Travis), it is
>> about preventing the followings.
>>
>> 1. JDK7 compilation errors. (Recently, 2 days ago and 5 days ago)
>> 2. Java static errors. (Not critical but more frequently.)
>> 3. Maven installation errors. (A month ago, it's reported in this mailing
>> list.)
>>
>> Scala 2.10 compilation errors are fixed nearly instantly. But, 1~3 were
>> not.
>> If SparkPullRequestBuilder can do the above 1~3, that's the best for us.
>> Do you think it is possible in some ways?
>>
>> By the way, as of today, Spark has 724 Java files and 96762 lines
>> (without comment/blank).
>> It's about 1/3 of Scala code. It's not small.
>> --------------------------------------------------------------------------
>> Language files blank comment code
>> --------------------------------------------------------------------------
>> Scala 2368 63578 124904 322518
>> Java 724 18569 23445
>> 96762
>>
>> Dongjoon.
>>
>>
>>
>> On Mon, May 23, 2016 at 12:20 PM, Michael Armbrust <
>> michael@databricks.com> wrote:
>>
>>> We did turn on travis a few years ago, but ended up turning it off
>>> because it was failing (I believe because of insufficient resources) which
>>> was confusing for developers. I wouldn't be opposed to turning it on if it
>>> provides more/faster signal, but its not obvious to me that it would. In
>>> particular, do we know that given the rate PRs are created if we will hit
>>> rate limits?
>>>
>>> Really my main feedback is, if the java linter is important we should
>>> probably have it as part of the canonical build process. I worry about
>>> having more than one set of CI infrastructure to maintain.
>>>
>>> On Mon, May 23, 2016 at 9:43 AM, Dongjoon Hyun <do...@apache.org>
>>> wrote:
>>>
>>>> Thank you, Steve and Hyukjin.
>>>>
>>>> And, don't worry, Ted.
>>>>
>>>> Travis launches new VMs for every PR.
>>>>
>>>> Apache Spark repository uses the following setting.
>>>>
>>>> VM: Google Compute Engine
>>>> OS: Ubuntu 14.04.3 LTS Server Edition 64bit
>>>> CPU: ~2 CORE
>>>> RAM: 7.5GB
>>>>
>>>> FYI, you can find more information about this here.
>>>>
>>>>
>>>> https://docs.travis-ci.com/user/ci-environment/#Virtualization-environments
>>>>
>>>> Dongjoon.
>>>>
>>>>
>>>>
>>>> On Mon, May 23, 2016 at 6:32 AM, Ted Yu <yu...@gmail.com> wrote:
>>>>
>>>>> Do you know if more than one PR would be verified on the same machine ?
>>>>>
>>>>> I wonder whether the 'mvn install' from two simultaneous PR builds may
>>>>> have conflict.
>>>>>
>>>>> On Sun, May 22, 2016 at 9:21 PM, Dongjoon Hyun <do...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> Thank you for feedback. Sure, correctly, that's the reason why the
>>>>>> current SparkPullRequestBuilder do not run `lint-java`. :-)
>>>>>>
>>>>>> In addition, that's the same reason why contributors are reluctant to
>>>>>> run `lint-java` and causes breaking on JDK7 builds.
>>>>>>
>>>>>> Such a tedious and time-consuming job should be done by CI without
>>>>>> human interventions.
>>>>>>
>>>>>> By the way, why do you think we need to wait for that? We should not
>>>>>> wait for any CIs, we should continue our own work.
>>>>>>
>>>>>> My proposal isn't for making you wait to watch the result. There are
>>>>>> two use cases I want for us to focus here.
>>>>>>
>>>>>> Case 1: When you make a PR to Spark PR queue.
>>>>>>
>>>>>> Travis CI will finish before SparkPullRequestBuilder.
>>>>>> We will run the followings in parallel mode.
>>>>>> 1. Current SparkPullRequestBuilder: JDK8 + sbt build + (no
>>>>>> Java Linter)
>>>>>> 2. Travis: JDK7 + mvn build + Java Linter
>>>>>> 3. Travis: JDK8 + mvn build + Java Linter
>>>>>> As we know, 1 is the longest time-consuming one which have lots
>>>>>> of works (except maven building or lint- java). You don't need to wait
>>>>>> more in many cases. Yes, in many cases, not all the cases.
>>>>>>
>>>>>>
>>>>>> Case 2: When you prepare a PR on your branch.
>>>>>>
>>>>>> If you are at the final commit (maybe already-squashed), just go
>>>>>> to case 1.
>>>>>>
>>>>>> However, usually, we makes lots of commits locally while making
>>>>>> preparing our PR.
>>>>>> And, finally we squashed them into one and send a PR to Spark.
>>>>>> I mean you can use Travis CI during preparing your PRs.
>>>>>> Again, don't wait for Travis CI. Just push it sometime or at
>>>>>> every commit, and continue your work.
>>>>>>
>>>>>> At the final stage when you finish your coding, squash your
>>>>>> commits into one,
>>>>>> and amend your commit title or messages, see the Travis CI.
>>>>>> Or, you can monitor Travis CI result on status menu bar.
>>>>>> If it shows green icon, you have nothing to do.
>>>>>>
>>>>>> https://docs.travis-ci.com/user/apps/
>>>>>>
>>>>>> To sum up, I think we don't need to wait for any CIs. It's like an
>>>>>> email. `Send and back to work.`
>>>>>>
>>>>>> Dongjoon.
>>>>>>
>>>>>>
>>>>>> On Sun, May 22, 2016 at 8:32 PM, Ted Yu <yu...@gmail.com> wrote:
>>>>>>
>>>>>>> Without Zinc, 'mvn -DskipTests clean install' takes ~30 minutes.
>>>>>>>
>>>>>>> Maybe not everyone is willing to wait that long.
>>>>>>>
>>>>>>> On Sun, May 22, 2016 at 1:30 PM, Dongjoon Hyun <do...@apache.org>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Oh, Sure. My bad!
>>>>>>>>
>>>>>>>> - For Oracle JDK7, mvn -DskipTests install and run `dev/lint-java`.
>>>>>>>> - For Oracle JDK8, mvn -DskipTests install and run `dev/lint-java`.
>>>>>>>>
>>>>>>>> Thank you, Ted.
>>>>>>>>
>>>>>>>> Dongjoon.
>>>>>>>>
>>>>>>>> On Sun, May 22, 2016 at 1:29 PM, Ted Yu <yu...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> The following line was repeated twice:
>>>>>>>>>
>>>>>>>>> - For Oracle JDK7, mvn -DskipTests install and run `dev/lint-java`.
>>>>>>>>>
>>>>>>>>> Did you intend to cover JDK 8 ?
>>>>>>>>>
>>>>>>>>> Cheers
>>>>>>>>>
>>>>>>>>> On Sun, May 22, 2016 at 1:25 PM, Dongjoon Hyun <
>>>>>>>>> dongjoon@apache.org> wrote:
>>>>>>>>>
>>>>>>>>>> Hi, All.
>>>>>>>>>>
>>>>>>>>>> I want to propose the followings.
>>>>>>>>>>
>>>>>>>>>> - Turn on Travis CI for Apache Spark PR queue.
>>>>>>>>>> - Recommend this for contributors, too
>>>>>>>>>>
>>>>>>>>>> Currently, Spark provides Travis CI configuration file to help
>>>>>>>>>> contributors check Scala/Java style conformance and JDK7/8 compilation
>>>>>>>>>> easily during their preparing pull requests. Please note that it's only
>>>>>>>>>> about static analysis.
>>>>>>>>>>
>>>>>>>>>> - For Oracle JDK7, mvn -DskipTests install and run
>>>>>>>>>> `dev/lint-java`.
>>>>>>>>>> - For Oracle JDK7, mvn -DskipTests install and run
>>>>>>>>>> `dev/lint-java`.
>>>>>>>>>> Scalastyle is included in the step 'mvn install', too.
>>>>>>>>>>
>>>>>>>>>> Yep, if you turn on your Travis CI configuration, you can already
>>>>>>>>>> see the results on your branches before making PR. I wrote this email to
>>>>>>>>>> prevent more failures proactively and community-widely.
>>>>>>>>>>
>>>>>>>>>> For stability, I have been monitoring that for two weeks. It
>>>>>>>>>> detects the failures or recovery on JDK7 builds or Java linter on Spark
>>>>>>>>>> master branch correctly. The only exceptional case I observed rarely is
>>>>>>>>>> `timeout` failure due to hangs of maven. But, as we know, it's happen in
>>>>>>>>>> our Jenkins SparkPullRequestBuilder, too. I think we can ignore that.
>>>>>>>>>>
>>>>>>>>>> I'm sure that this will save much more community's efforts on the
>>>>>>>>>> static errors by preventing them at the very early stage. But, there might
>>>>>>>>>> be another reason not to do this. I'm wondering about your thoughts.
>>>>>>>>>>
>>>>>>>>>> I can make a Apache INFRA Jira issue for this if there is some
>>>>>>>>>> consensus.
>>>>>>>>>>
>>>>>>>>>> Warmly,
>>>>>>>>>> Dongjoon.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Using Travis for JDK7/8 compilation and lint-java.
Posted by Dongjoon Hyun <do...@apache.org>.
Thank you, Shane!
I really hope that SparkPullRequestBuilder handle them if possible.
Dongjoon.
On Mon, May 23, 2016 at 1:24 PM, Dongjoon Hyun <do...@apache.org> wrote:
> Thank you for your opinion!
>
> Sure. I know that history and totally agree with all your concerns.
> I indeed has hesitated about sending this kind of suggestion for a while.
>
> If Travis CI cannot handle those simple jobs at this time again,
> we must turn off from Spark PR queue.
> We can see the result quickly in one or two days.
> To turn on/off, Spark have nothing to do. INFRA team will do that.
>
> In fact, the goal is not about using another CI (like Travis), it is about
> preventing the followings.
>
> 1. JDK7 compilation errors. (Recently, 2 days ago and 5 days ago)
> 2. Java static errors. (Not critical but more frequently.)
> 3. Maven installation errors. (A month ago, it's reported in this mailing
> list.)
>
> Scala 2.10 compilation errors are fixed nearly instantly. But, 1~3 were
> not.
> If SparkPullRequestBuilder can do the above 1~3, that's the best for us.
> Do you think it is possible in some ways?
>
> By the way, as of today, Spark has 724 Java files and 96762 lines (without
> comment/blank).
> It's about 1/3 of Scala code. It's not small.
> --------------------------------------------------------------------------
> Language files blank comment code
> --------------------------------------------------------------------------
> Scala 2368 63578 124904 322518
> Java 724 18569 23445
> 96762
>
> Dongjoon.
>
>
>
> On Mon, May 23, 2016 at 12:20 PM, Michael Armbrust <michael@databricks.com
> > wrote:
>
>> We did turn on travis a few years ago, but ended up turning it off
>> because it was failing (I believe because of insufficient resources) which
>> was confusing for developers. I wouldn't be opposed to turning it on if it
>> provides more/faster signal, but its not obvious to me that it would. In
>> particular, do we know that given the rate PRs are created if we will hit
>> rate limits?
>>
>> Really my main feedback is, if the java linter is important we should
>> probably have it as part of the canonical build process. I worry about
>> having more than one set of CI infrastructure to maintain.
>>
>> On Mon, May 23, 2016 at 9:43 AM, Dongjoon Hyun <do...@apache.org>
>> wrote:
>>
>>> Thank you, Steve and Hyukjin.
>>>
>>> And, don't worry, Ted.
>>>
>>> Travis launches new VMs for every PR.
>>>
>>> Apache Spark repository uses the following setting.
>>>
>>> VM: Google Compute Engine
>>> OS: Ubuntu 14.04.3 LTS Server Edition 64bit
>>> CPU: ~2 CORE
>>> RAM: 7.5GB
>>>
>>> FYI, you can find more information about this here.
>>>
>>>
>>> https://docs.travis-ci.com/user/ci-environment/#Virtualization-environments
>>>
>>> Dongjoon.
>>>
>>>
>>>
>>> On Mon, May 23, 2016 at 6:32 AM, Ted Yu <yu...@gmail.com> wrote:
>>>
>>>> Do you know if more than one PR would be verified on the same machine ?
>>>>
>>>> I wonder whether the 'mvn install' from two simultaneous PR builds may
>>>> have conflict.
>>>>
>>>> On Sun, May 22, 2016 at 9:21 PM, Dongjoon Hyun <do...@apache.org>
>>>> wrote:
>>>>
>>>>> Thank you for feedback. Sure, correctly, that's the reason why the
>>>>> current SparkPullRequestBuilder do not run `lint-java`. :-)
>>>>>
>>>>> In addition, that's the same reason why contributors are reluctant to
>>>>> run `lint-java` and causes breaking on JDK7 builds.
>>>>>
>>>>> Such a tedious and time-consuming job should be done by CI without
>>>>> human interventions.
>>>>>
>>>>> By the way, why do you think we need to wait for that? We should not
>>>>> wait for any CIs, we should continue our own work.
>>>>>
>>>>> My proposal isn't for making you wait to watch the result. There are
>>>>> two use cases I want for us to focus here.
>>>>>
>>>>> Case 1: When you make a PR to Spark PR queue.
>>>>>
>>>>> Travis CI will finish before SparkPullRequestBuilder.
>>>>> We will run the followings in parallel mode.
>>>>> 1. Current SparkPullRequestBuilder: JDK8 + sbt build + (no
>>>>> Java Linter)
>>>>> 2. Travis: JDK7 + mvn build + Java Linter
>>>>> 3. Travis: JDK8 + mvn build + Java Linter
>>>>> As we know, 1 is the longest time-consuming one which have lots
>>>>> of works (except maven building or lint- java). You don't need to wait
>>>>> more in many cases. Yes, in many cases, not all the cases.
>>>>>
>>>>>
>>>>> Case 2: When you prepare a PR on your branch.
>>>>>
>>>>> If you are at the final commit (maybe already-squashed), just go
>>>>> to case 1.
>>>>>
>>>>> However, usually, we makes lots of commits locally while making
>>>>> preparing our PR.
>>>>> And, finally we squashed them into one and send a PR to Spark.
>>>>> I mean you can use Travis CI during preparing your PRs.
>>>>> Again, don't wait for Travis CI. Just push it sometime or at every
>>>>> commit, and continue your work.
>>>>>
>>>>> At the final stage when you finish your coding, squash your
>>>>> commits into one,
>>>>> and amend your commit title or messages, see the Travis CI.
>>>>> Or, you can monitor Travis CI result on status menu bar.
>>>>> If it shows green icon, you have nothing to do.
>>>>>
>>>>> https://docs.travis-ci.com/user/apps/
>>>>>
>>>>> To sum up, I think we don't need to wait for any CIs. It's like an
>>>>> email. `Send and back to work.`
>>>>>
>>>>> Dongjoon.
>>>>>
>>>>>
>>>>> On Sun, May 22, 2016 at 8:32 PM, Ted Yu <yu...@gmail.com> wrote:
>>>>>
>>>>>> Without Zinc, 'mvn -DskipTests clean install' takes ~30 minutes.
>>>>>>
>>>>>> Maybe not everyone is willing to wait that long.
>>>>>>
>>>>>> On Sun, May 22, 2016 at 1:30 PM, Dongjoon Hyun <do...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>>> Oh, Sure. My bad!
>>>>>>>
>>>>>>> - For Oracle JDK7, mvn -DskipTests install and run `dev/lint-java`.
>>>>>>> - For Oracle JDK8, mvn -DskipTests install and run `dev/lint-java`.
>>>>>>>
>>>>>>> Thank you, Ted.
>>>>>>>
>>>>>>> Dongjoon.
>>>>>>>
>>>>>>> On Sun, May 22, 2016 at 1:29 PM, Ted Yu <yu...@gmail.com> wrote:
>>>>>>>
>>>>>>>> The following line was repeated twice:
>>>>>>>>
>>>>>>>> - For Oracle JDK7, mvn -DskipTests install and run `dev/lint-java`.
>>>>>>>>
>>>>>>>> Did you intend to cover JDK 8 ?
>>>>>>>>
>>>>>>>> Cheers
>>>>>>>>
>>>>>>>> On Sun, May 22, 2016 at 1:25 PM, Dongjoon Hyun <dongjoon@apache.org
>>>>>>>> > wrote:
>>>>>>>>
>>>>>>>>> Hi, All.
>>>>>>>>>
>>>>>>>>> I want to propose the followings.
>>>>>>>>>
>>>>>>>>> - Turn on Travis CI for Apache Spark PR queue.
>>>>>>>>> - Recommend this for contributors, too
>>>>>>>>>
>>>>>>>>> Currently, Spark provides Travis CI configuration file to help
>>>>>>>>> contributors check Scala/Java style conformance and JDK7/8 compilation
>>>>>>>>> easily during their preparing pull requests. Please note that it's only
>>>>>>>>> about static analysis.
>>>>>>>>>
>>>>>>>>> - For Oracle JDK7, mvn -DskipTests install and run `dev/lint-java`.
>>>>>>>>> - For Oracle JDK7, mvn -DskipTests install and run `dev/lint-java`.
>>>>>>>>> Scalastyle is included in the step 'mvn install', too.
>>>>>>>>>
>>>>>>>>> Yep, if you turn on your Travis CI configuration, you can already
>>>>>>>>> see the results on your branches before making PR. I wrote this email to
>>>>>>>>> prevent more failures proactively and community-widely.
>>>>>>>>>
>>>>>>>>> For stability, I have been monitoring that for two weeks. It
>>>>>>>>> detects the failures or recovery on JDK7 builds or Java linter on Spark
>>>>>>>>> master branch correctly. The only exceptional case I observed rarely is
>>>>>>>>> `timeout` failure due to hangs of maven. But, as we know, it's happen in
>>>>>>>>> our Jenkins SparkPullRequestBuilder, too. I think we can ignore that.
>>>>>>>>>
>>>>>>>>> I'm sure that this will save much more community's efforts on the
>>>>>>>>> static errors by preventing them at the very early stage. But, there might
>>>>>>>>> be another reason not to do this. I'm wondering about your thoughts.
>>>>>>>>>
>>>>>>>>> I can make a Apache INFRA Jira issue for this if there is some
>>>>>>>>> consensus.
>>>>>>>>>
>>>>>>>>> Warmly,
>>>>>>>>> Dongjoon.
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Using Travis for JDK7/8 compilation and lint-java.
Posted by Dongjoon Hyun <do...@apache.org>.
Thank you for your opinion!
Sure. I know that history and totally agree with all your concerns.
I indeed has hesitated about sending this kind of suggestion for a while.
If Travis CI cannot handle those simple jobs at this time again,
we must turn off from Spark PR queue.
We can see the result quickly in one or two days.
To turn on/off, Spark have nothing to do. INFRA team will do that.
In fact, the goal is not about using another CI (like Travis), it is about
preventing the followings.
1. JDK7 compilation errors. (Recently, 2 days ago and 5 days ago)
2. Java static errors. (Not critical but more frequently.)
3. Maven installation errors. (A month ago, it's reported in this mailing
list.)
Scala 2.10 compilation errors are fixed nearly instantly. But, 1~3 were not.
If SparkPullRequestBuilder can do the above 1~3, that's the best for us.
Do you think it is possible in some ways?
By the way, as of today, Spark has 724 Java files and 96762 lines (without
comment/blank).
It's about 1/3 of Scala code. It's not small.
--------------------------------------------------------------------------
Language files blank comment code
--------------------------------------------------------------------------
Scala 2368 63578 124904 322518
Java 724 18569 23445
96762
Dongjoon.
On Mon, May 23, 2016 at 12:20 PM, Michael Armbrust <mi...@databricks.com>
wrote:
> We did turn on travis a few years ago, but ended up turning it off because
> it was failing (I believe because of insufficient resources) which was
> confusing for developers. I wouldn't be opposed to turning it on if it
> provides more/faster signal, but its not obvious to me that it would. In
> particular, do we know that given the rate PRs are created if we will hit
> rate limits?
>
> Really my main feedback is, if the java linter is important we should
> probably have it as part of the canonical build process. I worry about
> having more than one set of CI infrastructure to maintain.
>
> On Mon, May 23, 2016 at 9:43 AM, Dongjoon Hyun <do...@apache.org>
> wrote:
>
>> Thank you, Steve and Hyukjin.
>>
>> And, don't worry, Ted.
>>
>> Travis launches new VMs for every PR.
>>
>> Apache Spark repository uses the following setting.
>>
>> VM: Google Compute Engine
>> OS: Ubuntu 14.04.3 LTS Server Edition 64bit
>> CPU: ~2 CORE
>> RAM: 7.5GB
>>
>> FYI, you can find more information about this here.
>>
>>
>> https://docs.travis-ci.com/user/ci-environment/#Virtualization-environments
>>
>> Dongjoon.
>>
>>
>>
>> On Mon, May 23, 2016 at 6:32 AM, Ted Yu <yu...@gmail.com> wrote:
>>
>>> Do you know if more than one PR would be verified on the same machine ?
>>>
>>> I wonder whether the 'mvn install' from two simultaneous PR builds may
>>> have conflict.
>>>
>>> On Sun, May 22, 2016 at 9:21 PM, Dongjoon Hyun <do...@apache.org>
>>> wrote:
>>>
>>>> Thank you for feedback. Sure, correctly, that's the reason why the
>>>> current SparkPullRequestBuilder do not run `lint-java`. :-)
>>>>
>>>> In addition, that's the same reason why contributors are reluctant to
>>>> run `lint-java` and causes breaking on JDK7 builds.
>>>>
>>>> Such a tedious and time-consuming job should be done by CI without
>>>> human interventions.
>>>>
>>>> By the way, why do you think we need to wait for that? We should not
>>>> wait for any CIs, we should continue our own work.
>>>>
>>>> My proposal isn't for making you wait to watch the result. There are
>>>> two use cases I want for us to focus here.
>>>>
>>>> Case 1: When you make a PR to Spark PR queue.
>>>>
>>>> Travis CI will finish before SparkPullRequestBuilder.
>>>> We will run the followings in parallel mode.
>>>> 1. Current SparkPullRequestBuilder: JDK8 + sbt build + (no
>>>> Java Linter)
>>>> 2. Travis: JDK7 + mvn build + Java Linter
>>>> 3. Travis: JDK8 + mvn build + Java Linter
>>>> As we know, 1 is the longest time-consuming one which have lots of
>>>> works (except maven building or lint- java). You don't need to wait more
>>>> in many cases. Yes, in many cases, not all the cases.
>>>>
>>>>
>>>> Case 2: When you prepare a PR on your branch.
>>>>
>>>> If you are at the final commit (maybe already-squashed), just go to
>>>> case 1.
>>>>
>>>> However, usually, we makes lots of commits locally while making
>>>> preparing our PR.
>>>> And, finally we squashed them into one and send a PR to Spark.
>>>> I mean you can use Travis CI during preparing your PRs.
>>>> Again, don't wait for Travis CI. Just push it sometime or at every
>>>> commit, and continue your work.
>>>>
>>>> At the final stage when you finish your coding, squash your commits
>>>> into one,
>>>> and amend your commit title or messages, see the Travis CI.
>>>> Or, you can monitor Travis CI result on status menu bar.
>>>> If it shows green icon, you have nothing to do.
>>>>
>>>> https://docs.travis-ci.com/user/apps/
>>>>
>>>> To sum up, I think we don't need to wait for any CIs. It's like an
>>>> email. `Send and back to work.`
>>>>
>>>> Dongjoon.
>>>>
>>>>
>>>> On Sun, May 22, 2016 at 8:32 PM, Ted Yu <yu...@gmail.com> wrote:
>>>>
>>>>> Without Zinc, 'mvn -DskipTests clean install' takes ~30 minutes.
>>>>>
>>>>> Maybe not everyone is willing to wait that long.
>>>>>
>>>>> On Sun, May 22, 2016 at 1:30 PM, Dongjoon Hyun <do...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> Oh, Sure. My bad!
>>>>>>
>>>>>> - For Oracle JDK7, mvn -DskipTests install and run `dev/lint-java`.
>>>>>> - For Oracle JDK8, mvn -DskipTests install and run `dev/lint-java`.
>>>>>>
>>>>>> Thank you, Ted.
>>>>>>
>>>>>> Dongjoon.
>>>>>>
>>>>>> On Sun, May 22, 2016 at 1:29 PM, Ted Yu <yu...@gmail.com> wrote:
>>>>>>
>>>>>>> The following line was repeated twice:
>>>>>>>
>>>>>>> - For Oracle JDK7, mvn -DskipTests install and run `dev/lint-java`.
>>>>>>>
>>>>>>> Did you intend to cover JDK 8 ?
>>>>>>>
>>>>>>> Cheers
>>>>>>>
>>>>>>> On Sun, May 22, 2016 at 1:25 PM, Dongjoon Hyun <do...@apache.org>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi, All.
>>>>>>>>
>>>>>>>> I want to propose the followings.
>>>>>>>>
>>>>>>>> - Turn on Travis CI for Apache Spark PR queue.
>>>>>>>> - Recommend this for contributors, too
>>>>>>>>
>>>>>>>> Currently, Spark provides Travis CI configuration file to help
>>>>>>>> contributors check Scala/Java style conformance and JDK7/8 compilation
>>>>>>>> easily during their preparing pull requests. Please note that it's only
>>>>>>>> about static analysis.
>>>>>>>>
>>>>>>>> - For Oracle JDK7, mvn -DskipTests install and run `dev/lint-java`.
>>>>>>>> - For Oracle JDK7, mvn -DskipTests install and run `dev/lint-java`.
>>>>>>>> Scalastyle is included in the step 'mvn install', too.
>>>>>>>>
>>>>>>>> Yep, if you turn on your Travis CI configuration, you can already
>>>>>>>> see the results on your branches before making PR. I wrote this email to
>>>>>>>> prevent more failures proactively and community-widely.
>>>>>>>>
>>>>>>>> For stability, I have been monitoring that for two weeks. It
>>>>>>>> detects the failures or recovery on JDK7 builds or Java linter on Spark
>>>>>>>> master branch correctly. The only exceptional case I observed rarely is
>>>>>>>> `timeout` failure due to hangs of maven. But, as we know, it's happen in
>>>>>>>> our Jenkins SparkPullRequestBuilder, too. I think we can ignore that.
>>>>>>>>
>>>>>>>> I'm sure that this will save much more community's efforts on the
>>>>>>>> static errors by preventing them at the very early stage. But, there might
>>>>>>>> be another reason not to do this. I'm wondering about your thoughts.
>>>>>>>>
>>>>>>>> I can make a Apache INFRA Jira issue for this if there is some
>>>>>>>> consensus.
>>>>>>>>
>>>>>>>> Warmly,
>>>>>>>> Dongjoon.
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Using Travis for JDK7/8 compilation and lint-java.
Posted by Michael Armbrust <mi...@databricks.com>.
We did turn on travis a few years ago, but ended up turning it off because
it was failing (I believe because of insufficient resources) which was
confusing for developers. I wouldn't be opposed to turning it on if it
provides more/faster signal, but its not obvious to me that it would. In
particular, do we know that given the rate PRs are created if we will hit
rate limits?
Really my main feedback is, if the java linter is important we should
probably have it as part of the canonical build process. I worry about
having more than one set of CI infrastructure to maintain.
On Mon, May 23, 2016 at 9:43 AM, Dongjoon Hyun <do...@apache.org> wrote:
> Thank you, Steve and Hyukjin.
>
> And, don't worry, Ted.
>
> Travis launches new VMs for every PR.
>
> Apache Spark repository uses the following setting.
>
> VM: Google Compute Engine
> OS: Ubuntu 14.04.3 LTS Server Edition 64bit
> CPU: ~2 CORE
> RAM: 7.5GB
>
> FYI, you can find more information about this here.
>
> https://docs.travis-ci.com/user/ci-environment/#Virtualization-environments
>
> Dongjoon.
>
>
>
> On Mon, May 23, 2016 at 6:32 AM, Ted Yu <yu...@gmail.com> wrote:
>
>> Do you know if more than one PR would be verified on the same machine ?
>>
>> I wonder whether the 'mvn install' from two simultaneous PR builds may
>> have conflict.
>>
>> On Sun, May 22, 2016 at 9:21 PM, Dongjoon Hyun <do...@apache.org>
>> wrote:
>>
>>> Thank you for feedback. Sure, correctly, that's the reason why the
>>> current SparkPullRequestBuilder do not run `lint-java`. :-)
>>>
>>> In addition, that's the same reason why contributors are reluctant to
>>> run `lint-java` and causes breaking on JDK7 builds.
>>>
>>> Such a tedious and time-consuming job should be done by CI without human
>>> interventions.
>>>
>>> By the way, why do you think we need to wait for that? We should not
>>> wait for any CIs, we should continue our own work.
>>>
>>> My proposal isn't for making you wait to watch the result. There are two
>>> use cases I want for us to focus here.
>>>
>>> Case 1: When you make a PR to Spark PR queue.
>>>
>>> Travis CI will finish before SparkPullRequestBuilder.
>>> We will run the followings in parallel mode.
>>> 1. Current SparkPullRequestBuilder: JDK8 + sbt build + (no
>>> Java Linter)
>>> 2. Travis: JDK7 + mvn build + Java Linter
>>> 3. Travis: JDK8 + mvn build + Java Linter
>>> As we know, 1 is the longest time-consuming one which have lots of
>>> works (except maven building or lint- java). You don't need to wait more
>>> in many cases. Yes, in many cases, not all the cases.
>>>
>>>
>>> Case 2: When you prepare a PR on your branch.
>>>
>>> If you are at the final commit (maybe already-squashed), just go to
>>> case 1.
>>>
>>> However, usually, we makes lots of commits locally while making
>>> preparing our PR.
>>> And, finally we squashed them into one and send a PR to Spark.
>>> I mean you can use Travis CI during preparing your PRs.
>>> Again, don't wait for Travis CI. Just push it sometime or at every
>>> commit, and continue your work.
>>>
>>> At the final stage when you finish your coding, squash your commits
>>> into one,
>>> and amend your commit title or messages, see the Travis CI.
>>> Or, you can monitor Travis CI result on status menu bar.
>>> If it shows green icon, you have nothing to do.
>>>
>>> https://docs.travis-ci.com/user/apps/
>>>
>>> To sum up, I think we don't need to wait for any CIs. It's like an
>>> email. `Send and back to work.`
>>>
>>> Dongjoon.
>>>
>>>
>>> On Sun, May 22, 2016 at 8:32 PM, Ted Yu <yu...@gmail.com> wrote:
>>>
>>>> Without Zinc, 'mvn -DskipTests clean install' takes ~30 minutes.
>>>>
>>>> Maybe not everyone is willing to wait that long.
>>>>
>>>> On Sun, May 22, 2016 at 1:30 PM, Dongjoon Hyun <do...@apache.org>
>>>> wrote:
>>>>
>>>>> Oh, Sure. My bad!
>>>>>
>>>>> - For Oracle JDK7, mvn -DskipTests install and run `dev/lint-java`.
>>>>> - For Oracle JDK8, mvn -DskipTests install and run `dev/lint-java`.
>>>>>
>>>>> Thank you, Ted.
>>>>>
>>>>> Dongjoon.
>>>>>
>>>>> On Sun, May 22, 2016 at 1:29 PM, Ted Yu <yu...@gmail.com> wrote:
>>>>>
>>>>>> The following line was repeated twice:
>>>>>>
>>>>>> - For Oracle JDK7, mvn -DskipTests install and run `dev/lint-java`.
>>>>>>
>>>>>> Did you intend to cover JDK 8 ?
>>>>>>
>>>>>> Cheers
>>>>>>
>>>>>> On Sun, May 22, 2016 at 1:25 PM, Dongjoon Hyun <do...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi, All.
>>>>>>>
>>>>>>> I want to propose the followings.
>>>>>>>
>>>>>>> - Turn on Travis CI for Apache Spark PR queue.
>>>>>>> - Recommend this for contributors, too
>>>>>>>
>>>>>>> Currently, Spark provides Travis CI configuration file to help
>>>>>>> contributors check Scala/Java style conformance and JDK7/8 compilation
>>>>>>> easily during their preparing pull requests. Please note that it's only
>>>>>>> about static analysis.
>>>>>>>
>>>>>>> - For Oracle JDK7, mvn -DskipTests install and run `dev/lint-java`.
>>>>>>> - For Oracle JDK7, mvn -DskipTests install and run `dev/lint-java`.
>>>>>>> Scalastyle is included in the step 'mvn install', too.
>>>>>>>
>>>>>>> Yep, if you turn on your Travis CI configuration, you can already
>>>>>>> see the results on your branches before making PR. I wrote this email to
>>>>>>> prevent more failures proactively and community-widely.
>>>>>>>
>>>>>>> For stability, I have been monitoring that for two weeks. It detects
>>>>>>> the failures or recovery on JDK7 builds or Java linter on Spark master
>>>>>>> branch correctly. The only exceptional case I observed rarely is `timeout`
>>>>>>> failure due to hangs of maven. But, as we know, it's happen in our Jenkins
>>>>>>> SparkPullRequestBuilder, too. I think we can ignore that.
>>>>>>>
>>>>>>> I'm sure that this will save much more community's efforts on the
>>>>>>> static errors by preventing them at the very early stage. But, there might
>>>>>>> be another reason not to do this. I'm wondering about your thoughts.
>>>>>>>
>>>>>>> I can make a Apache INFRA Jira issue for this if there is some
>>>>>>> consensus.
>>>>>>>
>>>>>>> Warmly,
>>>>>>> Dongjoon.
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Using Travis for JDK7/8 compilation and lint-java.
Posted by Dongjoon Hyun <do...@apache.org>.
Thank you, Steve and Hyukjin.
And, don't worry, Ted.
Travis launches new VMs for every PR.
Apache Spark repository uses the following setting.
VM: Google Compute Engine
OS: Ubuntu 14.04.3 LTS Server Edition 64bit
CPU: ~2 CORE
RAM: 7.5GB
FYI, you can find more information about this here.
https://docs.travis-ci.com/user/ci-environment/#Virtualization-environments
Dongjoon.
On Mon, May 23, 2016 at 6:32 AM, Ted Yu <yu...@gmail.com> wrote:
> Do you know if more than one PR would be verified on the same machine ?
>
> I wonder whether the 'mvn install' from two simultaneous PR builds may
> have conflict.
>
> On Sun, May 22, 2016 at 9:21 PM, Dongjoon Hyun <do...@apache.org>
> wrote:
>
>> Thank you for feedback. Sure, correctly, that's the reason why the
>> current SparkPullRequestBuilder do not run `lint-java`. :-)
>>
>> In addition, that's the same reason why contributors are reluctant to run
>> `lint-java` and causes breaking on JDK7 builds.
>>
>> Such a tedious and time-consuming job should be done by CI without human
>> interventions.
>>
>> By the way, why do you think we need to wait for that? We should not wait
>> for any CIs, we should continue our own work.
>>
>> My proposal isn't for making you wait to watch the result. There are two
>> use cases I want for us to focus here.
>>
>> Case 1: When you make a PR to Spark PR queue.
>>
>> Travis CI will finish before SparkPullRequestBuilder.
>> We will run the followings in parallel mode.
>> 1. Current SparkPullRequestBuilder: JDK8 + sbt build + (no Java
>> Linter)
>> 2. Travis: JDK7 + mvn build + Java Linter
>> 3. Travis: JDK8 + mvn build + Java Linter
>> As we know, 1 is the longest time-consuming one which have lots of
>> works (except maven building or lint- java). You don't need to wait more
>> in many cases. Yes, in many cases, not all the cases.
>>
>>
>> Case 2: When you prepare a PR on your branch.
>>
>> If you are at the final commit (maybe already-squashed), just go to
>> case 1.
>>
>> However, usually, we makes lots of commits locally while making
>> preparing our PR.
>> And, finally we squashed them into one and send a PR to Spark.
>> I mean you can use Travis CI during preparing your PRs.
>> Again, don't wait for Travis CI. Just push it sometime or at every
>> commit, and continue your work.
>>
>> At the final stage when you finish your coding, squash your commits
>> into one,
>> and amend your commit title or messages, see the Travis CI.
>> Or, you can monitor Travis CI result on status menu bar.
>> If it shows green icon, you have nothing to do.
>>
>> https://docs.travis-ci.com/user/apps/
>>
>> To sum up, I think we don't need to wait for any CIs. It's like an email.
>> `Send and back to work.`
>>
>> Dongjoon.
>>
>>
>> On Sun, May 22, 2016 at 8:32 PM, Ted Yu <yu...@gmail.com> wrote:
>>
>>> Without Zinc, 'mvn -DskipTests clean install' takes ~30 minutes.
>>>
>>> Maybe not everyone is willing to wait that long.
>>>
>>> On Sun, May 22, 2016 at 1:30 PM, Dongjoon Hyun <do...@apache.org>
>>> wrote:
>>>
>>>> Oh, Sure. My bad!
>>>>
>>>> - For Oracle JDK7, mvn -DskipTests install and run `dev/lint-java`.
>>>> - For Oracle JDK8, mvn -DskipTests install and run `dev/lint-java`.
>>>>
>>>> Thank you, Ted.
>>>>
>>>> Dongjoon.
>>>>
>>>> On Sun, May 22, 2016 at 1:29 PM, Ted Yu <yu...@gmail.com> wrote:
>>>>
>>>>> The following line was repeated twice:
>>>>>
>>>>> - For Oracle JDK7, mvn -DskipTests install and run `dev/lint-java`.
>>>>>
>>>>> Did you intend to cover JDK 8 ?
>>>>>
>>>>> Cheers
>>>>>
>>>>> On Sun, May 22, 2016 at 1:25 PM, Dongjoon Hyun <do...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> Hi, All.
>>>>>>
>>>>>> I want to propose the followings.
>>>>>>
>>>>>> - Turn on Travis CI for Apache Spark PR queue.
>>>>>> - Recommend this for contributors, too
>>>>>>
>>>>>> Currently, Spark provides Travis CI configuration file to help
>>>>>> contributors check Scala/Java style conformance and JDK7/8 compilation
>>>>>> easily during their preparing pull requests. Please note that it's only
>>>>>> about static analysis.
>>>>>>
>>>>>> - For Oracle JDK7, mvn -DskipTests install and run `dev/lint-java`.
>>>>>> - For Oracle JDK7, mvn -DskipTests install and run `dev/lint-java`.
>>>>>> Scalastyle is included in the step 'mvn install', too.
>>>>>>
>>>>>> Yep, if you turn on your Travis CI configuration, you can already see
>>>>>> the results on your branches before making PR. I wrote this email to
>>>>>> prevent more failures proactively and community-widely.
>>>>>>
>>>>>> For stability, I have been monitoring that for two weeks. It detects
>>>>>> the failures or recovery on JDK7 builds or Java linter on Spark master
>>>>>> branch correctly. The only exceptional case I observed rarely is `timeout`
>>>>>> failure due to hangs of maven. But, as we know, it's happen in our Jenkins
>>>>>> SparkPullRequestBuilder, too. I think we can ignore that.
>>>>>>
>>>>>> I'm sure that this will save much more community's efforts on the
>>>>>> static errors by preventing them at the very early stage. But, there might
>>>>>> be another reason not to do this. I'm wondering about your thoughts.
>>>>>>
>>>>>> I can make a Apache INFRA Jira issue for this if there is some
>>>>>> consensus.
>>>>>>
>>>>>> Warmly,
>>>>>> Dongjoon.
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: Using Travis for JDK7/8 compilation and lint-java.
Posted by Ted Yu <yu...@gmail.com>.
Do you know if more than one PR would be verified on the same machine ?
I wonder whether the 'mvn install' from two simultaneous PR builds may have
conflict.
On Sun, May 22, 2016 at 9:21 PM, Dongjoon Hyun <do...@apache.org> wrote:
> Thank you for feedback. Sure, correctly, that's the reason why the current
> SparkPullRequestBuilder do not run `lint-java`. :-)
>
> In addition, that's the same reason why contributors are reluctant to run
> `lint-java` and causes breaking on JDK7 builds.
>
> Such a tedious and time-consuming job should be done by CI without human
> interventions.
>
> By the way, why do you think we need to wait for that? We should not wait
> for any CIs, we should continue our own work.
>
> My proposal isn't for making you wait to watch the result. There are two
> use cases I want for us to focus here.
>
> Case 1: When you make a PR to Spark PR queue.
>
> Travis CI will finish before SparkPullRequestBuilder.
> We will run the followings in parallel mode.
> 1. Current SparkPullRequestBuilder: JDK8 + sbt build + (no Java
> Linter)
> 2. Travis: JDK7 + mvn build + Java Linter
> 3. Travis: JDK8 + mvn build + Java Linter
> As we know, 1 is the longest time-consuming one which have lots of
> works (except maven building or lint- java). You don't need to wait more
> in many cases. Yes, in many cases, not all the cases.
>
>
> Case 2: When you prepare a PR on your branch.
>
> If you are at the final commit (maybe already-squashed), just go to
> case 1.
>
> However, usually, we makes lots of commits locally while making
> preparing our PR.
> And, finally we squashed them into one and send a PR to Spark.
> I mean you can use Travis CI during preparing your PRs.
> Again, don't wait for Travis CI. Just push it sometime or at every
> commit, and continue your work.
>
> At the final stage when you finish your coding, squash your commits
> into one,
> and amend your commit title or messages, see the Travis CI.
> Or, you can monitor Travis CI result on status menu bar.
> If it shows green icon, you have nothing to do.
>
> https://docs.travis-ci.com/user/apps/
>
> To sum up, I think we don't need to wait for any CIs. It's like an email.
> `Send and back to work.`
>
> Dongjoon.
>
>
> On Sun, May 22, 2016 at 8:32 PM, Ted Yu <yu...@gmail.com> wrote:
>
>> Without Zinc, 'mvn -DskipTests clean install' takes ~30 minutes.
>>
>> Maybe not everyone is willing to wait that long.
>>
>> On Sun, May 22, 2016 at 1:30 PM, Dongjoon Hyun <do...@apache.org>
>> wrote:
>>
>>> Oh, Sure. My bad!
>>>
>>> - For Oracle JDK7, mvn -DskipTests install and run `dev/lint-java`.
>>> - For Oracle JDK8, mvn -DskipTests install and run `dev/lint-java`.
>>>
>>> Thank you, Ted.
>>>
>>> Dongjoon.
>>>
>>> On Sun, May 22, 2016 at 1:29 PM, Ted Yu <yu...@gmail.com> wrote:
>>>
>>>> The following line was repeated twice:
>>>>
>>>> - For Oracle JDK7, mvn -DskipTests install and run `dev/lint-java`.
>>>>
>>>> Did you intend to cover JDK 8 ?
>>>>
>>>> Cheers
>>>>
>>>> On Sun, May 22, 2016 at 1:25 PM, Dongjoon Hyun <do...@apache.org>
>>>> wrote:
>>>>
>>>>> Hi, All.
>>>>>
>>>>> I want to propose the followings.
>>>>>
>>>>> - Turn on Travis CI for Apache Spark PR queue.
>>>>> - Recommend this for contributors, too
>>>>>
>>>>> Currently, Spark provides Travis CI configuration file to help
>>>>> contributors check Scala/Java style conformance and JDK7/8 compilation
>>>>> easily during their preparing pull requests. Please note that it's only
>>>>> about static analysis.
>>>>>
>>>>> - For Oracle JDK7, mvn -DskipTests install and run `dev/lint-java`.
>>>>> - For Oracle JDK7, mvn -DskipTests install and run `dev/lint-java`.
>>>>> Scalastyle is included in the step 'mvn install', too.
>>>>>
>>>>> Yep, if you turn on your Travis CI configuration, you can already see
>>>>> the results on your branches before making PR. I wrote this email to
>>>>> prevent more failures proactively and community-widely.
>>>>>
>>>>> For stability, I have been monitoring that for two weeks. It detects
>>>>> the failures or recovery on JDK7 builds or Java linter on Spark master
>>>>> branch correctly. The only exceptional case I observed rarely is `timeout`
>>>>> failure due to hangs of maven. But, as we know, it's happen in our Jenkins
>>>>> SparkPullRequestBuilder, too. I think we can ignore that.
>>>>>
>>>>> I'm sure that this will save much more community's efforts on the
>>>>> static errors by preventing them at the very early stage. But, there might
>>>>> be another reason not to do this. I'm wondering about your thoughts.
>>>>>
>>>>> I can make a Apache INFRA Jira issue for this if there is some
>>>>> consensus.
>>>>>
>>>>> Warmly,
>>>>> Dongjoon.
>>>>>
>>>>
>>>>
>>>
>>
>
Re: Using Travis for JDK7/8 compilation and lint-java.
Posted by Dongjoon Hyun <do...@apache.org>.
Thank you for feedback. Sure, correctly, that's the reason why the current
SparkPullRequestBuilder do not run `lint-java`. :-)
In addition, that's the same reason why contributors are reluctant to run
`lint-java` and causes breaking on JDK7 builds.
Such a tedious and time-consuming job should be done by CI without human
interventions.
By the way, why do you think we need to wait for that? We should not wait
for any CIs, we should continue our own work.
My proposal isn't for making you wait to watch the result. There are two
use cases I want for us to focus here.
Case 1: When you make a PR to Spark PR queue.
Travis CI will finish before SparkPullRequestBuilder.
We will run the followings in parallel mode.
1. Current SparkPullRequestBuilder: JDK8 + sbt build + (no Java
Linter)
2. Travis: JDK7 + mvn build + Java Linter
3. Travis: JDK8 + mvn build + Java Linter
As we know, 1 is the longest time-consuming one which have lots of
works (except maven building or lint- java). You don't need to wait more
in many cases. Yes, in many cases, not all the cases.
Case 2: When you prepare a PR on your branch.
If you are at the final commit (maybe already-squashed), just go to
case 1.
However, usually, we makes lots of commits locally while making
preparing our PR.
And, finally we squashed them into one and send a PR to Spark.
I mean you can use Travis CI during preparing your PRs.
Again, don't wait for Travis CI. Just push it sometime or at every
commit, and continue your work.
At the final stage when you finish your coding, squash your commits
into one,
and amend your commit title or messages, see the Travis CI.
Or, you can monitor Travis CI result on status menu bar.
If it shows green icon, you have nothing to do.
https://docs.travis-ci.com/user/apps/
To sum up, I think we don't need to wait for any CIs. It's like an email.
`Send and back to work.`
Dongjoon.
On Sun, May 22, 2016 at 8:32 PM, Ted Yu <yu...@gmail.com> wrote:
> Without Zinc, 'mvn -DskipTests clean install' takes ~30 minutes.
>
> Maybe not everyone is willing to wait that long.
>
> On Sun, May 22, 2016 at 1:30 PM, Dongjoon Hyun <do...@apache.org>
> wrote:
>
>> Oh, Sure. My bad!
>>
>> - For Oracle JDK7, mvn -DskipTests install and run `dev/lint-java`.
>> - For Oracle JDK8, mvn -DskipTests install and run `dev/lint-java`.
>>
>> Thank you, Ted.
>>
>> Dongjoon.
>>
>> On Sun, May 22, 2016 at 1:29 PM, Ted Yu <yu...@gmail.com> wrote:
>>
>>> The following line was repeated twice:
>>>
>>> - For Oracle JDK7, mvn -DskipTests install and run `dev/lint-java`.
>>>
>>> Did you intend to cover JDK 8 ?
>>>
>>> Cheers
>>>
>>> On Sun, May 22, 2016 at 1:25 PM, Dongjoon Hyun <do...@apache.org>
>>> wrote:
>>>
>>>> Hi, All.
>>>>
>>>> I want to propose the followings.
>>>>
>>>> - Turn on Travis CI for Apache Spark PR queue.
>>>> - Recommend this for contributors, too
>>>>
>>>> Currently, Spark provides Travis CI configuration file to help
>>>> contributors check Scala/Java style conformance and JDK7/8 compilation
>>>> easily during their preparing pull requests. Please note that it's only
>>>> about static analysis.
>>>>
>>>> - For Oracle JDK7, mvn -DskipTests install and run `dev/lint-java`.
>>>> - For Oracle JDK7, mvn -DskipTests install and run `dev/lint-java`.
>>>> Scalastyle is included in the step 'mvn install', too.
>>>>
>>>> Yep, if you turn on your Travis CI configuration, you can already see
>>>> the results on your branches before making PR. I wrote this email to
>>>> prevent more failures proactively and community-widely.
>>>>
>>>> For stability, I have been monitoring that for two weeks. It detects
>>>> the failures or recovery on JDK7 builds or Java linter on Spark master
>>>> branch correctly. The only exceptional case I observed rarely is `timeout`
>>>> failure due to hangs of maven. But, as we know, it's happen in our Jenkins
>>>> SparkPullRequestBuilder, too. I think we can ignore that.
>>>>
>>>> I'm sure that this will save much more community's efforts on the
>>>> static errors by preventing them at the very early stage. But, there might
>>>> be another reason not to do this. I'm wondering about your thoughts.
>>>>
>>>> I can make a Apache INFRA Jira issue for this if there is some
>>>> consensus.
>>>>
>>>> Warmly,
>>>> Dongjoon.
>>>>
>>>
>>>
>>
>
Re: Using Travis for JDK7/8 compilation and lint-java.
Posted by Ted Yu <yu...@gmail.com>.
Without Zinc, 'mvn -DskipTests clean install' takes ~30 minutes.
Maybe not everyone is willing to wait that long.
On Sun, May 22, 2016 at 1:30 PM, Dongjoon Hyun <do...@apache.org> wrote:
> Oh, Sure. My bad!
>
> - For Oracle JDK7, mvn -DskipTests install and run `dev/lint-java`.
> - For Oracle JDK8, mvn -DskipTests install and run `dev/lint-java`.
>
> Thank you, Ted.
>
> Dongjoon.
>
> On Sun, May 22, 2016 at 1:29 PM, Ted Yu <yu...@gmail.com> wrote:
>
>> The following line was repeated twice:
>>
>> - For Oracle JDK7, mvn -DskipTests install and run `dev/lint-java`.
>>
>> Did you intend to cover JDK 8 ?
>>
>> Cheers
>>
>> On Sun, May 22, 2016 at 1:25 PM, Dongjoon Hyun <do...@apache.org>
>> wrote:
>>
>>> Hi, All.
>>>
>>> I want to propose the followings.
>>>
>>> - Turn on Travis CI for Apache Spark PR queue.
>>> - Recommend this for contributors, too
>>>
>>> Currently, Spark provides Travis CI configuration file to help
>>> contributors check Scala/Java style conformance and JDK7/8 compilation
>>> easily during their preparing pull requests. Please note that it's only
>>> about static analysis.
>>>
>>> - For Oracle JDK7, mvn -DskipTests install and run `dev/lint-java`.
>>> - For Oracle JDK7, mvn -DskipTests install and run `dev/lint-java`.
>>> Scalastyle is included in the step 'mvn install', too.
>>>
>>> Yep, if you turn on your Travis CI configuration, you can already see
>>> the results on your branches before making PR. I wrote this email to
>>> prevent more failures proactively and community-widely.
>>>
>>> For stability, I have been monitoring that for two weeks. It detects the
>>> failures or recovery on JDK7 builds or Java linter on Spark master branch
>>> correctly. The only exceptional case I observed rarely is `timeout` failure
>>> due to hangs of maven. But, as we know, it's happen in our Jenkins
>>> SparkPullRequestBuilder, too. I think we can ignore that.
>>>
>>> I'm sure that this will save much more community's efforts on the static
>>> errors by preventing them at the very early stage. But, there might be
>>> another reason not to do this. I'm wondering about your thoughts.
>>>
>>> I can make a Apache INFRA Jira issue for this if there is some consensus.
>>>
>>> Warmly,
>>> Dongjoon.
>>>
>>
>>
>
Re: Using Travis for JDK7/8 compilation and lint-java.
Posted by Dongjoon Hyun <do...@apache.org>.
Oh, Sure. My bad!
- For Oracle JDK7, mvn -DskipTests install and run `dev/lint-java`.
- For Oracle JDK8, mvn -DskipTests install and run `dev/lint-java`.
Thank you, Ted.
Dongjoon.
On Sun, May 22, 2016 at 1:29 PM, Ted Yu <yu...@gmail.com> wrote:
> The following line was repeated twice:
>
> - For Oracle JDK7, mvn -DskipTests install and run `dev/lint-java`.
>
> Did you intend to cover JDK 8 ?
>
> Cheers
>
> On Sun, May 22, 2016 at 1:25 PM, Dongjoon Hyun <do...@apache.org>
> wrote:
>
>> Hi, All.
>>
>> I want to propose the followings.
>>
>> - Turn on Travis CI for Apache Spark PR queue.
>> - Recommend this for contributors, too
>>
>> Currently, Spark provides Travis CI configuration file to help
>> contributors check Scala/Java style conformance and JDK7/8 compilation
>> easily during their preparing pull requests. Please note that it's only
>> about static analysis.
>>
>> - For Oracle JDK7, mvn -DskipTests install and run `dev/lint-java`.
>> - For Oracle JDK7, mvn -DskipTests install and run `dev/lint-java`.
>> Scalastyle is included in the step 'mvn install', too.
>>
>> Yep, if you turn on your Travis CI configuration, you can already see the
>> results on your branches before making PR. I wrote this email to prevent
>> more failures proactively and community-widely.
>>
>> For stability, I have been monitoring that for two weeks. It detects the
>> failures or recovery on JDK7 builds or Java linter on Spark master branch
>> correctly. The only exceptional case I observed rarely is `timeout` failure
>> due to hangs of maven. But, as we know, it's happen in our Jenkins
>> SparkPullRequestBuilder, too. I think we can ignore that.
>>
>> I'm sure that this will save much more community's efforts on the static
>> errors by preventing them at the very early stage. But, there might be
>> another reason not to do this. I'm wondering about your thoughts.
>>
>> I can make a Apache INFRA Jira issue for this if there is some consensus.
>>
>> Warmly,
>> Dongjoon.
>>
>
>
Re: Using Travis for JDK7/8 compilation and lint-java.
Posted by Ted Yu <yu...@gmail.com>.
The following line was repeated twice:
- For Oracle JDK7, mvn -DskipTests install and run `dev/lint-java`.
Did you intend to cover JDK 8 ?
Cheers
On Sun, May 22, 2016 at 1:25 PM, Dongjoon Hyun <do...@apache.org> wrote:
> Hi, All.
>
> I want to propose the followings.
>
> - Turn on Travis CI for Apache Spark PR queue.
> - Recommend this for contributors, too
>
> Currently, Spark provides Travis CI configuration file to help
> contributors check Scala/Java style conformance and JDK7/8 compilation
> easily during their preparing pull requests. Please note that it's only
> about static analysis.
>
> - For Oracle JDK7, mvn -DskipTests install and run `dev/lint-java`.
> - For Oracle JDK7, mvn -DskipTests install and run `dev/lint-java`.
> Scalastyle is included in the step 'mvn install', too.
>
> Yep, if you turn on your Travis CI configuration, you can already see the
> results on your branches before making PR. I wrote this email to prevent
> more failures proactively and community-widely.
>
> For stability, I have been monitoring that for two weeks. It detects the
> failures or recovery on JDK7 builds or Java linter on Spark master branch
> correctly. The only exceptional case I observed rarely is `timeout` failure
> due to hangs of maven. But, as we know, it's happen in our Jenkins
> SparkPullRequestBuilder, too. I think we can ignore that.
>
> I'm sure that this will save much more community's efforts on the static
> errors by preventing them at the very early stage. But, there might be
> another reason not to do this. I'm wondering about your thoughts.
>
> I can make a Apache INFRA Jira issue for this if there is some consensus.
>
> Warmly,
> Dongjoon.
>