You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Dan Halperin <dh...@google.com.INVALID> on 2017/01/05 17:24:53 UTC
Better developer instructions for using Maven?
Several folks seem to have been confused after BEAM-246, where we moved the
"slow things" into the release profile. I've started a discussion with
https://github.com/apache/beam/pull/1740 to see if there are things we can
do to fill these gaps.
Would love folks to chime in with opinions.
Dan
On Wed, Jan 4, 2017 at 1:34 PM, Jesse Anderson <je...@smokinghand.com>
wrote:
> @Eugene, yes that failed on the checkstyle.
>
> On Wed, Jan 4, 2017 at 1:27 PM Eugene Kirpichov
> <ki...@google.com.invalid> wrote:
>
> > Try just -Prelease.
> > On Wed, Jan 4, 2017 at 1:21 PM Jesse Anderson <je...@smokinghand.com>
> > wrote:
> >
> > > Fails because I don't have a secret key.
> > >
> > > On Wed, Jan 4, 2017 at 1:03 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
> > > wrote:
> > >
> > > > Hi Jesse,
> > > >
> > > > Could you try the same with:
> > > >
> > > > mvn verify -Prelease,apache-release
> > > >
> > > > ?
> > > >
> > > > Regards
> > > > JB
> > > >
> > > > On 01/04/2017 09:53 PM, Jesse Anderson wrote:
> > > > > For some reason, running "mvn verify" isn't running checkstyle on
> > > > > everything. I had checkstyle errors in beam-sdks-java-core that
> > weren't
> > > > > being found.
> > > > >
> > > > > I thought this was due to the extra parameters. I reran with the
> > plain
> > > > "mvn
> > > > > verify" and it still didn't find them. From the output, it doesn't
> > look
> > > > > like they're being run at all.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jesse
> > > > >
> > > >
> > > > --
> > > > Jean-Baptiste Onofré
> > > > jbonofre@apache.org
> > > > http://blog.nanthrax.net
> > > > Talend - http://www.talend.com
> > > >
> > >
> >
>
Re: Better developer instructions for using Maven?
Posted by Kenneth Knowles <kl...@google.com.INVALID>.
On Wed, Feb 22, 2017 at 10:04 AM, Robert Bradshaw <
robertwb@google.com.invalid> wrote:
> with their respective costs (e.g. runtime,
> dependencies) expected utility (how likely it is to find something) and
> actionability (how easy it would be for someone to fix the reported errors
> without asking us).
>
I'd add "detriment to newcomer morale / likelihood to deter new
contributors", though obviously not really quantifiable we could say "none,
little, some, lots". This isn't redundant with other criteria, as
checkstyle is pretty fast, has almost no false positives, has actionable
messages, and errors are easy to fix, but it is also nitpicky and a
needless deterrent. Or, if you mean "how likely it is to find a real error"
the score is zero. I am 100% in favor of checkstyle on the committed code,
just not for the uninitiated.
But yes, this is just my vote and rationale. I will certainly be happy with
the consensus of the community. I agree that a multifaceted survey like
Robert suggests might be helpful.
Re: Better developer instructions for using Maven?
Posted by Robert Bradshaw <ro...@google.com.INVALID>.
On Wed, Feb 22, 2017 at 7:51 AM, Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:
> Thanks Kenn, it's perfectly clear now ;)
>
That was Kenn's vote. I'm of the opposite opinion (at least I think checkstyle
should be done by default, possibly others). It's clear many people aren't
very happy with the current setup either. An email thread like this isn't
very good for tracking what consensus is though. Though there has been some
good discussion on this thread it seems we're spinning our wheels a bit
because it's hard to track where we even disagree.
It would be helpful if someone were to provide a definitive list of what
the possibilities are, with their respective costs (e.g. runtime,
dependencies) expected utility (how likely it is to find something) and
actionability (how easy it would be for someone to fix the reported errors
without asking us). We could then have a vote (not necessarily binding, but
to deduce consensus if any), something like
* Build (x sec)
robertwb, klk
* Unit tests (y sec)
robertwb, klk
* Checkstyle (z sec)
robertwb
* Findbugs
...
(likely in a new thread).
Regards
> JB
>
>
> On 02/22/2017 04:40 PM, Kenneth Knowles wrote:
>
>> On Wed, Feb 22, 2017 at 6:50 AM, Jean-Baptiste Onofré <jb...@nanthrax.net>
>> wrote:
>>
>> Can we finalize a decision about this ?
>>>
>>> So basically, the question is:
>>>
>>> 1. Do we enable checkstyle, findbugs, ... (all things increasing the
>>> build
>>> duration, tests excluded) by default or not ?
>>> 2. If we decide to enable it by default, can we use a 'quick' property
>>> (-Dquick) to bypass ?
>>>
>>>
>> To be extremely clear, I vote for:
>>
>> 1. Disable checkstyle, findbugs, javadoc, and everything else beyond the
>> minimum needed to compile and run unit tests.
>> 2. Add -Dall-checks to re-enable it all, including very slow checks.
>>
>> Kenn
>>
>>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
Re: Better developer instructions for using Maven?
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Thanks Kenn, it's perfectly clear now ;)
Regards
JB
On 02/22/2017 04:40 PM, Kenneth Knowles wrote:
> On Wed, Feb 22, 2017 at 6:50 AM, Jean-Baptiste Onofr� <jb...@nanthrax.net>
> wrote:
>
>> Can we finalize a decision about this ?
>>
>> So basically, the question is:
>>
>> 1. Do we enable checkstyle, findbugs, ... (all things increasing the build
>> duration, tests excluded) by default or not ?
>> 2. If we decide to enable it by default, can we use a 'quick' property
>> (-Dquick) to bypass ?
>>
>
> To be extremely clear, I vote for:
>
> 1. Disable checkstyle, findbugs, javadoc, and everything else beyond the
> minimum needed to compile and run unit tests.
> 2. Add -Dall-checks to re-enable it all, including very slow checks.
>
> Kenn
>
--
Jean-Baptiste Onofr�
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: Better developer instructions for using Maven?
Posted by Kenneth Knowles <kl...@google.com.INVALID>.
On Wed, Feb 22, 2017 at 6:50 AM, Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:
> Can we finalize a decision about this ?
>
> So basically, the question is:
>
> 1. Do we enable checkstyle, findbugs, ... (all things increasing the build
> duration, tests excluded) by default or not ?
> 2. If we decide to enable it by default, can we use a 'quick' property
> (-Dquick) to bypass ?
>
To be extremely clear, I vote for:
1. Disable checkstyle, findbugs, javadoc, and everything else beyond the
minimum needed to compile and run unit tests.
2. Add -Dall-checks to re-enable it all, including very slow checks.
Kenn
Re: Better developer instructions for using Maven?
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi guys,
sorry, I thought we got a consensus about the usage of quick but it
doesn't seem so. I reverted the corresponding commit.
Can we finalize a decision about this ?
So basically, the question is:
1. Do we enable checkstyle, findbugs, ... (all things increasing the
build duration, tests excluded) by default or not ?
2. If we decide to enable it by default, can we use a 'quick' property
(-Dquick) to bypass ?
Thanks, and sorry again
Regards
JB
On 02/21/2017 05:07 AM, Aviem Zur wrote:
> I agree, I'll change it to -Dquick
>
> On Tue, Feb 21, 2017 at 12:07 AM Davor Bonaci <da...@apache.org> wrote:
>
>> (I'm also in favor of not overloading existing flags; they have some
>> meaning/semantics that developers have come to expect.)
>>
>> On Sun, Feb 19, 2017 at 12:34 PM, Jean-Baptiste Onofr� <jb...@nanthrax.net>
>> wrote:
>>
>>> Thanks Aviem,
>>>
>>> Not sure if we should use skipTests as it really means unit tests and
>>> integration tests (in Karaf, skipTests skips the unit test, integration
>>> test, archetype itests and maven plugin invoker test, but it doesn't skip
>>> checkstyle, findbugs, etc, for that, we have a fastinstall property).
>>>
>>> Maybe it would make more sense to use a specific property like
>>> -DfastBuild=true.
>>>
>>> WDYT ?
>>>
>>> Regards
>>> JB
>>>
>>>
>>> On 02/19/2017 09:27 PM, Aviem Zur wrote:
>>>
>>>> I've created a PR which disables slow verifications if '-DskipTests' was
>>>> specified, otherwise runs them.
>>>> I think this satisfies all the considerations mentioned in this thread.
>>>>
>>>> PTAL: https://github.com/apache/beam/pull/2048
>>>> Ticket: https://issues.apache.org/jira/browse/BEAM-1513
>>>>
>>>> On Thu, Feb 16, 2017 at 3:05 PM Isma�l Mej�a <ie...@gmail.com> wrote:
>>>>
>>>> JB, Maybe I was not clear, when I talked about the tests I was thinking
>>>>> more about execute them in parallel in the same machine, this is not
>> the
>>>>> case today for some test suites, and for these the tests need to be
>>>>> refined
>>>>> to support this, and configured via the pom to execute the tests in
>>>>> parallel per method, class, etc. Of course we need to check if this is
>>>>> worth, because I can imagine that the more expensive time for example
>> in
>>>>> the IO case comes from starting the embedded versions of the IOs (e.g.
>>>>> HadoopMiniCluster, MongodExecutable, HBasetestingutility, etc) and not
>>>>> from
>>>>> the tests themselves but this has to be evaluated.
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Feb 15, 2017 at 5:46 PM, Jean-Baptiste Onofr� <jb@nanthrax.net
>>>
>>>>> wrote:
>>>>>
>>>>> On Jenkins it's possible to run several jobs in the same time but on
>>>>>> different executor. That's the easiest way.
>>>>>>
>>>>>> Regards
>>>>>> JB
>>>>>>
>>>>>> On Feb 15, 2017, 10:15, at 10:15, "Isma�l Mej�a" <ie...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> This question got lost in the discussion, but there is a small
>>>>>>> improvement
>>>>>>> that we can do:
>>>>>>>
>>>>>>> Just to check, are we doing parallel builds?
>>>>>>>>
>>>>>>>
>>>>>>> We are on jenkins, not in travis, there is an ongoing PR to fix this.
>>>>>>>
>>>>>>> What we can improve is to check if we can run some of the test suites
>>>>>>> in
>>>>>>> parallel to gain some extra time. For exemple most of the IOs and
>> some
>>>>>>> runners don't execute tests in parallel.
>>>>>>>
>>>>>>> Ismael
>>>>>>>
>>>>>>> (slightly related), is there a way to get change the timeout of
>> travis
>>>>>>> jobs). Because it is failing most of the time now because of this,
>> and
>>>>>>> it
>>>>>>> is quite noisey to have so many false positives.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Feb 10, 2017 at 8:00 PM, Robert Bradshaw <
>>>>>>> robertwb@google.com.invalid> wrote:
>>>>>>>
>>>>>>> On Fri, Feb 10, 2017 at 8:45 AM, Dan Halperin
>>>>>>>>
>>>>>>> <dhalperi@google.com.invalid
>>>>>>>
>>>>>>>>
>>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> On Fri, Feb 10, 2017 at 7:42 AM, Kenneth Knowles
>>>>>>>>>
>>>>>>>> <klk@google.com.invalid
>>>>>>>
>>>>>>>>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> On Feb 10, 2017 07:36, "Dan Halperin"
>>>>>>>>>>
>>>>>>>>> <dh...@google.com.invalid>
>>>>>>>
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Before we added checkstyle it was under a minute. Now it's over
>>>>>>>>>>
>>>>>>>>> five?
>>>>>>>
>>>>>>>> That's awful IMO
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Checkstyle didn't cause all that, did it?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> The "5 minutes" was going with Aviem's numbers after this change.
>>>>>>>>>
>>>>>>>> But
>>>>>>>
>>>>>>>> yes,
>>>>>>>>
>>>>>>>>> Checkstyle alone substantially (>+50%) the time from what it was
>>>>>>>>>
>>>>>>>> previously
>>>>>>>>
>>>>>>>>> to adding it back to the default build.
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Just to check, are we doing parallel builds?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> Noting that findbugs takes quite a lot more time. Javadoc and jar
>>>>>>>>>
>>>>>>>> are the
>>>>>>>
>>>>>>>> other two slow ones.
>>>>>>>>>>
>>>>>>>>>> RAT is fast. But it has very poor error messages, so we wouldn't
>>>>>>>>>>
>>>>>>>>> want a
>>>>>>>
>>>>>>>> new
>>>>>>>>>
>>>>>>>>>> contributor trying to figure out what is going on without our
>>>>>>>>>>
>>>>>>>>> help.
>>>>>>>
>>>>>>>>
>>>>>>>>>>
>>>>>>>>> There is a larger philosophical issue here: is there a point of
>>>>>>>>>
>>>>>>>> Jenkins
>>>>>>>
>>>>>>>> precommit testing? Why not just make `mvn install` run everything
>>>>>>>>>
>>>>>>>> that
>>>>>>>
>>>>>>>> Jenkins does? For that matter, why don't committers just push
>>>>>>>>>
>>>>>>>> directly to
>>>>>>>
>>>>>>>> master? Wouldn't that make everyone's life easier?
>>>>>>>>>
>>>>>>>>> I'd argue that's not true.
>>>>>>>>>
>>>>>>>>> 1. Developer productivity -- Jenkins should run many more checks
>>>>>>>>>
>>>>>>>> than
>>>>>>>
>>>>>>>> developers do. Especially time-, resource-, or setup- intensive
>>>>>>>>>
>>>>>>>> tasks.
>>>>>>>
>>>>>>>> 2. Automated enforcement -- Jenkins is better at running the right
>>>>>>>>>
>>>>>>>> commands
>>>>>>>>
>>>>>>>>> than we are.
>>>>>>>>> 3. Lower the barrier to entry -- individual developers need not
>>>>>>>>>
>>>>>>>> have a
>>>>>>>
>>>>>>>> running Spark/Flink/Apex/Dataflow setup in order to contribute
>>>>>>>>>
>>>>>>>> code.
>>>>>>>
>>>>>>>> 4. Focus on the user -- someone checking out the code and using it
>>>>>>>>>
>>>>>>>> for
>>>>>>>
>>>>>>>> the
>>>>>>>>
>>>>>>>>> first time does not care whether the code style checks or has the
>>>>>>>>>
>>>>>>>> right
>>>>>>>
>>>>>>>> licenses -- that should have been enforced by the Beam team before
>>>>>>>>> committing.
>>>>>>>>>
>>>>>>>>> We should be *very* choosy about what we enforce on every developer
>>>>>>>>>
>>>>>>>> every
>>>>>>>
>>>>>>>> time they go to compile. I probably compile Beam 50x-100x a day.
>>>>>>>>>
>>>>>>>> Literally,
>>>>>>>>
>>>>>>>>> the extra minutes you want to add here will cost me an hour daily.
>>>>>>>>>
>>>>>>>>>
>>>>>>>> By the same token of having a different bar for the Jenkins
>> presubmit
>>>>>>>>
>>>>>>> vs.
>>>>>>>
>>>>>>>> what's run locally, I think it makes a lot of sense to run a
>>>>>>>>
>>>>>>> different
>>>>>>>
>>>>>>>> command for iterative development than you run before creating a
>> pull
>>>>>>>> request. E.g. during development I'll often run only one test rather
>>>>>>>>
>>>>>>> than
>>>>>>>
>>>>>>>> the entire suite, but do run the entire suite occasionally (often
>>>>>>>>
>>>>>>> before
>>>>>>>
>>>>>>>> commit, especially before pushing).
>>>>>>>>
>>>>>>>> The contributors guild should give a suggested command to run before
>>>>>>>> creating a PR, right in the docs of how to create a PR, which may be
>>>>>>>>
>>>>>>> more
>>>>>>>
>>>>>>>> expensive than what you run during development. IMHO, this should be
>>>>>>>>
>>>>>>> fairly
>>>>>>>
>>>>>>>> comprehensive (certainly tests and checkstyle, maybe javadoc and
>>>>>>>>
>>>>>>> findbugs).
>>>>>>>
>>>>>>>> This should be the "default" command that the one-time-contributor
>>>>>>>>
>>>>>>> should
>>>>>>>
>>>>>>>> know. For those compiling 50x or more a day, I think the burden of
>>>>>>>>
>>>>>>> learning
>>>>>>>
>>>>>>>> a second (or more) cheaper commands is not high, and we could even
>>>>>>>>
>>>>>>> put such
>>>>>>>
>>>>>>>> a thing in the docs (and hopefully a common maven convention like
>>>>>>>>
>>>>>>> "mvn
>>>>>>>
>>>>>>>> test").
>>>>>>>>
>>>>>>>> I've listed the fraction of commits I think will break one of the
>>>>>>>>
>>>>>>> following
>>>>>>>
>>>>>>>> if that property is not tested:
>>>>>>>>>
>>>>>>>>> * compiling (100%)
>>>>>>>>> * tests (100%)
>>>>>>>>> * checkstyle (90%)
>>>>>>>>> * javadoc (30%)
>>>>>>>>> * findbugs (5%)
>>>>>>>>> * rat (1%)
>>>>>>>>>
>>>>>>>>> So you can see where I stand and why. I'm sorry that 1/20 PRs has
>>>>>>>>>
>>>>>>>> Apache
>>>>>>>
>>>>>>>> RAT catch a licensing issue or Findbugs catch a threading issue --
>>>>>>>>>
>>>>>>>> you
>>>>>>>
>>>>>>>> can
>>>>>>>>
>>>>>>>>> always get a larger set of the precommit checks using -Prelease,
>>>>>>>>>
>>>>>>>> though
>>>>>>>
>>>>>>>> of
>>>>>>>>
>>>>>>>>> course the integration tests and runnableonservice tests may catch
>>>>>>>>>
>>>>>>>> more
>>>>>>>
>>>>>>>> issues still. But I want my developer minutes back for the 95%+ of
>>>>>>>>>
>>>>>>>> the
>>>>>>>
>>>>>>>> rest.
>>>>>>>>>
>>>>>>>>> Dan
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>
>>>>
>>> --
>>> Jean-Baptiste Onofr�
>>> jbonofre@apache.org
>>> http://blog.nanthrax.net
>>> Talend - http://www.talend.com
>>>
>>
>
--
Jean-Baptiste Onofr�
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: Better developer instructions for using Maven?
Posted by Aviem Zur <av...@gmail.com>.
I agree, I'll change it to -Dquick
On Tue, Feb 21, 2017 at 12:07 AM Davor Bonaci <da...@apache.org> wrote:
> (I'm also in favor of not overloading existing flags; they have some
> meaning/semantics that developers have come to expect.)
>
> On Sun, Feb 19, 2017 at 12:34 PM, Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
>
> > Thanks Aviem,
> >
> > Not sure if we should use skipTests as it really means unit tests and
> > integration tests (in Karaf, skipTests skips the unit test, integration
> > test, archetype itests and maven plugin invoker test, but it doesn't skip
> > checkstyle, findbugs, etc, for that, we have a fastinstall property).
> >
> > Maybe it would make more sense to use a specific property like
> > -DfastBuild=true.
> >
> > WDYT ?
> >
> > Regards
> > JB
> >
> >
> > On 02/19/2017 09:27 PM, Aviem Zur wrote:
> >
> >> I've created a PR which disables slow verifications if '-DskipTests' was
> >> specified, otherwise runs them.
> >> I think this satisfies all the considerations mentioned in this thread.
> >>
> >> PTAL: https://github.com/apache/beam/pull/2048
> >> Ticket: https://issues.apache.org/jira/browse/BEAM-1513
> >>
> >> On Thu, Feb 16, 2017 at 3:05 PM Ismaël Mejía <ie...@gmail.com> wrote:
> >>
> >> JB, Maybe I was not clear, when I talked about the tests I was thinking
> >>> more about execute them in parallel in the same machine, this is not
> the
> >>> case today for some test suites, and for these the tests need to be
> >>> refined
> >>> to support this, and configured via the pom to execute the tests in
> >>> parallel per method, class, etc. Of course we need to check if this is
> >>> worth, because I can imagine that the more expensive time for example
> in
> >>> the IO case comes from starting the embedded versions of the IOs (e.g.
> >>> HadoopMiniCluster, MongodExecutable, HBasetestingutility, etc) and not
> >>> from
> >>> the tests themselves but this has to be evaluated.
> >>>
> >>>
> >>>
> >>> On Wed, Feb 15, 2017 at 5:46 PM, Jean-Baptiste Onofré <jb@nanthrax.net
> >
> >>> wrote:
> >>>
> >>> On Jenkins it's possible to run several jobs in the same time but on
> >>>> different executor. That's the easiest way.
> >>>>
> >>>> Regards
> >>>> JB
> >>>>
> >>>> On Feb 15, 2017, 10:15, at 10:15, "Ismaël Mejía" <ie...@gmail.com>
> >>>> wrote:
> >>>>
> >>>>> This question got lost in the discussion, but there is a small
> >>>>> improvement
> >>>>> that we can do:
> >>>>>
> >>>>> Just to check, are we doing parallel builds?
> >>>>>>
> >>>>>
> >>>>> We are on jenkins, not in travis, there is an ongoing PR to fix this.
> >>>>>
> >>>>> What we can improve is to check if we can run some of the test suites
> >>>>> in
> >>>>> parallel to gain some extra time. For exemple most of the IOs and
> some
> >>>>> runners don't execute tests in parallel.
> >>>>>
> >>>>> Ismael
> >>>>>
> >>>>> (slightly related), is there a way to get change the timeout of
> travis
> >>>>> jobs). Because it is failing most of the time now because of this,
> and
> >>>>> it
> >>>>> is quite noisey to have so many false positives.
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Fri, Feb 10, 2017 at 8:00 PM, Robert Bradshaw <
> >>>>> robertwb@google.com.invalid> wrote:
> >>>>>
> >>>>> On Fri, Feb 10, 2017 at 8:45 AM, Dan Halperin
> >>>>>>
> >>>>> <dhalperi@google.com.invalid
> >>>>>
> >>>>>>
> >>>>>>> wrote:
> >>>>>>
> >>>>>> On Fri, Feb 10, 2017 at 7:42 AM, Kenneth Knowles
> >>>>>>>
> >>>>>> <klk@google.com.invalid
> >>>>>
> >>>>>>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>> On Feb 10, 2017 07:36, "Dan Halperin"
> >>>>>>>>
> >>>>>>> <dh...@google.com.invalid>
> >>>>>
> >>>>>> wrote:
> >>>>>>>
> >>>>>>>>
> >>>>>>>> Before we added checkstyle it was under a minute. Now it's over
> >>>>>>>>
> >>>>>>> five?
> >>>>>
> >>>>>> That's awful IMO
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Checkstyle didn't cause all that, did it?
> >>>>>>>>
> >>>>>>>>
> >>>>>>> The "5 minutes" was going with Aviem's numbers after this change.
> >>>>>>>
> >>>>>> But
> >>>>>
> >>>>>> yes,
> >>>>>>
> >>>>>>> Checkstyle alone substantially (>+50%) the time from what it was
> >>>>>>>
> >>>>>> previously
> >>>>>>
> >>>>>>> to adding it back to the default build.
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>> Just to check, are we doing parallel builds?
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>> Noting that findbugs takes quite a lot more time. Javadoc and jar
> >>>>>>>
> >>>>>> are the
> >>>>>
> >>>>>> other two slow ones.
> >>>>>>>>
> >>>>>>>> RAT is fast. But it has very poor error messages, so we wouldn't
> >>>>>>>>
> >>>>>>> want a
> >>>>>
> >>>>>> new
> >>>>>>>
> >>>>>>>> contributor trying to figure out what is going on without our
> >>>>>>>>
> >>>>>>> help.
> >>>>>
> >>>>>>
> >>>>>>>>
> >>>>>>> There is a larger philosophical issue here: is there a point of
> >>>>>>>
> >>>>>> Jenkins
> >>>>>
> >>>>>> precommit testing? Why not just make `mvn install` run everything
> >>>>>>>
> >>>>>> that
> >>>>>
> >>>>>> Jenkins does? For that matter, why don't committers just push
> >>>>>>>
> >>>>>> directly to
> >>>>>
> >>>>>> master? Wouldn't that make everyone's life easier?
> >>>>>>>
> >>>>>>> I'd argue that's not true.
> >>>>>>>
> >>>>>>> 1. Developer productivity -- Jenkins should run many more checks
> >>>>>>>
> >>>>>> than
> >>>>>
> >>>>>> developers do. Especially time-, resource-, or setup- intensive
> >>>>>>>
> >>>>>> tasks.
> >>>>>
> >>>>>> 2. Automated enforcement -- Jenkins is better at running the right
> >>>>>>>
> >>>>>> commands
> >>>>>>
> >>>>>>> than we are.
> >>>>>>> 3. Lower the barrier to entry -- individual developers need not
> >>>>>>>
> >>>>>> have a
> >>>>>
> >>>>>> running Spark/Flink/Apex/Dataflow setup in order to contribute
> >>>>>>>
> >>>>>> code.
> >>>>>
> >>>>>> 4. Focus on the user -- someone checking out the code and using it
> >>>>>>>
> >>>>>> for
> >>>>>
> >>>>>> the
> >>>>>>
> >>>>>>> first time does not care whether the code style checks or has the
> >>>>>>>
> >>>>>> right
> >>>>>
> >>>>>> licenses -- that should have been enforced by the Beam team before
> >>>>>>> committing.
> >>>>>>>
> >>>>>>> We should be *very* choosy about what we enforce on every developer
> >>>>>>>
> >>>>>> every
> >>>>>
> >>>>>> time they go to compile. I probably compile Beam 50x-100x a day.
> >>>>>>>
> >>>>>> Literally,
> >>>>>>
> >>>>>>> the extra minutes you want to add here will cost me an hour daily.
> >>>>>>>
> >>>>>>>
> >>>>>> By the same token of having a different bar for the Jenkins
> presubmit
> >>>>>>
> >>>>> vs.
> >>>>>
> >>>>>> what's run locally, I think it makes a lot of sense to run a
> >>>>>>
> >>>>> different
> >>>>>
> >>>>>> command for iterative development than you run before creating a
> pull
> >>>>>> request. E.g. during development I'll often run only one test rather
> >>>>>>
> >>>>> than
> >>>>>
> >>>>>> the entire suite, but do run the entire suite occasionally (often
> >>>>>>
> >>>>> before
> >>>>>
> >>>>>> commit, especially before pushing).
> >>>>>>
> >>>>>> The contributors guild should give a suggested command to run before
> >>>>>> creating a PR, right in the docs of how to create a PR, which may be
> >>>>>>
> >>>>> more
> >>>>>
> >>>>>> expensive than what you run during development. IMHO, this should be
> >>>>>>
> >>>>> fairly
> >>>>>
> >>>>>> comprehensive (certainly tests and checkstyle, maybe javadoc and
> >>>>>>
> >>>>> findbugs).
> >>>>>
> >>>>>> This should be the "default" command that the one-time-contributor
> >>>>>>
> >>>>> should
> >>>>>
> >>>>>> know. For those compiling 50x or more a day, I think the burden of
> >>>>>>
> >>>>> learning
> >>>>>
> >>>>>> a second (or more) cheaper commands is not high, and we could even
> >>>>>>
> >>>>> put such
> >>>>>
> >>>>>> a thing in the docs (and hopefully a common maven convention like
> >>>>>>
> >>>>> "mvn
> >>>>>
> >>>>>> test").
> >>>>>>
> >>>>>> I've listed the fraction of commits I think will break one of the
> >>>>>>
> >>>>> following
> >>>>>
> >>>>>> if that property is not tested:
> >>>>>>>
> >>>>>>> * compiling (100%)
> >>>>>>> * tests (100%)
> >>>>>>> * checkstyle (90%)
> >>>>>>> * javadoc (30%)
> >>>>>>> * findbugs (5%)
> >>>>>>> * rat (1%)
> >>>>>>>
> >>>>>>> So you can see where I stand and why. I'm sorry that 1/20 PRs has
> >>>>>>>
> >>>>>> Apache
> >>>>>
> >>>>>> RAT catch a licensing issue or Findbugs catch a threading issue --
> >>>>>>>
> >>>>>> you
> >>>>>
> >>>>>> can
> >>>>>>
> >>>>>>> always get a larger set of the precommit checks using -Prelease,
> >>>>>>>
> >>>>>> though
> >>>>>
> >>>>>> of
> >>>>>>
> >>>>>>> course the integration tests and runnableonservice tests may catch
> >>>>>>>
> >>>>>> more
> >>>>>
> >>>>>> issues still. But I want my developer minutes back for the 95%+ of
> >>>>>>>
> >>>>>> the
> >>>>>
> >>>>>> rest.
> >>>>>>>
> >>>>>>> Dan
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>
> >>>
> >>
> > --
> > Jean-Baptiste Onofré
> > jbonofre@apache.org
> > http://blog.nanthrax.net
> > Talend - http://www.talend.com
> >
>
Re: Better developer instructions for using Maven?
Posted by Davor Bonaci <da...@apache.org>.
(I'm also in favor of not overloading existing flags; they have some
meaning/semantics that developers have come to expect.)
On Sun, Feb 19, 2017 at 12:34 PM, Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:
> Thanks Aviem,
>
> Not sure if we should use skipTests as it really means unit tests and
> integration tests (in Karaf, skipTests skips the unit test, integration
> test, archetype itests and maven plugin invoker test, but it doesn't skip
> checkstyle, findbugs, etc, for that, we have a fastinstall property).
>
> Maybe it would make more sense to use a specific property like
> -DfastBuild=true.
>
> WDYT ?
>
> Regards
> JB
>
>
> On 02/19/2017 09:27 PM, Aviem Zur wrote:
>
>> I've created a PR which disables slow verifications if '-DskipTests' was
>> specified, otherwise runs them.
>> I think this satisfies all the considerations mentioned in this thread.
>>
>> PTAL: https://github.com/apache/beam/pull/2048
>> Ticket: https://issues.apache.org/jira/browse/BEAM-1513
>>
>> On Thu, Feb 16, 2017 at 3:05 PM Ismaël Mejía <ie...@gmail.com> wrote:
>>
>> JB, Maybe I was not clear, when I talked about the tests I was thinking
>>> more about execute them in parallel in the same machine, this is not the
>>> case today for some test suites, and for these the tests need to be
>>> refined
>>> to support this, and configured via the pom to execute the tests in
>>> parallel per method, class, etc. Of course we need to check if this is
>>> worth, because I can imagine that the more expensive time for example in
>>> the IO case comes from starting the embedded versions of the IOs (e.g.
>>> HadoopMiniCluster, MongodExecutable, HBasetestingutility, etc) and not
>>> from
>>> the tests themselves but this has to be evaluated.
>>>
>>>
>>>
>>> On Wed, Feb 15, 2017 at 5:46 PM, Jean-Baptiste Onofré <jb...@nanthrax.net>
>>> wrote:
>>>
>>> On Jenkins it's possible to run several jobs in the same time but on
>>>> different executor. That's the easiest way.
>>>>
>>>> Regards
>>>> JB
>>>>
>>>> On Feb 15, 2017, 10:15, at 10:15, "Ismaël Mejía" <ie...@gmail.com>
>>>> wrote:
>>>>
>>>>> This question got lost in the discussion, but there is a small
>>>>> improvement
>>>>> that we can do:
>>>>>
>>>>> Just to check, are we doing parallel builds?
>>>>>>
>>>>>
>>>>> We are on jenkins, not in travis, there is an ongoing PR to fix this.
>>>>>
>>>>> What we can improve is to check if we can run some of the test suites
>>>>> in
>>>>> parallel to gain some extra time. For exemple most of the IOs and some
>>>>> runners don't execute tests in parallel.
>>>>>
>>>>> Ismael
>>>>>
>>>>> (slightly related), is there a way to get change the timeout of travis
>>>>> jobs). Because it is failing most of the time now because of this, and
>>>>> it
>>>>> is quite noisey to have so many false positives.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Feb 10, 2017 at 8:00 PM, Robert Bradshaw <
>>>>> robertwb@google.com.invalid> wrote:
>>>>>
>>>>> On Fri, Feb 10, 2017 at 8:45 AM, Dan Halperin
>>>>>>
>>>>> <dhalperi@google.com.invalid
>>>>>
>>>>>>
>>>>>>> wrote:
>>>>>>
>>>>>> On Fri, Feb 10, 2017 at 7:42 AM, Kenneth Knowles
>>>>>>>
>>>>>> <klk@google.com.invalid
>>>>>
>>>>>>
>>>>>>> wrote:
>>>>>>>
>>>>>>> On Feb 10, 2017 07:36, "Dan Halperin"
>>>>>>>>
>>>>>>> <dh...@google.com.invalid>
>>>>>
>>>>>> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>> Before we added checkstyle it was under a minute. Now it's over
>>>>>>>>
>>>>>>> five?
>>>>>
>>>>>> That's awful IMO
>>>>>>>>
>>>>>>>>
>>>>>>>> Checkstyle didn't cause all that, did it?
>>>>>>>>
>>>>>>>>
>>>>>>> The "5 minutes" was going with Aviem's numbers after this change.
>>>>>>>
>>>>>> But
>>>>>
>>>>>> yes,
>>>>>>
>>>>>>> Checkstyle alone substantially (>+50%) the time from what it was
>>>>>>>
>>>>>> previously
>>>>>>
>>>>>>> to adding it back to the default build.
>>>>>>>
>>>>>>
>>>>>>
>>>>>> Just to check, are we doing parallel builds?
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Noting that findbugs takes quite a lot more time. Javadoc and jar
>>>>>>>
>>>>>> are the
>>>>>
>>>>>> other two slow ones.
>>>>>>>>
>>>>>>>> RAT is fast. But it has very poor error messages, so we wouldn't
>>>>>>>>
>>>>>>> want a
>>>>>
>>>>>> new
>>>>>>>
>>>>>>>> contributor trying to figure out what is going on without our
>>>>>>>>
>>>>>>> help.
>>>>>
>>>>>>
>>>>>>>>
>>>>>>> There is a larger philosophical issue here: is there a point of
>>>>>>>
>>>>>> Jenkins
>>>>>
>>>>>> precommit testing? Why not just make `mvn install` run everything
>>>>>>>
>>>>>> that
>>>>>
>>>>>> Jenkins does? For that matter, why don't committers just push
>>>>>>>
>>>>>> directly to
>>>>>
>>>>>> master? Wouldn't that make everyone's life easier?
>>>>>>>
>>>>>>> I'd argue that's not true.
>>>>>>>
>>>>>>> 1. Developer productivity -- Jenkins should run many more checks
>>>>>>>
>>>>>> than
>>>>>
>>>>>> developers do. Especially time-, resource-, or setup- intensive
>>>>>>>
>>>>>> tasks.
>>>>>
>>>>>> 2. Automated enforcement -- Jenkins is better at running the right
>>>>>>>
>>>>>> commands
>>>>>>
>>>>>>> than we are.
>>>>>>> 3. Lower the barrier to entry -- individual developers need not
>>>>>>>
>>>>>> have a
>>>>>
>>>>>> running Spark/Flink/Apex/Dataflow setup in order to contribute
>>>>>>>
>>>>>> code.
>>>>>
>>>>>> 4. Focus on the user -- someone checking out the code and using it
>>>>>>>
>>>>>> for
>>>>>
>>>>>> the
>>>>>>
>>>>>>> first time does not care whether the code style checks or has the
>>>>>>>
>>>>>> right
>>>>>
>>>>>> licenses -- that should have been enforced by the Beam team before
>>>>>>> committing.
>>>>>>>
>>>>>>> We should be *very* choosy about what we enforce on every developer
>>>>>>>
>>>>>> every
>>>>>
>>>>>> time they go to compile. I probably compile Beam 50x-100x a day.
>>>>>>>
>>>>>> Literally,
>>>>>>
>>>>>>> the extra minutes you want to add here will cost me an hour daily.
>>>>>>>
>>>>>>>
>>>>>> By the same token of having a different bar for the Jenkins presubmit
>>>>>>
>>>>> vs.
>>>>>
>>>>>> what's run locally, I think it makes a lot of sense to run a
>>>>>>
>>>>> different
>>>>>
>>>>>> command for iterative development than you run before creating a pull
>>>>>> request. E.g. during development I'll often run only one test rather
>>>>>>
>>>>> than
>>>>>
>>>>>> the entire suite, but do run the entire suite occasionally (often
>>>>>>
>>>>> before
>>>>>
>>>>>> commit, especially before pushing).
>>>>>>
>>>>>> The contributors guild should give a suggested command to run before
>>>>>> creating a PR, right in the docs of how to create a PR, which may be
>>>>>>
>>>>> more
>>>>>
>>>>>> expensive than what you run during development. IMHO, this should be
>>>>>>
>>>>> fairly
>>>>>
>>>>>> comprehensive (certainly tests and checkstyle, maybe javadoc and
>>>>>>
>>>>> findbugs).
>>>>>
>>>>>> This should be the "default" command that the one-time-contributor
>>>>>>
>>>>> should
>>>>>
>>>>>> know. For those compiling 50x or more a day, I think the burden of
>>>>>>
>>>>> learning
>>>>>
>>>>>> a second (or more) cheaper commands is not high, and we could even
>>>>>>
>>>>> put such
>>>>>
>>>>>> a thing in the docs (and hopefully a common maven convention like
>>>>>>
>>>>> "mvn
>>>>>
>>>>>> test").
>>>>>>
>>>>>> I've listed the fraction of commits I think will break one of the
>>>>>>
>>>>> following
>>>>>
>>>>>> if that property is not tested:
>>>>>>>
>>>>>>> * compiling (100%)
>>>>>>> * tests (100%)
>>>>>>> * checkstyle (90%)
>>>>>>> * javadoc (30%)
>>>>>>> * findbugs (5%)
>>>>>>> * rat (1%)
>>>>>>>
>>>>>>> So you can see where I stand and why. I'm sorry that 1/20 PRs has
>>>>>>>
>>>>>> Apache
>>>>>
>>>>>> RAT catch a licensing issue or Findbugs catch a threading issue --
>>>>>>>
>>>>>> you
>>>>>
>>>>>> can
>>>>>>
>>>>>>> always get a larger set of the precommit checks using -Prelease,
>>>>>>>
>>>>>> though
>>>>>
>>>>>> of
>>>>>>
>>>>>>> course the integration tests and runnableonservice tests may catch
>>>>>>>
>>>>>> more
>>>>>
>>>>>> issues still. But I want my developer minutes back for the 95%+ of
>>>>>>>
>>>>>> the
>>>>>
>>>>>> rest.
>>>>>>>
>>>>>>> Dan
>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>>
>>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
Re: Better developer instructions for using Maven?
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Thanks Aviem,
Not sure if we should use skipTests as it really means unit tests and
integration tests (in Karaf, skipTests skips the unit test, integration
test, archetype itests and maven plugin invoker test, but it doesn't
skip checkstyle, findbugs, etc, for that, we have a fastinstall property).
Maybe it would make more sense to use a specific property like
-DfastBuild=true.
WDYT ?
Regards
JB
On 02/19/2017 09:27 PM, Aviem Zur wrote:
> I've created a PR which disables slow verifications if '-DskipTests' was
> specified, otherwise runs them.
> I think this satisfies all the considerations mentioned in this thread.
>
> PTAL: https://github.com/apache/beam/pull/2048
> Ticket: https://issues.apache.org/jira/browse/BEAM-1513
>
> On Thu, Feb 16, 2017 at 3:05 PM Isma�l Mej�a <ie...@gmail.com> wrote:
>
>> JB, Maybe I was not clear, when I talked about the tests I was thinking
>> more about execute them in parallel in the same machine, this is not the
>> case today for some test suites, and for these the tests need to be refined
>> to support this, and configured via the pom to execute the tests in
>> parallel per method, class, etc. Of course we need to check if this is
>> worth, because I can imagine that the more expensive time for example in
>> the IO case comes from starting the embedded versions of the IOs (e.g.
>> HadoopMiniCluster, MongodExecutable, HBasetestingutility, etc) and not from
>> the tests themselves but this has to be evaluated.
>>
>>
>>
>> On Wed, Feb 15, 2017 at 5:46 PM, Jean-Baptiste Onofr� <jb...@nanthrax.net>
>> wrote:
>>
>>> On Jenkins it's possible to run several jobs in the same time but on
>>> different executor. That's the easiest way.
>>>
>>> Regards
>>> JB
>>>
>>> On Feb 15, 2017, 10:15, at 10:15, "Isma�l Mej�a" <ie...@gmail.com>
>>> wrote:
>>>> This question got lost in the discussion, but there is a small
>>>> improvement
>>>> that we can do:
>>>>
>>>>> Just to check, are we doing parallel builds?
>>>>
>>>> We are on jenkins, not in travis, there is an ongoing PR to fix this.
>>>>
>>>> What we can improve is to check if we can run some of the test suites
>>>> in
>>>> parallel to gain some extra time. For exemple most of the IOs and some
>>>> runners don't execute tests in parallel.
>>>>
>>>> Ismael
>>>>
>>>> (slightly related), is there a way to get change the timeout of travis
>>>> jobs). Because it is failing most of the time now because of this, and
>>>> it
>>>> is quite noisey to have so many false positives.
>>>>
>>>>
>>>>
>>>>
>>>> On Fri, Feb 10, 2017 at 8:00 PM, Robert Bradshaw <
>>>> robertwb@google.com.invalid> wrote:
>>>>
>>>>> On Fri, Feb 10, 2017 at 8:45 AM, Dan Halperin
>>>> <dhalperi@google.com.invalid
>>>>>>
>>>>> wrote:
>>>>>
>>>>>> On Fri, Feb 10, 2017 at 7:42 AM, Kenneth Knowles
>>>> <klk@google.com.invalid
>>>>>>
>>>>>> wrote:
>>>>>>
>>>>>>> On Feb 10, 2017 07:36, "Dan Halperin"
>>>> <dh...@google.com.invalid>
>>>>>> wrote:
>>>>>>>
>>>>>>> Before we added checkstyle it was under a minute. Now it's over
>>>> five?
>>>>>>> That's awful IMO
>>>>>>>
>>>>>>>
>>>>>>> Checkstyle didn't cause all that, did it?
>>>>>>>
>>>>>>
>>>>>> The "5 minutes" was going with Aviem's numbers after this change.
>>>> But
>>>>> yes,
>>>>>> Checkstyle alone substantially (>+50%) the time from what it was
>>>>> previously
>>>>>> to adding it back to the default build.
>>>>>
>>>>>
>>>>> Just to check, are we doing parallel builds?
>>>>>
>>>>>
>>>>>>
>>>>>> Noting that findbugs takes quite a lot more time. Javadoc and jar
>>>> are the
>>>>>>> other two slow ones.
>>>>>>>
>>>>>>> RAT is fast. But it has very poor error messages, so we wouldn't
>>>> want a
>>>>>> new
>>>>>>> contributor trying to figure out what is going on without our
>>>> help.
>>>>>>>
>>>>>>
>>>>>> There is a larger philosophical issue here: is there a point of
>>>> Jenkins
>>>>>> precommit testing? Why not just make `mvn install` run everything
>>>> that
>>>>>> Jenkins does? For that matter, why don't committers just push
>>>> directly to
>>>>>> master? Wouldn't that make everyone's life easier?
>>>>>>
>>>>>> I'd argue that's not true.
>>>>>>
>>>>>> 1. Developer productivity -- Jenkins should run many more checks
>>>> than
>>>>>> developers do. Especially time-, resource-, or setup- intensive
>>>> tasks.
>>>>>> 2. Automated enforcement -- Jenkins is better at running the right
>>>>> commands
>>>>>> than we are.
>>>>>> 3. Lower the barrier to entry -- individual developers need not
>>>> have a
>>>>>> running Spark/Flink/Apex/Dataflow setup in order to contribute
>>>> code.
>>>>>> 4. Focus on the user -- someone checking out the code and using it
>>>> for
>>>>> the
>>>>>> first time does not care whether the code style checks or has the
>>>> right
>>>>>> licenses -- that should have been enforced by the Beam team before
>>>>>> committing.
>>>>>>
>>>>>> We should be *very* choosy about what we enforce on every developer
>>>> every
>>>>>> time they go to compile. I probably compile Beam 50x-100x a day.
>>>>> Literally,
>>>>>> the extra minutes you want to add here will cost me an hour daily.
>>>>>>
>>>>>
>>>>> By the same token of having a different bar for the Jenkins presubmit
>>>> vs.
>>>>> what's run locally, I think it makes a lot of sense to run a
>>>> different
>>>>> command for iterative development than you run before creating a pull
>>>>> request. E.g. during development I'll often run only one test rather
>>>> than
>>>>> the entire suite, but do run the entire suite occasionally (often
>>>> before
>>>>> commit, especially before pushing).
>>>>>
>>>>> The contributors guild should give a suggested command to run before
>>>>> creating a PR, right in the docs of how to create a PR, which may be
>>>> more
>>>>> expensive than what you run during development. IMHO, this should be
>>>> fairly
>>>>> comprehensive (certainly tests and checkstyle, maybe javadoc and
>>>> findbugs).
>>>>> This should be the "default" command that the one-time-contributor
>>>> should
>>>>> know. For those compiling 50x or more a day, I think the burden of
>>>> learning
>>>>> a second (or more) cheaper commands is not high, and we could even
>>>> put such
>>>>> a thing in the docs (and hopefully a common maven convention like
>>>> "mvn
>>>>> test").
>>>>>
>>>>> I've listed the fraction of commits I think will break one of the
>>>> following
>>>>>> if that property is not tested:
>>>>>>
>>>>>> * compiling (100%)
>>>>>> * tests (100%)
>>>>>> * checkstyle (90%)
>>>>>> * javadoc (30%)
>>>>>> * findbugs (5%)
>>>>>> * rat (1%)
>>>>>>
>>>>>> So you can see where I stand and why. I'm sorry that 1/20 PRs has
>>>> Apache
>>>>>> RAT catch a licensing issue or Findbugs catch a threading issue --
>>>> you
>>>>> can
>>>>>> always get a larger set of the precommit checks using -Prelease,
>>>> though
>>>>> of
>>>>>> course the integration tests and runnableonservice tests may catch
>>>> more
>>>>>> issues still. But I want my developer minutes back for the 95%+ of
>>>> the
>>>>>> rest.
>>>>>>
>>>>>> Dan
>>>>>>
>>>>>
>>>
>>
>
--
Jean-Baptiste Onofr�
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: Better developer instructions for using Maven?
Posted by Aviem Zur <av...@gmail.com>.
I've created a PR which disables slow verifications if '-DskipTests' was
specified, otherwise runs them.
I think this satisfies all the considerations mentioned in this thread.
PTAL: https://github.com/apache/beam/pull/2048
Ticket: https://issues.apache.org/jira/browse/BEAM-1513
On Thu, Feb 16, 2017 at 3:05 PM Ismaël Mejía <ie...@gmail.com> wrote:
> JB, Maybe I was not clear, when I talked about the tests I was thinking
> more about execute them in parallel in the same machine, this is not the
> case today for some test suites, and for these the tests need to be refined
> to support this, and configured via the pom to execute the tests in
> parallel per method, class, etc. Of course we need to check if this is
> worth, because I can imagine that the more expensive time for example in
> the IO case comes from starting the embedded versions of the IOs (e.g.
> HadoopMiniCluster, MongodExecutable, HBasetestingutility, etc) and not from
> the tests themselves but this has to be evaluated.
>
>
>
> On Wed, Feb 15, 2017 at 5:46 PM, Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
>
> > On Jenkins it's possible to run several jobs in the same time but on
> > different executor. That's the easiest way.
> >
> > Regards
> > JB
> >
> > On Feb 15, 2017, 10:15, at 10:15, "Ismaël Mejía" <ie...@gmail.com>
> > wrote:
> > >This question got lost in the discussion, but there is a small
> > >improvement
> > >that we can do:
> > >
> > >> Just to check, are we doing parallel builds?
> > >
> > >We are on jenkins, not in travis, there is an ongoing PR to fix this.
> > >
> > >What we can improve is to check if we can run some of the test suites
> > >in
> > >parallel to gain some extra time. For exemple most of the IOs and some
> > >runners don't execute tests in parallel.
> > >
> > >Ismael
> > >
> > >(slightly related), is there a way to get change the timeout of travis
> > >jobs). Because it is failing most of the time now because of this, and
> > >it
> > >is quite noisey to have so many false positives.
> > >
> > >
> > >
> > >
> > >On Fri, Feb 10, 2017 at 8:00 PM, Robert Bradshaw <
> > >robertwb@google.com.invalid> wrote:
> > >
> > >> On Fri, Feb 10, 2017 at 8:45 AM, Dan Halperin
> > ><dhalperi@google.com.invalid
> > >> >
> > >> wrote:
> > >>
> > >> > On Fri, Feb 10, 2017 at 7:42 AM, Kenneth Knowles
> > ><klk@google.com.invalid
> > >> >
> > >> > wrote:
> > >> >
> > >> > > On Feb 10, 2017 07:36, "Dan Halperin"
> > ><dh...@google.com.invalid>
> > >> > wrote:
> > >> > >
> > >> > > Before we added checkstyle it was under a minute. Now it's over
> > >five?
> > >> > > That's awful IMO
> > >> > >
> > >> > >
> > >> > > Checkstyle didn't cause all that, did it?
> > >> > >
> > >> >
> > >> > The "5 minutes" was going with Aviem's numbers after this change.
> > >But
> > >> yes,
> > >> > Checkstyle alone substantially (>+50%) the time from what it was
> > >> previously
> > >> > to adding it back to the default build.
> > >>
> > >>
> > >> Just to check, are we doing parallel builds?
> > >>
> > >>
> > >> >
> > >> > Noting that findbugs takes quite a lot more time. Javadoc and jar
> > >are the
> > >> > > other two slow ones.
> > >> > >
> > >> > > RAT is fast. But it has very poor error messages, so we wouldn't
> > >want a
> > >> > new
> > >> > > contributor trying to figure out what is going on without our
> > >help.
> > >> > >
> > >> >
> > >> > There is a larger philosophical issue here: is there a point of
> > >Jenkins
> > >> > precommit testing? Why not just make `mvn install` run everything
> > >that
> > >> > Jenkins does? For that matter, why don't committers just push
> > >directly to
> > >> > master? Wouldn't that make everyone's life easier?
> > >> >
> > >> > I'd argue that's not true.
> > >> >
> > >> > 1. Developer productivity -- Jenkins should run many more checks
> > >than
> > >> > developers do. Especially time-, resource-, or setup- intensive
> > >tasks.
> > >> > 2. Automated enforcement -- Jenkins is better at running the right
> > >> commands
> > >> > than we are.
> > >> > 3. Lower the barrier to entry -- individual developers need not
> > >have a
> > >> > running Spark/Flink/Apex/Dataflow setup in order to contribute
> > >code.
> > >> > 4. Focus on the user -- someone checking out the code and using it
> > >for
> > >> the
> > >> > first time does not care whether the code style checks or has the
> > >right
> > >> > licenses -- that should have been enforced by the Beam team before
> > >> > committing.
> > >> >
> > >> > We should be *very* choosy about what we enforce on every developer
> > >every
> > >> > time they go to compile. I probably compile Beam 50x-100x a day.
> > >> Literally,
> > >> > the extra minutes you want to add here will cost me an hour daily.
> > >> >
> > >>
> > >> By the same token of having a different bar for the Jenkins presubmit
> > >vs.
> > >> what's run locally, I think it makes a lot of sense to run a
> > >different
> > >> command for iterative development than you run before creating a pull
> > >> request. E.g. during development I'll often run only one test rather
> > >than
> > >> the entire suite, but do run the entire suite occasionally (often
> > >before
> > >> commit, especially before pushing).
> > >>
> > >> The contributors guild should give a suggested command to run before
> > >> creating a PR, right in the docs of how to create a PR, which may be
> > >more
> > >> expensive than what you run during development. IMHO, this should be
> > >fairly
> > >> comprehensive (certainly tests and checkstyle, maybe javadoc and
> > >findbugs).
> > >> This should be the "default" command that the one-time-contributor
> > >should
> > >> know. For those compiling 50x or more a day, I think the burden of
> > >learning
> > >> a second (or more) cheaper commands is not high, and we could even
> > >put such
> > >> a thing in the docs (and hopefully a common maven convention like
> > >"mvn
> > >> test").
> > >>
> > >> I've listed the fraction of commits I think will break one of the
> > >following
> > >> > if that property is not tested:
> > >> >
> > >> > * compiling (100%)
> > >> > * tests (100%)
> > >> > * checkstyle (90%)
> > >> > * javadoc (30%)
> > >> > * findbugs (5%)
> > >> > * rat (1%)
> > >> >
> > >> > So you can see where I stand and why. I'm sorry that 1/20 PRs has
> > >Apache
> > >> > RAT catch a licensing issue or Findbugs catch a threading issue --
> > >you
> > >> can
> > >> > always get a larger set of the precommit checks using -Prelease,
> > >though
> > >> of
> > >> > course the integration tests and runnableonservice tests may catch
> > >more
> > >> > issues still. But I want my developer minutes back for the 95%+ of
> > >the
> > >> > rest.
> > >> >
> > >> > Dan
> > >> >
> > >>
> >
>
Re: Better developer instructions for using Maven?
Posted by Ismaël Mejía <ie...@gmail.com>.
JB, Maybe I was not clear, when I talked about the tests I was thinking
more about execute them in parallel in the same machine, this is not the
case today for some test suites, and for these the tests need to be refined
to support this, and configured via the pom to execute the tests in
parallel per method, class, etc. Of course we need to check if this is
worth, because I can imagine that the more expensive time for example in
the IO case comes from starting the embedded versions of the IOs (e.g.
HadoopMiniCluster, MongodExecutable, HBasetestingutility, etc) and not from
the tests themselves but this has to be evaluated.
On Wed, Feb 15, 2017 at 5:46 PM, Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:
> On Jenkins it's possible to run several jobs in the same time but on
> different executor. That's the easiest way.
>
> Regards
> JB
>
> On Feb 15, 2017, 10:15, at 10:15, "Ismaël Mejía" <ie...@gmail.com>
> wrote:
> >This question got lost in the discussion, but there is a small
> >improvement
> >that we can do:
> >
> >> Just to check, are we doing parallel builds?
> >
> >We are on jenkins, not in travis, there is an ongoing PR to fix this.
> >
> >What we can improve is to check if we can run some of the test suites
> >in
> >parallel to gain some extra time. For exemple most of the IOs and some
> >runners don't execute tests in parallel.
> >
> >Ismael
> >
> >(slightly related), is there a way to get change the timeout of travis
> >jobs). Because it is failing most of the time now because of this, and
> >it
> >is quite noisey to have so many false positives.
> >
> >
> >
> >
> >On Fri, Feb 10, 2017 at 8:00 PM, Robert Bradshaw <
> >robertwb@google.com.invalid> wrote:
> >
> >> On Fri, Feb 10, 2017 at 8:45 AM, Dan Halperin
> ><dhalperi@google.com.invalid
> >> >
> >> wrote:
> >>
> >> > On Fri, Feb 10, 2017 at 7:42 AM, Kenneth Knowles
> ><klk@google.com.invalid
> >> >
> >> > wrote:
> >> >
> >> > > On Feb 10, 2017 07:36, "Dan Halperin"
> ><dh...@google.com.invalid>
> >> > wrote:
> >> > >
> >> > > Before we added checkstyle it was under a minute. Now it's over
> >five?
> >> > > That's awful IMO
> >> > >
> >> > >
> >> > > Checkstyle didn't cause all that, did it?
> >> > >
> >> >
> >> > The "5 minutes" was going with Aviem's numbers after this change.
> >But
> >> yes,
> >> > Checkstyle alone substantially (>+50%) the time from what it was
> >> previously
> >> > to adding it back to the default build.
> >>
> >>
> >> Just to check, are we doing parallel builds?
> >>
> >>
> >> >
> >> > Noting that findbugs takes quite a lot more time. Javadoc and jar
> >are the
> >> > > other two slow ones.
> >> > >
> >> > > RAT is fast. But it has very poor error messages, so we wouldn't
> >want a
> >> > new
> >> > > contributor trying to figure out what is going on without our
> >help.
> >> > >
> >> >
> >> > There is a larger philosophical issue here: is there a point of
> >Jenkins
> >> > precommit testing? Why not just make `mvn install` run everything
> >that
> >> > Jenkins does? For that matter, why don't committers just push
> >directly to
> >> > master? Wouldn't that make everyone's life easier?
> >> >
> >> > I'd argue that's not true.
> >> >
> >> > 1. Developer productivity -- Jenkins should run many more checks
> >than
> >> > developers do. Especially time-, resource-, or setup- intensive
> >tasks.
> >> > 2. Automated enforcement -- Jenkins is better at running the right
> >> commands
> >> > than we are.
> >> > 3. Lower the barrier to entry -- individual developers need not
> >have a
> >> > running Spark/Flink/Apex/Dataflow setup in order to contribute
> >code.
> >> > 4. Focus on the user -- someone checking out the code and using it
> >for
> >> the
> >> > first time does not care whether the code style checks or has the
> >right
> >> > licenses -- that should have been enforced by the Beam team before
> >> > committing.
> >> >
> >> > We should be *very* choosy about what we enforce on every developer
> >every
> >> > time they go to compile. I probably compile Beam 50x-100x a day.
> >> Literally,
> >> > the extra minutes you want to add here will cost me an hour daily.
> >> >
> >>
> >> By the same token of having a different bar for the Jenkins presubmit
> >vs.
> >> what's run locally, I think it makes a lot of sense to run a
> >different
> >> command for iterative development than you run before creating a pull
> >> request. E.g. during development I'll often run only one test rather
> >than
> >> the entire suite, but do run the entire suite occasionally (often
> >before
> >> commit, especially before pushing).
> >>
> >> The contributors guild should give a suggested command to run before
> >> creating a PR, right in the docs of how to create a PR, which may be
> >more
> >> expensive than what you run during development. IMHO, this should be
> >fairly
> >> comprehensive (certainly tests and checkstyle, maybe javadoc and
> >findbugs).
> >> This should be the "default" command that the one-time-contributor
> >should
> >> know. For those compiling 50x or more a day, I think the burden of
> >learning
> >> a second (or more) cheaper commands is not high, and we could even
> >put such
> >> a thing in the docs (and hopefully a common maven convention like
> >"mvn
> >> test").
> >>
> >> I've listed the fraction of commits I think will break one of the
> >following
> >> > if that property is not tested:
> >> >
> >> > * compiling (100%)
> >> > * tests (100%)
> >> > * checkstyle (90%)
> >> > * javadoc (30%)
> >> > * findbugs (5%)
> >> > * rat (1%)
> >> >
> >> > So you can see where I stand and why. I'm sorry that 1/20 PRs has
> >Apache
> >> > RAT catch a licensing issue or Findbugs catch a threading issue --
> >you
> >> can
> >> > always get a larger set of the precommit checks using -Prelease,
> >though
> >> of
> >> > course the integration tests and runnableonservice tests may catch
> >more
> >> > issues still. But I want my developer minutes back for the 95%+ of
> >the
> >> > rest.
> >> >
> >> > Dan
> >> >
> >>
>
Re: Better developer instructions for using Maven?
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
On Jenkins it's possible to run several jobs in the same time but on different executor. That's the easiest way.
Regards
JB
On Feb 15, 2017, 10:15, at 10:15, "Ismaël Mejía" <ie...@gmail.com> wrote:
>This question got lost in the discussion, but there is a small
>improvement
>that we can do:
>
>> Just to check, are we doing parallel builds?
>
>We are on jenkins, not in travis, there is an ongoing PR to fix this.
>
>What we can improve is to check if we can run some of the test suites
>in
>parallel to gain some extra time. For exemple most of the IOs and some
>runners don't execute tests in parallel.
>
>Ismael
>
>(slightly related), is there a way to get change the timeout of travis
>jobs). Because it is failing most of the time now because of this, and
>it
>is quite noisey to have so many false positives.
>
>
>
>
>On Fri, Feb 10, 2017 at 8:00 PM, Robert Bradshaw <
>robertwb@google.com.invalid> wrote:
>
>> On Fri, Feb 10, 2017 at 8:45 AM, Dan Halperin
><dhalperi@google.com.invalid
>> >
>> wrote:
>>
>> > On Fri, Feb 10, 2017 at 7:42 AM, Kenneth Knowles
><klk@google.com.invalid
>> >
>> > wrote:
>> >
>> > > On Feb 10, 2017 07:36, "Dan Halperin"
><dh...@google.com.invalid>
>> > wrote:
>> > >
>> > > Before we added checkstyle it was under a minute. Now it's over
>five?
>> > > That's awful IMO
>> > >
>> > >
>> > > Checkstyle didn't cause all that, did it?
>> > >
>> >
>> > The "5 minutes" was going with Aviem's numbers after this change.
>But
>> yes,
>> > Checkstyle alone substantially (>+50%) the time from what it was
>> previously
>> > to adding it back to the default build.
>>
>>
>> Just to check, are we doing parallel builds?
>>
>>
>> >
>> > Noting that findbugs takes quite a lot more time. Javadoc and jar
>are the
>> > > other two slow ones.
>> > >
>> > > RAT is fast. But it has very poor error messages, so we wouldn't
>want a
>> > new
>> > > contributor trying to figure out what is going on without our
>help.
>> > >
>> >
>> > There is a larger philosophical issue here: is there a point of
>Jenkins
>> > precommit testing? Why not just make `mvn install` run everything
>that
>> > Jenkins does? For that matter, why don't committers just push
>directly to
>> > master? Wouldn't that make everyone's life easier?
>> >
>> > I'd argue that's not true.
>> >
>> > 1. Developer productivity -- Jenkins should run many more checks
>than
>> > developers do. Especially time-, resource-, or setup- intensive
>tasks.
>> > 2. Automated enforcement -- Jenkins is better at running the right
>> commands
>> > than we are.
>> > 3. Lower the barrier to entry -- individual developers need not
>have a
>> > running Spark/Flink/Apex/Dataflow setup in order to contribute
>code.
>> > 4. Focus on the user -- someone checking out the code and using it
>for
>> the
>> > first time does not care whether the code style checks or has the
>right
>> > licenses -- that should have been enforced by the Beam team before
>> > committing.
>> >
>> > We should be *very* choosy about what we enforce on every developer
>every
>> > time they go to compile. I probably compile Beam 50x-100x a day.
>> Literally,
>> > the extra minutes you want to add here will cost me an hour daily.
>> >
>>
>> By the same token of having a different bar for the Jenkins presubmit
>vs.
>> what's run locally, I think it makes a lot of sense to run a
>different
>> command for iterative development than you run before creating a pull
>> request. E.g. during development I'll often run only one test rather
>than
>> the entire suite, but do run the entire suite occasionally (often
>before
>> commit, especially before pushing).
>>
>> The contributors guild should give a suggested command to run before
>> creating a PR, right in the docs of how to create a PR, which may be
>more
>> expensive than what you run during development. IMHO, this should be
>fairly
>> comprehensive (certainly tests and checkstyle, maybe javadoc and
>findbugs).
>> This should be the "default" command that the one-time-contributor
>should
>> know. For those compiling 50x or more a day, I think the burden of
>learning
>> a second (or more) cheaper commands is not high, and we could even
>put such
>> a thing in the docs (and hopefully a common maven convention like
>"mvn
>> test").
>>
>> I've listed the fraction of commits I think will break one of the
>following
>> > if that property is not tested:
>> >
>> > * compiling (100%)
>> > * tests (100%)
>> > * checkstyle (90%)
>> > * javadoc (30%)
>> > * findbugs (5%)
>> > * rat (1%)
>> >
>> > So you can see where I stand and why. I'm sorry that 1/20 PRs has
>Apache
>> > RAT catch a licensing issue or Findbugs catch a threading issue --
>you
>> can
>> > always get a larger set of the precommit checks using -Prelease,
>though
>> of
>> > course the integration tests and runnableonservice tests may catch
>more
>> > issues still. But I want my developer minutes back for the 95%+ of
>the
>> > rest.
>> >
>> > Dan
>> >
>>
Re: Better developer instructions for using Maven?
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
I didn't read correctly Aviem's email: I thought checkstyle + rat + findbugs just add 1 mn 50 to the build time.
Agree that 5 mn time extend is too long and it's better to have on a specific profile
Regards
JB
On Feb 10, 2017, 11:36, at 11:36, Dan Halperin <dh...@google.com.INVALID> wrote:
>Before we added checkstyle it was under a minute. Now it's over five?
>That's awful IMO
>On Fri, Feb 10, 2017 at 07:14 Aviem Zur <av...@gmail.com> wrote:
>
>> Opened JIRA ticket: https://issues.apache.org/jira/browse/BEAM-1457
>>
>> On Fri, Feb 10, 2017 at 4:54 PM Jean-Baptiste Onofré
><jb...@nanthrax.net>
>> wrote:
>>
>> > Yeah. Agree. Time extend is not huge and it's worth to add it in
>verify
>> > phase.
>> >
>> > Regards
>> > JB
>> >
>> > On Feb 10, 2017, 10:13, at 10:13, Aviem Zur <av...@gmail.com>
>wrote:
>> > >This goes back to the original discussion in this thread - reduce
>the
>> > >amount of things pull requesters should know and keep the maven
>command
>> > >in
>> > >the PR checklist as: 'mvn clean verify'.
>> > >
>> > >So if rat and findbugs do not take that long to run I think they
>should
>> > >be
>> > >run by 'mvn clean verify'
>> > >
>> > >I ran a quick test on my laptop to see how much time they add to
>the
>> > >build
>> > >(of the entire project):
>> > >
>> > >'mvn clean install -DskipTests' => Total time: 03:51 min
>> > >'mvn clean install apache-rat:check findbugs:check -DskipTests'
>=>
>> > >Total
>> > >time: 05:29 min (Added 01:38 min)
>> > >'mvn clean install' => Total time: 09:37 min
>> > >'mvn clean install apache-rat:check findbugs:check' => Total time:
>> > >11:13
>> > >min (Added 01:36 min)
>> > >
>> > >Are these times reasonable enough to add rat and findbugs to the
>> > >default
>> > >build?
>> > >
>> > >On Fri, Feb 10, 2017 at 1:55 PM Jean-Baptiste Onofré
><jb...@nanthrax.net>
>> > >wrote:
>> > >
>> > >> Hi
>> > >>
>> > >> We discussed about that at the beginning of the project. We
>agreed to
>> > >> execute rat and findbugs in a specific profile to reduce the
>build
>> > >time for
>> > >> dev.
>> > >>
>> > >> That's why I do mvn clean install -Prelease before submitting a
>PR
>> > >and
>> > >> just clean install when I'm developing.
>> > >>
>> > >> No problem to change that.
>> > >>
>> > >> Regards
>> > >> JB
>> > >>
>> > >> On Feb 10, 2017, 07:51, at 07:51, Aviem Zur <av...@gmail.com>
>> > >wrote:
>> > >> >Can we consider adding rat-plugin and findbugs to the default
>verify
>> > >> >phase?
>> > >> >Currently they only run when the `release` profile is enabled.
>> > >> >
>> > >> >On Thu, Jan 26, 2017 at 11:42 AM Aljoscha Krettek
>> > ><al...@apache.org>
>> > >> >wrote:
>> > >> >
>> > >> >> +1 to what Dan said
>> > >> >>
>> > >> >> On Wed, 25 Jan 2017 at 21:40 Kenneth Knowles
>> > ><kl...@google.com.invalid>
>> > >> >> wrote:
>> > >> >>
>> > >> >> > +1
>> > >> >> >
>> > >> >> > On Jan 25, 2017 11:15, "Jean-Baptiste Onofré"
><jb...@nanthrax.net>
>> > >> >wrote:
>> > >> >> >
>> > >> >> > > +1
>> > >> >> > >
>> > >> >> > > It sounds good to me.
>> > >> >> > >
>> > >> >> > > Thanks Dan !
>> > >> >> > >
>> > >> >> > > Regards
>> > >> >> > > JB
>> > >> >> > >
>> > >> >> > > On Jan 25, 2017, 19:39, at 19:39, Dan Halperin
>> > >> >> > <dh...@google.com.INVALID>
>> > >> >> > > wrote:
>> > >> >> > > >Here is my summary of the threads:
>> > >> >> > > >
>> > >> >> > > >Overwhelming agreement:
>> > >> >> > > >
>> > >> >> > > >- rename `release` to something more appropriate.
>> > >> >> > > >- add `checkstyle` to the default build (it's basically
>a
>> > >> >compile
>> > >> >> > > >error)
>> > >> >> > > >- add more information to contributor guide
>> > >> >> > > >
>> > >> >> > > >Reasonable agreement
>> > >> >> > > >
>> > >> >> > > >- don't update the github instructions to make passing
>`mvn
>> > >> >verify
>> > >> >> > > >-P<all
>> > >> >> > > >checks>` mandatory. Maybe add a hint that this is a good
>> > >proxy
>> > >> >for
>> > >> >> what
>> > >> >> > > >Jenkins will run.
>> > >> >> > > >
>> > >> >> > > >Unresolved:
>> > >> >> > > >
>> > >> >> > > >- whether all checks should be in `mvn verify`
>> > >> >> > > >- whether `mvn test` is useful for most workflows
>> > >> >> > > >
>> > >> >> > > >I'll propose to proceed with the overwhelmingly
>agreed-upon
>> > >> >changes,
>> > >> >> > > >and as
>> > >> >> > > >we see increasingly many new contributors re-evaluate
>the
>> > >> >remaining
>> > >> >> > > >issues.
>> > >> >> > > >
>> > >> >> > > >Thanks,
>> > >> >> > > >Dan
>> > >> >> > > >
>> > >> >> > > >On Tue, Jan 24, 2017 at 12:51 PM, Jean-Baptiste Onofré
>> > >> >> > > ><jb...@nanthrax.net>
>> > >> >> > > >wrote:
>> > >> >> > > >
>> > >> >> > > >> +1 to at least update the contribution guide and
>improve
>> > >the
>> > >> >profile
>> > >> >> > > >name.
>> > >> >> > > >>
>> > >> >> > > >> Regards
>> > >> >> > > >> JB
>> > >> >> > > >>
>> > >> >> > > >>
>> > >> >> > > >> On 01/24/2017 09:49 PM, Kenneth Knowles wrote:
>> > >> >> > > >>
>> > >> >> > > >>> My impression is that we don't have consensus on
>whether
>> > >all
>> > >> >checks
>> > >> >> > > >or
>> > >> >> > > >>> minimal checks should be the default, or whether we
>can
>> > >have
>> > >> >both
>> > >> >> > > >via `mvn
>> > >> >> > > >>> test` and `mvn verify`.
>> > >> >> > > >>>
>> > >> >> > > >>> But that doesn't prevent us from giving -P release a
>> > >better
>> > >> >name
>> > >> >> and
>> > >> >> > > >>> mentioning it in the dev guide and in some manner in
>our
>> > >PR
>> > >> >> > > >template.
>> > >> >> > > >>>
>> > >> >> > > >>> Right now we are living with the combination of the
>bad
>> > >> >aspects -
>> > >> >> > > >default
>> > >> >> > > >>> is not thorough but not actually fast and a thorough
>check
>> > >is
>> > >> >> > > >>> undocumented.
>> > >> >> > > >>>
>> > >> >> > > >>> On Tue, Jan 24, 2017 at 2:22 AM, Ismaël Mejía
>> > >> ><ie...@gmail.com>
>> > >> >> > > >wrote:
>> > >> >> > > >>>
>> > >> >> > > >>> I just wanted to know if we have achieved some
>consensus
>> > >> >about
>> > >> >> this,
>> > >> >> > > >I
>> > >> >> > > >>>> just
>> > >> >> > > >>>> saw this PR that reminded me about this discussion.
>> > >> >> > > >>>>
>> > >> >> > > >>>> https://github.com/apache/beam/pull/1829
>> > >> >> > > >>>>
>> > >> >> > > >>>> It is important that we mention the existing
>profiles
>> > >(and
>> > >> >the
>> > >> >> > > >intended
>> > >> >> > > >>>> checks) in the contribution guide (e.g. -Prelease
>(or
>> > >> >-Pall-checks
>> > >> >> > > >>>> triggers
>> > >> >> > > >>>> these validations).
>> > >> >> > > >>>>
>> > >> >> > > >>>> I can add this to the guide if you like once we
>define
>> > >the
>> > >> >checks
>> > >> >> > > >per
>> > >> >> > > >>>> stage/profile.
>> > >> >> > > >>>>
>> > >> >> > > >>>> Ismaël
>> > >> >> > > >>>>
>> > >> >> > > >>>>
>> > >> >> > > >>>> On Wed, Jan 11, 2017 at 8:12 AM, Aviem Zur
>> > >> ><av...@gmail.com>
>> > >> >> > > >wrote:
>> > >> >> > > >>>>
>> > >> >> > > >>>> I agree with Dan and Lukasz.
>> > >> >> > > >>>>> Developers should not be expected to know
>beforehand
>> > >which
>> > >> >> > > >specific
>> > >> >> > > >>>>> profiles to run.
>> > >> >> > > >>>>> The phase specified in the PR instructions
>(`verify`)
>> > >> >should run
>> > >> >> > > >all the
>> > >> >> > > >>>>> relevant verifications and be the "slower" build,
>while
>> > >a
>> > >> >> > > >preceding
>> > >> >> > > >>>>> lifecycle, such as `test`, should run the "faster"
>> > >> >verifications.
>> > >> >> > > >>>>>
>> > >> >> > > >>>>> Aviem.
>> > >> >> > > >>>>>
>> > >> >> > > >>>>> On Mon, Jan 9, 2017 at 7:57 PM Robert Bradshaw
>> > >> >> > > >>>>>
>> > >> >> > > >>>> <robertwb@google.com.invalid
>> > >> >> > > >>>>
>> > >> >> > > >>>>>
>> > >> >> > > >>>>>> wrote:
>> > >> >> > > >>>>>
>> > >> >> > > >>>>> On Mon, Jan 9, 2017 at 3:49 AM, Aljoscha Krettek
>> > >> >> > > ><al...@apache.org>
>> > >> >> > > >>>>>> wrote:
>> > >> >> > > >>>>>>
>> > >> >> > > >>>>>>> I also usually prefer "mvn verify" to to the
>expected
>> > >> >thing but
>> > >> >> > > >I see
>> > >> >> > > >>>>>>>
>> > >> >> > > >>>>>> that
>> > >> >> > > >>>>>>
>> > >> >> > > >>>>>>> quick iteration times are key.
>> > >> >> > > >>>>>>>
>> > >> >> > > >>>>>>
>> > >> >> > > >>>>>> I see
>> > >> >> > > >>>>>> https://maven.apache.org/guides/introduction/
>> > >> >> > > >>>>>>
>> > >> >> > > >>>>> introduction-to-the-lifecycle.html
>> > >> >> > > >>>>>
>> > >> >> > > >>>>>>
>> > >> >> > > >>>>>> verify - run any checks on results of
>integration
>> > >> >tests to
>> > >> >> > > >ensure
>> > >> >> > > >>>>>> quality criteria are met
>> > >> >> > > >>>>>>
>> > >> >> > > >>>>>> Of course our integration tests are long enough
>that we
>> > >> >> shouldn't
>> > >> >> > > >be
>> > >> >> > > >>>>>> putting all of them here, but I too would expect
>> > >> >checkstyle.
>> > >> >> > > >>>>>>
>> > >> >> > > >>>>>> Perhaps we could introduce a verify-fast or
>somesuch
>> > >for
>> > >> >fast
>> > >> >> > > >(but
>> > >> >> > > >>>>>> lower coverage) turnaround time. I would expect
>"mvn
>> > >> >verify
>> > >> >> test"
>> > >> >> > > >to
>> > >> >> > > >>>>>> pass before submitting a PR, and would want to run
>that
>> > >> >before
>> > >> >> > > >asking
>> > >> >> > > >>>>>> others to look at it. I think this should be our
>> > >criteria
>> > >> >(i.e.
>> > >> >> > > >what
>> > >> >> > > >>>>>> will a new but maven-savvy user run before pushing
>> > >their
>> > >> >code).
>> > >> >> > > >>>>>>
>> > >> >> > > >>>>>> As long as the pre-commit hooks still check
>everything
>> > >I'm
>> > >> >ok
>> > >> >> > > >with
>> > >> >> > > >>>>>>>
>> > >> >> > > >>>>>> making
>> > >> >> > > >>>>>
>> > >> >> > > >>>>>> the default a little more lightweight.
>> > >> >> > > >>>>>>>
>> > >> >> > > >>>>>>
>> > >> >> > > >>>>>> The fact that our pre-commit hooks take a long
>time to
>> > >run
>> > >> >does
>> > >> >> > > >change
>> > >> >> > > >>>>>> things. Nothing more annoying than seeing that
>your PR
>> > >> >failed 3
>> > >> >> > > >hours
>> > >> >> > > >>>>>> later because you had some trailing whitespace...
>> > >> >> > > >>>>>>
>> > >> >> > > >>>>>> On Thu, 5 Jan 2017 at 21:49 Lukasz Cwik
>> > >> >> > > ><lc...@google.com.invalid>
>> > >> >> > > >>>>>>>
>> > >> >> > > >>>>>> wrote:
>> > >> >> > > >>>>>>
>> > >> >> > > >>>>>>>
>> > >> >> > > >>>>>>> I was hoping that the default mvn verify would be
>the
>> > >> >slow
>> > >> >> build
>> > >> >> > > >>>>>>>>
>> > >> >> > > >>>>>>> and a
>> > >> >> > > >>>>
>> > >> >> > > >>>>> profile could be enabled that would skip checks to
>make
>> > >> >things
>> > >> >> > > >>>>>>>>
>> > >> >> > > >>>>>>> faster
>> > >> >> > > >>>>
>> > >> >> > > >>>>> for
>> > >> >> > > >>>>>>
>> > >> >> > > >>>>>>> regular contributors. This way a person doesn't
>need
>> > >to
>> > >> >have
>> > >> >> > > >>>>>>>>
>> > >> >> > > >>>>>>> detailed
>> > >> >> > > >>>>
>> > >> >> > > >>>>> knowledge of all our profiles and what they do
>> > >(typically
>> > >> >mvn
>> > >> >> > > >>>>>>>>
>> > >> >> > > >>>>>>> verify)
>> > >> >> > > >>>>
>> > >> >> > > >>>>> will
>> > >> >> > > >>>>>>
>> > >> >> > > >>>>>>> do the right thing most of the time.
>> > >> >> > > >>>>>>>>
>> > >> >> > > >>>>>>>> On Thu, Jan 5, 2017 at 9:30 AM, Dan Halperin
>> > >> >> > > >>>>>>>>
>> > >> >> > > >>>>>>> <dh...@google.com.invalid>
>> > >> >> > > >>>>>>
>> > >> >> > > >>>>>>> wrote:
>> > >> >> > > >>>>>>>>
>> > >> >> > > >>>>>>>> On Thu, Jan 5, 2017 at 9:28 AM, Jesse Anderson <
>> > >> >> > > >>>>>>>>>
>> > >> >> > > >>>>>>>> jesse@smokinghand.com
>> > >> >> > > >>>>>
>> > >> >> > > >>>>>>
>> > >> >> > > >>>>>>> wrote:
>> > >> >> > > >>>>>>>>>
>> > >> >> > > >>>>>>>>> @dan are you saying that mvn verify isn't doing
>> > >> >checkstyle
>> > >> >> > > >>>>>>>>>>
>> > >> >> > > >>>>>>>>> anymore?
>> > >> >> > > >>>>>
>> > >> >> > > >>>>>>
>> > >> >> > > >>>>>>>>>
>> > >> >> > > >>>>>>>>> `mvn verify` alone should not be running
>checkstyle,
>> > >if
>> > >> >> > > >modules
>> > >> >> > > >>>>>>>>>
>> > >> >> > > >>>>>>>> are
>> > >> >> > > >>>>
>> > >> >> > > >>>>> configured correctly.
>> > >> >> > > >>>>>>>>>
>> > >> >> > > >>>>>>>>>
>> > >> >> > > >>>>>>>>> Some of
>> > >> >> > > >>>>>>>>>> the checkstyles are still running for a few
>> > >modules.
>> > >> >Also,
>> > >> >> > > >the
>> > >> >> > > >>>>>>>>>>
>> > >> >> > > >>>>>>>>> contribution
>> > >> >> > > >>>>>>>>>
>> > >> >> > > >>>>>>>>>> docs will need to change.
>> > >> >> > > >>>>>>>>>>
>> > >> >> > > >>>>>>>>>
>> > >> >> > > >>>>>>>>>
>> > >> >> > > >>>>>>>>> Yes. The PR includes discussion of these other
>> > >needed
>> > >> >> changes,
>> > >> >> > > >>>>>>>>> unfortunately one PR can't change two
>repositories.
>> > >> >> > > >>>>>>>>>
>> > >> >> > > >>>>>>>>> Please continue the discussion on the PR, then
>I
>> > >will
>> > >> >> > > >summarize it
>> > >> >> > > >>>>>>>>>
>> > >> >> > > >>>>>>>> back
>> > >> >> > > >>>>>>
>> > >> >> > > >>>>>>> into the dev thread.
>> > >> >> > > >>>>>>>>>
>> > >> >> > > >>>>>>>>> Thanks,
>> > >> >> > > >>>>>>>>> Dan
>> > >> >> > > >>>>>>>>>
>> > >> >> > > >>>>>>>>>
>> > >> >> > > >>>>>>>>> They say to run mvn verify before commits.
>> > >> >> > > >>>>>>>>>>
>> > >> >> > > >>>>>>>>>> On Thu, Jan 5, 2017 at 9:25 AM Dan Halperin
>> > >> >> > > >>>>>>>>>>
>> > >> >> > > >>>>>>>>> <dhalperi@google.com.invalid
>> > >> >> > > >>>>>>>>
>> > >> >> > > >>>>>>>>>
>> > >> >> > > >>>>>>>>>> wrote:
>> > >> >> > > >>>>>>>>>>
>> > >> >> > > >>>>>>>>>> Several folks seem to have been confused after
>> > >> >BEAM-246,
>> > >> >> > > >where
>> > >> >> > > >>>>>>>>>>>
>> > >> >> > > >>>>>>>>>> we
>> > >> >> > > >>>>>
>> > >> >> > > >>>>>> moved
>> > >> >> > > >>>>>>>>
>> > >> >> > > >>>>>>>>> the
>> > >> >> > > >>>>>>>>>>
>> > >> >> > > >>>>>>>>>>> "slow things" into the release profile. I've
>> > >started
>> > >> >a
>> > >> >> > > >>>>>>>>>>>
>> > >> >> > > >>>>>>>>>> discussion
>> > >> >> > > >>>>>
>> > >> >> > > >>>>>> with
>> > >> >> > > >>>>>>>>
>> > >> >> > > >>>>>>>>> https://github.com/apache/beam/pull/1740 to see
>if
>> > >> >there are
>> > >> >> > > >>>>>>>>>>>
>> > >> >> > > >>>>>>>>>> things
>> > >> >> > > >>>>>>
>> > >> >> > > >>>>>>> we
>> > >> >> > > >>>>>>>>
>> > >> >> > > >>>>>>>>> can
>> > >> >> > > >>>>>>>>>>
>> > >> >> > > >>>>>>>>>>> do to fill these gaps.
>> > >> >> > > >>>>>>>>>>>
>> > >> >> > > >>>>>>>>>>> Would love folks to chime in with opinions.
>> > >> >> > > >>>>>>>>>>>
>> > >> >> > > >>>>>>>>>>> Dan
>> > >> >> > > >>>>>>>>>>>
>> > >> >> > > >>>>>>>>>>> On Wed, Jan 4, 2017 at 1:34 PM, Jesse
>Anderson <
>> > >> >> > > >>>>>>>>>>>
>> > >> >> > > >>>>>>>>>> jesse@smokinghand.com>
>> > >> >> > > >>>>>>>>
>> > >> >> > > >>>>>>>>> wrote:
>> > >> >> > > >>>>>>>>>>>
>> > >> >> > > >>>>>>>>>>> @Eugene, yes that failed on the checkstyle.
>> > >> >> > > >>>>>>>>>>>>
>> > >> >> > > >>>>>>>>>>>> On Wed, Jan 4, 2017 at 1:27 PM Eugene
>Kirpichov
>> > >> >> > > >>>>>>>>>>>> <ki...@google.com.invalid> wrote:
>> > >> >> > > >>>>>>>>>>>>
>> > >> >> > > >>>>>>>>>>>> Try just -Prelease.
>> > >> >> > > >>>>>>>>>>>>> On Wed, Jan 4, 2017 at 1:21 PM Jesse
>Anderson <
>> > >> >> > > >>>>>>>>>>>>>
>> > >> >> > > >>>>>>>>>>>> jesse@smokinghand.com
>> > >> >> > > >>>>>>>>>
>> > >> >> > > >>>>>>>>>>
>> > >> >> > > >>>>>>>>>>> wrote:
>> > >> >> > > >>>>>>>>>>>>>
>> > >> >> > > >>>>>>>>>>>>> Fails because I don't have a secret key.
>> > >> >> > > >>>>>>>>>>>>>>
>> > >> >> > > >>>>>>>>>>>>>> On Wed, Jan 4, 2017 at 1:03 PM
>Jean-Baptiste
>> > >> >Onofré <
>> > >> >> > > >>>>>>>>>>>>>>
>> > >> >> > > >>>>>>>>>>>>> jb@nanthrax.net
>> > >> >> > > >>>>>>>>>>
>> > >> >> > > >>>>>>>>>>>
>> > >> >> > > >>>>>>>>>>>> wrote:
>> > >> >> > > >>>>>>>>>>>>>>
>> > >> >> > > >>>>>>>>>>>>>> Hi Jesse,
>> > >> >> > > >>>>>>>>>>>>>>>
>> > >> >> > > >>>>>>>>>>>>>>> Could you try the same with:
>> > >> >> > > >>>>>>>>>>>>>>>
>> > >> >> > > >>>>>>>>>>>>>>> mvn verify -Prelease,apache-release
>> > >> >> > > >>>>>>>>>>>>>>>
>> > >> >> > > >>>>>>>>>>>>>>> ?
>> > >> >> > > >>>>>>>>>>>>>>>
>> > >> >> > > >>>>>>>>>>>>>>> Regards
>> > >> >> > > >>>>>>>>>>>>>>> JB
>> > >> >> > > >>>>>>>>>>>>>>>
>> > >> >> > > >>>>>>>>>>>>>>> On 01/04/2017 09:53 PM, Jesse Anderson
>wrote:
>> > >> >> > > >>>>>>>>>>>>>>>
>> > >> >> > > >>>>>>>>>>>>>>>> For some reason, running "mvn verify"
>isn't
>> > >> >running
>> > >> >> > > >>>>>>>>>>>>>>>>
>> > >> >> > > >>>>>>>>>>>>>>> checkstyle
>> > >> >> > > >>>>>>>>>
>> > >> >> > > >>>>>>>>>> on
>> > >> >> > > >>>>>>>>>>
>> > >> >> > > >>>>>>>>>>> everything. I had checkstyle errors in
>> > >> >> > > >>>>>>>>>>>>>>>>
>> > >> >> > > >>>>>>>>>>>>>>> beam-sdks-java-core
>> > >> >> > > >>>>>>
>> > >> >> > > >>>>>>> that
>> > >> >> > > >>>>>>>>>
>> > >> >> > > >>>>>>>>>> weren't
>> > >> >> > > >>>>>>>>>>>>>
>> > >> >> > > >>>>>>>>>>>>>> being found.
>> > >> >> > > >>>>>>>>>>>>>>>>
>> > >> >> > > >>>>>>>>>>>>>>>> I thought this was due to the extra
>> > >parameters.
>> > >> >I
>> > >> >> > > >>>>>>>>>>>>>>>>
>> > >> >> > > >>>>>>>>>>>>>>> reran
>> > >> >> > > >>>>>
>> > >> >> > > >>>>>> with
>> > >> >> > > >>>>>>>>
>> > >> >> > > >>>>>>>>> the
>> > >> >> > > >>>>>>>>>>
>> > >> >> > > >>>>>>>>>>> plain
>> > >> >> > > >>>>>>>>>>>>>
>> > >> >> > > >>>>>>>>>>>>>> "mvn
>> > >> >> > > >>>>>>>>>>>>>>>
>> > >> >> > > >>>>>>>>>>>>>>>> verify" and it still didn't find them.
>From
>> > >the
>> > >> >> > > >>>>>>>>>>>>>>>>
>> > >> >> > > >>>>>>>>>>>>>>> output,
>> > >> >> > > >>>>>
>> > >> >> > > >>>>>> it
>> > >> >> > > >>>>>>
>> > >> >> > > >>>>>>> doesn't
>> > >> >> > > >>>>>>>>>>>
>> > >> >> > > >>>>>>>>>>>> look
>> > >> >> > > >>>>>>>>>>>>>
>> > >> >> > > >>>>>>>>>>>>>> like they're being run at all.
>> > >> >> > > >>>>>>>>>>>>>>>>
>> > >> >> > > >>>>>>>>>>>>>>>> Thanks,
>> > >> >> > > >>>>>>>>>>>>>>>>
>> > >> >> > > >>>>>>>>>>>>>>>> Jesse
>> > >> >> > > >>>>>>>>>>>>>>>>
>> > >> >> > > >>>>>>>>>>>>>>>>
>> > >> >> > > >>>>>>>>>>>>>>> --
>> > >> >> > > >>>>>>>>>>>>>>> Jean-Baptiste Onofré
>> > >> >> > > >>>>>>>>>>>>>>> jbonofre@apache.org
>> > >> >> > > >>>>>>>>>>>>>>> http://blog.nanthrax.net
>> > >> >> > > >>>>>>>>>>>>>>> Talend - http://www.talend.com
>> > >> >> > > >>>>>>>>>>>>>>>
>> > >> >> > > >>>>>>>>>>>>>>>
>> > >> >> > > >>>>>>>>>>>>>>
>> > >> >> > > >>>>>>>>>>>>>
>> > >> >> > > >>>>>>>>>>>>
>> > >> >> > > >>>>>>>>>>>
>> > >> >> > > >>>>>>>>>>
>> > >> >> > > >>>>>>>>>
>> > >> >> > > >>>>>>>>
>> > >> >> > > >>>>>>
>> > >> >> > > >>>>>
>> > >> >> > > >>>>
>> > >> >> > > >>>
>> > >> >> > > >> --
>> > >> >> > > >> Jean-Baptiste Onofré
>> > >> >> > > >> jbonofre@apache.org
>> > >> >> > > >> http://blog.nanthrax.net
>> > >> >> > > >> Talend - http://www.talend.com
>> > >> >> > > >>
>> > >> >> > >
>> > >> >> >
>> > >> >>
>> > >>
>> >
>>
Re: Better developer instructions for using Maven?
Posted by Aviem Zur <av...@gmail.com>.
How about we change the slower verification profile from being opt-in to
being opt-out?
Instead of casual contributors having to know about the release profile,
have the default verification verify everything needed and regular
developers that want just to install without all the verifications will
specify something like "-Dquick".
This way a casual contributor does not need any prior knowledge to verify
their branch while we can realistically expect an experienced developer to
know the opt-out argument (Same as they know the opt-in one today).
On Fri, Feb 10, 2017 at 8:29 PM Kenneth Knowles <kl...@google.com.invalid>
wrote:
> Agree!
>
> Here I'm just responding to clarify and reiterate, since the threading
> might appear as though I am the "you" in "the extra minutes you want to add
> here will cost me an hour daily" and "you can see where I stand and why"
> but I don't hold that opinion of that "you". I have said many of the same
> things as Dan previously (and I'm the one who separated slow stuff back in
> the day).
>
> So is there a path to improve it further?
>
> Kenn
>
> On Feb 10, 2017 9:28 AM, "Aljoscha Krettek" <al...@apache.org> wrote:
>
> I'm with Dan on this. The iteration time should be cut down as low as
> possible and we have Jenkins to ensure that tests pass.
>
> As a side note, there are IntelliJ plugins for Checkstyle and Findbugs and
> my personal setup highlights Checkstyle violations as errors in the code so
> I can immediately see them and fix them.
>
>
> I'll chat you up about this! I have both plugins but have not found a
> satisfactory way of surfacing the things to fix with low latency or
> immediately.
>
> Kenn
>
>
> On Fri, 10 Feb 2017 at 17:45 Dan Halperin <dh...@google.com.invalid>
> wrote:
>
> > On Fri, Feb 10, 2017 at 7:42 AM, Kenneth Knowles <klk@google.com.invalid
> >
> > wrote:
> >
> > > On Feb 10, 2017 07:36, "Dan Halperin" <dh...@google.com.invalid>
> > wrote:
> > >
> > > Before we added checkstyle it was under a minute. Now it's over five?
> > > That's awful IMO
> > >
> > >
> > > Checkstyle didn't cause all that, did it?
> > >
> >
> > The "5 minutes" was going with Aviem's numbers after this change. But
> yes,
> > Checkstyle alone substantially (>+50%) the time from what it was
> previously
> > to adding it back to the default build.
> >
> > Noting that findbugs takes quite a lot more time. Javadoc and jar are the
> > > other two slow ones.
> > >
> > > RAT is fast. But it has very poor error messages, so we wouldn't want a
> > new
> > > contributor trying to figure out what is going on without our help.
> > >
> >
> > There is a larger philosophical issue here: is there a point of Jenkins
> > precommit testing? Why not just make `mvn install` run everything that
> > Jenkins does? For that matter, why don't committers just push directly to
> > master? Wouldn't that make everyone's life easier?
> >
> > I'd argue that's not true.
> >
> > 1. Developer productivity -- Jenkins should run many more checks than
> > developers do. Especially time-, resource-, or setup- intensive tasks.
> > 2. Automated enforcement -- Jenkins is better at running the right
> commands
> > than we are.
> > 3. Lower the barrier to entry -- individual developers need not have a
> > running Spark/Flink/Apex/Dataflow setup in order to contribute code.
> > 4. Focus on the user -- someone checking out the code and using it for
> the
> > first time does not care whether the code style checks or has the right
> > licenses -- that should have been enforced by the Beam team before
> > committing.
> >
> > We should be *very* choosy about what we enforce on every developer every
> > time they go to compile. I probably compile Beam 50x-100x a day.
> Literally,
> >.
> >
> > I've listed the fraction of commits I think will break one of the
> following
> > if that property is not tested:
> >
> > * compiling (100%)
> > * tests (100%)
> > * checkstyle (90%)
> > * javadoc (30%)
> > * findbugs (5%)
> > * rat (1%)
> >
> > So . I'm sorry that 1/20 PRs has Apache
> > RAT catch a licensing issue or Findbugs catch a threading issue -- you
> can
> > always get a larger set of the precommit checks using -Prelease, though
> of
> > course the integration tests and runnableonservice tests may catch more
> > issues still. But I want my developer minutes back for the 95%+ of the
> > rest.
> >
> > Dan
> >
> >
> > >
> > >
> > > On Fri, Feb 10, 2017 at 07:14 Aviem Zur <av...@gmail.com> wrote:
> > >
> > > > Opened JIRA ticket: https://issues.apache.org/jira/browse/BEAM-1457
> > > >
> > > > On Fri, Feb 10, 2017 at 4:54 PM Jean-Baptiste Onofré <
> jb@nanthrax.net>
> > > > wrote:
> > > >
> > > > > Yeah. Agree. Time extend is not huge and it's worth to add it in
> > verify
> > > > > phase.
> > > > >
> > > > > Regards
> > > > > JB
> > > > >
> > > > > On Feb 10, 2017, 10:13, at 10:13, Aviem Zur <av...@gmail.com>
> > > wrote:
> > > > > >This goes back to the original discussion in this thread - reduce
> > the
> > > > > >amount of things pull requesters should know and keep the maven
> > > command
> > > > > >in
> > > > > >the PR checklist as: 'mvn clean verify'.
> > > > > >
> > > > > >So if rat and findbugs do not take that long to run I think they
> > > should
> > > > > >be
> > > > > >run by 'mvn clean verify'
> > > > > >
> > > > > >I ran a quick test on my laptop to see how much time they add to
> the
> > > > > >build
> > > > > >(of the entire project):
> > > > > >
> > > > > >'mvn clean install -DskipTests' => Total time: 03:51 min
> > > > > >'mvn clean install apache-rat:check findbugs:check -DskipTests'
> =>
> > > > > >Total
> > > > > >time: 05:29 min (Added 01:38 min)
> > > > > >'mvn clean install' => Total time: 09:37 min
> > > > > >'mvn clean install apache-rat:check findbugs:check' => Total time:
> > > > > >11:13
> > > > > >min (Added 01:36 min)
> > > > > >
> > > > > >Are these times reasonable enough to add rat and findbugs to the
> > > > > >default
> > > > > >build?
> > > > > >
> > > > > >On Fri, Feb 10, 2017 at 1:55 PM Jean-Baptiste Onofré <
> > jb@nanthrax.net
> > > >
> > > > > >wrote:
> > > > > >
> > > > > >> Hi
> > > > > >>
> > > > > >> We discussed about that at the beginning of the project. We
> agreed
> > > to
> > > > > >> execute rat and findbugs in a specific profile to reduce the
> build
> > > > > >time for
> > > > > >> dev.
> > > > > >>
> > > > > >> That's why I do mvn clean install -Prelease before submitting a
> PR
> > > > > >and
> > > > > >> just clean install when I'm developing.
> > > > > >>
> > > > > >> No problem to change that.
> > > > > >>
> > > > > >> Regards
> > > > > >> JB
> > > > > >>
> > > > > >> On Feb 10, 2017, 07:51, at 07:51, Aviem Zur <aviemzur@gmail.com
> >
> > > > > >wrote:
> > > > > >> >Can we consider adding rat-plugin and findbugs to the default
> > > verify
> > > > > >> >phase?
> > > > > >> >Currently they only run when the `release` profile is enabled.
> > > > > >> >
> > > > > >> >On Thu, Jan 26, 2017 at 11:42 AM Aljoscha Krettek
> > > > > ><al...@apache.org>
> > > > > >> >wrote:
> > > > > >> >
> > > > > >> >> +1 to what Dan said
> > > > > >> >>
> > > > > >> >> On Wed, 25 Jan 2017 at 21:40 Kenneth Knowles
> > > > > ><kl...@google.com.invalid>
> > > > > >> >> wrote:
> > > > > >> >>
> > > > > >> >> > +1
> > > > > >> >> >
> > > > > >> >> > On Jan 25, 2017 11:15, "Jean-Baptiste Onofré" <
> > jb@nanthrax.net
> > > >
> > > > > >> >wrote:
> > > > > >> >> >
> > > > > >> >> > > +1
> > > > > >> >> > >
> > > > > >> >> > > It sounds good to me.
> > > > > >> >> > >
> > > > > >> >> > > Thanks Dan !
> > > > > >> >> > >
> > > > > >> >> > > Regards
> > > > > >> >> > > JB
> > > > > >> >> > >
> > > > > >> >> > > On Jan 25, 2017, 19:39, at 19:39, Dan Halperin
> > > > > >> >> > <dh...@google.com.INVALID>
> > > > > >> >> > > wrote:
> > > > > >> >> > > >Here is my summary of the threads:
> > > > > >> >> > > >
> > > > > >> >> > > >Overwhelming agreement:
> > > > > >> >> > > >
> > > > > >> >> > > >- rename `release` to something more appropriate.
> > > > > >> >> > > >- add `checkstyle` to the default build (it's basically
> a
> > > > > >> >compile
> > > > > >> >> > > >error)
> > > > > >> >> > > >- add more information to contributor guide
> > > > > >> >> > > >
> > > > > >> >> > > >Reasonable agreement
> > > > > >> >> > > >
> > > > > >> >> > > >- don't update the github instructions to make passing
> > `mvn
> > > > > >> >verify
> > > > > >> >> > > >-P<all
> > > > > >> >> > > >checks>` mandatory. Maybe add a hint that this is a good
> > > > > >proxy
> > > > > >> >for
> > > > > >> >> what
> > > > > >> >> > > >Jenkins will run.
> > > > > >> >> > > >
> > > > > >> >> > > >Unresolved:
> > > > > >> >> > > >
> > > > > >> >> > > >- whether all checks should be in `mvn verify`
> > > > > >> >> > > >- whether `mvn test` is useful for most workflows
> > > > > >> >> > > >
> > > > > >> >> > > >I'll propose to proceed with the overwhelmingly
> > agreed-upon
> > > > > >> >changes,
> > > > > >> >> > > >and as
> > > > > >> >> > > >we see increasingly many new contributors re-evaluate
> the
> > > > > >> >remaining
> > > > > >> >> > > >issues.
> > > > > >> >> > > >
> > > > > >> >> > > >Thanks,
> > > > > >> >> > > >Dan
> > > > > >> >> > > >
> > > > > >> >> > > >On Tue, Jan 24, 2017 at 12:51 PM, Jean-Baptiste Onofré
> > > > > >> >> > > ><jb...@nanthrax.net>
> > > > > >> >> > > >wrote:
> > > > > >> >> > > >
> > > > > >> >> > > >> +1 to at least update the contribution guide and
> improve
> > > > > >the
> > > > > >> >profile
> > > > > >> >> > > >name.
> > > > > >> >> > > >>
> > > > > >> >> > > >> Regards
> > > > > >> >> > > >> JB
> > > > > >> >> > > >>
> > > > > >> >> > > >>
> > > > > >> >> > > >> On 01/24/2017 09:49 PM, Kenneth Knowles wrote:
> > > > > >> >> > > >>
> > > > > >> >> > > >>> My impression is that we don't have consensus on
> > whether
> > > > > >all
> > > > > >> >checks
> > > > > >> >> > > >or
> > > > > >> >> > > >>> minimal checks should be the default, or whether we
> can
> > > > > >have
> > > > > >> >both
> > > > > >> >> > > >via `mvn
> > > > > >> >> > > >>> test` and `mvn verify`.
> > > > > >> >> > > >>>
> > > > > >> >> > > >>> But that doesn't prevent us from giving -P release a
> > > > > >better
> > > > > >> >name
> > > > > >> >> and
> > > > > >> >> > > >>> mentioning it in the dev guide and in some manner in
> > our
> > > > > >PR
> > > > > >> >> > > >template.
> > > > > >> >> > > >>>
> > > > > >> >> > > >>> Right now we are living with the combination of the
> bad
> > > > > >> >aspects -
> > > > > >> >> > > >default
> > > > > >> >> > > >>> is not thorough but not actually fast and a thorough
> > > check
> > > > > >is
> > > > > >> >> > > >>> undocumented.
> > > > > >> >> > > >>>
> > > > > >> >> > > >>> On Tue, Jan 24, 2017 at 2:22 AM, Ismaël Mejía
> > > > > >> ><ie...@gmail.com>
> > > > > >> >> > > >wrote:
> > > > > >> >> > > >>>
> > > > > >> >> > > >>> I just wanted to know if we have achieved some
> > consensus
> > > > > >> >about
> > > > > >> >> this,
> > > > > >> >> > > >I
> > > > > >> >> > > >>>> just
> > > > > >> >> > > >>>> saw this PR that reminded me about this discussion.
> > > > > >> >> > > >>>>
> > > > > >> >> > > >>>> https://github.com/apache/beam/pull/1829
> > > > > >> >> > > >>>>
> > > > > >> >> > > >>>> It is important that we mention the existing
> profiles
> > > > > >(and
> > > > > >> >the
> > > > > >> >> > > >intended
> > > > > >> >> > > >>>> checks) in the contribution guide (e.g. -Prelease
> (or
> > > > > >> >-Pall-checks
> > > > > >> >> > > >>>> triggers
> > > > > >> >> > > >>>> these validations).
> > > > > >> >> > > >>>>
> > > > > >> >> > > >>>> I can add this to the guide if you like once we
> define
> > > > > >the
> > > > > >> >checks
> > > > > >> >> > > >per
> > > > > >> >> > > >>>> stage/profile.
> > > > > >> >> > > >>>>
> > > > > >> >> > > >>>> Ismaël
> > > > > >> >> > > >>>>
> > > > > >> >> > > >>>>
> > > > > >> >> > > >>>> On Wed, Jan 11, 2017 at 8:12 AM, Aviem Zur
> > > > > >> ><av...@gmail.com>
> > > > > >> >> > > >wrote:
> > > > > >> >> > > >>>>
> > > > > >> >> > > >>>> I agree with Dan and Lukasz.
> > > > > >> >> > > >>>>> Developers should not be expected to know
> beforehand
> > > > > >which
> > > > > >> >> > > >specific
> > > > > >> >> > > >>>>> profiles to run.
> > > > > >> >> > > >>>>> The phase specified in the PR instructions
> (`verify`)
> > > > > >> >should run
> > > > > >> >> > > >all the
> > > > > >> >> > > >>>>> relevant verifications and be the "slower" build,
> > while
> > > > > >a
> > > > > >> >> > > >preceding
> > > > > >> >> > > >>>>> lifecycle, such as `test`, should run the "faster"
> > > > > >> >verifications.
> > > > > >> >> > > >>>>>
> > > > > >> >> > > >>>>> Aviem.
> > > > > >> >> > > >>>>>
> > > > > >> >> > > >>>>> On Mon, Jan 9, 2017 at 7:57 PM Robert Bradshaw
> > > > > >> >> > > >>>>>
> > > > > >> >> > > >>>> <robertwb@google.com.invalid
> > > > > >> >> > > >>>>
> > > > > >> >> > > >>>>>
> > > > > >> >> > > >>>>>> wrote:
> > > > > >> >> > > >>>>>
> > > > > >> >> > > >>>>> On Mon, Jan 9, 2017 at 3:49 AM, Aljoscha Krettek
> > > > > >> >> > > ><al...@apache.org>
> > > > > >> >> > > >>>>>> wrote:
> > > > > >> >> > > >>>>>>
> > > > > >> >> > > >>>>>>> I also usually prefer "mvn verify" to to the
> > expected
> > > > > >> >thing but
> > > > > >> >> > > >I see
> > > > > >> >> > > >>>>>>>
> > > > > >> >> > > >>>>>> that
> > > > > >> >> > > >>>>>>
> > > > > >> >> > > >>>>>>> quick iteration times are key.
> > > > > >> >> > > >>>>>>>
> > > > > >> >> > > >>>>>>
> > > > > >> >> > > >>>>>> I see
> > > > > >> >> > > >>>>>> https://maven.apache.org/guides/introduction/
> > > > > >> >> > > >>>>>>
> > > > > >> >> > > >>>>> introduction-to-the-lifecycle.html
> > > > > >> >> > > >>>>>
> > > > > >> >> > > >>>>>>
> > > > > >> >> > > >>>>>> verify - run any checks on results of
> > integration
> > > > > >> >tests to
> > > > > >> >> > > >ensure
> > > > > >> >> > > >>>>>> quality criteria are met
> > > > > >> >> > > >>>>>>
> > > > > >> >> > > >>>>>> Of course our integration tests are long enough
> that
> > > we
> > > > > >> >> shouldn't
> > > > > >> >> > > >be
> > > > > >> >> > > >>>>>> putting all of them here, but I too would expect
> > > > > >> >checkstyle.
> > > > > >> >> > > >>>>>>
> > > > > >> >> > > >>>>>> Perhaps we could introduce a verify-fast or
> somesuch
> > > > > >for
> > > > > >> >fast
> > > > > >> >> > > >(but
> > > > > >> >> > > >>>>>> lower coverage) turnaround time. I would expect
> "mvn
> > > > > >> >verify
> > > > > >> >> test"
> > > > > >> >> > > >to
> > > > > >> >> > > >>>>>> pass before submitting a PR, and would want to run
> > > that
> > > > > >> >before
> > > > > >> >> > > >asking
> > > > > >> >> > > >>>>>> others to look at it. I think this should be our
> > > > > >criteria
> > > > > >> >(i.e.
> > > > > >> >> > > >what
> > > > > >> >> > > >>>>>> will a new but maven-savvy user run before pushing
> > > > > >their
> > > > > >> >code).
> > > > > >> >> > > >>>>>>
> > > > > >> >> > > >>>>>> As long as the pre-commit hooks still check
> > everything
> > > > > >I'm
> > > > > >> >ok
> > > > > >> >> > > >with
> > > > > >> >> > > >>>>>>>
> > > > > >> >> > > >>>>>> making
> > > > > >> >> > > >>>>>
> > > > > >> >> > > >>>>>> the default a little more lightweight.
> > > > > >> >> > > >>>>>>>
> > > > > >> >> > > >>>>>>
> > > > > >> >> > > >>>>>> The fact that our pre-commit hooks take a long
> time
> > to
> > > > > >run
> > > > > >> >does
> > > > > >> >> > > >change
> > > > > >> >> > > >>>>>> things. Nothing more annoying than seeing that
> your
> > PR
> > > > > >> >failed 3
> > > > > >> >> > > >hours
> > > > > >> >> > > >>>>>> later because you had some trailing whitespace...
> > > > > >> >> > > >>>>>>
> > > > > >> >> > > >>>>>> On Thu, 5 Jan 2017 at 21:49 Lukasz Cwik
> > > > > >> >> > > ><lc...@google.com.invalid>
> > > > > >> >> > > >>>>>>>
> > > > > >> >> > > >>>>>> wrote:
> > > > > >> >> > > >>>>>>
> > > > > >> >> > > >>>>>>>
> > > > > >> >> > > >>>>>>> I was hoping that the default mvn verify would be
> > the
> > > > > >> >slow
> > > > > >> >> build
> > > > > >> >> > > >>>>>>>>
> > > > > >> >> > > >>>>>>> and a
> > > > > >> >> > > >>>>
> > > > > >> >> > > >>>>> profile could be enabled that would skip checks to
> > make
> > > > > >> >things
> > > > > >> >> > > >>>>>>>>
> > > > > >> >> > > >>>>>>> faster
> > > > > >> >> > > >>>>
> > > > > >> >> > > >>>>> for
> > > > > >> >> > > >>>>>>
> > > > > >> >> > > >>>>>>> regular contributors. This way a person doesn't
> > need
> > > > > >to
> > > > > >> >have
> > > > > >> >> > > >>>>>>>>
> > > > > >> >> > > >>>>>>> detailed
> > > > > >> >> > > >>>>
> > > > > >> >> > > >>>>> knowledge of all our profiles and what they do
> > > > > >(typically
> > > > > >> >mvn
> > > > > >> >> > > >>>>>>>>
> > > > > >> >> > > >>>>>>> verify)
> > > > > >> >> > > >>>>
> > > > > >> >> > > >>>>> will
> > > > > >> >> > > >>>>>>
> > > > > >> >> > > >>>>>>> do the right thing most of the time.
> > > > > >> >> > > >>>>>>>>
> > > > > >> >> > > >>>>>>>> On Thu, Jan 5, 2017 at 9:30 AM, Dan Halperin
> > > > > >> >> > > >>>>>>>>
> > > > > >> >> > > >>>>>>> <dh...@google.com.invalid>
> > > > > >> >> > > >>>>>>
> > > > > >> >> > > >>>>>>> wrote:
> > > > > >> >> > > >>>>>>>>
> > > > > >> >> > > >>>>>>>> On Thu, Jan 5, 2017 at 9:28 AM, Jesse Anderson <
> > > > > >> >> > > >>>>>>>>>
> > > > > >> >> > > >>>>>>>> jesse@smokinghand.com
> > > > > >> >> > > >>>>>
> > > > > >> >> > > >>>>>>
> > > > > >> >> > > >>>>>>> wrote:
> > > > > >> >> > > >>>>>>>>>
> > > > > >> >> > > >>>>>>>>> @dan are you saying that mvn verify isn't doing
> > > > > >> >checkstyle
> > > > > >> >> > > >>>>>>>>>>
> > > > > >> >> > > >>>>>>>>> anymore?
> > > > > >> >> > > >>>>>
> > > > > >> >> > > >>>>>>
> > > > > >> >> > > >>>>>>>>>
> > > > > >> >> > > >>>>>>>>> `mvn verify` alone should not be running
> > > checkstyle,
> > > > > >if
> > > > > >> >> > > >modules
> > > > > >> >> > > >>>>>>>>>
> > > > > >> >> > > >>>>>>>> are
> > > > > >> >> > > >>>>
> > > > > >> >> > > >>>>> configured correctly.
> > > > > >> >> > > >>>>>>>>>
> > > > > >> >> > > >>>>>>>>>
> > > > > >> >> > > >>>>>>>>> Some of
> > > > > >> >> > > >>>>>>>>>> the checkstyles are still running for a few
> > > > > >modules.
> > > > > >> >Also,
> > > > > >> >> > > >the
> > > > > >> >> > > >>>>>>>>>>
> > > > > >> >> > > >>>>>>>>> contribution
> > > > > >> >> > > >>>>>>>>>
> > > > > >> >> > > >>>>>>>>>> docs will need to change.
> > > > > >> >> > > >>>>>>>>>>
> > > > > >> >> > > >>>>>>>>>
> > > > > >> >> > > >>>>>>>>>
> > > > > >> >> > > >>>>>>>>> Yes. The PR includes discussion of these other
> > > > > >needed
> > > > > >> >> changes,
> > > > > >> >> > > >>>>>>>>> unfortunately one PR can't change two
> > repositories.
> > > > > >> >> > > >>>>>>>>>
> > > > > >> >> > > >>>>>>>>> Please continue the discussion on the PR, then
> I
> > > > > >will
> > > > > >> >> > > >summarize it
> > > > > >> >> > > >>>>>>>>>
> > > > > >> >> > > >>>>>>>> back
> > > > > >> >> > > >>>>>>
> > > > > >> >> > > >>>>>>> into the dev thread.
> > > > > >> >> > > >>>>>>>>>
> > > > > >> >> > > >>>>>>>>> Thanks,
> > > > > >> >> > > >>>>>>>>> Dan
> > > > > >> >> > > >>>>>>>>>
> > > > > >> >> > > >>>>>>>>>
> > > > > >> >> > > >>>>>>>>> They say to run mvn verify before commits.
> > > > > >> >> > > >>>>>>>>>>
> > > > > >> >> > > >>>>>>>>>> On Thu, Jan 5, 2017 at 9:25 AM Dan Halperin
> > > > > >> >> > > >>>>>>>>>>
> > > > > >> >> > > >>>>>>>>> <dhalperi@google.com.invalid
> > > > > >> >> > > >>>>>>>>
> > > > > >> >> > > >>>>>>>>>
> > > > > >> >> > > >>>>>>>>>> wrote:
> > > > > >> >> > > >>>>>>>>>>
> > > > > >> >> > > >>>>>>>>>> Several folks seem to have been confused after
> > > > > >> >BEAM-246,
> > > > > >> >> > > >where
> > > > > >> >> > > >>>>>>>>>>>
> > > > > >> >> > > >>>>>>>>>> we
> > > > > >> >> > > >>>>>
> > > > > >> >> > > >>>>>> moved
> > > > > >> >> > > >>>>>>>>
> > > > > >> >> > > >>>>>>>>> the
> > > > > >> >> > > >>>>>>>>>>
> > > > > >> >> > > >>>>>>>>>>> "slow things" into the release profile. I've
> > > > > >started
> > > > > >> >a
> > > > > >> >> > > >>>>>>>>>>>
> > > > > >> >> > > >>>>>>>>>> discussion
> > > > > >> >> > > >>>>>
> > > > > >> >> > > >>>>>> with
> > > > > >> >> > > >>>>>>>>
> > > > > >> >> > > >>>>>>>>> https://github.com/apache/beam/pull/1740 to
> see
> > if
> > > > > >> >there are
> > > > > >> >> > > >>>>>>>>>>>
> > > > > >> >> > > >>>>>>>>>> things
> > > > > >> >> > > >>>>>>
> > > > > >> >> > > >>>>>>> we
> > > > > >> >> > > >>>>>>>>
> > > > > >> >> > > >>>>>>>>> can
> > > > > >> >> > > >>>>>>>>>>
> > > > > >> >> > > >>>>>>>>>>> do to fill these gaps.
> > > > > >> >> > > >>>>>>>>>>>
> > > > > >> >> > > >>>>>>>>>>> Would love folks to chime in with opinions.
> > > > > >> >> > > >>>>>>>>>>>
> > > > > >> >> > > >>>>>>>>>>> Dan
> > > > > >> >> > > >>>>>>>>>>>
> > > > > >> >> > > >>>>>>>>>>> On Wed, Jan 4, 2017 at 1:34 PM, Jesse
> Anderson
> > <
> > > > > >> >> > > >>>>>>>>>>>
> > > > > >> >> > > >>>>>>>>>> jesse@smokinghand.com>
> > > > > >> >> > > >>>>>>>>
> > > > > >> >> > > >>>>>>>>> wrote:
> > > > > >> >> > > >>>>>>>>>>>
> > > > > >> >> > > >>>>>>>>>>> @Eugene, yes that failed on the checkstyle.
> > > > > >> >> > > >>>>>>>>>>>>
> > > > > >> >> > > >>>>>>>>>>>> On Wed, Jan 4, 2017 at 1:27 PM Eugene
> > Kirpichov
> > > > > >> >> > > >>>>>>>>>>>> <ki...@google.com.invalid> wrote:
> > > > > >> >> > > >>>>>>>>>>>>
> > > > > >> >> > > >>>>>>>>>>>> Try just -Prelease.
> > > > > >> >> > > >>>>>>>>>>>>> On Wed, Jan 4, 2017 at 1:21 PM Jesse
> > Anderson <
> > > > > >> >> > > >>>>>>>>>>>>>
> > > > > >> >> > > >>>>>>>>>>>> jesse@smokinghand.com
> > > > > >> >> > > >>>>>>>>>
> > > > > >> >> > > >>>>>>>>>>
> > > > > >> >> > > >>>>>>>>>>> wrote:
> > > > > >> >> > > >>>>>>>>>>>>>
> > > > > >> >> > > >>>>>>>>>>>>> Fails because I don't have a secret key.
> > > > > >> >> > > >>>>>>>>>>>>>>
> > > > > >> >> > > >>>>>>>>>>>>>> On Wed, Jan 4, 2017 at 1:03 PM
> Jean-Baptiste
> > > > > >> >Onofré <
> > > > > >> >> > > >>>>>>>>>>>>>>
> > > > > >> >> > > >>>>>>>>>>>>> jb@nanthrax.net
> > > > > >> >> > > >>>>>>>>>>
> > > > > >> >> > > >>>>>>>>>>>
> > > > > >> >> > > >>>>>>>>>>>> wrote:
> > > > > >> >> > > >>>>>>>>>>>>>>
> > > > > >> >> > > >>>>>>>>>>>>>> Hi Jesse,
> > > > > >> >> > > >>>>>>>>>>>>>>>
> > > > > >> >> > > >>>>>>>>>>>>>>> Could you try the same with:
> > > > > >> >> > > >>>>>>>>>>>>>>>
> > > > > >> >> > > >>>>>>>>>>>>>>> mvn verify -Prelease,apache-release
> > > > > >> >> > > >>>>>>>>>>>>>>>
> > > > > >> >> > > >>>>>>>>>>>>>>> ?
> > > > > >> >> > > >>>>>>>>>>>>>>>
> > > > > >> >> > > >>>>>>>>>>>>>>> Regards
> > > > > >> >> > > >>>>>>>>>>>>>>> JB
> > > > > >> >> > > >>>>>>>>>>>>>>>
> > > > > >> >> > > >>>>>>>>>>>>>>> On 01/04/2017 09:53 PM, Jesse Anderson
> > wrote:
> > > > > >> >> > > >>>>>>>>>>>>>>>
> > > > > >> >> > > >>>>>>>>>>>>>>>> For some reason, running "mvn verify"
> > isn't
> > > > > >> >running
> > > > > >> >> > > >>>>>>>>>>>>>>>>
> > > > > >> >> > > >>>>>>>>>>>>>>> checkstyle
> > > > > >> >> > > >>>>>>>>>
> > > > > >> >> > > >>>>>>>>>> on
> > > > > >> >> > > >>>>>>>>>>
> > > > > >> >> > > >>>>>>>>>>> everything. I had checkstyle errors in
> > > > > >> >> > > >>>>>>>>>>>>>>>>
> > > > > >> >> > > >>>>>>>>>>>>>>> beam-sdks-java-core
> > > > > >> >> > > >>>>>>
> > > > > >> >> > > >>>>>>> that
> > > > > >> >> > > >>>>>>>>>
> > > > > >> >> > > >>>>>>>>>> weren't
> > > > > >> >> > > >>>>>>>>>>>>>
> > > > > >> >> > > >>>>>>>>>>>>>> being found.
> > > > > >> >> > > >>>>>>>>>>>>>>>>
> > > > > >> >> > > >>>>>>>>>>>>>>>> I thought this was due to the extra
> > > > > >parameters.
> > > > > >> >I
> > > > > >> >> > > >>>>>>>>>>>>>>>>
> > > > > >> >> > > >>>>>>>>>>>>>>> reran
> > > > > >> >> > > >>>>>
> > > > > >> >> > > >>>>>> with
> > > > > >> >> > > >>>>>>>>
> > > > > >> >> > > >>>>>>>>> the
> > > > > >> >> > > >>>>>>>>>>
> > > > > >> >> > > >>>>>>>>>>> plain
> > > > > >> >> > > >>>>>>>>>>>>>
> > > > > >> >> > > >>>>>>>>>>>>>> "mvn
> > > > > >> >> > > >>>>>>>>>>>>>>>
> > > > > >> >> > > >>>>>>>>>>>>>>>> verify" and it still didn't find them.
> > From
> > > > > >the
> > > > > >> >> > > >>>>>>>>>>>>>>>>
> > > > > >> >> > > >>>>>>>>>>>>>>> output,
> > > > > >> >> > > >>>>>
> > > > > >> >> > > >>>>>> it
> > > > > >> >> > > >>>>>>
> > > > > >> >> > > >>>>>>> doesn't
> > > > > >> >> > > >>>>>>>>>>>
> > > > > >> >> > > >>>>>>>>>>>> look
> > > > > >> >> > > >>>>>>>>>>>>>
> > > > > >> >> > > >>>>>>>>>>>>>> like they're being run at all.
> > > > > >> >> > > >>>>>>>>>>>>>>>>
> > > > > >> >> > > >>>>>>>>>>>>>>>> Thanks,
> > > > > >> >> > > >>>>>>>>>>>>>>>>
> > > > > >> >> > > >>>>>>>>>>>>>>>> Jesse
> > > > > >> >> > > >>>>>>>>>>>>>>>>
> > > > > >> >> > > >>>>>>>>>>>>>>>>
> > > > > >> >> > > >>>>>>>>>>>>>>> --
> > > > > >> >> > > >>>>>>>>>>>>>>> Jean-Baptiste Onofré
> > > > > >> >> > > >>>>>>>>>>>>>>> jbonofre@apache.org
> > > > > >> >> > > >>>>>>>>>>>>>>> http://blog.nanthrax.net
> > > > > >> >> > > >>>>>>>>>>>>>>> Talend - http://www.talend.com
> > > > > >> >> > > >>>>>>>>>>>>>>>
> > > > > >> >> > > >>>>>>>>>>>>>>>
> > > > > >> >> > > >>>>>>>>>>>>>>
> > > > > >> >> > > >>>>>>>>>>>>>
> > > > > >> >> > > >>>>>>>>>>>>
> > > > > >> >> > > >>>>>>>>>>>
> > > > > >> >> > > >>>>>>>>>>
> > > > > >> >> > > >>>>>>>>>
> > > > > >> >> > > >>>>>>>>
> > > > > >> >> > > >>>>>>
> > > > > >> >> > > >>>>>
> > > > > >> >> > > >>>>
> > > > > >> >> > > >>>
> > > > > >> >> > > >> --
> > > > > >> >> > > >> Jean-Baptiste Onofré
> > > > > >> >> > > >> jbonofre@apache.org
> > > > > >> >> > > >> http://blog.nanthrax.net
> > > > > >> >> > > >> Talend - http://www.talend.com
> > > > > >> >> > > >>
> > > > > >> >> > >
> > > > > >> >> >
> > > > > >> >>
> > > > > >>
> > > > >
> > > >
> > >
> >
>
Re: Better developer instructions for using Maven?
Posted by Kenneth Knowles <kl...@google.com.INVALID>.
Agree!
Here I'm just responding to clarify and reiterate, since the threading
might appear as though I am the "you" in "the extra minutes you want to add
here will cost me an hour daily" and "you can see where I stand and why"
but I don't hold that opinion of that "you". I have said many of the same
things as Dan previously (and I'm the one who separated slow stuff back in
the day).
So is there a path to improve it further?
Kenn
On Feb 10, 2017 9:28 AM, "Aljoscha Krettek" <al...@apache.org> wrote:
I'm with Dan on this. The iteration time should be cut down as low as
possible and we have Jenkins to ensure that tests pass.
As a side note, there are IntelliJ plugins for Checkstyle and Findbugs and
my personal setup highlights Checkstyle violations as errors in the code so
I can immediately see them and fix them.
I'll chat you up about this! I have both plugins but have not found a
satisfactory way of surfacing the things to fix with low latency or
immediately.
Kenn
On Fri, 10 Feb 2017 at 17:45 Dan Halperin <dh...@google.com.invalid>
wrote:
> On Fri, Feb 10, 2017 at 7:42 AM, Kenneth Knowles <kl...@google.com.invalid>
> wrote:
>
> > On Feb 10, 2017 07:36, "Dan Halperin" <dh...@google.com.invalid>
> wrote:
> >
> > Before we added checkstyle it was under a minute. Now it's over five?
> > That's awful IMO
> >
> >
> > Checkstyle didn't cause all that, did it?
> >
>
> The "5 minutes" was going with Aviem's numbers after this change. But yes,
> Checkstyle alone substantially (>+50%) the time from what it was
previously
> to adding it back to the default build.
>
> Noting that findbugs takes quite a lot more time. Javadoc and jar are the
> > other two slow ones.
> >
> > RAT is fast. But it has very poor error messages, so we wouldn't want a
> new
> > contributor trying to figure out what is going on without our help.
> >
>
> There is a larger philosophical issue here: is there a point of Jenkins
> precommit testing? Why not just make `mvn install` run everything that
> Jenkins does? For that matter, why don't committers just push directly to
> master? Wouldn't that make everyone's life easier?
>
> I'd argue that's not true.
>
> 1. Developer productivity -- Jenkins should run many more checks than
> developers do. Especially time-, resource-, or setup- intensive tasks.
> 2. Automated enforcement -- Jenkins is better at running the right
commands
> than we are.
> 3. Lower the barrier to entry -- individual developers need not have a
> running Spark/Flink/Apex/Dataflow setup in order to contribute code.
> 4. Focus on the user -- someone checking out the code and using it for the
> first time does not care whether the code style checks or has the right
> licenses -- that should have been enforced by the Beam team before
> committing.
>
> We should be *very* choosy about what we enforce on every developer every
> time they go to compile. I probably compile Beam 50x-100x a day.
Literally,
>.
>
> I've listed the fraction of commits I think will break one of the
following
> if that property is not tested:
>
> * compiling (100%)
> * tests (100%)
> * checkstyle (90%)
> * javadoc (30%)
> * findbugs (5%)
> * rat (1%)
>
> So . I'm sorry that 1/20 PRs has Apache
> RAT catch a licensing issue or Findbugs catch a threading issue -- you can
> always get a larger set of the precommit checks using -Prelease, though of
> course the integration tests and runnableonservice tests may catch more
> issues still. But I want my developer minutes back for the 95%+ of the
> rest.
>
> Dan
>
>
> >
> >
> > On Fri, Feb 10, 2017 at 07:14 Aviem Zur <av...@gmail.com> wrote:
> >
> > > Opened JIRA ticket: https://issues.apache.org/jira/browse/BEAM-1457
> > >
> > > On Fri, Feb 10, 2017 at 4:54 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
> > > wrote:
> > >
> > > > Yeah. Agree. Time extend is not huge and it's worth to add it in
> verify
> > > > phase.
> > > >
> > > > Regards
> > > > JB
> > > >
> > > > On Feb 10, 2017, 10:13, at 10:13, Aviem Zur <av...@gmail.com>
> > wrote:
> > > > >This goes back to the original discussion in this thread - reduce
> the
> > > > >amount of things pull requesters should know and keep the maven
> > command
> > > > >in
> > > > >the PR checklist as: 'mvn clean verify'.
> > > > >
> > > > >So if rat and findbugs do not take that long to run I think they
> > should
> > > > >be
> > > > >run by 'mvn clean verify'
> > > > >
> > > > >I ran a quick test on my laptop to see how much time they add to
the
> > > > >build
> > > > >(of the entire project):
> > > > >
> > > > >'mvn clean install -DskipTests' => Total time: 03:51 min
> > > > >'mvn clean install apache-rat:check findbugs:check -DskipTests' =>
> > > > >Total
> > > > >time: 05:29 min (Added 01:38 min)
> > > > >'mvn clean install' => Total time: 09:37 min
> > > > >'mvn clean install apache-rat:check findbugs:check' => Total time:
> > > > >11:13
> > > > >min (Added 01:36 min)
> > > > >
> > > > >Are these times reasonable enough to add rat and findbugs to the
> > > > >default
> > > > >build?
> > > > >
> > > > >On Fri, Feb 10, 2017 at 1:55 PM Jean-Baptiste Onofré <
> jb@nanthrax.net
> > >
> > > > >wrote:
> > > > >
> > > > >> Hi
> > > > >>
> > > > >> We discussed about that at the beginning of the project. We
agreed
> > to
> > > > >> execute rat and findbugs in a specific profile to reduce the
build
> > > > >time for
> > > > >> dev.
> > > > >>
> > > > >> That's why I do mvn clean install -Prelease before submitting a
PR
> > > > >and
> > > > >> just clean install when I'm developing.
> > > > >>
> > > > >> No problem to change that.
> > > > >>
> > > > >> Regards
> > > > >> JB
> > > > >>
> > > > >> On Feb 10, 2017, 07:51, at 07:51, Aviem Zur <av...@gmail.com>
> > > > >wrote:
> > > > >> >Can we consider adding rat-plugin and findbugs to the default
> > verify
> > > > >> >phase?
> > > > >> >Currently they only run when the `release` profile is enabled.
> > > > >> >
> > > > >> >On Thu, Jan 26, 2017 at 11:42 AM Aljoscha Krettek
> > > > ><al...@apache.org>
> > > > >> >wrote:
> > > > >> >
> > > > >> >> +1 to what Dan said
> > > > >> >>
> > > > >> >> On Wed, 25 Jan 2017 at 21:40 Kenneth Knowles
> > > > ><kl...@google.com.invalid>
> > > > >> >> wrote:
> > > > >> >>
> > > > >> >> > +1
> > > > >> >> >
> > > > >> >> > On Jan 25, 2017 11:15, "Jean-Baptiste Onofré" <
> jb@nanthrax.net
> > >
> > > > >> >wrote:
> > > > >> >> >
> > > > >> >> > > +1
> > > > >> >> > >
> > > > >> >> > > It sounds good to me.
> > > > >> >> > >
> > > > >> >> > > Thanks Dan !
> > > > >> >> > >
> > > > >> >> > > Regards
> > > > >> >> > > JB
> > > > >> >> > >
> > > > >> >> > > On Jan 25, 2017, 19:39, at 19:39, Dan Halperin
> > > > >> >> > <dh...@google.com.INVALID>
> > > > >> >> > > wrote:
> > > > >> >> > > >Here is my summary of the threads:
> > > > >> >> > > >
> > > > >> >> > > >Overwhelming agreement:
> > > > >> >> > > >
> > > > >> >> > > >- rename `release` to something more appropriate.
> > > > >> >> > > >- add `checkstyle` to the default build (it's basically a
> > > > >> >compile
> > > > >> >> > > >error)
> > > > >> >> > > >- add more information to contributor guide
> > > > >> >> > > >
> > > > >> >> > > >Reasonable agreement
> > > > >> >> > > >
> > > > >> >> > > >- don't update the github instructions to make passing
> `mvn
> > > > >> >verify
> > > > >> >> > > >-P<all
> > > > >> >> > > >checks>` mandatory. Maybe add a hint that this is a good
> > > > >proxy
> > > > >> >for
> > > > >> >> what
> > > > >> >> > > >Jenkins will run.
> > > > >> >> > > >
> > > > >> >> > > >Unresolved:
> > > > >> >> > > >
> > > > >> >> > > >- whether all checks should be in `mvn verify`
> > > > >> >> > > >- whether `mvn test` is useful for most workflows
> > > > >> >> > > >
> > > > >> >> > > >I'll propose to proceed with the overwhelmingly
> agreed-upon
> > > > >> >changes,
> > > > >> >> > > >and as
> > > > >> >> > > >we see increasingly many new contributors re-evaluate the
> > > > >> >remaining
> > > > >> >> > > >issues.
> > > > >> >> > > >
> > > > >> >> > > >Thanks,
> > > > >> >> > > >Dan
> > > > >> >> > > >
> > > > >> >> > > >On Tue, Jan 24, 2017 at 12:51 PM, Jean-Baptiste Onofré
> > > > >> >> > > ><jb...@nanthrax.net>
> > > > >> >> > > >wrote:
> > > > >> >> > > >
> > > > >> >> > > >> +1 to at least update the contribution guide and
improve
> > > > >the
> > > > >> >profile
> > > > >> >> > > >name.
> > > > >> >> > > >>
> > > > >> >> > > >> Regards
> > > > >> >> > > >> JB
> > > > >> >> > > >>
> > > > >> >> > > >>
> > > > >> >> > > >> On 01/24/2017 09:49 PM, Kenneth Knowles wrote:
> > > > >> >> > > >>
> > > > >> >> > > >>> My impression is that we don't have consensus on
> whether
> > > > >all
> > > > >> >checks
> > > > >> >> > > >or
> > > > >> >> > > >>> minimal checks should be the default, or whether we
can
> > > > >have
> > > > >> >both
> > > > >> >> > > >via `mvn
> > > > >> >> > > >>> test` and `mvn verify`.
> > > > >> >> > > >>>
> > > > >> >> > > >>> But that doesn't prevent us from giving -P release a
> > > > >better
> > > > >> >name
> > > > >> >> and
> > > > >> >> > > >>> mentioning it in the dev guide and in some manner in
> our
> > > > >PR
> > > > >> >> > > >template.
> > > > >> >> > > >>>
> > > > >> >> > > >>> Right now we are living with the combination of the
bad
> > > > >> >aspects -
> > > > >> >> > > >default
> > > > >> >> > > >>> is not thorough but not actually fast and a thorough
> > check
> > > > >is
> > > > >> >> > > >>> undocumented.
> > > > >> >> > > >>>
> > > > >> >> > > >>> On Tue, Jan 24, 2017 at 2:22 AM, Ismaël Mejía
> > > > >> ><ie...@gmail.com>
> > > > >> >> > > >wrote:
> > > > >> >> > > >>>
> > > > >> >> > > >>> I just wanted to know if we have achieved some
> consensus
> > > > >> >about
> > > > >> >> this,
> > > > >> >> > > >I
> > > > >> >> > > >>>> just
> > > > >> >> > > >>>> saw this PR that reminded me about this discussion.
> > > > >> >> > > >>>>
> > > > >> >> > > >>>> https://github.com/apache/beam/pull/1829
> > > > >> >> > > >>>>
> > > > >> >> > > >>>> It is important that we mention the existing profiles
> > > > >(and
> > > > >> >the
> > > > >> >> > > >intended
> > > > >> >> > > >>>> checks) in the contribution guide (e.g. -Prelease (or
> > > > >> >-Pall-checks
> > > > >> >> > > >>>> triggers
> > > > >> >> > > >>>> these validations).
> > > > >> >> > > >>>>
> > > > >> >> > > >>>> I can add this to the guide if you like once we
define
> > > > >the
> > > > >> >checks
> > > > >> >> > > >per
> > > > >> >> > > >>>> stage/profile.
> > > > >> >> > > >>>>
> > > > >> >> > > >>>> Ismaël
> > > > >> >> > > >>>>
> > > > >> >> > > >>>>
> > > > >> >> > > >>>> On Wed, Jan 11, 2017 at 8:12 AM, Aviem Zur
> > > > >> ><av...@gmail.com>
> > > > >> >> > > >wrote:
> > > > >> >> > > >>>>
> > > > >> >> > > >>>> I agree with Dan and Lukasz.
> > > > >> >> > > >>>>> Developers should not be expected to know beforehand
> > > > >which
> > > > >> >> > > >specific
> > > > >> >> > > >>>>> profiles to run.
> > > > >> >> > > >>>>> The phase specified in the PR instructions
(`verify`)
> > > > >> >should run
> > > > >> >> > > >all the
> > > > >> >> > > >>>>> relevant verifications and be the "slower" build,
> while
> > > > >a
> > > > >> >> > > >preceding
> > > > >> >> > > >>>>> lifecycle, such as `test`, should run the "faster"
> > > > >> >verifications.
> > > > >> >> > > >>>>>
> > > > >> >> > > >>>>> Aviem.
> > > > >> >> > > >>>>>
> > > > >> >> > > >>>>> On Mon, Jan 9, 2017 at 7:57 PM Robert Bradshaw
> > > > >> >> > > >>>>>
> > > > >> >> > > >>>> <robertwb@google.com.invalid
> > > > >> >> > > >>>>
> > > > >> >> > > >>>>>
> > > > >> >> > > >>>>>> wrote:
> > > > >> >> > > >>>>>
> > > > >> >> > > >>>>> On Mon, Jan 9, 2017 at 3:49 AM, Aljoscha Krettek
> > > > >> >> > > ><al...@apache.org>
> > > > >> >> > > >>>>>> wrote:
> > > > >> >> > > >>>>>>
> > > > >> >> > > >>>>>>> I also usually prefer "mvn verify" to to the
> expected
> > > > >> >thing but
> > > > >> >> > > >I see
> > > > >> >> > > >>>>>>>
> > > > >> >> > > >>>>>> that
> > > > >> >> > > >>>>>>
> > > > >> >> > > >>>>>>> quick iteration times are key.
> > > > >> >> > > >>>>>>>
> > > > >> >> > > >>>>>>
> > > > >> >> > > >>>>>> I see
> > > > >> >> > > >>>>>> https://maven.apache.org/guides/introduction/
> > > > >> >> > > >>>>>>
> > > > >> >> > > >>>>> introduction-to-the-lifecycle.html
> > > > >> >> > > >>>>>
> > > > >> >> > > >>>>>>
> > > > >> >> > > >>>>>> verify - run any checks on results of
> integration
> > > > >> >tests to
> > > > >> >> > > >ensure
> > > > >> >> > > >>>>>> quality criteria are met
> > > > >> >> > > >>>>>>
> > > > >> >> > > >>>>>> Of course our integration tests are long enough
that
> > we
> > > > >> >> shouldn't
> > > > >> >> > > >be
> > > > >> >> > > >>>>>> putting all of them here, but I too would expect
> > > > >> >checkstyle.
> > > > >> >> > > >>>>>>
> > > > >> >> > > >>>>>> Perhaps we could introduce a verify-fast or
somesuch
> > > > >for
> > > > >> >fast
> > > > >> >> > > >(but
> > > > >> >> > > >>>>>> lower coverage) turnaround time. I would expect
"mvn
> > > > >> >verify
> > > > >> >> test"
> > > > >> >> > > >to
> > > > >> >> > > >>>>>> pass before submitting a PR, and would want to run
> > that
> > > > >> >before
> > > > >> >> > > >asking
> > > > >> >> > > >>>>>> others to look at it. I think this should be our
> > > > >criteria
> > > > >> >(i.e.
> > > > >> >> > > >what
> > > > >> >> > > >>>>>> will a new but maven-savvy user run before pushing
> > > > >their
> > > > >> >code).
> > > > >> >> > > >>>>>>
> > > > >> >> > > >>>>>> As long as the pre-commit hooks still check
> everything
> > > > >I'm
> > > > >> >ok
> > > > >> >> > > >with
> > > > >> >> > > >>>>>>>
> > > > >> >> > > >>>>>> making
> > > > >> >> > > >>>>>
> > > > >> >> > > >>>>>> the default a little more lightweight.
> > > > >> >> > > >>>>>>>
> > > > >> >> > > >>>>>>
> > > > >> >> > > >>>>>> The fact that our pre-commit hooks take a long time
> to
> > > > >run
> > > > >> >does
> > > > >> >> > > >change
> > > > >> >> > > >>>>>> things. Nothing more annoying than seeing that your
> PR
> > > > >> >failed 3
> > > > >> >> > > >hours
> > > > >> >> > > >>>>>> later because you had some trailing whitespace...
> > > > >> >> > > >>>>>>
> > > > >> >> > > >>>>>> On Thu, 5 Jan 2017 at 21:49 Lukasz Cwik
> > > > >> >> > > ><lc...@google.com.invalid>
> > > > >> >> > > >>>>>>>
> > > > >> >> > > >>>>>> wrote:
> > > > >> >> > > >>>>>>
> > > > >> >> > > >>>>>>>
> > > > >> >> > > >>>>>>> I was hoping that the default mvn verify would be
> the
> > > > >> >slow
> > > > >> >> build
> > > > >> >> > > >>>>>>>>
> > > > >> >> > > >>>>>>> and a
> > > > >> >> > > >>>>
> > > > >> >> > > >>>>> profile could be enabled that would skip checks to
> make
> > > > >> >things
> > > > >> >> > > >>>>>>>>
> > > > >> >> > > >>>>>>> faster
> > > > >> >> > > >>>>
> > > > >> >> > > >>>>> for
> > > > >> >> > > >>>>>>
> > > > >> >> > > >>>>>>> regular contributors. This way a person doesn't
> need
> > > > >to
> > > > >> >have
> > > > >> >> > > >>>>>>>>
> > > > >> >> > > >>>>>>> detailed
> > > > >> >> > > >>>>
> > > > >> >> > > >>>>> knowledge of all our profiles and what they do
> > > > >(typically
> > > > >> >mvn
> > > > >> >> > > >>>>>>>>
> > > > >> >> > > >>>>>>> verify)
> > > > >> >> > > >>>>
> > > > >> >> > > >>>>> will
> > > > >> >> > > >>>>>>
> > > > >> >> > > >>>>>>> do the right thing most of the time.
> > > > >> >> > > >>>>>>>>
> > > > >> >> > > >>>>>>>> On Thu, Jan 5, 2017 at 9:30 AM, Dan Halperin
> > > > >> >> > > >>>>>>>>
> > > > >> >> > > >>>>>>> <dh...@google.com.invalid>
> > > > >> >> > > >>>>>>
> > > > >> >> > > >>>>>>> wrote:
> > > > >> >> > > >>>>>>>>
> > > > >> >> > > >>>>>>>> On Thu, Jan 5, 2017 at 9:28 AM, Jesse Anderson <
> > > > >> >> > > >>>>>>>>>
> > > > >> >> > > >>>>>>>> jesse@smokinghand.com
> > > > >> >> > > >>>>>
> > > > >> >> > > >>>>>>
> > > > >> >> > > >>>>>>> wrote:
> > > > >> >> > > >>>>>>>>>
> > > > >> >> > > >>>>>>>>> @dan are you saying that mvn verify isn't doing
> > > > >> >checkstyle
> > > > >> >> > > >>>>>>>>>>
> > > > >> >> > > >>>>>>>>> anymore?
> > > > >> >> > > >>>>>
> > > > >> >> > > >>>>>>
> > > > >> >> > > >>>>>>>>>
> > > > >> >> > > >>>>>>>>> `mvn verify` alone should not be running
> > checkstyle,
> > > > >if
> > > > >> >> > > >modules
> > > > >> >> > > >>>>>>>>>
> > > > >> >> > > >>>>>>>> are
> > > > >> >> > > >>>>
> > > > >> >> > > >>>>> configured correctly.
> > > > >> >> > > >>>>>>>>>
> > > > >> >> > > >>>>>>>>>
> > > > >> >> > > >>>>>>>>> Some of
> > > > >> >> > > >>>>>>>>>> the checkstyles are still running for a few
> > > > >modules.
> > > > >> >Also,
> > > > >> >> > > >the
> > > > >> >> > > >>>>>>>>>>
> > > > >> >> > > >>>>>>>>> contribution
> > > > >> >> > > >>>>>>>>>
> > > > >> >> > > >>>>>>>>>> docs will need to change.
> > > > >> >> > > >>>>>>>>>>
> > > > >> >> > > >>>>>>>>>
> > > > >> >> > > >>>>>>>>>
> > > > >> >> > > >>>>>>>>> Yes. The PR includes discussion of these other
> > > > >needed
> > > > >> >> changes,
> > > > >> >> > > >>>>>>>>> unfortunately one PR can't change two
> repositories.
> > > > >> >> > > >>>>>>>>>
> > > > >> >> > > >>>>>>>>> Please continue the discussion on the PR, then I
> > > > >will
> > > > >> >> > > >summarize it
> > > > >> >> > > >>>>>>>>>
> > > > >> >> > > >>>>>>>> back
> > > > >> >> > > >>>>>>
> > > > >> >> > > >>>>>>> into the dev thread.
> > > > >> >> > > >>>>>>>>>
> > > > >> >> > > >>>>>>>>> Thanks,
> > > > >> >> > > >>>>>>>>> Dan
> > > > >> >> > > >>>>>>>>>
> > > > >> >> > > >>>>>>>>>
> > > > >> >> > > >>>>>>>>> They say to run mvn verify before commits.
> > > > >> >> > > >>>>>>>>>>
> > > > >> >> > > >>>>>>>>>> On Thu, Jan 5, 2017 at 9:25 AM Dan Halperin
> > > > >> >> > > >>>>>>>>>>
> > > > >> >> > > >>>>>>>>> <dhalperi@google.com.invalid
> > > > >> >> > > >>>>>>>>
> > > > >> >> > > >>>>>>>>>
> > > > >> >> > > >>>>>>>>>> wrote:
> > > > >> >> > > >>>>>>>>>>
> > > > >> >> > > >>>>>>>>>> Several folks seem to have been confused after
> > > > >> >BEAM-246,
> > > > >> >> > > >where
> > > > >> >> > > >>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>> we
> > > > >> >> > > >>>>>
> > > > >> >> > > >>>>>> moved
> > > > >> >> > > >>>>>>>>
> > > > >> >> > > >>>>>>>>> the
> > > > >> >> > > >>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>> "slow things" into the release profile. I've
> > > > >started
> > > > >> >a
> > > > >> >> > > >>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>> discussion
> > > > >> >> > > >>>>>
> > > > >> >> > > >>>>>> with
> > > > >> >> > > >>>>>>>>
> > > > >> >> > > >>>>>>>>> https://github.com/apache/beam/pull/1740 to see
> if
> > > > >> >there are
> > > > >> >> > > >>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>> things
> > > > >> >> > > >>>>>>
> > > > >> >> > > >>>>>>> we
> > > > >> >> > > >>>>>>>>
> > > > >> >> > > >>>>>>>>> can
> > > > >> >> > > >>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>> do to fill these gaps.
> > > > >> >> > > >>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>> Would love folks to chime in with opinions.
> > > > >> >> > > >>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>> Dan
> > > > >> >> > > >>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>> On Wed, Jan 4, 2017 at 1:34 PM, Jesse Anderson
> <
> > > > >> >> > > >>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>> jesse@smokinghand.com>
> > > > >> >> > > >>>>>>>>
> > > > >> >> > > >>>>>>>>> wrote:
> > > > >> >> > > >>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>> @Eugene, yes that failed on the checkstyle.
> > > > >> >> > > >>>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>>> On Wed, Jan 4, 2017 at 1:27 PM Eugene
> Kirpichov
> > > > >> >> > > >>>>>>>>>>>> <ki...@google.com.invalid> wrote:
> > > > >> >> > > >>>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>>> Try just -Prelease.
> > > > >> >> > > >>>>>>>>>>>>> On Wed, Jan 4, 2017 at 1:21 PM Jesse
> Anderson <
> > > > >> >> > > >>>>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>>> jesse@smokinghand.com
> > > > >> >> > > >>>>>>>>>
> > > > >> >> > > >>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>> wrote:
> > > > >> >> > > >>>>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>>>> Fails because I don't have a secret key.
> > > > >> >> > > >>>>>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>>>>> On Wed, Jan 4, 2017 at 1:03 PM
Jean-Baptiste
> > > > >> >Onofré <
> > > > >> >> > > >>>>>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>>>> jb@nanthrax.net
> > > > >> >> > > >>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>>> wrote:
> > > > >> >> > > >>>>>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>>>>> Hi Jesse,
> > > > >> >> > > >>>>>>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>>>>>> Could you try the same with:
> > > > >> >> > > >>>>>>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>>>>>> mvn verify -Prelease,apache-release
> > > > >> >> > > >>>>>>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>>>>>> ?
> > > > >> >> > > >>>>>>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>>>>>> Regards
> > > > >> >> > > >>>>>>>>>>>>>>> JB
> > > > >> >> > > >>>>>>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>>>>>> On 01/04/2017 09:53 PM, Jesse Anderson
> wrote:
> > > > >> >> > > >>>>>>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>>>>>>> For some reason, running "mvn verify"
> isn't
> > > > >> >running
> > > > >> >> > > >>>>>>>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>>>>>> checkstyle
> > > > >> >> > > >>>>>>>>>
> > > > >> >> > > >>>>>>>>>> on
> > > > >> >> > > >>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>> everything. I had checkstyle errors in
> > > > >> >> > > >>>>>>>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>>>>>> beam-sdks-java-core
> > > > >> >> > > >>>>>>
> > > > >> >> > > >>>>>>> that
> > > > >> >> > > >>>>>>>>>
> > > > >> >> > > >>>>>>>>>> weren't
> > > > >> >> > > >>>>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>>>>> being found.
> > > > >> >> > > >>>>>>>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>>>>>>> I thought this was due to the extra
> > > > >parameters.
> > > > >> >I
> > > > >> >> > > >>>>>>>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>>>>>> reran
> > > > >> >> > > >>>>>
> > > > >> >> > > >>>>>> with
> > > > >> >> > > >>>>>>>>
> > > > >> >> > > >>>>>>>>> the
> > > > >> >> > > >>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>> plain
> > > > >> >> > > >>>>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>>>>> "mvn
> > > > >> >> > > >>>>>>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>>>>>>> verify" and it still didn't find them.
> From
> > > > >the
> > > > >> >> > > >>>>>>>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>>>>>> output,
> > > > >> >> > > >>>>>
> > > > >> >> > > >>>>>> it
> > > > >> >> > > >>>>>>
> > > > >> >> > > >>>>>>> doesn't
> > > > >> >> > > >>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>>> look
> > > > >> >> > > >>>>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>>>>> like they're being run at all.
> > > > >> >> > > >>>>>>>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>>>>>>> Thanks,
> > > > >> >> > > >>>>>>>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>>>>>>> Jesse
> > > > >> >> > > >>>>>>>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>>>>>> --
> > > > >> >> > > >>>>>>>>>>>>>>> Jean-Baptiste Onofré
> > > > >> >> > > >>>>>>>>>>>>>>> jbonofre@apache.org
> > > > >> >> > > >>>>>>>>>>>>>>> http://blog.nanthrax.net
> > > > >> >> > > >>>>>>>>>>>>>>> Talend - http://www.talend.com
> > > > >> >> > > >>>>>>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>
> > > > >> >> > > >>>>>>>>>
> > > > >> >> > > >>>>>>>>
> > > > >> >> > > >>>>>>
> > > > >> >> > > >>>>>
> > > > >> >> > > >>>>
> > > > >> >> > > >>>
> > > > >> >> > > >> --
> > > > >> >> > > >> Jean-Baptiste Onofré
> > > > >> >> > > >> jbonofre@apache.org
> > > > >> >> > > >> http://blog.nanthrax.net
> > > > >> >> > > >> Talend - http://www.talend.com
> > > > >> >> > > >>
> > > > >> >> > >
> > > > >> >> >
> > > > >> >>
> > > > >>
> > > >
> > >
> >
>
Re: Better developer instructions for using Maven?
Posted by Aljoscha Krettek <al...@apache.org>.
I'm with Dan on this. The iteration time should be cut down as low as
possible and we have Jenkins to ensure that tests pass.
As a side note, there are IntelliJ plugins for Checkstyle and Findbugs and
my personal setup highlights Checkstyle violations as errors in the code so
I can immediately see them and fix them.
On Fri, 10 Feb 2017 at 17:45 Dan Halperin <dh...@google.com.invalid>
wrote:
> On Fri, Feb 10, 2017 at 7:42 AM, Kenneth Knowles <kl...@google.com.invalid>
> wrote:
>
> > On Feb 10, 2017 07:36, "Dan Halperin" <dh...@google.com.invalid>
> wrote:
> >
> > Before we added checkstyle it was under a minute. Now it's over five?
> > That's awful IMO
> >
> >
> > Checkstyle didn't cause all that, did it?
> >
>
> The "5 minutes" was going with Aviem's numbers after this change. But yes,
> Checkstyle alone substantially (>+50%) the time from what it was previously
> to adding it back to the default build.
>
> Noting that findbugs takes quite a lot more time. Javadoc and jar are the
> > other two slow ones.
> >
> > RAT is fast. But it has very poor error messages, so we wouldn't want a
> new
> > contributor trying to figure out what is going on without our help.
> >
>
> There is a larger philosophical issue here: is there a point of Jenkins
> precommit testing? Why not just make `mvn install` run everything that
> Jenkins does? For that matter, why don't committers just push directly to
> master? Wouldn't that make everyone's life easier?
>
> I'd argue that's not true.
>
> 1. Developer productivity -- Jenkins should run many more checks than
> developers do. Especially time-, resource-, or setup- intensive tasks.
> 2. Automated enforcement -- Jenkins is better at running the right commands
> than we are.
> 3. Lower the barrier to entry -- individual developers need not have a
> running Spark/Flink/Apex/Dataflow setup in order to contribute code.
> 4. Focus on the user -- someone checking out the code and using it for the
> first time does not care whether the code style checks or has the right
> licenses -- that should have been enforced by the Beam team before
> committing.
>
> We should be *very* choosy about what we enforce on every developer every
> time they go to compile. I probably compile Beam 50x-100x a day. Literally,
> the extra minutes you want to add here will cost me an hour daily.
>
> I've listed the fraction of commits I think will break one of the following
> if that property is not tested:
>
> * compiling (100%)
> * tests (100%)
> * checkstyle (90%)
> * javadoc (30%)
> * findbugs (5%)
> * rat (1%)
>
> So you can see where I stand and why. I'm sorry that 1/20 PRs has Apache
> RAT catch a licensing issue or Findbugs catch a threading issue -- you can
> always get a larger set of the precommit checks using -Prelease, though of
> course the integration tests and runnableonservice tests may catch more
> issues still. But I want my developer minutes back for the 95%+ of the
> rest.
>
> Dan
>
>
> >
> >
> > On Fri, Feb 10, 2017 at 07:14 Aviem Zur <av...@gmail.com> wrote:
> >
> > > Opened JIRA ticket: https://issues.apache.org/jira/browse/BEAM-1457
> > >
> > > On Fri, Feb 10, 2017 at 4:54 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
> > > wrote:
> > >
> > > > Yeah. Agree. Time extend is not huge and it's worth to add it in
> verify
> > > > phase.
> > > >
> > > > Regards
> > > > JB
> > > >
> > > > On Feb 10, 2017, 10:13, at 10:13, Aviem Zur <av...@gmail.com>
> > wrote:
> > > > >This goes back to the original discussion in this thread - reduce
> the
> > > > >amount of things pull requesters should know and keep the maven
> > command
> > > > >in
> > > > >the PR checklist as: 'mvn clean verify'.
> > > > >
> > > > >So if rat and findbugs do not take that long to run I think they
> > should
> > > > >be
> > > > >run by 'mvn clean verify'
> > > > >
> > > > >I ran a quick test on my laptop to see how much time they add to the
> > > > >build
> > > > >(of the entire project):
> > > > >
> > > > >'mvn clean install -DskipTests' => Total time: 03:51 min
> > > > >'mvn clean install apache-rat:check findbugs:check -DskipTests' =>
> > > > >Total
> > > > >time: 05:29 min (Added 01:38 min)
> > > > >'mvn clean install' => Total time: 09:37 min
> > > > >'mvn clean install apache-rat:check findbugs:check' => Total time:
> > > > >11:13
> > > > >min (Added 01:36 min)
> > > > >
> > > > >Are these times reasonable enough to add rat and findbugs to the
> > > > >default
> > > > >build?
> > > > >
> > > > >On Fri, Feb 10, 2017 at 1:55 PM Jean-Baptiste Onofré <
> jb@nanthrax.net
> > >
> > > > >wrote:
> > > > >
> > > > >> Hi
> > > > >>
> > > > >> We discussed about that at the beginning of the project. We agreed
> > to
> > > > >> execute rat and findbugs in a specific profile to reduce the build
> > > > >time for
> > > > >> dev.
> > > > >>
> > > > >> That's why I do mvn clean install -Prelease before submitting a PR
> > > > >and
> > > > >> just clean install when I'm developing.
> > > > >>
> > > > >> No problem to change that.
> > > > >>
> > > > >> Regards
> > > > >> JB
> > > > >>
> > > > >> On Feb 10, 2017, 07:51, at 07:51, Aviem Zur <av...@gmail.com>
> > > > >wrote:
> > > > >> >Can we consider adding rat-plugin and findbugs to the default
> > verify
> > > > >> >phase?
> > > > >> >Currently they only run when the `release` profile is enabled.
> > > > >> >
> > > > >> >On Thu, Jan 26, 2017 at 11:42 AM Aljoscha Krettek
> > > > ><al...@apache.org>
> > > > >> >wrote:
> > > > >> >
> > > > >> >> +1 to what Dan said
> > > > >> >>
> > > > >> >> On Wed, 25 Jan 2017 at 21:40 Kenneth Knowles
> > > > ><kl...@google.com.invalid>
> > > > >> >> wrote:
> > > > >> >>
> > > > >> >> > +1
> > > > >> >> >
> > > > >> >> > On Jan 25, 2017 11:15, "Jean-Baptiste Onofré" <
> jb@nanthrax.net
> > >
> > > > >> >wrote:
> > > > >> >> >
> > > > >> >> > > +1
> > > > >> >> > >
> > > > >> >> > > It sounds good to me.
> > > > >> >> > >
> > > > >> >> > > Thanks Dan !
> > > > >> >> > >
> > > > >> >> > > Regards
> > > > >> >> > > JB
> > > > >> >> > >
> > > > >> >> > > On Jan 25, 2017, 19:39, at 19:39, Dan Halperin
> > > > >> >> > <dh...@google.com.INVALID>
> > > > >> >> > > wrote:
> > > > >> >> > > >Here is my summary of the threads:
> > > > >> >> > > >
> > > > >> >> > > >Overwhelming agreement:
> > > > >> >> > > >
> > > > >> >> > > >- rename `release` to something more appropriate.
> > > > >> >> > > >- add `checkstyle` to the default build (it's basically a
> > > > >> >compile
> > > > >> >> > > >error)
> > > > >> >> > > >- add more information to contributor guide
> > > > >> >> > > >
> > > > >> >> > > >Reasonable agreement
> > > > >> >> > > >
> > > > >> >> > > >- don't update the github instructions to make passing
> `mvn
> > > > >> >verify
> > > > >> >> > > >-P<all
> > > > >> >> > > >checks>` mandatory. Maybe add a hint that this is a good
> > > > >proxy
> > > > >> >for
> > > > >> >> what
> > > > >> >> > > >Jenkins will run.
> > > > >> >> > > >
> > > > >> >> > > >Unresolved:
> > > > >> >> > > >
> > > > >> >> > > >- whether all checks should be in `mvn verify`
> > > > >> >> > > >- whether `mvn test` is useful for most workflows
> > > > >> >> > > >
> > > > >> >> > > >I'll propose to proceed with the overwhelmingly
> agreed-upon
> > > > >> >changes,
> > > > >> >> > > >and as
> > > > >> >> > > >we see increasingly many new contributors re-evaluate the
> > > > >> >remaining
> > > > >> >> > > >issues.
> > > > >> >> > > >
> > > > >> >> > > >Thanks,
> > > > >> >> > > >Dan
> > > > >> >> > > >
> > > > >> >> > > >On Tue, Jan 24, 2017 at 12:51 PM, Jean-Baptiste Onofré
> > > > >> >> > > ><jb...@nanthrax.net>
> > > > >> >> > > >wrote:
> > > > >> >> > > >
> > > > >> >> > > >> +1 to at least update the contribution guide and improve
> > > > >the
> > > > >> >profile
> > > > >> >> > > >name.
> > > > >> >> > > >>
> > > > >> >> > > >> Regards
> > > > >> >> > > >> JB
> > > > >> >> > > >>
> > > > >> >> > > >>
> > > > >> >> > > >> On 01/24/2017 09:49 PM, Kenneth Knowles wrote:
> > > > >> >> > > >>
> > > > >> >> > > >>> My impression is that we don't have consensus on
> whether
> > > > >all
> > > > >> >checks
> > > > >> >> > > >or
> > > > >> >> > > >>> minimal checks should be the default, or whether we can
> > > > >have
> > > > >> >both
> > > > >> >> > > >via `mvn
> > > > >> >> > > >>> test` and `mvn verify`.
> > > > >> >> > > >>>
> > > > >> >> > > >>> But that doesn't prevent us from giving -P release a
> > > > >better
> > > > >> >name
> > > > >> >> and
> > > > >> >> > > >>> mentioning it in the dev guide and in some manner in
> our
> > > > >PR
> > > > >> >> > > >template.
> > > > >> >> > > >>>
> > > > >> >> > > >>> Right now we are living with the combination of the bad
> > > > >> >aspects -
> > > > >> >> > > >default
> > > > >> >> > > >>> is not thorough but not actually fast and a thorough
> > check
> > > > >is
> > > > >> >> > > >>> undocumented.
> > > > >> >> > > >>>
> > > > >> >> > > >>> On Tue, Jan 24, 2017 at 2:22 AM, Ismaël Mejía
> > > > >> ><ie...@gmail.com>
> > > > >> >> > > >wrote:
> > > > >> >> > > >>>
> > > > >> >> > > >>> I just wanted to know if we have achieved some
> consensus
> > > > >> >about
> > > > >> >> this,
> > > > >> >> > > >I
> > > > >> >> > > >>>> just
> > > > >> >> > > >>>> saw this PR that reminded me about this discussion.
> > > > >> >> > > >>>>
> > > > >> >> > > >>>> https://github.com/apache/beam/pull/1829
> > > > >> >> > > >>>>
> > > > >> >> > > >>>> It is important that we mention the existing profiles
> > > > >(and
> > > > >> >the
> > > > >> >> > > >intended
> > > > >> >> > > >>>> checks) in the contribution guide (e.g. -Prelease (or
> > > > >> >-Pall-checks
> > > > >> >> > > >>>> triggers
> > > > >> >> > > >>>> these validations).
> > > > >> >> > > >>>>
> > > > >> >> > > >>>> I can add this to the guide if you like once we define
> > > > >the
> > > > >> >checks
> > > > >> >> > > >per
> > > > >> >> > > >>>> stage/profile.
> > > > >> >> > > >>>>
> > > > >> >> > > >>>> Ismaël
> > > > >> >> > > >>>>
> > > > >> >> > > >>>>
> > > > >> >> > > >>>> On Wed, Jan 11, 2017 at 8:12 AM, Aviem Zur
> > > > >> ><av...@gmail.com>
> > > > >> >> > > >wrote:
> > > > >> >> > > >>>>
> > > > >> >> > > >>>> I agree with Dan and Lukasz.
> > > > >> >> > > >>>>> Developers should not be expected to know beforehand
> > > > >which
> > > > >> >> > > >specific
> > > > >> >> > > >>>>> profiles to run.
> > > > >> >> > > >>>>> The phase specified in the PR instructions (`verify`)
> > > > >> >should run
> > > > >> >> > > >all the
> > > > >> >> > > >>>>> relevant verifications and be the "slower" build,
> while
> > > > >a
> > > > >> >> > > >preceding
> > > > >> >> > > >>>>> lifecycle, such as `test`, should run the "faster"
> > > > >> >verifications.
> > > > >> >> > > >>>>>
> > > > >> >> > > >>>>> Aviem.
> > > > >> >> > > >>>>>
> > > > >> >> > > >>>>> On Mon, Jan 9, 2017 at 7:57 PM Robert Bradshaw
> > > > >> >> > > >>>>>
> > > > >> >> > > >>>> <robertwb@google.com.invalid
> > > > >> >> > > >>>>
> > > > >> >> > > >>>>>
> > > > >> >> > > >>>>>> wrote:
> > > > >> >> > > >>>>>
> > > > >> >> > > >>>>> On Mon, Jan 9, 2017 at 3:49 AM, Aljoscha Krettek
> > > > >> >> > > ><al...@apache.org>
> > > > >> >> > > >>>>>> wrote:
> > > > >> >> > > >>>>>>
> > > > >> >> > > >>>>>>> I also usually prefer "mvn verify" to to the
> expected
> > > > >> >thing but
> > > > >> >> > > >I see
> > > > >> >> > > >>>>>>>
> > > > >> >> > > >>>>>> that
> > > > >> >> > > >>>>>>
> > > > >> >> > > >>>>>>> quick iteration times are key.
> > > > >> >> > > >>>>>>>
> > > > >> >> > > >>>>>>
> > > > >> >> > > >>>>>> I see
> > > > >> >> > > >>>>>> https://maven.apache.org/guides/introduction/
> > > > >> >> > > >>>>>>
> > > > >> >> > > >>>>> introduction-to-the-lifecycle.html
> > > > >> >> > > >>>>>
> > > > >> >> > > >>>>>>
> > > > >> >> > > >>>>>> verify - run any checks on results of
> integration
> > > > >> >tests to
> > > > >> >> > > >ensure
> > > > >> >> > > >>>>>> quality criteria are met
> > > > >> >> > > >>>>>>
> > > > >> >> > > >>>>>> Of course our integration tests are long enough that
> > we
> > > > >> >> shouldn't
> > > > >> >> > > >be
> > > > >> >> > > >>>>>> putting all of them here, but I too would expect
> > > > >> >checkstyle.
> > > > >> >> > > >>>>>>
> > > > >> >> > > >>>>>> Perhaps we could introduce a verify-fast or somesuch
> > > > >for
> > > > >> >fast
> > > > >> >> > > >(but
> > > > >> >> > > >>>>>> lower coverage) turnaround time. I would expect "mvn
> > > > >> >verify
> > > > >> >> test"
> > > > >> >> > > >to
> > > > >> >> > > >>>>>> pass before submitting a PR, and would want to run
> > that
> > > > >> >before
> > > > >> >> > > >asking
> > > > >> >> > > >>>>>> others to look at it. I think this should be our
> > > > >criteria
> > > > >> >(i.e.
> > > > >> >> > > >what
> > > > >> >> > > >>>>>> will a new but maven-savvy user run before pushing
> > > > >their
> > > > >> >code).
> > > > >> >> > > >>>>>>
> > > > >> >> > > >>>>>> As long as the pre-commit hooks still check
> everything
> > > > >I'm
> > > > >> >ok
> > > > >> >> > > >with
> > > > >> >> > > >>>>>>>
> > > > >> >> > > >>>>>> making
> > > > >> >> > > >>>>>
> > > > >> >> > > >>>>>> the default a little more lightweight.
> > > > >> >> > > >>>>>>>
> > > > >> >> > > >>>>>>
> > > > >> >> > > >>>>>> The fact that our pre-commit hooks take a long time
> to
> > > > >run
> > > > >> >does
> > > > >> >> > > >change
> > > > >> >> > > >>>>>> things. Nothing more annoying than seeing that your
> PR
> > > > >> >failed 3
> > > > >> >> > > >hours
> > > > >> >> > > >>>>>> later because you had some trailing whitespace...
> > > > >> >> > > >>>>>>
> > > > >> >> > > >>>>>> On Thu, 5 Jan 2017 at 21:49 Lukasz Cwik
> > > > >> >> > > ><lc...@google.com.invalid>
> > > > >> >> > > >>>>>>>
> > > > >> >> > > >>>>>> wrote:
> > > > >> >> > > >>>>>>
> > > > >> >> > > >>>>>>>
> > > > >> >> > > >>>>>>> I was hoping that the default mvn verify would be
> the
> > > > >> >slow
> > > > >> >> build
> > > > >> >> > > >>>>>>>>
> > > > >> >> > > >>>>>>> and a
> > > > >> >> > > >>>>
> > > > >> >> > > >>>>> profile could be enabled that would skip checks to
> make
> > > > >> >things
> > > > >> >> > > >>>>>>>>
> > > > >> >> > > >>>>>>> faster
> > > > >> >> > > >>>>
> > > > >> >> > > >>>>> for
> > > > >> >> > > >>>>>>
> > > > >> >> > > >>>>>>> regular contributors. This way a person doesn't
> need
> > > > >to
> > > > >> >have
> > > > >> >> > > >>>>>>>>
> > > > >> >> > > >>>>>>> detailed
> > > > >> >> > > >>>>
> > > > >> >> > > >>>>> knowledge of all our profiles and what they do
> > > > >(typically
> > > > >> >mvn
> > > > >> >> > > >>>>>>>>
> > > > >> >> > > >>>>>>> verify)
> > > > >> >> > > >>>>
> > > > >> >> > > >>>>> will
> > > > >> >> > > >>>>>>
> > > > >> >> > > >>>>>>> do the right thing most of the time.
> > > > >> >> > > >>>>>>>>
> > > > >> >> > > >>>>>>>> On Thu, Jan 5, 2017 at 9:30 AM, Dan Halperin
> > > > >> >> > > >>>>>>>>
> > > > >> >> > > >>>>>>> <dh...@google.com.invalid>
> > > > >> >> > > >>>>>>
> > > > >> >> > > >>>>>>> wrote:
> > > > >> >> > > >>>>>>>>
> > > > >> >> > > >>>>>>>> On Thu, Jan 5, 2017 at 9:28 AM, Jesse Anderson <
> > > > >> >> > > >>>>>>>>>
> > > > >> >> > > >>>>>>>> jesse@smokinghand.com
> > > > >> >> > > >>>>>
> > > > >> >> > > >>>>>>
> > > > >> >> > > >>>>>>> wrote:
> > > > >> >> > > >>>>>>>>>
> > > > >> >> > > >>>>>>>>> @dan are you saying that mvn verify isn't doing
> > > > >> >checkstyle
> > > > >> >> > > >>>>>>>>>>
> > > > >> >> > > >>>>>>>>> anymore?
> > > > >> >> > > >>>>>
> > > > >> >> > > >>>>>>
> > > > >> >> > > >>>>>>>>>
> > > > >> >> > > >>>>>>>>> `mvn verify` alone should not be running
> > checkstyle,
> > > > >if
> > > > >> >> > > >modules
> > > > >> >> > > >>>>>>>>>
> > > > >> >> > > >>>>>>>> are
> > > > >> >> > > >>>>
> > > > >> >> > > >>>>> configured correctly.
> > > > >> >> > > >>>>>>>>>
> > > > >> >> > > >>>>>>>>>
> > > > >> >> > > >>>>>>>>> Some of
> > > > >> >> > > >>>>>>>>>> the checkstyles are still running for a few
> > > > >modules.
> > > > >> >Also,
> > > > >> >> > > >the
> > > > >> >> > > >>>>>>>>>>
> > > > >> >> > > >>>>>>>>> contribution
> > > > >> >> > > >>>>>>>>>
> > > > >> >> > > >>>>>>>>>> docs will need to change.
> > > > >> >> > > >>>>>>>>>>
> > > > >> >> > > >>>>>>>>>
> > > > >> >> > > >>>>>>>>>
> > > > >> >> > > >>>>>>>>> Yes. The PR includes discussion of these other
> > > > >needed
> > > > >> >> changes,
> > > > >> >> > > >>>>>>>>> unfortunately one PR can't change two
> repositories.
> > > > >> >> > > >>>>>>>>>
> > > > >> >> > > >>>>>>>>> Please continue the discussion on the PR, then I
> > > > >will
> > > > >> >> > > >summarize it
> > > > >> >> > > >>>>>>>>>
> > > > >> >> > > >>>>>>>> back
> > > > >> >> > > >>>>>>
> > > > >> >> > > >>>>>>> into the dev thread.
> > > > >> >> > > >>>>>>>>>
> > > > >> >> > > >>>>>>>>> Thanks,
> > > > >> >> > > >>>>>>>>> Dan
> > > > >> >> > > >>>>>>>>>
> > > > >> >> > > >>>>>>>>>
> > > > >> >> > > >>>>>>>>> They say to run mvn verify before commits.
> > > > >> >> > > >>>>>>>>>>
> > > > >> >> > > >>>>>>>>>> On Thu, Jan 5, 2017 at 9:25 AM Dan Halperin
> > > > >> >> > > >>>>>>>>>>
> > > > >> >> > > >>>>>>>>> <dhalperi@google.com.invalid
> > > > >> >> > > >>>>>>>>
> > > > >> >> > > >>>>>>>>>
> > > > >> >> > > >>>>>>>>>> wrote:
> > > > >> >> > > >>>>>>>>>>
> > > > >> >> > > >>>>>>>>>> Several folks seem to have been confused after
> > > > >> >BEAM-246,
> > > > >> >> > > >where
> > > > >> >> > > >>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>> we
> > > > >> >> > > >>>>>
> > > > >> >> > > >>>>>> moved
> > > > >> >> > > >>>>>>>>
> > > > >> >> > > >>>>>>>>> the
> > > > >> >> > > >>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>> "slow things" into the release profile. I've
> > > > >started
> > > > >> >a
> > > > >> >> > > >>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>> discussion
> > > > >> >> > > >>>>>
> > > > >> >> > > >>>>>> with
> > > > >> >> > > >>>>>>>>
> > > > >> >> > > >>>>>>>>> https://github.com/apache/beam/pull/1740 to see
> if
> > > > >> >there are
> > > > >> >> > > >>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>> things
> > > > >> >> > > >>>>>>
> > > > >> >> > > >>>>>>> we
> > > > >> >> > > >>>>>>>>
> > > > >> >> > > >>>>>>>>> can
> > > > >> >> > > >>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>> do to fill these gaps.
> > > > >> >> > > >>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>> Would love folks to chime in with opinions.
> > > > >> >> > > >>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>> Dan
> > > > >> >> > > >>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>> On Wed, Jan 4, 2017 at 1:34 PM, Jesse Anderson
> <
> > > > >> >> > > >>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>> jesse@smokinghand.com>
> > > > >> >> > > >>>>>>>>
> > > > >> >> > > >>>>>>>>> wrote:
> > > > >> >> > > >>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>> @Eugene, yes that failed on the checkstyle.
> > > > >> >> > > >>>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>>> On Wed, Jan 4, 2017 at 1:27 PM Eugene
> Kirpichov
> > > > >> >> > > >>>>>>>>>>>> <ki...@google.com.invalid> wrote:
> > > > >> >> > > >>>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>>> Try just -Prelease.
> > > > >> >> > > >>>>>>>>>>>>> On Wed, Jan 4, 2017 at 1:21 PM Jesse
> Anderson <
> > > > >> >> > > >>>>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>>> jesse@smokinghand.com
> > > > >> >> > > >>>>>>>>>
> > > > >> >> > > >>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>> wrote:
> > > > >> >> > > >>>>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>>>> Fails because I don't have a secret key.
> > > > >> >> > > >>>>>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>>>>> On Wed, Jan 4, 2017 at 1:03 PM Jean-Baptiste
> > > > >> >Onofré <
> > > > >> >> > > >>>>>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>>>> jb@nanthrax.net
> > > > >> >> > > >>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>>> wrote:
> > > > >> >> > > >>>>>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>>>>> Hi Jesse,
> > > > >> >> > > >>>>>>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>>>>>> Could you try the same with:
> > > > >> >> > > >>>>>>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>>>>>> mvn verify -Prelease,apache-release
> > > > >> >> > > >>>>>>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>>>>>> ?
> > > > >> >> > > >>>>>>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>>>>>> Regards
> > > > >> >> > > >>>>>>>>>>>>>>> JB
> > > > >> >> > > >>>>>>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>>>>>> On 01/04/2017 09:53 PM, Jesse Anderson
> wrote:
> > > > >> >> > > >>>>>>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>>>>>>> For some reason, running "mvn verify"
> isn't
> > > > >> >running
> > > > >> >> > > >>>>>>>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>>>>>> checkstyle
> > > > >> >> > > >>>>>>>>>
> > > > >> >> > > >>>>>>>>>> on
> > > > >> >> > > >>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>> everything. I had checkstyle errors in
> > > > >> >> > > >>>>>>>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>>>>>> beam-sdks-java-core
> > > > >> >> > > >>>>>>
> > > > >> >> > > >>>>>>> that
> > > > >> >> > > >>>>>>>>>
> > > > >> >> > > >>>>>>>>>> weren't
> > > > >> >> > > >>>>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>>>>> being found.
> > > > >> >> > > >>>>>>>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>>>>>>> I thought this was due to the extra
> > > > >parameters.
> > > > >> >I
> > > > >> >> > > >>>>>>>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>>>>>> reran
> > > > >> >> > > >>>>>
> > > > >> >> > > >>>>>> with
> > > > >> >> > > >>>>>>>>
> > > > >> >> > > >>>>>>>>> the
> > > > >> >> > > >>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>> plain
> > > > >> >> > > >>>>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>>>>> "mvn
> > > > >> >> > > >>>>>>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>>>>>>> verify" and it still didn't find them.
> From
> > > > >the
> > > > >> >> > > >>>>>>>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>>>>>> output,
> > > > >> >> > > >>>>>
> > > > >> >> > > >>>>>> it
> > > > >> >> > > >>>>>>
> > > > >> >> > > >>>>>>> doesn't
> > > > >> >> > > >>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>>> look
> > > > >> >> > > >>>>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>>>>> like they're being run at all.
> > > > >> >> > > >>>>>>>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>>>>>>> Thanks,
> > > > >> >> > > >>>>>>>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>>>>>>> Jesse
> > > > >> >> > > >>>>>>>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>>>>>> --
> > > > >> >> > > >>>>>>>>>>>>>>> Jean-Baptiste Onofré
> > > > >> >> > > >>>>>>>>>>>>>>> jbonofre@apache.org
> > > > >> >> > > >>>>>>>>>>>>>>> http://blog.nanthrax.net
> > > > >> >> > > >>>>>>>>>>>>>>> Talend - http://www.talend.com
> > > > >> >> > > >>>>>>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>>
> > > > >> >> > > >>>>>>>>>>
> > > > >> >> > > >>>>>>>>>
> > > > >> >> > > >>>>>>>>
> > > > >> >> > > >>>>>>
> > > > >> >> > > >>>>>
> > > > >> >> > > >>>>
> > > > >> >> > > >>>
> > > > >> >> > > >> --
> > > > >> >> > > >> Jean-Baptiste Onofré
> > > > >> >> > > >> jbonofre@apache.org
> > > > >> >> > > >> http://blog.nanthrax.net
> > > > >> >> > > >> Talend - http://www.talend.com
> > > > >> >> > > >>
> > > > >> >> > >
> > > > >> >> >
> > > > >> >>
> > > > >>
> > > >
> > >
> >
>
Re: Better developer instructions for using Maven?
Posted by Ismaël Mejía <ie...@gmail.com>.
This question got lost in the discussion, but there is a small improvement
that we can do:
> Just to check, are we doing parallel builds?
We are on jenkins, not in travis, there is an ongoing PR to fix this.
What we can improve is to check if we can run some of the test suites in
parallel to gain some extra time. For exemple most of the IOs and some
runners don't execute tests in parallel.
Ismael
(slightly related), is there a way to get change the timeout of travis
jobs). Because it is failing most of the time now because of this, and it
is quite noisey to have so many false positives.
On Fri, Feb 10, 2017 at 8:00 PM, Robert Bradshaw <
robertwb@google.com.invalid> wrote:
> On Fri, Feb 10, 2017 at 8:45 AM, Dan Halperin <dhalperi@google.com.invalid
> >
> wrote:
>
> > On Fri, Feb 10, 2017 at 7:42 AM, Kenneth Knowles <klk@google.com.invalid
> >
> > wrote:
> >
> > > On Feb 10, 2017 07:36, "Dan Halperin" <dh...@google.com.invalid>
> > wrote:
> > >
> > > Before we added checkstyle it was under a minute. Now it's over five?
> > > That's awful IMO
> > >
> > >
> > > Checkstyle didn't cause all that, did it?
> > >
> >
> > The "5 minutes" was going with Aviem's numbers after this change. But
> yes,
> > Checkstyle alone substantially (>+50%) the time from what it was
> previously
> > to adding it back to the default build.
>
>
> Just to check, are we doing parallel builds?
>
>
> >
> > Noting that findbugs takes quite a lot more time. Javadoc and jar are the
> > > other two slow ones.
> > >
> > > RAT is fast. But it has very poor error messages, so we wouldn't want a
> > new
> > > contributor trying to figure out what is going on without our help.
> > >
> >
> > There is a larger philosophical issue here: is there a point of Jenkins
> > precommit testing? Why not just make `mvn install` run everything that
> > Jenkins does? For that matter, why don't committers just push directly to
> > master? Wouldn't that make everyone's life easier?
> >
> > I'd argue that's not true.
> >
> > 1. Developer productivity -- Jenkins should run many more checks than
> > developers do. Especially time-, resource-, or setup- intensive tasks.
> > 2. Automated enforcement -- Jenkins is better at running the right
> commands
> > than we are.
> > 3. Lower the barrier to entry -- individual developers need not have a
> > running Spark/Flink/Apex/Dataflow setup in order to contribute code.
> > 4. Focus on the user -- someone checking out the code and using it for
> the
> > first time does not care whether the code style checks or has the right
> > licenses -- that should have been enforced by the Beam team before
> > committing.
> >
> > We should be *very* choosy about what we enforce on every developer every
> > time they go to compile. I probably compile Beam 50x-100x a day.
> Literally,
> > the extra minutes you want to add here will cost me an hour daily.
> >
>
> By the same token of having a different bar for the Jenkins presubmit vs.
> what's run locally, I think it makes a lot of sense to run a different
> command for iterative development than you run before creating a pull
> request. E.g. during development I'll often run only one test rather than
> the entire suite, but do run the entire suite occasionally (often before
> commit, especially before pushing).
>
> The contributors guild should give a suggested command to run before
> creating a PR, right in the docs of how to create a PR, which may be more
> expensive than what you run during development. IMHO, this should be fairly
> comprehensive (certainly tests and checkstyle, maybe javadoc and findbugs).
> This should be the "default" command that the one-time-contributor should
> know. For those compiling 50x or more a day, I think the burden of learning
> a second (or more) cheaper commands is not high, and we could even put such
> a thing in the docs (and hopefully a common maven convention like "mvn
> test").
>
> I've listed the fraction of commits I think will break one of the following
> > if that property is not tested:
> >
> > * compiling (100%)
> > * tests (100%)
> > * checkstyle (90%)
> > * javadoc (30%)
> > * findbugs (5%)
> > * rat (1%)
> >
> > So you can see where I stand and why. I'm sorry that 1/20 PRs has Apache
> > RAT catch a licensing issue or Findbugs catch a threading issue -- you
> can
> > always get a larger set of the precommit checks using -Prelease, though
> of
> > course the integration tests and runnableonservice tests may catch more
> > issues still. But I want my developer minutes back for the 95%+ of the
> > rest.
> >
> > Dan
> >
>
Re: Better developer instructions for using Maven?
Posted by Robert Bradshaw <ro...@google.com.INVALID>.
On Fri, Feb 10, 2017 at 11:54 AM, Davor Bonaci <da...@google.com.invalid>
wrote:
> I think Dan's framework of thinking is right -- what is the probability of
> something finding a real issue, vs. the cost of running that all the time.
>
> Obviously, we cannot run *everything* all the time. There's an infinite
> number of things to run and infinite matrix of configurations. Many tests
> are not even possible to execute locally like testing scale, performance,
> compatibility across platforms, etc. So, we run locally *the least*,
> slightly more in presubmit, slightly more in post-submit, even more
> nightly, and as much as possible for each release. Of course, it is great
> to catch issues as soon as possible, but the cost is prohibitive in terms
> of time and productivity.
>
> This may sound a bit tangential so far... but, I think Aviem's (and others)
> point of view come from a social aspect. As a contributor, nobody wants to
> be "embarrassed" by proposing a pull request that is broken and has obvious
> problems. One might think of themselves as failing to do a good job in this
> particular case. So, instead, I might want to default to running more
> things locally to prevent the "embarrassment".
>
+1
It's also disheartening to think a PR is ready for folks to look at, only
to notice (later, when you've moved on to doing something else and a
Jenkins issue comes in) that you have, say, some trivial checkstyle
whitespace issue.
Also, in my experience, newcomers usually value direction and the ability
to algorithmically, privately verify their code quite a bit.
On the other hand, having to start up a spark cluster or create GCP
credentials is a higher burden that is often better offloaded to something
like Jenkins.
Looking more broadly, however, there's a huge difference in things that the
> default local execution *and* pre-commit don't exercise. It is really huge
> and many breakages are caught in post commit, nightlies, or, at the very
> end, cherrypicked into the release. So, I'd like to argue that nobody
> should ever be embarrassed by proposing a pull request. It is normal,
> expected, reasonable that Jenkins will catch issues -- that's why we have
> it. Instead, think in terms of "error budget" -- increase individual
> productivity by sacrificing occasional breakages. It is also completely
> normal that a PR causes a post-commit breakage at times, or that a release
> is delayed because of some PR -- this is all normal and fits into your
> "error budget".
>
<aside>In my opinion, post-commits should catch virtually nothing, or at
least nothing deterministic. Once we have a mergebot, there's no reason we
should't test virtually everything we do on those runs as a last step
before every merge instead/as well.</aside>.
One easy improvement that I know Jason is looking at is to separate the
> precommit signal into multiple signals. So, instead of a one "red" signal,
> contributors can get five "greens" and one "red", which may help decrease
> this social impact.
>
That would be great.
> Hopefully, this is a convincing argument to use this framework in deciding
> this matter. On the lower-lever issue, I think rat and findbugs have a low
> probability of finding an issue in most cases.
>
> (Also, an explicit +1 to Kenn's point of view of getting people to the PR
> so we can work with them, as opposed to blocking them locally before they
> interact with us.)
>
We should also factor in how (naively) actionable the errors from the
various tools are. Linters are typically pretty explicit about what needs
to be changed, other tools sometimes less so.
> On Fri, Feb 10, 2017 at 11:50 AM, Kenneth Knowles <kl...@google.com.invalid>
> wrote:
>
> > Since the discussion has returned to the thread rather than Dan's PR, I
> > want to paraphrase the point I feel strongest about here:
> >
> > *For a new contributor, I want to minimize the distance between them
> > deciding to hack and becoming our friends.*
> >
> > So I don't want them to have to learn much, if anything, about our idioms
> > prior to opening a PR. That includes checkstyle, findbugs, javadoc, RAT,
> > (any others?). Once a PR is open, we can welcome them to the community,
> > help them understand any failure that happened via Jenkins, tell them the
> > command to run the more thorough test, or even issue little fix PRs
> against
> > their branch. If they get blocked by nitpicky or confusing failures in
> > their console or while hacking, they might (reasonably, IMO) decide to go
> > do something else.
> >
> > Folks other than newcomers can learn a repertoire of commands, like
> Robert
> > says. So we shouldn't consider them (aka "us") so much when deciding
> > whether "fast" or "slow" is the default, as long as we can explicitly
> > choose.
> >
> > Kenn
> >
> > On Fri, Feb 10, 2017 at 11:00 AM, Robert Bradshaw <
> > robertwb@google.com.invalid> wrote:
> >
> > > > 1. Developer productivity -- Jenkins should run many more checks than
> > > > developers do. Especially time-, resource-, or setup- intensive
> tasks.
> > > > 2. Automated enforcement -- Jenkins is better at running the right
> > > commands
> > > > than we are.
> > > > 3. Lower the barrier to entry -- individual developers need not have
> a
> > > > running Spark/Flink/Apex/Dataflow setup in order to contribute code.
> > > > 4. Focus on the user -- someone checking out the code and using it
> for
> > > the
> > > > first time does not care whether the code style checks or has the
> right
> > > > licenses -- that should have been enforced by the Beam team before
> > > > committing.
> > > >
> > > > We should be *very* choosy about what we enforce on every developer
> > every
> > > > time they go to compile. I probably compile Beam 50x-100x a day.
> > > Literally,
> > > > the extra minutes you want to add here will cost me an hour daily.
> > > >
> > >
> > > By the same token of having a different bar for the Jenkins presubmit
> vs.
> > > what's run locally, I think it makes a lot of sense to run a different
> > > command for iterative development than you run before creating a pull
> > > request. E.g. during development I'll often run only one test rather
> than
> > > the entire suite, but do run the entire suite occasionally (often
> before
> > > commit, especially before pushing).
> > >
> > > The contributors guild should give a suggested command to run before
> > > creating a PR, right in the docs of how to create a PR, which may be
> more
> > > expensive than what you run during development. IMHO, this should be
> > fairly
> > > comprehensive (certainly tests and checkstyle, maybe javadoc and
> > findbugs).
> > > This should be the "default" command that the one-time-contributor
> should
> > > know. For those compiling 50x or more a day, I think the burden of
> > learning
> > > a second (or more) cheaper commands is not high, and we could even put
> > such
> > > a thing in the docs (and hopefully a common maven convention like "mvn
> > > test").
> >
> >
> >
> > Kenn
> >
> >
> > > I've listed the fraction of commits I think will break one of the
> > following
> > > > if that property is not tested:
> > > >
> > > > * compiling (100%)
> > > > * tests (100%)
> > > > * checkstyle (90%)
> > > > * javadoc (30%)
> > > > * findbugs (5%)
> > > > * rat (1%)
> > > >
> > > > So you can see where I stand and why. I'm sorry that 1/20 PRs has
> > Apache
> > > > RAT catch a licensing issue or Findbugs catch a threading issue --
> you
> > > can
> > > > always get a larger set of the precommit checks using -Prelease,
> though
> > > of
> > > > course the integration tests and runnableonservice tests may catch
> more
> > > > issues still. But I want my developer minutes back for the 95%+ of
> the
> > > > rest.
> > > >
> > > > Dan
> > > >
> > >
> >
>
Re: Better developer instructions for using Maven?
Posted by Davor Bonaci <da...@google.com.INVALID>.
I think Dan's framework of thinking is right -- what is the probability of
something finding a real issue, vs. the cost of running that all the time.
Obviously, we cannot run *everything* all the time. There's an infinite
number of things to run and infinite matrix of configurations. Many tests
are not even possible to execute locally like testing scale, performance,
compatibility across platforms, etc. So, we run locally *the least*,
slightly more in presubmit, slightly more in post-submit, even more
nightly, and as much as possible for each release. Of course, it is great
to catch issues as soon as possible, but the cost is prohibitive in terms
of time and productivity.
This may sound a bit tangential so far... but, I think Aviem's (and others)
point of view come from a social aspect. As a contributor, nobody wants to
be "embarrassed" by proposing a pull request that is broken and has obvious
problems. One might think of themselves as failing to do a good job in this
particular case. So, instead, I might want to default to running more
things locally to prevent the "embarrassment".
Looking more broadly, however, there's a huge difference in things that the
default local execution *and* pre-commit don't exercise. It is really huge
and many breakages are caught in post commit, nightlies, or, at the very
end, cherrypicked into the release. So, I'd like to argue that nobody
should ever be embarrassed by proposing a pull request. It is normal,
expected, reasonable that Jenkins will catch issues -- that's why we have
it. Instead, think in terms of "error budget" -- increase individual
productivity by sacrificing occasional breakages. It is also completely
normal that a PR causes a post-commit breakage at times, or that a release
is delayed because of some PR -- this is all normal and fits into your
"error budget".
One easy improvement that I know Jason is looking at is to separate the
precommit signal into multiple signals. So, instead of a one "red" signal,
contributors can get five "greens" and one "red", which may help decrease
this social impact.
Hopefully, this is a convincing argument to use this framework in deciding
this matter. On the lower-lever issue, I think rat and findbugs have a low
probability of finding an issue in most cases.
(Also, an explicit +1 to Kenn's point of view of getting people to the PR
so we can work with them, as opposed to blocking them locally before they
interact with us.)
On Fri, Feb 10, 2017 at 11:50 AM, Kenneth Knowles <kl...@google.com.invalid>
wrote:
> Since the discussion has returned to the thread rather than Dan's PR, I
> want to paraphrase the point I feel strongest about here:
>
> *For a new contributor, I want to minimize the distance between them
> deciding to hack and becoming our friends.*
>
> So I don't want them to have to learn much, if anything, about our idioms
> prior to opening a PR. That includes checkstyle, findbugs, javadoc, RAT,
> (any others?). Once a PR is open, we can welcome them to the community,
> help them understand any failure that happened via Jenkins, tell them the
> command to run the more thorough test, or even issue little fix PRs against
> their branch. If they get blocked by nitpicky or confusing failures in
> their console or while hacking, they might (reasonably, IMO) decide to go
> do something else.
>
> Folks other than newcomers can learn a repertoire of commands, like Robert
> says. So we shouldn't consider them (aka "us") so much when deciding
> whether "fast" or "slow" is the default, as long as we can explicitly
> choose.
>
> Kenn
>
> On Fri, Feb 10, 2017 at 11:00 AM, Robert Bradshaw <
> robertwb@google.com.invalid> wrote:
>
> > > 1. Developer productivity -- Jenkins should run many more checks than
> > > developers do. Especially time-, resource-, or setup- intensive tasks.
> > > 2. Automated enforcement -- Jenkins is better at running the right
> > commands
> > > than we are.
> > > 3. Lower the barrier to entry -- individual developers need not have a
> > > running Spark/Flink/Apex/Dataflow setup in order to contribute code.
> > > 4. Focus on the user -- someone checking out the code and using it for
> > the
> > > first time does not care whether the code style checks or has the right
> > > licenses -- that should have been enforced by the Beam team before
> > > committing.
> > >
> > > We should be *very* choosy about what we enforce on every developer
> every
> > > time they go to compile. I probably compile Beam 50x-100x a day.
> > Literally,
> > > the extra minutes you want to add here will cost me an hour daily.
> > >
> >
> > By the same token of having a different bar for the Jenkins presubmit vs.
> > what's run locally, I think it makes a lot of sense to run a different
> > command for iterative development than you run before creating a pull
> > request. E.g. during development I'll often run only one test rather than
> > the entire suite, but do run the entire suite occasionally (often before
> > commit, especially before pushing).
> >
> > The contributors guild should give a suggested command to run before
> > creating a PR, right in the docs of how to create a PR, which may be more
> > expensive than what you run during development. IMHO, this should be
> fairly
> > comprehensive (certainly tests and checkstyle, maybe javadoc and
> findbugs).
> > This should be the "default" command that the one-time-contributor should
> > know. For those compiling 50x or more a day, I think the burden of
> learning
> > a second (or more) cheaper commands is not high, and we could even put
> such
> > a thing in the docs (and hopefully a common maven convention like "mvn
> > test").
>
>
>
> Kenn
>
>
> > I've listed the fraction of commits I think will break one of the
> following
> > > if that property is not tested:
> > >
> > > * compiling (100%)
> > > * tests (100%)
> > > * checkstyle (90%)
> > > * javadoc (30%)
> > > * findbugs (5%)
> > > * rat (1%)
> > >
> > > So you can see where I stand and why. I'm sorry that 1/20 PRs has
> Apache
> > > RAT catch a licensing issue or Findbugs catch a threading issue -- you
> > can
> > > always get a larger set of the precommit checks using -Prelease, though
> > of
> > > course the integration tests and runnableonservice tests may catch more
> > > issues still. But I want my developer minutes back for the 95%+ of the
> > > rest.
> > >
> > > Dan
> > >
> >
>
Re: Better developer instructions for using Maven?
Posted by Kenneth Knowles <kl...@google.com.INVALID>.
Since the discussion has returned to the thread rather than Dan's PR, I
want to paraphrase the point I feel strongest about here:
*For a new contributor, I want to minimize the distance between them
deciding to hack and becoming our friends.*
So I don't want them to have to learn much, if anything, about our idioms
prior to opening a PR. That includes checkstyle, findbugs, javadoc, RAT,
(any others?). Once a PR is open, we can welcome them to the community,
help them understand any failure that happened via Jenkins, tell them the
command to run the more thorough test, or even issue little fix PRs against
their branch. If they get blocked by nitpicky or confusing failures in
their console or while hacking, they might (reasonably, IMO) decide to go
do something else.
Folks other than newcomers can learn a repertoire of commands, like Robert
says. So we shouldn't consider them (aka "us") so much when deciding
whether "fast" or "slow" is the default, as long as we can explicitly
choose.
Kenn
On Fri, Feb 10, 2017 at 11:00 AM, Robert Bradshaw <
robertwb@google.com.invalid> wrote:
> > 1. Developer productivity -- Jenkins should run many more checks than
> > developers do. Especially time-, resource-, or setup- intensive tasks.
> > 2. Automated enforcement -- Jenkins is better at running the right
> commands
> > than we are.
> > 3. Lower the barrier to entry -- individual developers need not have a
> > running Spark/Flink/Apex/Dataflow setup in order to contribute code.
> > 4. Focus on the user -- someone checking out the code and using it for
> the
> > first time does not care whether the code style checks or has the right
> > licenses -- that should have been enforced by the Beam team before
> > committing.
> >
> > We should be *very* choosy about what we enforce on every developer every
> > time they go to compile. I probably compile Beam 50x-100x a day.
> Literally,
> > the extra minutes you want to add here will cost me an hour daily.
> >
>
> By the same token of having a different bar for the Jenkins presubmit vs.
> what's run locally, I think it makes a lot of sense to run a different
> command for iterative development than you run before creating a pull
> request. E.g. during development I'll often run only one test rather than
> the entire suite, but do run the entire suite occasionally (often before
> commit, especially before pushing).
>
> The contributors guild should give a suggested command to run before
> creating a PR, right in the docs of how to create a PR, which may be more
> expensive than what you run during development. IMHO, this should be fairly
> comprehensive (certainly tests and checkstyle, maybe javadoc and findbugs).
> This should be the "default" command that the one-time-contributor should
> know. For those compiling 50x or more a day, I think the burden of learning
> a second (or more) cheaper commands is not high, and we could even put such
> a thing in the docs (and hopefully a common maven convention like "mvn
> test").
Kenn
> I've listed the fraction of commits I think will break one of the following
> > if that property is not tested:
> >
> > * compiling (100%)
> > * tests (100%)
> > * checkstyle (90%)
> > * javadoc (30%)
> > * findbugs (5%)
> > * rat (1%)
> >
> > So you can see where I stand and why. I'm sorry that 1/20 PRs has Apache
> > RAT catch a licensing issue or Findbugs catch a threading issue -- you
> can
> > always get a larger set of the precommit checks using -Prelease, though
> of
> > course the integration tests and runnableonservice tests may catch more
> > issues still. But I want my developer minutes back for the 95%+ of the
> > rest.
> >
> > Dan
> >
>
Re: Better developer instructions for using Maven?
Posted by Robert Bradshaw <ro...@google.com.INVALID>.
On Fri, Feb 10, 2017 at 8:45 AM, Dan Halperin <dh...@google.com.invalid>
wrote:
> On Fri, Feb 10, 2017 at 7:42 AM, Kenneth Knowles <kl...@google.com.invalid>
> wrote:
>
> > On Feb 10, 2017 07:36, "Dan Halperin" <dh...@google.com.invalid>
> wrote:
> >
> > Before we added checkstyle it was under a minute. Now it's over five?
> > That's awful IMO
> >
> >
> > Checkstyle didn't cause all that, did it?
> >
>
> The "5 minutes" was going with Aviem's numbers after this change. But yes,
> Checkstyle alone substantially (>+50%) the time from what it was previously
> to adding it back to the default build.
Just to check, are we doing parallel builds?
>
> Noting that findbugs takes quite a lot more time. Javadoc and jar are the
> > other two slow ones.
> >
> > RAT is fast. But it has very poor error messages, so we wouldn't want a
> new
> > contributor trying to figure out what is going on without our help.
> >
>
> There is a larger philosophical issue here: is there a point of Jenkins
> precommit testing? Why not just make `mvn install` run everything that
> Jenkins does? For that matter, why don't committers just push directly to
> master? Wouldn't that make everyone's life easier?
>
> I'd argue that's not true.
>
> 1. Developer productivity -- Jenkins should run many more checks than
> developers do. Especially time-, resource-, or setup- intensive tasks.
> 2. Automated enforcement -- Jenkins is better at running the right commands
> than we are.
> 3. Lower the barrier to entry -- individual developers need not have a
> running Spark/Flink/Apex/Dataflow setup in order to contribute code.
> 4. Focus on the user -- someone checking out the code and using it for the
> first time does not care whether the code style checks or has the right
> licenses -- that should have been enforced by the Beam team before
> committing.
>
> We should be *very* choosy about what we enforce on every developer every
> time they go to compile. I probably compile Beam 50x-100x a day. Literally,
> the extra minutes you want to add here will cost me an hour daily.
>
By the same token of having a different bar for the Jenkins presubmit vs.
what's run locally, I think it makes a lot of sense to run a different
command for iterative development than you run before creating a pull
request. E.g. during development I'll often run only one test rather than
the entire suite, but do run the entire suite occasionally (often before
commit, especially before pushing).
The contributors guild should give a suggested command to run before
creating a PR, right in the docs of how to create a PR, which may be more
expensive than what you run during development. IMHO, this should be fairly
comprehensive (certainly tests and checkstyle, maybe javadoc and findbugs).
This should be the "default" command that the one-time-contributor should
know. For those compiling 50x or more a day, I think the burden of learning
a second (or more) cheaper commands is not high, and we could even put such
a thing in the docs (and hopefully a common maven convention like "mvn
test").
I've listed the fraction of commits I think will break one of the following
> if that property is not tested:
>
> * compiling (100%)
> * tests (100%)
> * checkstyle (90%)
> * javadoc (30%)
> * findbugs (5%)
> * rat (1%)
>
> So you can see where I stand and why. I'm sorry that 1/20 PRs has Apache
> RAT catch a licensing issue or Findbugs catch a threading issue -- you can
> always get a larger set of the precommit checks using -Prelease, though of
> course the integration tests and runnableonservice tests may catch more
> issues still. But I want my developer minutes back for the 95%+ of the
> rest.
>
> Dan
>
Re: Better developer instructions for using Maven?
Posted by Dan Halperin <dh...@google.com.INVALID>.
On Fri, Feb 10, 2017 at 7:42 AM, Kenneth Knowles <kl...@google.com.invalid>
wrote:
> On Feb 10, 2017 07:36, "Dan Halperin" <dh...@google.com.invalid> wrote:
>
> Before we added checkstyle it was under a minute. Now it's over five?
> That's awful IMO
>
>
> Checkstyle didn't cause all that, did it?
>
The "5 minutes" was going with Aviem's numbers after this change. But yes,
Checkstyle alone substantially (>+50%) the time from what it was previously
to adding it back to the default build.
Noting that findbugs takes quite a lot more time. Javadoc and jar are the
> other two slow ones.
>
> RAT is fast. But it has very poor error messages, so we wouldn't want a new
> contributor trying to figure out what is going on without our help.
>
There is a larger philosophical issue here: is there a point of Jenkins
precommit testing? Why not just make `mvn install` run everything that
Jenkins does? For that matter, why don't committers just push directly to
master? Wouldn't that make everyone's life easier?
I'd argue that's not true.
1. Developer productivity -- Jenkins should run many more checks than
developers do. Especially time-, resource-, or setup- intensive tasks.
2. Automated enforcement -- Jenkins is better at running the right commands
than we are.
3. Lower the barrier to entry -- individual developers need not have a
running Spark/Flink/Apex/Dataflow setup in order to contribute code.
4. Focus on the user -- someone checking out the code and using it for the
first time does not care whether the code style checks or has the right
licenses -- that should have been enforced by the Beam team before
committing.
We should be *very* choosy about what we enforce on every developer every
time they go to compile. I probably compile Beam 50x-100x a day. Literally,
the extra minutes you want to add here will cost me an hour daily.
I've listed the fraction of commits I think will break one of the following
if that property is not tested:
* compiling (100%)
* tests (100%)
* checkstyle (90%)
* javadoc (30%)
* findbugs (5%)
* rat (1%)
So you can see where I stand and why. I'm sorry that 1/20 PRs has Apache
RAT catch a licensing issue or Findbugs catch a threading issue -- you can
always get a larger set of the precommit checks using -Prelease, though of
course the integration tests and runnableonservice tests may catch more
issues still. But I want my developer minutes back for the 95%+ of the rest.
Dan
>
>
> On Fri, Feb 10, 2017 at 07:14 Aviem Zur <av...@gmail.com> wrote:
>
> > Opened JIRA ticket: https://issues.apache.org/jira/browse/BEAM-1457
> >
> > On Fri, Feb 10, 2017 at 4:54 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
> > wrote:
> >
> > > Yeah. Agree. Time extend is not huge and it's worth to add it in verify
> > > phase.
> > >
> > > Regards
> > > JB
> > >
> > > On Feb 10, 2017, 10:13, at 10:13, Aviem Zur <av...@gmail.com>
> wrote:
> > > >This goes back to the original discussion in this thread - reduce the
> > > >amount of things pull requesters should know and keep the maven
> command
> > > >in
> > > >the PR checklist as: 'mvn clean verify'.
> > > >
> > > >So if rat and findbugs do not take that long to run I think they
> should
> > > >be
> > > >run by 'mvn clean verify'
> > > >
> > > >I ran a quick test on my laptop to see how much time they add to the
> > > >build
> > > >(of the entire project):
> > > >
> > > >'mvn clean install -DskipTests' => Total time: 03:51 min
> > > >'mvn clean install apache-rat:check findbugs:check -DskipTests' =>
> > > >Total
> > > >time: 05:29 min (Added 01:38 min)
> > > >'mvn clean install' => Total time: 09:37 min
> > > >'mvn clean install apache-rat:check findbugs:check' => Total time:
> > > >11:13
> > > >min (Added 01:36 min)
> > > >
> > > >Are these times reasonable enough to add rat and findbugs to the
> > > >default
> > > >build?
> > > >
> > > >On Fri, Feb 10, 2017 at 1:55 PM Jean-Baptiste Onofré <jb@nanthrax.net
> >
> > > >wrote:
> > > >
> > > >> Hi
> > > >>
> > > >> We discussed about that at the beginning of the project. We agreed
> to
> > > >> execute rat and findbugs in a specific profile to reduce the build
> > > >time for
> > > >> dev.
> > > >>
> > > >> That's why I do mvn clean install -Prelease before submitting a PR
> > > >and
> > > >> just clean install when I'm developing.
> > > >>
> > > >> No problem to change that.
> > > >>
> > > >> Regards
> > > >> JB
> > > >>
> > > >> On Feb 10, 2017, 07:51, at 07:51, Aviem Zur <av...@gmail.com>
> > > >wrote:
> > > >> >Can we consider adding rat-plugin and findbugs to the default
> verify
> > > >> >phase?
> > > >> >Currently they only run when the `release` profile is enabled.
> > > >> >
> > > >> >On Thu, Jan 26, 2017 at 11:42 AM Aljoscha Krettek
> > > ><al...@apache.org>
> > > >> >wrote:
> > > >> >
> > > >> >> +1 to what Dan said
> > > >> >>
> > > >> >> On Wed, 25 Jan 2017 at 21:40 Kenneth Knowles
> > > ><kl...@google.com.invalid>
> > > >> >> wrote:
> > > >> >>
> > > >> >> > +1
> > > >> >> >
> > > >> >> > On Jan 25, 2017 11:15, "Jean-Baptiste Onofré" <jb@nanthrax.net
> >
> > > >> >wrote:
> > > >> >> >
> > > >> >> > > +1
> > > >> >> > >
> > > >> >> > > It sounds good to me.
> > > >> >> > >
> > > >> >> > > Thanks Dan !
> > > >> >> > >
> > > >> >> > > Regards
> > > >> >> > > JB
> > > >> >> > >
> > > >> >> > > On Jan 25, 2017, 19:39, at 19:39, Dan Halperin
> > > >> >> > <dh...@google.com.INVALID>
> > > >> >> > > wrote:
> > > >> >> > > >Here is my summary of the threads:
> > > >> >> > > >
> > > >> >> > > >Overwhelming agreement:
> > > >> >> > > >
> > > >> >> > > >- rename `release` to something more appropriate.
> > > >> >> > > >- add `checkstyle` to the default build (it's basically a
> > > >> >compile
> > > >> >> > > >error)
> > > >> >> > > >- add more information to contributor guide
> > > >> >> > > >
> > > >> >> > > >Reasonable agreement
> > > >> >> > > >
> > > >> >> > > >- don't update the github instructions to make passing `mvn
> > > >> >verify
> > > >> >> > > >-P<all
> > > >> >> > > >checks>` mandatory. Maybe add a hint that this is a good
> > > >proxy
> > > >> >for
> > > >> >> what
> > > >> >> > > >Jenkins will run.
> > > >> >> > > >
> > > >> >> > > >Unresolved:
> > > >> >> > > >
> > > >> >> > > >- whether all checks should be in `mvn verify`
> > > >> >> > > >- whether `mvn test` is useful for most workflows
> > > >> >> > > >
> > > >> >> > > >I'll propose to proceed with the overwhelmingly agreed-upon
> > > >> >changes,
> > > >> >> > > >and as
> > > >> >> > > >we see increasingly many new contributors re-evaluate the
> > > >> >remaining
> > > >> >> > > >issues.
> > > >> >> > > >
> > > >> >> > > >Thanks,
> > > >> >> > > >Dan
> > > >> >> > > >
> > > >> >> > > >On Tue, Jan 24, 2017 at 12:51 PM, Jean-Baptiste Onofré
> > > >> >> > > ><jb...@nanthrax.net>
> > > >> >> > > >wrote:
> > > >> >> > > >
> > > >> >> > > >> +1 to at least update the contribution guide and improve
> > > >the
> > > >> >profile
> > > >> >> > > >name.
> > > >> >> > > >>
> > > >> >> > > >> Regards
> > > >> >> > > >> JB
> > > >> >> > > >>
> > > >> >> > > >>
> > > >> >> > > >> On 01/24/2017 09:49 PM, Kenneth Knowles wrote:
> > > >> >> > > >>
> > > >> >> > > >>> My impression is that we don't have consensus on whether
> > > >all
> > > >> >checks
> > > >> >> > > >or
> > > >> >> > > >>> minimal checks should be the default, or whether we can
> > > >have
> > > >> >both
> > > >> >> > > >via `mvn
> > > >> >> > > >>> test` and `mvn verify`.
> > > >> >> > > >>>
> > > >> >> > > >>> But that doesn't prevent us from giving -P release a
> > > >better
> > > >> >name
> > > >> >> and
> > > >> >> > > >>> mentioning it in the dev guide and in some manner in our
> > > >PR
> > > >> >> > > >template.
> > > >> >> > > >>>
> > > >> >> > > >>> Right now we are living with the combination of the bad
> > > >> >aspects -
> > > >> >> > > >default
> > > >> >> > > >>> is not thorough but not actually fast and a thorough
> check
> > > >is
> > > >> >> > > >>> undocumented.
> > > >> >> > > >>>
> > > >> >> > > >>> On Tue, Jan 24, 2017 at 2:22 AM, Ismaël Mejía
> > > >> ><ie...@gmail.com>
> > > >> >> > > >wrote:
> > > >> >> > > >>>
> > > >> >> > > >>> I just wanted to know if we have achieved some consensus
> > > >> >about
> > > >> >> this,
> > > >> >> > > >I
> > > >> >> > > >>>> just
> > > >> >> > > >>>> saw this PR that reminded me about this discussion.
> > > >> >> > > >>>>
> > > >> >> > > >>>> https://github.com/apache/beam/pull/1829
> > > >> >> > > >>>>
> > > >> >> > > >>>> It is important that we mention the existing profiles
> > > >(and
> > > >> >the
> > > >> >> > > >intended
> > > >> >> > > >>>> checks) in the contribution guide (e.g. -Prelease (or
> > > >> >-Pall-checks
> > > >> >> > > >>>> triggers
> > > >> >> > > >>>> these validations).
> > > >> >> > > >>>>
> > > >> >> > > >>>> I can add this to the guide if you like once we define
> > > >the
> > > >> >checks
> > > >> >> > > >per
> > > >> >> > > >>>> stage/profile.
> > > >> >> > > >>>>
> > > >> >> > > >>>> Ismaël
> > > >> >> > > >>>>
> > > >> >> > > >>>>
> > > >> >> > > >>>> On Wed, Jan 11, 2017 at 8:12 AM, Aviem Zur
> > > >> ><av...@gmail.com>
> > > >> >> > > >wrote:
> > > >> >> > > >>>>
> > > >> >> > > >>>> I agree with Dan and Lukasz.
> > > >> >> > > >>>>> Developers should not be expected to know beforehand
> > > >which
> > > >> >> > > >specific
> > > >> >> > > >>>>> profiles to run.
> > > >> >> > > >>>>> The phase specified in the PR instructions (`verify`)
> > > >> >should run
> > > >> >> > > >all the
> > > >> >> > > >>>>> relevant verifications and be the "slower" build, while
> > > >a
> > > >> >> > > >preceding
> > > >> >> > > >>>>> lifecycle, such as `test`, should run the "faster"
> > > >> >verifications.
> > > >> >> > > >>>>>
> > > >> >> > > >>>>> Aviem.
> > > >> >> > > >>>>>
> > > >> >> > > >>>>> On Mon, Jan 9, 2017 at 7:57 PM Robert Bradshaw
> > > >> >> > > >>>>>
> > > >> >> > > >>>> <robertwb@google.com.invalid
> > > >> >> > > >>>>
> > > >> >> > > >>>>>
> > > >> >> > > >>>>>> wrote:
> > > >> >> > > >>>>>
> > > >> >> > > >>>>> On Mon, Jan 9, 2017 at 3:49 AM, Aljoscha Krettek
> > > >> >> > > ><al...@apache.org>
> > > >> >> > > >>>>>> wrote:
> > > >> >> > > >>>>>>
> > > >> >> > > >>>>>>> I also usually prefer "mvn verify" to to the expected
> > > >> >thing but
> > > >> >> > > >I see
> > > >> >> > > >>>>>>>
> > > >> >> > > >>>>>> that
> > > >> >> > > >>>>>>
> > > >> >> > > >>>>>>> quick iteration times are key.
> > > >> >> > > >>>>>>>
> > > >> >> > > >>>>>>
> > > >> >> > > >>>>>> I see
> > > >> >> > > >>>>>> https://maven.apache.org/guides/introduction/
> > > >> >> > > >>>>>>
> > > >> >> > > >>>>> introduction-to-the-lifecycle.html
> > > >> >> > > >>>>>
> > > >> >> > > >>>>>>
> > > >> >> > > >>>>>> verify - run any checks on results of integration
> > > >> >tests to
> > > >> >> > > >ensure
> > > >> >> > > >>>>>> quality criteria are met
> > > >> >> > > >>>>>>
> > > >> >> > > >>>>>> Of course our integration tests are long enough that
> we
> > > >> >> shouldn't
> > > >> >> > > >be
> > > >> >> > > >>>>>> putting all of them here, but I too would expect
> > > >> >checkstyle.
> > > >> >> > > >>>>>>
> > > >> >> > > >>>>>> Perhaps we could introduce a verify-fast or somesuch
> > > >for
> > > >> >fast
> > > >> >> > > >(but
> > > >> >> > > >>>>>> lower coverage) turnaround time. I would expect "mvn
> > > >> >verify
> > > >> >> test"
> > > >> >> > > >to
> > > >> >> > > >>>>>> pass before submitting a PR, and would want to run
> that
> > > >> >before
> > > >> >> > > >asking
> > > >> >> > > >>>>>> others to look at it. I think this should be our
> > > >criteria
> > > >> >(i.e.
> > > >> >> > > >what
> > > >> >> > > >>>>>> will a new but maven-savvy user run before pushing
> > > >their
> > > >> >code).
> > > >> >> > > >>>>>>
> > > >> >> > > >>>>>> As long as the pre-commit hooks still check everything
> > > >I'm
> > > >> >ok
> > > >> >> > > >with
> > > >> >> > > >>>>>>>
> > > >> >> > > >>>>>> making
> > > >> >> > > >>>>>
> > > >> >> > > >>>>>> the default a little more lightweight.
> > > >> >> > > >>>>>>>
> > > >> >> > > >>>>>>
> > > >> >> > > >>>>>> The fact that our pre-commit hooks take a long time to
> > > >run
> > > >> >does
> > > >> >> > > >change
> > > >> >> > > >>>>>> things. Nothing more annoying than seeing that your PR
> > > >> >failed 3
> > > >> >> > > >hours
> > > >> >> > > >>>>>> later because you had some trailing whitespace...
> > > >> >> > > >>>>>>
> > > >> >> > > >>>>>> On Thu, 5 Jan 2017 at 21:49 Lukasz Cwik
> > > >> >> > > ><lc...@google.com.invalid>
> > > >> >> > > >>>>>>>
> > > >> >> > > >>>>>> wrote:
> > > >> >> > > >>>>>>
> > > >> >> > > >>>>>>>
> > > >> >> > > >>>>>>> I was hoping that the default mvn verify would be the
> > > >> >slow
> > > >> >> build
> > > >> >> > > >>>>>>>>
> > > >> >> > > >>>>>>> and a
> > > >> >> > > >>>>
> > > >> >> > > >>>>> profile could be enabled that would skip checks to make
> > > >> >things
> > > >> >> > > >>>>>>>>
> > > >> >> > > >>>>>>> faster
> > > >> >> > > >>>>
> > > >> >> > > >>>>> for
> > > >> >> > > >>>>>>
> > > >> >> > > >>>>>>> regular contributors. This way a person doesn't need
> > > >to
> > > >> >have
> > > >> >> > > >>>>>>>>
> > > >> >> > > >>>>>>> detailed
> > > >> >> > > >>>>
> > > >> >> > > >>>>> knowledge of all our profiles and what they do
> > > >(typically
> > > >> >mvn
> > > >> >> > > >>>>>>>>
> > > >> >> > > >>>>>>> verify)
> > > >> >> > > >>>>
> > > >> >> > > >>>>> will
> > > >> >> > > >>>>>>
> > > >> >> > > >>>>>>> do the right thing most of the time.
> > > >> >> > > >>>>>>>>
> > > >> >> > > >>>>>>>> On Thu, Jan 5, 2017 at 9:30 AM, Dan Halperin
> > > >> >> > > >>>>>>>>
> > > >> >> > > >>>>>>> <dh...@google.com.invalid>
> > > >> >> > > >>>>>>
> > > >> >> > > >>>>>>> wrote:
> > > >> >> > > >>>>>>>>
> > > >> >> > > >>>>>>>> On Thu, Jan 5, 2017 at 9:28 AM, Jesse Anderson <
> > > >> >> > > >>>>>>>>>
> > > >> >> > > >>>>>>>> jesse@smokinghand.com
> > > >> >> > > >>>>>
> > > >> >> > > >>>>>>
> > > >> >> > > >>>>>>> wrote:
> > > >> >> > > >>>>>>>>>
> > > >> >> > > >>>>>>>>> @dan are you saying that mvn verify isn't doing
> > > >> >checkstyle
> > > >> >> > > >>>>>>>>>>
> > > >> >> > > >>>>>>>>> anymore?
> > > >> >> > > >>>>>
> > > >> >> > > >>>>>>
> > > >> >> > > >>>>>>>>>
> > > >> >> > > >>>>>>>>> `mvn verify` alone should not be running
> checkstyle,
> > > >if
> > > >> >> > > >modules
> > > >> >> > > >>>>>>>>>
> > > >> >> > > >>>>>>>> are
> > > >> >> > > >>>>
> > > >> >> > > >>>>> configured correctly.
> > > >> >> > > >>>>>>>>>
> > > >> >> > > >>>>>>>>>
> > > >> >> > > >>>>>>>>> Some of
> > > >> >> > > >>>>>>>>>> the checkstyles are still running for a few
> > > >modules.
> > > >> >Also,
> > > >> >> > > >the
> > > >> >> > > >>>>>>>>>>
> > > >> >> > > >>>>>>>>> contribution
> > > >> >> > > >>>>>>>>>
> > > >> >> > > >>>>>>>>>> docs will need to change.
> > > >> >> > > >>>>>>>>>>
> > > >> >> > > >>>>>>>>>
> > > >> >> > > >>>>>>>>>
> > > >> >> > > >>>>>>>>> Yes. The PR includes discussion of these other
> > > >needed
> > > >> >> changes,
> > > >> >> > > >>>>>>>>> unfortunately one PR can't change two repositories.
> > > >> >> > > >>>>>>>>>
> > > >> >> > > >>>>>>>>> Please continue the discussion on the PR, then I
> > > >will
> > > >> >> > > >summarize it
> > > >> >> > > >>>>>>>>>
> > > >> >> > > >>>>>>>> back
> > > >> >> > > >>>>>>
> > > >> >> > > >>>>>>> into the dev thread.
> > > >> >> > > >>>>>>>>>
> > > >> >> > > >>>>>>>>> Thanks,
> > > >> >> > > >>>>>>>>> Dan
> > > >> >> > > >>>>>>>>>
> > > >> >> > > >>>>>>>>>
> > > >> >> > > >>>>>>>>> They say to run mvn verify before commits.
> > > >> >> > > >>>>>>>>>>
> > > >> >> > > >>>>>>>>>> On Thu, Jan 5, 2017 at 9:25 AM Dan Halperin
> > > >> >> > > >>>>>>>>>>
> > > >> >> > > >>>>>>>>> <dhalperi@google.com.invalid
> > > >> >> > > >>>>>>>>
> > > >> >> > > >>>>>>>>>
> > > >> >> > > >>>>>>>>>> wrote:
> > > >> >> > > >>>>>>>>>>
> > > >> >> > > >>>>>>>>>> Several folks seem to have been confused after
> > > >> >BEAM-246,
> > > >> >> > > >where
> > > >> >> > > >>>>>>>>>>>
> > > >> >> > > >>>>>>>>>> we
> > > >> >> > > >>>>>
> > > >> >> > > >>>>>> moved
> > > >> >> > > >>>>>>>>
> > > >> >> > > >>>>>>>>> the
> > > >> >> > > >>>>>>>>>>
> > > >> >> > > >>>>>>>>>>> "slow things" into the release profile. I've
> > > >started
> > > >> >a
> > > >> >> > > >>>>>>>>>>>
> > > >> >> > > >>>>>>>>>> discussion
> > > >> >> > > >>>>>
> > > >> >> > > >>>>>> with
> > > >> >> > > >>>>>>>>
> > > >> >> > > >>>>>>>>> https://github.com/apache/beam/pull/1740 to see if
> > > >> >there are
> > > >> >> > > >>>>>>>>>>>
> > > >> >> > > >>>>>>>>>> things
> > > >> >> > > >>>>>>
> > > >> >> > > >>>>>>> we
> > > >> >> > > >>>>>>>>
> > > >> >> > > >>>>>>>>> can
> > > >> >> > > >>>>>>>>>>
> > > >> >> > > >>>>>>>>>>> do to fill these gaps.
> > > >> >> > > >>>>>>>>>>>
> > > >> >> > > >>>>>>>>>>> Would love folks to chime in with opinions.
> > > >> >> > > >>>>>>>>>>>
> > > >> >> > > >>>>>>>>>>> Dan
> > > >> >> > > >>>>>>>>>>>
> > > >> >> > > >>>>>>>>>>> On Wed, Jan 4, 2017 at 1:34 PM, Jesse Anderson <
> > > >> >> > > >>>>>>>>>>>
> > > >> >> > > >>>>>>>>>> jesse@smokinghand.com>
> > > >> >> > > >>>>>>>>
> > > >> >> > > >>>>>>>>> wrote:
> > > >> >> > > >>>>>>>>>>>
> > > >> >> > > >>>>>>>>>>> @Eugene, yes that failed on the checkstyle.
> > > >> >> > > >>>>>>>>>>>>
> > > >> >> > > >>>>>>>>>>>> On Wed, Jan 4, 2017 at 1:27 PM Eugene Kirpichov
> > > >> >> > > >>>>>>>>>>>> <ki...@google.com.invalid> wrote:
> > > >> >> > > >>>>>>>>>>>>
> > > >> >> > > >>>>>>>>>>>> Try just -Prelease.
> > > >> >> > > >>>>>>>>>>>>> On Wed, Jan 4, 2017 at 1:21 PM Jesse Anderson <
> > > >> >> > > >>>>>>>>>>>>>
> > > >> >> > > >>>>>>>>>>>> jesse@smokinghand.com
> > > >> >> > > >>>>>>>>>
> > > >> >> > > >>>>>>>>>>
> > > >> >> > > >>>>>>>>>>> wrote:
> > > >> >> > > >>>>>>>>>>>>>
> > > >> >> > > >>>>>>>>>>>>> Fails because I don't have a secret key.
> > > >> >> > > >>>>>>>>>>>>>>
> > > >> >> > > >>>>>>>>>>>>>> On Wed, Jan 4, 2017 at 1:03 PM Jean-Baptiste
> > > >> >Onofré <
> > > >> >> > > >>>>>>>>>>>>>>
> > > >> >> > > >>>>>>>>>>>>> jb@nanthrax.net
> > > >> >> > > >>>>>>>>>>
> > > >> >> > > >>>>>>>>>>>
> > > >> >> > > >>>>>>>>>>>> wrote:
> > > >> >> > > >>>>>>>>>>>>>>
> > > >> >> > > >>>>>>>>>>>>>> Hi Jesse,
> > > >> >> > > >>>>>>>>>>>>>>>
> > > >> >> > > >>>>>>>>>>>>>>> Could you try the same with:
> > > >> >> > > >>>>>>>>>>>>>>>
> > > >> >> > > >>>>>>>>>>>>>>> mvn verify -Prelease,apache-release
> > > >> >> > > >>>>>>>>>>>>>>>
> > > >> >> > > >>>>>>>>>>>>>>> ?
> > > >> >> > > >>>>>>>>>>>>>>>
> > > >> >> > > >>>>>>>>>>>>>>> Regards
> > > >> >> > > >>>>>>>>>>>>>>> JB
> > > >> >> > > >>>>>>>>>>>>>>>
> > > >> >> > > >>>>>>>>>>>>>>> On 01/04/2017 09:53 PM, Jesse Anderson wrote:
> > > >> >> > > >>>>>>>>>>>>>>>
> > > >> >> > > >>>>>>>>>>>>>>>> For some reason, running "mvn verify" isn't
> > > >> >running
> > > >> >> > > >>>>>>>>>>>>>>>>
> > > >> >> > > >>>>>>>>>>>>>>> checkstyle
> > > >> >> > > >>>>>>>>>
> > > >> >> > > >>>>>>>>>> on
> > > >> >> > > >>>>>>>>>>
> > > >> >> > > >>>>>>>>>>> everything. I had checkstyle errors in
> > > >> >> > > >>>>>>>>>>>>>>>>
> > > >> >> > > >>>>>>>>>>>>>>> beam-sdks-java-core
> > > >> >> > > >>>>>>
> > > >> >> > > >>>>>>> that
> > > >> >> > > >>>>>>>>>
> > > >> >> > > >>>>>>>>>> weren't
> > > >> >> > > >>>>>>>>>>>>>
> > > >> >> > > >>>>>>>>>>>>>> being found.
> > > >> >> > > >>>>>>>>>>>>>>>>
> > > >> >> > > >>>>>>>>>>>>>>>> I thought this was due to the extra
> > > >parameters.
> > > >> >I
> > > >> >> > > >>>>>>>>>>>>>>>>
> > > >> >> > > >>>>>>>>>>>>>>> reran
> > > >> >> > > >>>>>
> > > >> >> > > >>>>>> with
> > > >> >> > > >>>>>>>>
> > > >> >> > > >>>>>>>>> the
> > > >> >> > > >>>>>>>>>>
> > > >> >> > > >>>>>>>>>>> plain
> > > >> >> > > >>>>>>>>>>>>>
> > > >> >> > > >>>>>>>>>>>>>> "mvn
> > > >> >> > > >>>>>>>>>>>>>>>
> > > >> >> > > >>>>>>>>>>>>>>>> verify" and it still didn't find them. From
> > > >the
> > > >> >> > > >>>>>>>>>>>>>>>>
> > > >> >> > > >>>>>>>>>>>>>>> output,
> > > >> >> > > >>>>>
> > > >> >> > > >>>>>> it
> > > >> >> > > >>>>>>
> > > >> >> > > >>>>>>> doesn't
> > > >> >> > > >>>>>>>>>>>
> > > >> >> > > >>>>>>>>>>>> look
> > > >> >> > > >>>>>>>>>>>>>
> > > >> >> > > >>>>>>>>>>>>>> like they're being run at all.
> > > >> >> > > >>>>>>>>>>>>>>>>
> > > >> >> > > >>>>>>>>>>>>>>>> Thanks,
> > > >> >> > > >>>>>>>>>>>>>>>>
> > > >> >> > > >>>>>>>>>>>>>>>> Jesse
> > > >> >> > > >>>>>>>>>>>>>>>>
> > > >> >> > > >>>>>>>>>>>>>>>>
> > > >> >> > > >>>>>>>>>>>>>>> --
> > > >> >> > > >>>>>>>>>>>>>>> Jean-Baptiste Onofré
> > > >> >> > > >>>>>>>>>>>>>>> jbonofre@apache.org
> > > >> >> > > >>>>>>>>>>>>>>> http://blog.nanthrax.net
> > > >> >> > > >>>>>>>>>>>>>>> Talend - http://www.talend.com
> > > >> >> > > >>>>>>>>>>>>>>>
> > > >> >> > > >>>>>>>>>>>>>>>
> > > >> >> > > >>>>>>>>>>>>>>
> > > >> >> > > >>>>>>>>>>>>>
> > > >> >> > > >>>>>>>>>>>>
> > > >> >> > > >>>>>>>>>>>
> > > >> >> > > >>>>>>>>>>
> > > >> >> > > >>>>>>>>>
> > > >> >> > > >>>>>>>>
> > > >> >> > > >>>>>>
> > > >> >> > > >>>>>
> > > >> >> > > >>>>
> > > >> >> > > >>>
> > > >> >> > > >> --
> > > >> >> > > >> Jean-Baptiste Onofré
> > > >> >> > > >> jbonofre@apache.org
> > > >> >> > > >> http://blog.nanthrax.net
> > > >> >> > > >> Talend - http://www.talend.com
> > > >> >> > > >>
> > > >> >> > >
> > > >> >> >
> > > >> >>
> > > >>
> > >
> >
>
Re: Better developer instructions for using Maven?
Posted by Kenneth Knowles <kl...@google.com.INVALID>.
On Feb 10, 2017 07:36, "Dan Halperin" <dh...@google.com.invalid> wrote:
Before we added checkstyle it was under a minute. Now it's over five?
That's awful IMO
Checkstyle didn't cause all that, did it?
Noting that findbugs takes quite a lot more time. Javadoc and jar are the
other two slow ones.
RAT is fast. But it has very poor error messages, so we wouldn't want a new
contributor trying to figure out what is going on without our help.
On Fri, Feb 10, 2017 at 07:14 Aviem Zur <av...@gmail.com> wrote:
> Opened JIRA ticket: https://issues.apache.org/jira/browse/BEAM-1457
>
> On Fri, Feb 10, 2017 at 4:54 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
>
> > Yeah. Agree. Time extend is not huge and it's worth to add it in verify
> > phase.
> >
> > Regards
> > JB
> >
> > On Feb 10, 2017, 10:13, at 10:13, Aviem Zur <av...@gmail.com> wrote:
> > >This goes back to the original discussion in this thread - reduce the
> > >amount of things pull requesters should know and keep the maven command
> > >in
> > >the PR checklist as: 'mvn clean verify'.
> > >
> > >So if rat and findbugs do not take that long to run I think they should
> > >be
> > >run by 'mvn clean verify'
> > >
> > >I ran a quick test on my laptop to see how much time they add to the
> > >build
> > >(of the entire project):
> > >
> > >'mvn clean install -DskipTests' => Total time: 03:51 min
> > >'mvn clean install apache-rat:check findbugs:check -DskipTests' =>
> > >Total
> > >time: 05:29 min (Added 01:38 min)
> > >'mvn clean install' => Total time: 09:37 min
> > >'mvn clean install apache-rat:check findbugs:check' => Total time:
> > >11:13
> > >min (Added 01:36 min)
> > >
> > >Are these times reasonable enough to add rat and findbugs to the
> > >default
> > >build?
> > >
> > >On Fri, Feb 10, 2017 at 1:55 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
> > >wrote:
> > >
> > >> Hi
> > >>
> > >> We discussed about that at the beginning of the project. We agreed to
> > >> execute rat and findbugs in a specific profile to reduce the build
> > >time for
> > >> dev.
> > >>
> > >> That's why I do mvn clean install -Prelease before submitting a PR
> > >and
> > >> just clean install when I'm developing.
> > >>
> > >> No problem to change that.
> > >>
> > >> Regards
> > >> JB
> > >>
> > >> On Feb 10, 2017, 07:51, at 07:51, Aviem Zur <av...@gmail.com>
> > >wrote:
> > >> >Can we consider adding rat-plugin and findbugs to the default verify
> > >> >phase?
> > >> >Currently they only run when the `release` profile is enabled.
> > >> >
> > >> >On Thu, Jan 26, 2017 at 11:42 AM Aljoscha Krettek
> > ><al...@apache.org>
> > >> >wrote:
> > >> >
> > >> >> +1 to what Dan said
> > >> >>
> > >> >> On Wed, 25 Jan 2017 at 21:40 Kenneth Knowles
> > ><kl...@google.com.invalid>
> > >> >> wrote:
> > >> >>
> > >> >> > +1
> > >> >> >
> > >> >> > On Jan 25, 2017 11:15, "Jean-Baptiste Onofré" <jb...@nanthrax.net>
> > >> >wrote:
> > >> >> >
> > >> >> > > +1
> > >> >> > >
> > >> >> > > It sounds good to me.
> > >> >> > >
> > >> >> > > Thanks Dan !
> > >> >> > >
> > >> >> > > Regards
> > >> >> > > JB
> > >> >> > >
> > >> >> > > On Jan 25, 2017, 19:39, at 19:39, Dan Halperin
> > >> >> > <dh...@google.com.INVALID>
> > >> >> > > wrote:
> > >> >> > > >Here is my summary of the threads:
> > >> >> > > >
> > >> >> > > >Overwhelming agreement:
> > >> >> > > >
> > >> >> > > >- rename `release` to something more appropriate.
> > >> >> > > >- add `checkstyle` to the default build (it's basically a
> > >> >compile
> > >> >> > > >error)
> > >> >> > > >- add more information to contributor guide
> > >> >> > > >
> > >> >> > > >Reasonable agreement
> > >> >> > > >
> > >> >> > > >- don't update the github instructions to make passing `mvn
> > >> >verify
> > >> >> > > >-P<all
> > >> >> > > >checks>` mandatory. Maybe add a hint that this is a good
> > >proxy
> > >> >for
> > >> >> what
> > >> >> > > >Jenkins will run.
> > >> >> > > >
> > >> >> > > >Unresolved:
> > >> >> > > >
> > >> >> > > >- whether all checks should be in `mvn verify`
> > >> >> > > >- whether `mvn test` is useful for most workflows
> > >> >> > > >
> > >> >> > > >I'll propose to proceed with the overwhelmingly agreed-upon
> > >> >changes,
> > >> >> > > >and as
> > >> >> > > >we see increasingly many new contributors re-evaluate the
> > >> >remaining
> > >> >> > > >issues.
> > >> >> > > >
> > >> >> > > >Thanks,
> > >> >> > > >Dan
> > >> >> > > >
> > >> >> > > >On Tue, Jan 24, 2017 at 12:51 PM, Jean-Baptiste Onofré
> > >> >> > > ><jb...@nanthrax.net>
> > >> >> > > >wrote:
> > >> >> > > >
> > >> >> > > >> +1 to at least update the contribution guide and improve
> > >the
> > >> >profile
> > >> >> > > >name.
> > >> >> > > >>
> > >> >> > > >> Regards
> > >> >> > > >> JB
> > >> >> > > >>
> > >> >> > > >>
> > >> >> > > >> On 01/24/2017 09:49 PM, Kenneth Knowles wrote:
> > >> >> > > >>
> > >> >> > > >>> My impression is that we don't have consensus on whether
> > >all
> > >> >checks
> > >> >> > > >or
> > >> >> > > >>> minimal checks should be the default, or whether we can
> > >have
> > >> >both
> > >> >> > > >via `mvn
> > >> >> > > >>> test` and `mvn verify`.
> > >> >> > > >>>
> > >> >> > > >>> But that doesn't prevent us from giving -P release a
> > >better
> > >> >name
> > >> >> and
> > >> >> > > >>> mentioning it in the dev guide and in some manner in our
> > >PR
> > >> >> > > >template.
> > >> >> > > >>>
> > >> >> > > >>> Right now we are living with the combination of the bad
> > >> >aspects -
> > >> >> > > >default
> > >> >> > > >>> is not thorough but not actually fast and a thorough check
> > >is
> > >> >> > > >>> undocumented.
> > >> >> > > >>>
> > >> >> > > >>> On Tue, Jan 24, 2017 at 2:22 AM, Ismaël Mejía
> > >> ><ie...@gmail.com>
> > >> >> > > >wrote:
> > >> >> > > >>>
> > >> >> > > >>> I just wanted to know if we have achieved some consensus
> > >> >about
> > >> >> this,
> > >> >> > > >I
> > >> >> > > >>>> just
> > >> >> > > >>>> saw this PR that reminded me about this discussion.
> > >> >> > > >>>>
> > >> >> > > >>>> https://github.com/apache/beam/pull/1829
> > >> >> > > >>>>
> > >> >> > > >>>> It is important that we mention the existing profiles
> > >(and
> > >> >the
> > >> >> > > >intended
> > >> >> > > >>>> checks) in the contribution guide (e.g. -Prelease (or
> > >> >-Pall-checks
> > >> >> > > >>>> triggers
> > >> >> > > >>>> these validations).
> > >> >> > > >>>>
> > >> >> > > >>>> I can add this to the guide if you like once we define
> > >the
> > >> >checks
> > >> >> > > >per
> > >> >> > > >>>> stage/profile.
> > >> >> > > >>>>
> > >> >> > > >>>> Ismaël
> > >> >> > > >>>>
> > >> >> > > >>>>
> > >> >> > > >>>> On Wed, Jan 11, 2017 at 8:12 AM, Aviem Zur
> > >> ><av...@gmail.com>
> > >> >> > > >wrote:
> > >> >> > > >>>>
> > >> >> > > >>>> I agree with Dan and Lukasz.
> > >> >> > > >>>>> Developers should not be expected to know beforehand
> > >which
> > >> >> > > >specific
> > >> >> > > >>>>> profiles to run.
> > >> >> > > >>>>> The phase specified in the PR instructions (`verify`)
> > >> >should run
> > >> >> > > >all the
> > >> >> > > >>>>> relevant verifications and be the "slower" build, while
> > >a
> > >> >> > > >preceding
> > >> >> > > >>>>> lifecycle, such as `test`, should run the "faster"
> > >> >verifications.
> > >> >> > > >>>>>
> > >> >> > > >>>>> Aviem.
> > >> >> > > >>>>>
> > >> >> > > >>>>> On Mon, Jan 9, 2017 at 7:57 PM Robert Bradshaw
> > >> >> > > >>>>>
> > >> >> > > >>>> <robertwb@google.com.invalid
> > >> >> > > >>>>
> > >> >> > > >>>>>
> > >> >> > > >>>>>> wrote:
> > >> >> > > >>>>>
> > >> >> > > >>>>> On Mon, Jan 9, 2017 at 3:49 AM, Aljoscha Krettek
> > >> >> > > ><al...@apache.org>
> > >> >> > > >>>>>> wrote:
> > >> >> > > >>>>>>
> > >> >> > > >>>>>>> I also usually prefer "mvn verify" to to the expected
> > >> >thing but
> > >> >> > > >I see
> > >> >> > > >>>>>>>
> > >> >> > > >>>>>> that
> > >> >> > > >>>>>>
> > >> >> > > >>>>>>> quick iteration times are key.
> > >> >> > > >>>>>>>
> > >> >> > > >>>>>>
> > >> >> > > >>>>>> I see
> > >> >> > > >>>>>> https://maven.apache.org/guides/introduction/
> > >> >> > > >>>>>>
> > >> >> > > >>>>> introduction-to-the-lifecycle.html
> > >> >> > > >>>>>
> > >> >> > > >>>>>>
> > >> >> > > >>>>>> verify - run any checks on results of integration
> > >> >tests to
> > >> >> > > >ensure
> > >> >> > > >>>>>> quality criteria are met
> > >> >> > > >>>>>>
> > >> >> > > >>>>>> Of course our integration tests are long enough that we
> > >> >> shouldn't
> > >> >> > > >be
> > >> >> > > >>>>>> putting all of them here, but I too would expect
> > >> >checkstyle.
> > >> >> > > >>>>>>
> > >> >> > > >>>>>> Perhaps we could introduce a verify-fast or somesuch
> > >for
> > >> >fast
> > >> >> > > >(but
> > >> >> > > >>>>>> lower coverage) turnaround time. I would expect "mvn
> > >> >verify
> > >> >> test"
> > >> >> > > >to
> > >> >> > > >>>>>> pass before submitting a PR, and would want to run that
> > >> >before
> > >> >> > > >asking
> > >> >> > > >>>>>> others to look at it. I think this should be our
> > >criteria
> > >> >(i.e.
> > >> >> > > >what
> > >> >> > > >>>>>> will a new but maven-savvy user run before pushing
> > >their
> > >> >code).
> > >> >> > > >>>>>>
> > >> >> > > >>>>>> As long as the pre-commit hooks still check everything
> > >I'm
> > >> >ok
> > >> >> > > >with
> > >> >> > > >>>>>>>
> > >> >> > > >>>>>> making
> > >> >> > > >>>>>
> > >> >> > > >>>>>> the default a little more lightweight.
> > >> >> > > >>>>>>>
> > >> >> > > >>>>>>
> > >> >> > > >>>>>> The fact that our pre-commit hooks take a long time to
> > >run
> > >> >does
> > >> >> > > >change
> > >> >> > > >>>>>> things. Nothing more annoying than seeing that your PR
> > >> >failed 3
> > >> >> > > >hours
> > >> >> > > >>>>>> later because you had some trailing whitespace...
> > >> >> > > >>>>>>
> > >> >> > > >>>>>> On Thu, 5 Jan 2017 at 21:49 Lukasz Cwik
> > >> >> > > ><lc...@google.com.invalid>
> > >> >> > > >>>>>>>
> > >> >> > > >>>>>> wrote:
> > >> >> > > >>>>>>
> > >> >> > > >>>>>>>
> > >> >> > > >>>>>>> I was hoping that the default mvn verify would be the
> > >> >slow
> > >> >> build
> > >> >> > > >>>>>>>>
> > >> >> > > >>>>>>> and a
> > >> >> > > >>>>
> > >> >> > > >>>>> profile could be enabled that would skip checks to make
> > >> >things
> > >> >> > > >>>>>>>>
> > >> >> > > >>>>>>> faster
> > >> >> > > >>>>
> > >> >> > > >>>>> for
> > >> >> > > >>>>>>
> > >> >> > > >>>>>>> regular contributors. This way a person doesn't need
> > >to
> > >> >have
> > >> >> > > >>>>>>>>
> > >> >> > > >>>>>>> detailed
> > >> >> > > >>>>
> > >> >> > > >>>>> knowledge of all our profiles and what they do
> > >(typically
> > >> >mvn
> > >> >> > > >>>>>>>>
> > >> >> > > >>>>>>> verify)
> > >> >> > > >>>>
> > >> >> > > >>>>> will
> > >> >> > > >>>>>>
> > >> >> > > >>>>>>> do the right thing most of the time.
> > >> >> > > >>>>>>>>
> > >> >> > > >>>>>>>> On Thu, Jan 5, 2017 at 9:30 AM, Dan Halperin
> > >> >> > > >>>>>>>>
> > >> >> > > >>>>>>> <dh...@google.com.invalid>
> > >> >> > > >>>>>>
> > >> >> > > >>>>>>> wrote:
> > >> >> > > >>>>>>>>
> > >> >> > > >>>>>>>> On Thu, Jan 5, 2017 at 9:28 AM, Jesse Anderson <
> > >> >> > > >>>>>>>>>
> > >> >> > > >>>>>>>> jesse@smokinghand.com
> > >> >> > > >>>>>
> > >> >> > > >>>>>>
> > >> >> > > >>>>>>> wrote:
> > >> >> > > >>>>>>>>>
> > >> >> > > >>>>>>>>> @dan are you saying that mvn verify isn't doing
> > >> >checkstyle
> > >> >> > > >>>>>>>>>>
> > >> >> > > >>>>>>>>> anymore?
> > >> >> > > >>>>>
> > >> >> > > >>>>>>
> > >> >> > > >>>>>>>>>
> > >> >> > > >>>>>>>>> `mvn verify` alone should not be running checkstyle,
> > >if
> > >> >> > > >modules
> > >> >> > > >>>>>>>>>
> > >> >> > > >>>>>>>> are
> > >> >> > > >>>>
> > >> >> > > >>>>> configured correctly.
> > >> >> > > >>>>>>>>>
> > >> >> > > >>>>>>>>>
> > >> >> > > >>>>>>>>> Some of
> > >> >> > > >>>>>>>>>> the checkstyles are still running for a few
> > >modules.
> > >> >Also,
> > >> >> > > >the
> > >> >> > > >>>>>>>>>>
> > >> >> > > >>>>>>>>> contribution
> > >> >> > > >>>>>>>>>
> > >> >> > > >>>>>>>>>> docs will need to change.
> > >> >> > > >>>>>>>>>>
> > >> >> > > >>>>>>>>>
> > >> >> > > >>>>>>>>>
> > >> >> > > >>>>>>>>> Yes. The PR includes discussion of these other
> > >needed
> > >> >> changes,
> > >> >> > > >>>>>>>>> unfortunately one PR can't change two repositories.
> > >> >> > > >>>>>>>>>
> > >> >> > > >>>>>>>>> Please continue the discussion on the PR, then I
> > >will
> > >> >> > > >summarize it
> > >> >> > > >>>>>>>>>
> > >> >> > > >>>>>>>> back
> > >> >> > > >>>>>>
> > >> >> > > >>>>>>> into the dev thread.
> > >> >> > > >>>>>>>>>
> > >> >> > > >>>>>>>>> Thanks,
> > >> >> > > >>>>>>>>> Dan
> > >> >> > > >>>>>>>>>
> > >> >> > > >>>>>>>>>
> > >> >> > > >>>>>>>>> They say to run mvn verify before commits.
> > >> >> > > >>>>>>>>>>
> > >> >> > > >>>>>>>>>> On Thu, Jan 5, 2017 at 9:25 AM Dan Halperin
> > >> >> > > >>>>>>>>>>
> > >> >> > > >>>>>>>>> <dhalperi@google.com.invalid
> > >> >> > > >>>>>>>>
> > >> >> > > >>>>>>>>>
> > >> >> > > >>>>>>>>>> wrote:
> > >> >> > > >>>>>>>>>>
> > >> >> > > >>>>>>>>>> Several folks seem to have been confused after
> > >> >BEAM-246,
> > >> >> > > >where
> > >> >> > > >>>>>>>>>>>
> > >> >> > > >>>>>>>>>> we
> > >> >> > > >>>>>
> > >> >> > > >>>>>> moved
> > >> >> > > >>>>>>>>
> > >> >> > > >>>>>>>>> the
> > >> >> > > >>>>>>>>>>
> > >> >> > > >>>>>>>>>>> "slow things" into the release profile. I've
> > >started
> > >> >a
> > >> >> > > >>>>>>>>>>>
> > >> >> > > >>>>>>>>>> discussion
> > >> >> > > >>>>>
> > >> >> > > >>>>>> with
> > >> >> > > >>>>>>>>
> > >> >> > > >>>>>>>>> https://github.com/apache/beam/pull/1740 to see if
> > >> >there are
> > >> >> > > >>>>>>>>>>>
> > >> >> > > >>>>>>>>>> things
> > >> >> > > >>>>>>
> > >> >> > > >>>>>>> we
> > >> >> > > >>>>>>>>
> > >> >> > > >>>>>>>>> can
> > >> >> > > >>>>>>>>>>
> > >> >> > > >>>>>>>>>>> do to fill these gaps.
> > >> >> > > >>>>>>>>>>>
> > >> >> > > >>>>>>>>>>> Would love folks to chime in with opinions.
> > >> >> > > >>>>>>>>>>>
> > >> >> > > >>>>>>>>>>> Dan
> > >> >> > > >>>>>>>>>>>
> > >> >> > > >>>>>>>>>>> On Wed, Jan 4, 2017 at 1:34 PM, Jesse Anderson <
> > >> >> > > >>>>>>>>>>>
> > >> >> > > >>>>>>>>>> jesse@smokinghand.com>
> > >> >> > > >>>>>>>>
> > >> >> > > >>>>>>>>> wrote:
> > >> >> > > >>>>>>>>>>>
> > >> >> > > >>>>>>>>>>> @Eugene, yes that failed on the checkstyle.
> > >> >> > > >>>>>>>>>>>>
> > >> >> > > >>>>>>>>>>>> On Wed, Jan 4, 2017 at 1:27 PM Eugene Kirpichov
> > >> >> > > >>>>>>>>>>>> <ki...@google.com.invalid> wrote:
> > >> >> > > >>>>>>>>>>>>
> > >> >> > > >>>>>>>>>>>> Try just -Prelease.
> > >> >> > > >>>>>>>>>>>>> On Wed, Jan 4, 2017 at 1:21 PM Jesse Anderson <
> > >> >> > > >>>>>>>>>>>>>
> > >> >> > > >>>>>>>>>>>> jesse@smokinghand.com
> > >> >> > > >>>>>>>>>
> > >> >> > > >>>>>>>>>>
> > >> >> > > >>>>>>>>>>> wrote:
> > >> >> > > >>>>>>>>>>>>>
> > >> >> > > >>>>>>>>>>>>> Fails because I don't have a secret key.
> > >> >> > > >>>>>>>>>>>>>>
> > >> >> > > >>>>>>>>>>>>>> On Wed, Jan 4, 2017 at 1:03 PM Jean-Baptiste
> > >> >Onofré <
> > >> >> > > >>>>>>>>>>>>>>
> > >> >> > > >>>>>>>>>>>>> jb@nanthrax.net
> > >> >> > > >>>>>>>>>>
> > >> >> > > >>>>>>>>>>>
> > >> >> > > >>>>>>>>>>>> wrote:
> > >> >> > > >>>>>>>>>>>>>>
> > >> >> > > >>>>>>>>>>>>>> Hi Jesse,
> > >> >> > > >>>>>>>>>>>>>>>
> > >> >> > > >>>>>>>>>>>>>>> Could you try the same with:
> > >> >> > > >>>>>>>>>>>>>>>
> > >> >> > > >>>>>>>>>>>>>>> mvn verify -Prelease,apache-release
> > >> >> > > >>>>>>>>>>>>>>>
> > >> >> > > >>>>>>>>>>>>>>> ?
> > >> >> > > >>>>>>>>>>>>>>>
> > >> >> > > >>>>>>>>>>>>>>> Regards
> > >> >> > > >>>>>>>>>>>>>>> JB
> > >> >> > > >>>>>>>>>>>>>>>
> > >> >> > > >>>>>>>>>>>>>>> On 01/04/2017 09:53 PM, Jesse Anderson wrote:
> > >> >> > > >>>>>>>>>>>>>>>
> > >> >> > > >>>>>>>>>>>>>>>> For some reason, running "mvn verify" isn't
> > >> >running
> > >> >> > > >>>>>>>>>>>>>>>>
> > >> >> > > >>>>>>>>>>>>>>> checkstyle
> > >> >> > > >>>>>>>>>
> > >> >> > > >>>>>>>>>> on
> > >> >> > > >>>>>>>>>>
> > >> >> > > >>>>>>>>>>> everything. I had checkstyle errors in
> > >> >> > > >>>>>>>>>>>>>>>>
> > >> >> > > >>>>>>>>>>>>>>> beam-sdks-java-core
> > >> >> > > >>>>>>
> > >> >> > > >>>>>>> that
> > >> >> > > >>>>>>>>>
> > >> >> > > >>>>>>>>>> weren't
> > >> >> > > >>>>>>>>>>>>>
> > >> >> > > >>>>>>>>>>>>>> being found.
> > >> >> > > >>>>>>>>>>>>>>>>
> > >> >> > > >>>>>>>>>>>>>>>> I thought this was due to the extra
> > >parameters.
> > >> >I
> > >> >> > > >>>>>>>>>>>>>>>>
> > >> >> > > >>>>>>>>>>>>>>> reran
> > >> >> > > >>>>>
> > >> >> > > >>>>>> with
> > >> >> > > >>>>>>>>
> > >> >> > > >>>>>>>>> the
> > >> >> > > >>>>>>>>>>
> > >> >> > > >>>>>>>>>>> plain
> > >> >> > > >>>>>>>>>>>>>
> > >> >> > > >>>>>>>>>>>>>> "mvn
> > >> >> > > >>>>>>>>>>>>>>>
> > >> >> > > >>>>>>>>>>>>>>>> verify" and it still didn't find them. From
> > >the
> > >> >> > > >>>>>>>>>>>>>>>>
> > >> >> > > >>>>>>>>>>>>>>> output,
> > >> >> > > >>>>>
> > >> >> > > >>>>>> it
> > >> >> > > >>>>>>
> > >> >> > > >>>>>>> doesn't
> > >> >> > > >>>>>>>>>>>
> > >> >> > > >>>>>>>>>>>> look
> > >> >> > > >>>>>>>>>>>>>
> > >> >> > > >>>>>>>>>>>>>> like they're being run at all.
> > >> >> > > >>>>>>>>>>>>>>>>
> > >> >> > > >>>>>>>>>>>>>>>> Thanks,
> > >> >> > > >>>>>>>>>>>>>>>>
> > >> >> > > >>>>>>>>>>>>>>>> Jesse
> > >> >> > > >>>>>>>>>>>>>>>>
> > >> >> > > >>>>>>>>>>>>>>>>
> > >> >> > > >>>>>>>>>>>>>>> --
> > >> >> > > >>>>>>>>>>>>>>> Jean-Baptiste Onofré
> > >> >> > > >>>>>>>>>>>>>>> jbonofre@apache.org
> > >> >> > > >>>>>>>>>>>>>>> http://blog.nanthrax.net
> > >> >> > > >>>>>>>>>>>>>>> Talend - http://www.talend.com
> > >> >> > > >>>>>>>>>>>>>>>
> > >> >> > > >>>>>>>>>>>>>>>
> > >> >> > > >>>>>>>>>>>>>>
> > >> >> > > >>>>>>>>>>>>>
> > >> >> > > >>>>>>>>>>>>
> > >> >> > > >>>>>>>>>>>
> > >> >> > > >>>>>>>>>>
> > >> >> > > >>>>>>>>>
> > >> >> > > >>>>>>>>
> > >> >> > > >>>>>>
> > >> >> > > >>>>>
> > >> >> > > >>>>
> > >> >> > > >>>
> > >> >> > > >> --
> > >> >> > > >> Jean-Baptiste Onofré
> > >> >> > > >> jbonofre@apache.org
> > >> >> > > >> http://blog.nanthrax.net
> > >> >> > > >> Talend - http://www.talend.com
> > >> >> > > >>
> > >> >> > >
> > >> >> >
> > >> >>
> > >>
> >
>
Re: Better developer instructions for using Maven?
Posted by Dan Halperin <dh...@google.com.INVALID>.
Before we added checkstyle it was under a minute. Now it's over five?
That's awful IMO
On Fri, Feb 10, 2017 at 07:14 Aviem Zur <av...@gmail.com> wrote:
> Opened JIRA ticket: https://issues.apache.org/jira/browse/BEAM-1457
>
> On Fri, Feb 10, 2017 at 4:54 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
>
> > Yeah. Agree. Time extend is not huge and it's worth to add it in verify
> > phase.
> >
> > Regards
> > JB
> >
> > On Feb 10, 2017, 10:13, at 10:13, Aviem Zur <av...@gmail.com> wrote:
> > >This goes back to the original discussion in this thread - reduce the
> > >amount of things pull requesters should know and keep the maven command
> > >in
> > >the PR checklist as: 'mvn clean verify'.
> > >
> > >So if rat and findbugs do not take that long to run I think they should
> > >be
> > >run by 'mvn clean verify'
> > >
> > >I ran a quick test on my laptop to see how much time they add to the
> > >build
> > >(of the entire project):
> > >
> > >'mvn clean install -DskipTests' => Total time: 03:51 min
> > >'mvn clean install apache-rat:check findbugs:check -DskipTests' =>
> > >Total
> > >time: 05:29 min (Added 01:38 min)
> > >'mvn clean install' => Total time: 09:37 min
> > >'mvn clean install apache-rat:check findbugs:check' => Total time:
> > >11:13
> > >min (Added 01:36 min)
> > >
> > >Are these times reasonable enough to add rat and findbugs to the
> > >default
> > >build?
> > >
> > >On Fri, Feb 10, 2017 at 1:55 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
> > >wrote:
> > >
> > >> Hi
> > >>
> > >> We discussed about that at the beginning of the project. We agreed to
> > >> execute rat and findbugs in a specific profile to reduce the build
> > >time for
> > >> dev.
> > >>
> > >> That's why I do mvn clean install -Prelease before submitting a PR
> > >and
> > >> just clean install when I'm developing.
> > >>
> > >> No problem to change that.
> > >>
> > >> Regards
> > >> JB
> > >>
> > >> On Feb 10, 2017, 07:51, at 07:51, Aviem Zur <av...@gmail.com>
> > >wrote:
> > >> >Can we consider adding rat-plugin and findbugs to the default verify
> > >> >phase?
> > >> >Currently they only run when the `release` profile is enabled.
> > >> >
> > >> >On Thu, Jan 26, 2017 at 11:42 AM Aljoscha Krettek
> > ><al...@apache.org>
> > >> >wrote:
> > >> >
> > >> >> +1 to what Dan said
> > >> >>
> > >> >> On Wed, 25 Jan 2017 at 21:40 Kenneth Knowles
> > ><kl...@google.com.invalid>
> > >> >> wrote:
> > >> >>
> > >> >> > +1
> > >> >> >
> > >> >> > On Jan 25, 2017 11:15, "Jean-Baptiste Onofré" <jb...@nanthrax.net>
> > >> >wrote:
> > >> >> >
> > >> >> > > +1
> > >> >> > >
> > >> >> > > It sounds good to me.
> > >> >> > >
> > >> >> > > Thanks Dan !
> > >> >> > >
> > >> >> > > Regards
> > >> >> > > JB
> > >> >> > >
> > >> >> > > On Jan 25, 2017, 19:39, at 19:39, Dan Halperin
> > >> >> > <dh...@google.com.INVALID>
> > >> >> > > wrote:
> > >> >> > > >Here is my summary of the threads:
> > >> >> > > >
> > >> >> > > >Overwhelming agreement:
> > >> >> > > >
> > >> >> > > >- rename `release` to something more appropriate.
> > >> >> > > >- add `checkstyle` to the default build (it's basically a
> > >> >compile
> > >> >> > > >error)
> > >> >> > > >- add more information to contributor guide
> > >> >> > > >
> > >> >> > > >Reasonable agreement
> > >> >> > > >
> > >> >> > > >- don't update the github instructions to make passing `mvn
> > >> >verify
> > >> >> > > >-P<all
> > >> >> > > >checks>` mandatory. Maybe add a hint that this is a good
> > >proxy
> > >> >for
> > >> >> what
> > >> >> > > >Jenkins will run.
> > >> >> > > >
> > >> >> > > >Unresolved:
> > >> >> > > >
> > >> >> > > >- whether all checks should be in `mvn verify`
> > >> >> > > >- whether `mvn test` is useful for most workflows
> > >> >> > > >
> > >> >> > > >I'll propose to proceed with the overwhelmingly agreed-upon
> > >> >changes,
> > >> >> > > >and as
> > >> >> > > >we see increasingly many new contributors re-evaluate the
> > >> >remaining
> > >> >> > > >issues.
> > >> >> > > >
> > >> >> > > >Thanks,
> > >> >> > > >Dan
> > >> >> > > >
> > >> >> > > >On Tue, Jan 24, 2017 at 12:51 PM, Jean-Baptiste Onofré
> > >> >> > > ><jb...@nanthrax.net>
> > >> >> > > >wrote:
> > >> >> > > >
> > >> >> > > >> +1 to at least update the contribution guide and improve
> > >the
> > >> >profile
> > >> >> > > >name.
> > >> >> > > >>
> > >> >> > > >> Regards
> > >> >> > > >> JB
> > >> >> > > >>
> > >> >> > > >>
> > >> >> > > >> On 01/24/2017 09:49 PM, Kenneth Knowles wrote:
> > >> >> > > >>
> > >> >> > > >>> My impression is that we don't have consensus on whether
> > >all
> > >> >checks
> > >> >> > > >or
> > >> >> > > >>> minimal checks should be the default, or whether we can
> > >have
> > >> >both
> > >> >> > > >via `mvn
> > >> >> > > >>> test` and `mvn verify`.
> > >> >> > > >>>
> > >> >> > > >>> But that doesn't prevent us from giving -P release a
> > >better
> > >> >name
> > >> >> and
> > >> >> > > >>> mentioning it in the dev guide and in some manner in our
> > >PR
> > >> >> > > >template.
> > >> >> > > >>>
> > >> >> > > >>> Right now we are living with the combination of the bad
> > >> >aspects -
> > >> >> > > >default
> > >> >> > > >>> is not thorough but not actually fast and a thorough check
> > >is
> > >> >> > > >>> undocumented.
> > >> >> > > >>>
> > >> >> > > >>> On Tue, Jan 24, 2017 at 2:22 AM, Ismaël Mejía
> > >> ><ie...@gmail.com>
> > >> >> > > >wrote:
> > >> >> > > >>>
> > >> >> > > >>> I just wanted to know if we have achieved some consensus
> > >> >about
> > >> >> this,
> > >> >> > > >I
> > >> >> > > >>>> just
> > >> >> > > >>>> saw this PR that reminded me about this discussion.
> > >> >> > > >>>>
> > >> >> > > >>>> https://github.com/apache/beam/pull/1829
> > >> >> > > >>>>
> > >> >> > > >>>> It is important that we mention the existing profiles
> > >(and
> > >> >the
> > >> >> > > >intended
> > >> >> > > >>>> checks) in the contribution guide (e.g. -Prelease (or
> > >> >-Pall-checks
> > >> >> > > >>>> triggers
> > >> >> > > >>>> these validations).
> > >> >> > > >>>>
> > >> >> > > >>>> I can add this to the guide if you like once we define
> > >the
> > >> >checks
> > >> >> > > >per
> > >> >> > > >>>> stage/profile.
> > >> >> > > >>>>
> > >> >> > > >>>> Ismaël
> > >> >> > > >>>>
> > >> >> > > >>>>
> > >> >> > > >>>> On Wed, Jan 11, 2017 at 8:12 AM, Aviem Zur
> > >> ><av...@gmail.com>
> > >> >> > > >wrote:
> > >> >> > > >>>>
> > >> >> > > >>>> I agree with Dan and Lukasz.
> > >> >> > > >>>>> Developers should not be expected to know beforehand
> > >which
> > >> >> > > >specific
> > >> >> > > >>>>> profiles to run.
> > >> >> > > >>>>> The phase specified in the PR instructions (`verify`)
> > >> >should run
> > >> >> > > >all the
> > >> >> > > >>>>> relevant verifications and be the "slower" build, while
> > >a
> > >> >> > > >preceding
> > >> >> > > >>>>> lifecycle, such as `test`, should run the "faster"
> > >> >verifications.
> > >> >> > > >>>>>
> > >> >> > > >>>>> Aviem.
> > >> >> > > >>>>>
> > >> >> > > >>>>> On Mon, Jan 9, 2017 at 7:57 PM Robert Bradshaw
> > >> >> > > >>>>>
> > >> >> > > >>>> <robertwb@google.com.invalid
> > >> >> > > >>>>
> > >> >> > > >>>>>
> > >> >> > > >>>>>> wrote:
> > >> >> > > >>>>>
> > >> >> > > >>>>> On Mon, Jan 9, 2017 at 3:49 AM, Aljoscha Krettek
> > >> >> > > ><al...@apache.org>
> > >> >> > > >>>>>> wrote:
> > >> >> > > >>>>>>
> > >> >> > > >>>>>>> I also usually prefer "mvn verify" to to the expected
> > >> >thing but
> > >> >> > > >I see
> > >> >> > > >>>>>>>
> > >> >> > > >>>>>> that
> > >> >> > > >>>>>>
> > >> >> > > >>>>>>> quick iteration times are key.
> > >> >> > > >>>>>>>
> > >> >> > > >>>>>>
> > >> >> > > >>>>>> I see
> > >> >> > > >>>>>> https://maven.apache.org/guides/introduction/
> > >> >> > > >>>>>>
> > >> >> > > >>>>> introduction-to-the-lifecycle.html
> > >> >> > > >>>>>
> > >> >> > > >>>>>>
> > >> >> > > >>>>>> verify - run any checks on results of integration
> > >> >tests to
> > >> >> > > >ensure
> > >> >> > > >>>>>> quality criteria are met
> > >> >> > > >>>>>>
> > >> >> > > >>>>>> Of course our integration tests are long enough that we
> > >> >> shouldn't
> > >> >> > > >be
> > >> >> > > >>>>>> putting all of them here, but I too would expect
> > >> >checkstyle.
> > >> >> > > >>>>>>
> > >> >> > > >>>>>> Perhaps we could introduce a verify-fast or somesuch
> > >for
> > >> >fast
> > >> >> > > >(but
> > >> >> > > >>>>>> lower coverage) turnaround time. I would expect "mvn
> > >> >verify
> > >> >> test"
> > >> >> > > >to
> > >> >> > > >>>>>> pass before submitting a PR, and would want to run that
> > >> >before
> > >> >> > > >asking
> > >> >> > > >>>>>> others to look at it. I think this should be our
> > >criteria
> > >> >(i.e.
> > >> >> > > >what
> > >> >> > > >>>>>> will a new but maven-savvy user run before pushing
> > >their
> > >> >code).
> > >> >> > > >>>>>>
> > >> >> > > >>>>>> As long as the pre-commit hooks still check everything
> > >I'm
> > >> >ok
> > >> >> > > >with
> > >> >> > > >>>>>>>
> > >> >> > > >>>>>> making
> > >> >> > > >>>>>
> > >> >> > > >>>>>> the default a little more lightweight.
> > >> >> > > >>>>>>>
> > >> >> > > >>>>>>
> > >> >> > > >>>>>> The fact that our pre-commit hooks take a long time to
> > >run
> > >> >does
> > >> >> > > >change
> > >> >> > > >>>>>> things. Nothing more annoying than seeing that your PR
> > >> >failed 3
> > >> >> > > >hours
> > >> >> > > >>>>>> later because you had some trailing whitespace...
> > >> >> > > >>>>>>
> > >> >> > > >>>>>> On Thu, 5 Jan 2017 at 21:49 Lukasz Cwik
> > >> >> > > ><lc...@google.com.invalid>
> > >> >> > > >>>>>>>
> > >> >> > > >>>>>> wrote:
> > >> >> > > >>>>>>
> > >> >> > > >>>>>>>
> > >> >> > > >>>>>>> I was hoping that the default mvn verify would be the
> > >> >slow
> > >> >> build
> > >> >> > > >>>>>>>>
> > >> >> > > >>>>>>> and a
> > >> >> > > >>>>
> > >> >> > > >>>>> profile could be enabled that would skip checks to make
> > >> >things
> > >> >> > > >>>>>>>>
> > >> >> > > >>>>>>> faster
> > >> >> > > >>>>
> > >> >> > > >>>>> for
> > >> >> > > >>>>>>
> > >> >> > > >>>>>>> regular contributors. This way a person doesn't need
> > >to
> > >> >have
> > >> >> > > >>>>>>>>
> > >> >> > > >>>>>>> detailed
> > >> >> > > >>>>
> > >> >> > > >>>>> knowledge of all our profiles and what they do
> > >(typically
> > >> >mvn
> > >> >> > > >>>>>>>>
> > >> >> > > >>>>>>> verify)
> > >> >> > > >>>>
> > >> >> > > >>>>> will
> > >> >> > > >>>>>>
> > >> >> > > >>>>>>> do the right thing most of the time.
> > >> >> > > >>>>>>>>
> > >> >> > > >>>>>>>> On Thu, Jan 5, 2017 at 9:30 AM, Dan Halperin
> > >> >> > > >>>>>>>>
> > >> >> > > >>>>>>> <dh...@google.com.invalid>
> > >> >> > > >>>>>>
> > >> >> > > >>>>>>> wrote:
> > >> >> > > >>>>>>>>
> > >> >> > > >>>>>>>> On Thu, Jan 5, 2017 at 9:28 AM, Jesse Anderson <
> > >> >> > > >>>>>>>>>
> > >> >> > > >>>>>>>> jesse@smokinghand.com
> > >> >> > > >>>>>
> > >> >> > > >>>>>>
> > >> >> > > >>>>>>> wrote:
> > >> >> > > >>>>>>>>>
> > >> >> > > >>>>>>>>> @dan are you saying that mvn verify isn't doing
> > >> >checkstyle
> > >> >> > > >>>>>>>>>>
> > >> >> > > >>>>>>>>> anymore?
> > >> >> > > >>>>>
> > >> >> > > >>>>>>
> > >> >> > > >>>>>>>>>
> > >> >> > > >>>>>>>>> `mvn verify` alone should not be running checkstyle,
> > >if
> > >> >> > > >modules
> > >> >> > > >>>>>>>>>
> > >> >> > > >>>>>>>> are
> > >> >> > > >>>>
> > >> >> > > >>>>> configured correctly.
> > >> >> > > >>>>>>>>>
> > >> >> > > >>>>>>>>>
> > >> >> > > >>>>>>>>> Some of
> > >> >> > > >>>>>>>>>> the checkstyles are still running for a few
> > >modules.
> > >> >Also,
> > >> >> > > >the
> > >> >> > > >>>>>>>>>>
> > >> >> > > >>>>>>>>> contribution
> > >> >> > > >>>>>>>>>
> > >> >> > > >>>>>>>>>> docs will need to change.
> > >> >> > > >>>>>>>>>>
> > >> >> > > >>>>>>>>>
> > >> >> > > >>>>>>>>>
> > >> >> > > >>>>>>>>> Yes. The PR includes discussion of these other
> > >needed
> > >> >> changes,
> > >> >> > > >>>>>>>>> unfortunately one PR can't change two repositories.
> > >> >> > > >>>>>>>>>
> > >> >> > > >>>>>>>>> Please continue the discussion on the PR, then I
> > >will
> > >> >> > > >summarize it
> > >> >> > > >>>>>>>>>
> > >> >> > > >>>>>>>> back
> > >> >> > > >>>>>>
> > >> >> > > >>>>>>> into the dev thread.
> > >> >> > > >>>>>>>>>
> > >> >> > > >>>>>>>>> Thanks,
> > >> >> > > >>>>>>>>> Dan
> > >> >> > > >>>>>>>>>
> > >> >> > > >>>>>>>>>
> > >> >> > > >>>>>>>>> They say to run mvn verify before commits.
> > >> >> > > >>>>>>>>>>
> > >> >> > > >>>>>>>>>> On Thu, Jan 5, 2017 at 9:25 AM Dan Halperin
> > >> >> > > >>>>>>>>>>
> > >> >> > > >>>>>>>>> <dhalperi@google.com.invalid
> > >> >> > > >>>>>>>>
> > >> >> > > >>>>>>>>>
> > >> >> > > >>>>>>>>>> wrote:
> > >> >> > > >>>>>>>>>>
> > >> >> > > >>>>>>>>>> Several folks seem to have been confused after
> > >> >BEAM-246,
> > >> >> > > >where
> > >> >> > > >>>>>>>>>>>
> > >> >> > > >>>>>>>>>> we
> > >> >> > > >>>>>
> > >> >> > > >>>>>> moved
> > >> >> > > >>>>>>>>
> > >> >> > > >>>>>>>>> the
> > >> >> > > >>>>>>>>>>
> > >> >> > > >>>>>>>>>>> "slow things" into the release profile. I've
> > >started
> > >> >a
> > >> >> > > >>>>>>>>>>>
> > >> >> > > >>>>>>>>>> discussion
> > >> >> > > >>>>>
> > >> >> > > >>>>>> with
> > >> >> > > >>>>>>>>
> > >> >> > > >>>>>>>>> https://github.com/apache/beam/pull/1740 to see if
> > >> >there are
> > >> >> > > >>>>>>>>>>>
> > >> >> > > >>>>>>>>>> things
> > >> >> > > >>>>>>
> > >> >> > > >>>>>>> we
> > >> >> > > >>>>>>>>
> > >> >> > > >>>>>>>>> can
> > >> >> > > >>>>>>>>>>
> > >> >> > > >>>>>>>>>>> do to fill these gaps.
> > >> >> > > >>>>>>>>>>>
> > >> >> > > >>>>>>>>>>> Would love folks to chime in with opinions.
> > >> >> > > >>>>>>>>>>>
> > >> >> > > >>>>>>>>>>> Dan
> > >> >> > > >>>>>>>>>>>
> > >> >> > > >>>>>>>>>>> On Wed, Jan 4, 2017 at 1:34 PM, Jesse Anderson <
> > >> >> > > >>>>>>>>>>>
> > >> >> > > >>>>>>>>>> jesse@smokinghand.com>
> > >> >> > > >>>>>>>>
> > >> >> > > >>>>>>>>> wrote:
> > >> >> > > >>>>>>>>>>>
> > >> >> > > >>>>>>>>>>> @Eugene, yes that failed on the checkstyle.
> > >> >> > > >>>>>>>>>>>>
> > >> >> > > >>>>>>>>>>>> On Wed, Jan 4, 2017 at 1:27 PM Eugene Kirpichov
> > >> >> > > >>>>>>>>>>>> <ki...@google.com.invalid> wrote:
> > >> >> > > >>>>>>>>>>>>
> > >> >> > > >>>>>>>>>>>> Try just -Prelease.
> > >> >> > > >>>>>>>>>>>>> On Wed, Jan 4, 2017 at 1:21 PM Jesse Anderson <
> > >> >> > > >>>>>>>>>>>>>
> > >> >> > > >>>>>>>>>>>> jesse@smokinghand.com
> > >> >> > > >>>>>>>>>
> > >> >> > > >>>>>>>>>>
> > >> >> > > >>>>>>>>>>> wrote:
> > >> >> > > >>>>>>>>>>>>>
> > >> >> > > >>>>>>>>>>>>> Fails because I don't have a secret key.
> > >> >> > > >>>>>>>>>>>>>>
> > >> >> > > >>>>>>>>>>>>>> On Wed, Jan 4, 2017 at 1:03 PM Jean-Baptiste
> > >> >Onofré <
> > >> >> > > >>>>>>>>>>>>>>
> > >> >> > > >>>>>>>>>>>>> jb@nanthrax.net
> > >> >> > > >>>>>>>>>>
> > >> >> > > >>>>>>>>>>>
> > >> >> > > >>>>>>>>>>>> wrote:
> > >> >> > > >>>>>>>>>>>>>>
> > >> >> > > >>>>>>>>>>>>>> Hi Jesse,
> > >> >> > > >>>>>>>>>>>>>>>
> > >> >> > > >>>>>>>>>>>>>>> Could you try the same with:
> > >> >> > > >>>>>>>>>>>>>>>
> > >> >> > > >>>>>>>>>>>>>>> mvn verify -Prelease,apache-release
> > >> >> > > >>>>>>>>>>>>>>>
> > >> >> > > >>>>>>>>>>>>>>> ?
> > >> >> > > >>>>>>>>>>>>>>>
> > >> >> > > >>>>>>>>>>>>>>> Regards
> > >> >> > > >>>>>>>>>>>>>>> JB
> > >> >> > > >>>>>>>>>>>>>>>
> > >> >> > > >>>>>>>>>>>>>>> On 01/04/2017 09:53 PM, Jesse Anderson wrote:
> > >> >> > > >>>>>>>>>>>>>>>
> > >> >> > > >>>>>>>>>>>>>>>> For some reason, running "mvn verify" isn't
> > >> >running
> > >> >> > > >>>>>>>>>>>>>>>>
> > >> >> > > >>>>>>>>>>>>>>> checkstyle
> > >> >> > > >>>>>>>>>
> > >> >> > > >>>>>>>>>> on
> > >> >> > > >>>>>>>>>>
> > >> >> > > >>>>>>>>>>> everything. I had checkstyle errors in
> > >> >> > > >>>>>>>>>>>>>>>>
> > >> >> > > >>>>>>>>>>>>>>> beam-sdks-java-core
> > >> >> > > >>>>>>
> > >> >> > > >>>>>>> that
> > >> >> > > >>>>>>>>>
> > >> >> > > >>>>>>>>>> weren't
> > >> >> > > >>>>>>>>>>>>>
> > >> >> > > >>>>>>>>>>>>>> being found.
> > >> >> > > >>>>>>>>>>>>>>>>
> > >> >> > > >>>>>>>>>>>>>>>> I thought this was due to the extra
> > >parameters.
> > >> >I
> > >> >> > > >>>>>>>>>>>>>>>>
> > >> >> > > >>>>>>>>>>>>>>> reran
> > >> >> > > >>>>>
> > >> >> > > >>>>>> with
> > >> >> > > >>>>>>>>
> > >> >> > > >>>>>>>>> the
> > >> >> > > >>>>>>>>>>
> > >> >> > > >>>>>>>>>>> plain
> > >> >> > > >>>>>>>>>>>>>
> > >> >> > > >>>>>>>>>>>>>> "mvn
> > >> >> > > >>>>>>>>>>>>>>>
> > >> >> > > >>>>>>>>>>>>>>>> verify" and it still didn't find them. From
> > >the
> > >> >> > > >>>>>>>>>>>>>>>>
> > >> >> > > >>>>>>>>>>>>>>> output,
> > >> >> > > >>>>>
> > >> >> > > >>>>>> it
> > >> >> > > >>>>>>
> > >> >> > > >>>>>>> doesn't
> > >> >> > > >>>>>>>>>>>
> > >> >> > > >>>>>>>>>>>> look
> > >> >> > > >>>>>>>>>>>>>
> > >> >> > > >>>>>>>>>>>>>> like they're being run at all.
> > >> >> > > >>>>>>>>>>>>>>>>
> > >> >> > > >>>>>>>>>>>>>>>> Thanks,
> > >> >> > > >>>>>>>>>>>>>>>>
> > >> >> > > >>>>>>>>>>>>>>>> Jesse
> > >> >> > > >>>>>>>>>>>>>>>>
> > >> >> > > >>>>>>>>>>>>>>>>
> > >> >> > > >>>>>>>>>>>>>>> --
> > >> >> > > >>>>>>>>>>>>>>> Jean-Baptiste Onofré
> > >> >> > > >>>>>>>>>>>>>>> jbonofre@apache.org
> > >> >> > > >>>>>>>>>>>>>>> http://blog.nanthrax.net
> > >> >> > > >>>>>>>>>>>>>>> Talend - http://www.talend.com
> > >> >> > > >>>>>>>>>>>>>>>
> > >> >> > > >>>>>>>>>>>>>>>
> > >> >> > > >>>>>>>>>>>>>>
> > >> >> > > >>>>>>>>>>>>>
> > >> >> > > >>>>>>>>>>>>
> > >> >> > > >>>>>>>>>>>
> > >> >> > > >>>>>>>>>>
> > >> >> > > >>>>>>>>>
> > >> >> > > >>>>>>>>
> > >> >> > > >>>>>>
> > >> >> > > >>>>>
> > >> >> > > >>>>
> > >> >> > > >>>
> > >> >> > > >> --
> > >> >> > > >> Jean-Baptiste Onofré
> > >> >> > > >> jbonofre@apache.org
> > >> >> > > >> http://blog.nanthrax.net
> > >> >> > > >> Talend - http://www.talend.com
> > >> >> > > >>
> > >> >> > >
> > >> >> >
> > >> >>
> > >>
> >
>
Re: Better developer instructions for using Maven?
Posted by Aviem Zur <av...@gmail.com>.
Opened JIRA ticket: https://issues.apache.org/jira/browse/BEAM-1457
On Fri, Feb 10, 2017 at 4:54 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:
> Yeah. Agree. Time extend is not huge and it's worth to add it in verify
> phase.
>
> Regards
> JB
>
> On Feb 10, 2017, 10:13, at 10:13, Aviem Zur <av...@gmail.com> wrote:
> >This goes back to the original discussion in this thread - reduce the
> >amount of things pull requesters should know and keep the maven command
> >in
> >the PR checklist as: 'mvn clean verify'.
> >
> >So if rat and findbugs do not take that long to run I think they should
> >be
> >run by 'mvn clean verify'
> >
> >I ran a quick test on my laptop to see how much time they add to the
> >build
> >(of the entire project):
> >
> >'mvn clean install -DskipTests' => Total time: 03:51 min
> >'mvn clean install apache-rat:check findbugs:check -DskipTests' =>
> >Total
> >time: 05:29 min (Added 01:38 min)
> >'mvn clean install' => Total time: 09:37 min
> >'mvn clean install apache-rat:check findbugs:check' => Total time:
> >11:13
> >min (Added 01:36 min)
> >
> >Are these times reasonable enough to add rat and findbugs to the
> >default
> >build?
> >
> >On Fri, Feb 10, 2017 at 1:55 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
> >wrote:
> >
> >> Hi
> >>
> >> We discussed about that at the beginning of the project. We agreed to
> >> execute rat and findbugs in a specific profile to reduce the build
> >time for
> >> dev.
> >>
> >> That's why I do mvn clean install -Prelease before submitting a PR
> >and
> >> just clean install when I'm developing.
> >>
> >> No problem to change that.
> >>
> >> Regards
> >> JB
> >>
> >> On Feb 10, 2017, 07:51, at 07:51, Aviem Zur <av...@gmail.com>
> >wrote:
> >> >Can we consider adding rat-plugin and findbugs to the default verify
> >> >phase?
> >> >Currently they only run when the `release` profile is enabled.
> >> >
> >> >On Thu, Jan 26, 2017 at 11:42 AM Aljoscha Krettek
> ><al...@apache.org>
> >> >wrote:
> >> >
> >> >> +1 to what Dan said
> >> >>
> >> >> On Wed, 25 Jan 2017 at 21:40 Kenneth Knowles
> ><kl...@google.com.invalid>
> >> >> wrote:
> >> >>
> >> >> > +1
> >> >> >
> >> >> > On Jan 25, 2017 11:15, "Jean-Baptiste Onofré" <jb...@nanthrax.net>
> >> >wrote:
> >> >> >
> >> >> > > +1
> >> >> > >
> >> >> > > It sounds good to me.
> >> >> > >
> >> >> > > Thanks Dan !
> >> >> > >
> >> >> > > Regards
> >> >> > > JB
> >> >> > >
> >> >> > > On Jan 25, 2017, 19:39, at 19:39, Dan Halperin
> >> >> > <dh...@google.com.INVALID>
> >> >> > > wrote:
> >> >> > > >Here is my summary of the threads:
> >> >> > > >
> >> >> > > >Overwhelming agreement:
> >> >> > > >
> >> >> > > >- rename `release` to something more appropriate.
> >> >> > > >- add `checkstyle` to the default build (it's basically a
> >> >compile
> >> >> > > >error)
> >> >> > > >- add more information to contributor guide
> >> >> > > >
> >> >> > > >Reasonable agreement
> >> >> > > >
> >> >> > > >- don't update the github instructions to make passing `mvn
> >> >verify
> >> >> > > >-P<all
> >> >> > > >checks>` mandatory. Maybe add a hint that this is a good
> >proxy
> >> >for
> >> >> what
> >> >> > > >Jenkins will run.
> >> >> > > >
> >> >> > > >Unresolved:
> >> >> > > >
> >> >> > > >- whether all checks should be in `mvn verify`
> >> >> > > >- whether `mvn test` is useful for most workflows
> >> >> > > >
> >> >> > > >I'll propose to proceed with the overwhelmingly agreed-upon
> >> >changes,
> >> >> > > >and as
> >> >> > > >we see increasingly many new contributors re-evaluate the
> >> >remaining
> >> >> > > >issues.
> >> >> > > >
> >> >> > > >Thanks,
> >> >> > > >Dan
> >> >> > > >
> >> >> > > >On Tue, Jan 24, 2017 at 12:51 PM, Jean-Baptiste Onofré
> >> >> > > ><jb...@nanthrax.net>
> >> >> > > >wrote:
> >> >> > > >
> >> >> > > >> +1 to at least update the contribution guide and improve
> >the
> >> >profile
> >> >> > > >name.
> >> >> > > >>
> >> >> > > >> Regards
> >> >> > > >> JB
> >> >> > > >>
> >> >> > > >>
> >> >> > > >> On 01/24/2017 09:49 PM, Kenneth Knowles wrote:
> >> >> > > >>
> >> >> > > >>> My impression is that we don't have consensus on whether
> >all
> >> >checks
> >> >> > > >or
> >> >> > > >>> minimal checks should be the default, or whether we can
> >have
> >> >both
> >> >> > > >via `mvn
> >> >> > > >>> test` and `mvn verify`.
> >> >> > > >>>
> >> >> > > >>> But that doesn't prevent us from giving -P release a
> >better
> >> >name
> >> >> and
> >> >> > > >>> mentioning it in the dev guide and in some manner in our
> >PR
> >> >> > > >template.
> >> >> > > >>>
> >> >> > > >>> Right now we are living with the combination of the bad
> >> >aspects -
> >> >> > > >default
> >> >> > > >>> is not thorough but not actually fast and a thorough check
> >is
> >> >> > > >>> undocumented.
> >> >> > > >>>
> >> >> > > >>> On Tue, Jan 24, 2017 at 2:22 AM, Ismaël Mejía
> >> ><ie...@gmail.com>
> >> >> > > >wrote:
> >> >> > > >>>
> >> >> > > >>> I just wanted to know if we have achieved some consensus
> >> >about
> >> >> this,
> >> >> > > >I
> >> >> > > >>>> just
> >> >> > > >>>> saw this PR that reminded me about this discussion.
> >> >> > > >>>>
> >> >> > > >>>> https://github.com/apache/beam/pull/1829
> >> >> > > >>>>
> >> >> > > >>>> It is important that we mention the existing profiles
> >(and
> >> >the
> >> >> > > >intended
> >> >> > > >>>> checks) in the contribution guide (e.g. -Prelease (or
> >> >-Pall-checks
> >> >> > > >>>> triggers
> >> >> > > >>>> these validations).
> >> >> > > >>>>
> >> >> > > >>>> I can add this to the guide if you like once we define
> >the
> >> >checks
> >> >> > > >per
> >> >> > > >>>> stage/profile.
> >> >> > > >>>>
> >> >> > > >>>> Ismaël
> >> >> > > >>>>
> >> >> > > >>>>
> >> >> > > >>>> On Wed, Jan 11, 2017 at 8:12 AM, Aviem Zur
> >> ><av...@gmail.com>
> >> >> > > >wrote:
> >> >> > > >>>>
> >> >> > > >>>> I agree with Dan and Lukasz.
> >> >> > > >>>>> Developers should not be expected to know beforehand
> >which
> >> >> > > >specific
> >> >> > > >>>>> profiles to run.
> >> >> > > >>>>> The phase specified in the PR instructions (`verify`)
> >> >should run
> >> >> > > >all the
> >> >> > > >>>>> relevant verifications and be the "slower" build, while
> >a
> >> >> > > >preceding
> >> >> > > >>>>> lifecycle, such as `test`, should run the "faster"
> >> >verifications.
> >> >> > > >>>>>
> >> >> > > >>>>> Aviem.
> >> >> > > >>>>>
> >> >> > > >>>>> On Mon, Jan 9, 2017 at 7:57 PM Robert Bradshaw
> >> >> > > >>>>>
> >> >> > > >>>> <robertwb@google.com.invalid
> >> >> > > >>>>
> >> >> > > >>>>>
> >> >> > > >>>>>> wrote:
> >> >> > > >>>>>
> >> >> > > >>>>> On Mon, Jan 9, 2017 at 3:49 AM, Aljoscha Krettek
> >> >> > > ><al...@apache.org>
> >> >> > > >>>>>> wrote:
> >> >> > > >>>>>>
> >> >> > > >>>>>>> I also usually prefer "mvn verify" to to the expected
> >> >thing but
> >> >> > > >I see
> >> >> > > >>>>>>>
> >> >> > > >>>>>> that
> >> >> > > >>>>>>
> >> >> > > >>>>>>> quick iteration times are key.
> >> >> > > >>>>>>>
> >> >> > > >>>>>>
> >> >> > > >>>>>> I see
> >> >> > > >>>>>> https://maven.apache.org/guides/introduction/
> >> >> > > >>>>>>
> >> >> > > >>>>> introduction-to-the-lifecycle.html
> >> >> > > >>>>>
> >> >> > > >>>>>>
> >> >> > > >>>>>> verify - run any checks on results of integration
> >> >tests to
> >> >> > > >ensure
> >> >> > > >>>>>> quality criteria are met
> >> >> > > >>>>>>
> >> >> > > >>>>>> Of course our integration tests are long enough that we
> >> >> shouldn't
> >> >> > > >be
> >> >> > > >>>>>> putting all of them here, but I too would expect
> >> >checkstyle.
> >> >> > > >>>>>>
> >> >> > > >>>>>> Perhaps we could introduce a verify-fast or somesuch
> >for
> >> >fast
> >> >> > > >(but
> >> >> > > >>>>>> lower coverage) turnaround time. I would expect "mvn
> >> >verify
> >> >> test"
> >> >> > > >to
> >> >> > > >>>>>> pass before submitting a PR, and would want to run that
> >> >before
> >> >> > > >asking
> >> >> > > >>>>>> others to look at it. I think this should be our
> >criteria
> >> >(i.e.
> >> >> > > >what
> >> >> > > >>>>>> will a new but maven-savvy user run before pushing
> >their
> >> >code).
> >> >> > > >>>>>>
> >> >> > > >>>>>> As long as the pre-commit hooks still check everything
> >I'm
> >> >ok
> >> >> > > >with
> >> >> > > >>>>>>>
> >> >> > > >>>>>> making
> >> >> > > >>>>>
> >> >> > > >>>>>> the default a little more lightweight.
> >> >> > > >>>>>>>
> >> >> > > >>>>>>
> >> >> > > >>>>>> The fact that our pre-commit hooks take a long time to
> >run
> >> >does
> >> >> > > >change
> >> >> > > >>>>>> things. Nothing more annoying than seeing that your PR
> >> >failed 3
> >> >> > > >hours
> >> >> > > >>>>>> later because you had some trailing whitespace...
> >> >> > > >>>>>>
> >> >> > > >>>>>> On Thu, 5 Jan 2017 at 21:49 Lukasz Cwik
> >> >> > > ><lc...@google.com.invalid>
> >> >> > > >>>>>>>
> >> >> > > >>>>>> wrote:
> >> >> > > >>>>>>
> >> >> > > >>>>>>>
> >> >> > > >>>>>>> I was hoping that the default mvn verify would be the
> >> >slow
> >> >> build
> >> >> > > >>>>>>>>
> >> >> > > >>>>>>> and a
> >> >> > > >>>>
> >> >> > > >>>>> profile could be enabled that would skip checks to make
> >> >things
> >> >> > > >>>>>>>>
> >> >> > > >>>>>>> faster
> >> >> > > >>>>
> >> >> > > >>>>> for
> >> >> > > >>>>>>
> >> >> > > >>>>>>> regular contributors. This way a person doesn't need
> >to
> >> >have
> >> >> > > >>>>>>>>
> >> >> > > >>>>>>> detailed
> >> >> > > >>>>
> >> >> > > >>>>> knowledge of all our profiles and what they do
> >(typically
> >> >mvn
> >> >> > > >>>>>>>>
> >> >> > > >>>>>>> verify)
> >> >> > > >>>>
> >> >> > > >>>>> will
> >> >> > > >>>>>>
> >> >> > > >>>>>>> do the right thing most of the time.
> >> >> > > >>>>>>>>
> >> >> > > >>>>>>>> On Thu, Jan 5, 2017 at 9:30 AM, Dan Halperin
> >> >> > > >>>>>>>>
> >> >> > > >>>>>>> <dh...@google.com.invalid>
> >> >> > > >>>>>>
> >> >> > > >>>>>>> wrote:
> >> >> > > >>>>>>>>
> >> >> > > >>>>>>>> On Thu, Jan 5, 2017 at 9:28 AM, Jesse Anderson <
> >> >> > > >>>>>>>>>
> >> >> > > >>>>>>>> jesse@smokinghand.com
> >> >> > > >>>>>
> >> >> > > >>>>>>
> >> >> > > >>>>>>> wrote:
> >> >> > > >>>>>>>>>
> >> >> > > >>>>>>>>> @dan are you saying that mvn verify isn't doing
> >> >checkstyle
> >> >> > > >>>>>>>>>>
> >> >> > > >>>>>>>>> anymore?
> >> >> > > >>>>>
> >> >> > > >>>>>>
> >> >> > > >>>>>>>>>
> >> >> > > >>>>>>>>> `mvn verify` alone should not be running checkstyle,
> >if
> >> >> > > >modules
> >> >> > > >>>>>>>>>
> >> >> > > >>>>>>>> are
> >> >> > > >>>>
> >> >> > > >>>>> configured correctly.
> >> >> > > >>>>>>>>>
> >> >> > > >>>>>>>>>
> >> >> > > >>>>>>>>> Some of
> >> >> > > >>>>>>>>>> the checkstyles are still running for a few
> >modules.
> >> >Also,
> >> >> > > >the
> >> >> > > >>>>>>>>>>
> >> >> > > >>>>>>>>> contribution
> >> >> > > >>>>>>>>>
> >> >> > > >>>>>>>>>> docs will need to change.
> >> >> > > >>>>>>>>>>
> >> >> > > >>>>>>>>>
> >> >> > > >>>>>>>>>
> >> >> > > >>>>>>>>> Yes. The PR includes discussion of these other
> >needed
> >> >> changes,
> >> >> > > >>>>>>>>> unfortunately one PR can't change two repositories.
> >> >> > > >>>>>>>>>
> >> >> > > >>>>>>>>> Please continue the discussion on the PR, then I
> >will
> >> >> > > >summarize it
> >> >> > > >>>>>>>>>
> >> >> > > >>>>>>>> back
> >> >> > > >>>>>>
> >> >> > > >>>>>>> into the dev thread.
> >> >> > > >>>>>>>>>
> >> >> > > >>>>>>>>> Thanks,
> >> >> > > >>>>>>>>> Dan
> >> >> > > >>>>>>>>>
> >> >> > > >>>>>>>>>
> >> >> > > >>>>>>>>> They say to run mvn verify before commits.
> >> >> > > >>>>>>>>>>
> >> >> > > >>>>>>>>>> On Thu, Jan 5, 2017 at 9:25 AM Dan Halperin
> >> >> > > >>>>>>>>>>
> >> >> > > >>>>>>>>> <dhalperi@google.com.invalid
> >> >> > > >>>>>>>>
> >> >> > > >>>>>>>>>
> >> >> > > >>>>>>>>>> wrote:
> >> >> > > >>>>>>>>>>
> >> >> > > >>>>>>>>>> Several folks seem to have been confused after
> >> >BEAM-246,
> >> >> > > >where
> >> >> > > >>>>>>>>>>>
> >> >> > > >>>>>>>>>> we
> >> >> > > >>>>>
> >> >> > > >>>>>> moved
> >> >> > > >>>>>>>>
> >> >> > > >>>>>>>>> the
> >> >> > > >>>>>>>>>>
> >> >> > > >>>>>>>>>>> "slow things" into the release profile. I've
> >started
> >> >a
> >> >> > > >>>>>>>>>>>
> >> >> > > >>>>>>>>>> discussion
> >> >> > > >>>>>
> >> >> > > >>>>>> with
> >> >> > > >>>>>>>>
> >> >> > > >>>>>>>>> https://github.com/apache/beam/pull/1740 to see if
> >> >there are
> >> >> > > >>>>>>>>>>>
> >> >> > > >>>>>>>>>> things
> >> >> > > >>>>>>
> >> >> > > >>>>>>> we
> >> >> > > >>>>>>>>
> >> >> > > >>>>>>>>> can
> >> >> > > >>>>>>>>>>
> >> >> > > >>>>>>>>>>> do to fill these gaps.
> >> >> > > >>>>>>>>>>>
> >> >> > > >>>>>>>>>>> Would love folks to chime in with opinions.
> >> >> > > >>>>>>>>>>>
> >> >> > > >>>>>>>>>>> Dan
> >> >> > > >>>>>>>>>>>
> >> >> > > >>>>>>>>>>> On Wed, Jan 4, 2017 at 1:34 PM, Jesse Anderson <
> >> >> > > >>>>>>>>>>>
> >> >> > > >>>>>>>>>> jesse@smokinghand.com>
> >> >> > > >>>>>>>>
> >> >> > > >>>>>>>>> wrote:
> >> >> > > >>>>>>>>>>>
> >> >> > > >>>>>>>>>>> @Eugene, yes that failed on the checkstyle.
> >> >> > > >>>>>>>>>>>>
> >> >> > > >>>>>>>>>>>> On Wed, Jan 4, 2017 at 1:27 PM Eugene Kirpichov
> >> >> > > >>>>>>>>>>>> <ki...@google.com.invalid> wrote:
> >> >> > > >>>>>>>>>>>>
> >> >> > > >>>>>>>>>>>> Try just -Prelease.
> >> >> > > >>>>>>>>>>>>> On Wed, Jan 4, 2017 at 1:21 PM Jesse Anderson <
> >> >> > > >>>>>>>>>>>>>
> >> >> > > >>>>>>>>>>>> jesse@smokinghand.com
> >> >> > > >>>>>>>>>
> >> >> > > >>>>>>>>>>
> >> >> > > >>>>>>>>>>> wrote:
> >> >> > > >>>>>>>>>>>>>
> >> >> > > >>>>>>>>>>>>> Fails because I don't have a secret key.
> >> >> > > >>>>>>>>>>>>>>
> >> >> > > >>>>>>>>>>>>>> On Wed, Jan 4, 2017 at 1:03 PM Jean-Baptiste
> >> >Onofré <
> >> >> > > >>>>>>>>>>>>>>
> >> >> > > >>>>>>>>>>>>> jb@nanthrax.net
> >> >> > > >>>>>>>>>>
> >> >> > > >>>>>>>>>>>
> >> >> > > >>>>>>>>>>>> wrote:
> >> >> > > >>>>>>>>>>>>>>
> >> >> > > >>>>>>>>>>>>>> Hi Jesse,
> >> >> > > >>>>>>>>>>>>>>>
> >> >> > > >>>>>>>>>>>>>>> Could you try the same with:
> >> >> > > >>>>>>>>>>>>>>>
> >> >> > > >>>>>>>>>>>>>>> mvn verify -Prelease,apache-release
> >> >> > > >>>>>>>>>>>>>>>
> >> >> > > >>>>>>>>>>>>>>> ?
> >> >> > > >>>>>>>>>>>>>>>
> >> >> > > >>>>>>>>>>>>>>> Regards
> >> >> > > >>>>>>>>>>>>>>> JB
> >> >> > > >>>>>>>>>>>>>>>
> >> >> > > >>>>>>>>>>>>>>> On 01/04/2017 09:53 PM, Jesse Anderson wrote:
> >> >> > > >>>>>>>>>>>>>>>
> >> >> > > >>>>>>>>>>>>>>>> For some reason, running "mvn verify" isn't
> >> >running
> >> >> > > >>>>>>>>>>>>>>>>
> >> >> > > >>>>>>>>>>>>>>> checkstyle
> >> >> > > >>>>>>>>>
> >> >> > > >>>>>>>>>> on
> >> >> > > >>>>>>>>>>
> >> >> > > >>>>>>>>>>> everything. I had checkstyle errors in
> >> >> > > >>>>>>>>>>>>>>>>
> >> >> > > >>>>>>>>>>>>>>> beam-sdks-java-core
> >> >> > > >>>>>>
> >> >> > > >>>>>>> that
> >> >> > > >>>>>>>>>
> >> >> > > >>>>>>>>>> weren't
> >> >> > > >>>>>>>>>>>>>
> >> >> > > >>>>>>>>>>>>>> being found.
> >> >> > > >>>>>>>>>>>>>>>>
> >> >> > > >>>>>>>>>>>>>>>> I thought this was due to the extra
> >parameters.
> >> >I
> >> >> > > >>>>>>>>>>>>>>>>
> >> >> > > >>>>>>>>>>>>>>> reran
> >> >> > > >>>>>
> >> >> > > >>>>>> with
> >> >> > > >>>>>>>>
> >> >> > > >>>>>>>>> the
> >> >> > > >>>>>>>>>>
> >> >> > > >>>>>>>>>>> plain
> >> >> > > >>>>>>>>>>>>>
> >> >> > > >>>>>>>>>>>>>> "mvn
> >> >> > > >>>>>>>>>>>>>>>
> >> >> > > >>>>>>>>>>>>>>>> verify" and it still didn't find them. From
> >the
> >> >> > > >>>>>>>>>>>>>>>>
> >> >> > > >>>>>>>>>>>>>>> output,
> >> >> > > >>>>>
> >> >> > > >>>>>> it
> >> >> > > >>>>>>
> >> >> > > >>>>>>> doesn't
> >> >> > > >>>>>>>>>>>
> >> >> > > >>>>>>>>>>>> look
> >> >> > > >>>>>>>>>>>>>
> >> >> > > >>>>>>>>>>>>>> like they're being run at all.
> >> >> > > >>>>>>>>>>>>>>>>
> >> >> > > >>>>>>>>>>>>>>>> Thanks,
> >> >> > > >>>>>>>>>>>>>>>>
> >> >> > > >>>>>>>>>>>>>>>> Jesse
> >> >> > > >>>>>>>>>>>>>>>>
> >> >> > > >>>>>>>>>>>>>>>>
> >> >> > > >>>>>>>>>>>>>>> --
> >> >> > > >>>>>>>>>>>>>>> Jean-Baptiste Onofré
> >> >> > > >>>>>>>>>>>>>>> jbonofre@apache.org
> >> >> > > >>>>>>>>>>>>>>> http://blog.nanthrax.net
> >> >> > > >>>>>>>>>>>>>>> Talend - http://www.talend.com
> >> >> > > >>>>>>>>>>>>>>>
> >> >> > > >>>>>>>>>>>>>>>
> >> >> > > >>>>>>>>>>>>>>
> >> >> > > >>>>>>>>>>>>>
> >> >> > > >>>>>>>>>>>>
> >> >> > > >>>>>>>>>>>
> >> >> > > >>>>>>>>>>
> >> >> > > >>>>>>>>>
> >> >> > > >>>>>>>>
> >> >> > > >>>>>>
> >> >> > > >>>>>
> >> >> > > >>>>
> >> >> > > >>>
> >> >> > > >> --
> >> >> > > >> Jean-Baptiste Onofré
> >> >> > > >> jbonofre@apache.org
> >> >> > > >> http://blog.nanthrax.net
> >> >> > > >> Talend - http://www.talend.com
> >> >> > > >>
> >> >> > >
> >> >> >
> >> >>
> >>
>
Re: Better developer instructions for using Maven?
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Yeah. Agree. Time extend is not huge and it's worth to add it in verify phase.
Regards
JB
On Feb 10, 2017, 10:13, at 10:13, Aviem Zur <av...@gmail.com> wrote:
>This goes back to the original discussion in this thread - reduce the
>amount of things pull requesters should know and keep the maven command
>in
>the PR checklist as: 'mvn clean verify'.
>
>So if rat and findbugs do not take that long to run I think they should
>be
>run by 'mvn clean verify'
>
>I ran a quick test on my laptop to see how much time they add to the
>build
>(of the entire project):
>
>'mvn clean install -DskipTests' => Total time: 03:51 min
>'mvn clean install apache-rat:check findbugs:check -DskipTests' =>
>Total
>time: 05:29 min (Added 01:38 min)
>'mvn clean install' => Total time: 09:37 min
>'mvn clean install apache-rat:check findbugs:check' => Total time:
>11:13
>min (Added 01:36 min)
>
>Are these times reasonable enough to add rat and findbugs to the
>default
>build?
>
>On Fri, Feb 10, 2017 at 1:55 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
>wrote:
>
>> Hi
>>
>> We discussed about that at the beginning of the project. We agreed to
>> execute rat and findbugs in a specific profile to reduce the build
>time for
>> dev.
>>
>> That's why I do mvn clean install -Prelease before submitting a PR
>and
>> just clean install when I'm developing.
>>
>> No problem to change that.
>>
>> Regards
>> JB
>>
>> On Feb 10, 2017, 07:51, at 07:51, Aviem Zur <av...@gmail.com>
>wrote:
>> >Can we consider adding rat-plugin and findbugs to the default verify
>> >phase?
>> >Currently they only run when the `release` profile is enabled.
>> >
>> >On Thu, Jan 26, 2017 at 11:42 AM Aljoscha Krettek
><al...@apache.org>
>> >wrote:
>> >
>> >> +1 to what Dan said
>> >>
>> >> On Wed, 25 Jan 2017 at 21:40 Kenneth Knowles
><kl...@google.com.invalid>
>> >> wrote:
>> >>
>> >> > +1
>> >> >
>> >> > On Jan 25, 2017 11:15, "Jean-Baptiste Onofré" <jb...@nanthrax.net>
>> >wrote:
>> >> >
>> >> > > +1
>> >> > >
>> >> > > It sounds good to me.
>> >> > >
>> >> > > Thanks Dan !
>> >> > >
>> >> > > Regards
>> >> > > JB
>> >> > >
>> >> > > On Jan 25, 2017, 19:39, at 19:39, Dan Halperin
>> >> > <dh...@google.com.INVALID>
>> >> > > wrote:
>> >> > > >Here is my summary of the threads:
>> >> > > >
>> >> > > >Overwhelming agreement:
>> >> > > >
>> >> > > >- rename `release` to something more appropriate.
>> >> > > >- add `checkstyle` to the default build (it's basically a
>> >compile
>> >> > > >error)
>> >> > > >- add more information to contributor guide
>> >> > > >
>> >> > > >Reasonable agreement
>> >> > > >
>> >> > > >- don't update the github instructions to make passing `mvn
>> >verify
>> >> > > >-P<all
>> >> > > >checks>` mandatory. Maybe add a hint that this is a good
>proxy
>> >for
>> >> what
>> >> > > >Jenkins will run.
>> >> > > >
>> >> > > >Unresolved:
>> >> > > >
>> >> > > >- whether all checks should be in `mvn verify`
>> >> > > >- whether `mvn test` is useful for most workflows
>> >> > > >
>> >> > > >I'll propose to proceed with the overwhelmingly agreed-upon
>> >changes,
>> >> > > >and as
>> >> > > >we see increasingly many new contributors re-evaluate the
>> >remaining
>> >> > > >issues.
>> >> > > >
>> >> > > >Thanks,
>> >> > > >Dan
>> >> > > >
>> >> > > >On Tue, Jan 24, 2017 at 12:51 PM, Jean-Baptiste Onofré
>> >> > > ><jb...@nanthrax.net>
>> >> > > >wrote:
>> >> > > >
>> >> > > >> +1 to at least update the contribution guide and improve
>the
>> >profile
>> >> > > >name.
>> >> > > >>
>> >> > > >> Regards
>> >> > > >> JB
>> >> > > >>
>> >> > > >>
>> >> > > >> On 01/24/2017 09:49 PM, Kenneth Knowles wrote:
>> >> > > >>
>> >> > > >>> My impression is that we don't have consensus on whether
>all
>> >checks
>> >> > > >or
>> >> > > >>> minimal checks should be the default, or whether we can
>have
>> >both
>> >> > > >via `mvn
>> >> > > >>> test` and `mvn verify`.
>> >> > > >>>
>> >> > > >>> But that doesn't prevent us from giving -P release a
>better
>> >name
>> >> and
>> >> > > >>> mentioning it in the dev guide and in some manner in our
>PR
>> >> > > >template.
>> >> > > >>>
>> >> > > >>> Right now we are living with the combination of the bad
>> >aspects -
>> >> > > >default
>> >> > > >>> is not thorough but not actually fast and a thorough check
>is
>> >> > > >>> undocumented.
>> >> > > >>>
>> >> > > >>> On Tue, Jan 24, 2017 at 2:22 AM, Ismaël Mejía
>> ><ie...@gmail.com>
>> >> > > >wrote:
>> >> > > >>>
>> >> > > >>> I just wanted to know if we have achieved some consensus
>> >about
>> >> this,
>> >> > > >I
>> >> > > >>>> just
>> >> > > >>>> saw this PR that reminded me about this discussion.
>> >> > > >>>>
>> >> > > >>>> https://github.com/apache/beam/pull/1829
>> >> > > >>>>
>> >> > > >>>> It is important that we mention the existing profiles
>(and
>> >the
>> >> > > >intended
>> >> > > >>>> checks) in the contribution guide (e.g. -Prelease (or
>> >-Pall-checks
>> >> > > >>>> triggers
>> >> > > >>>> these validations).
>> >> > > >>>>
>> >> > > >>>> I can add this to the guide if you like once we define
>the
>> >checks
>> >> > > >per
>> >> > > >>>> stage/profile.
>> >> > > >>>>
>> >> > > >>>> Ismaël
>> >> > > >>>>
>> >> > > >>>>
>> >> > > >>>> On Wed, Jan 11, 2017 at 8:12 AM, Aviem Zur
>> ><av...@gmail.com>
>> >> > > >wrote:
>> >> > > >>>>
>> >> > > >>>> I agree with Dan and Lukasz.
>> >> > > >>>>> Developers should not be expected to know beforehand
>which
>> >> > > >specific
>> >> > > >>>>> profiles to run.
>> >> > > >>>>> The phase specified in the PR instructions (`verify`)
>> >should run
>> >> > > >all the
>> >> > > >>>>> relevant verifications and be the "slower" build, while
>a
>> >> > > >preceding
>> >> > > >>>>> lifecycle, such as `test`, should run the "faster"
>> >verifications.
>> >> > > >>>>>
>> >> > > >>>>> Aviem.
>> >> > > >>>>>
>> >> > > >>>>> On Mon, Jan 9, 2017 at 7:57 PM Robert Bradshaw
>> >> > > >>>>>
>> >> > > >>>> <robertwb@google.com.invalid
>> >> > > >>>>
>> >> > > >>>>>
>> >> > > >>>>>> wrote:
>> >> > > >>>>>
>> >> > > >>>>> On Mon, Jan 9, 2017 at 3:49 AM, Aljoscha Krettek
>> >> > > ><al...@apache.org>
>> >> > > >>>>>> wrote:
>> >> > > >>>>>>
>> >> > > >>>>>>> I also usually prefer "mvn verify" to to the expected
>> >thing but
>> >> > > >I see
>> >> > > >>>>>>>
>> >> > > >>>>>> that
>> >> > > >>>>>>
>> >> > > >>>>>>> quick iteration times are key.
>> >> > > >>>>>>>
>> >> > > >>>>>>
>> >> > > >>>>>> I see
>> >> > > >>>>>> https://maven.apache.org/guides/introduction/
>> >> > > >>>>>>
>> >> > > >>>>> introduction-to-the-lifecycle.html
>> >> > > >>>>>
>> >> > > >>>>>>
>> >> > > >>>>>> verify - run any checks on results of integration
>> >tests to
>> >> > > >ensure
>> >> > > >>>>>> quality criteria are met
>> >> > > >>>>>>
>> >> > > >>>>>> Of course our integration tests are long enough that we
>> >> shouldn't
>> >> > > >be
>> >> > > >>>>>> putting all of them here, but I too would expect
>> >checkstyle.
>> >> > > >>>>>>
>> >> > > >>>>>> Perhaps we could introduce a verify-fast or somesuch
>for
>> >fast
>> >> > > >(but
>> >> > > >>>>>> lower coverage) turnaround time. I would expect "mvn
>> >verify
>> >> test"
>> >> > > >to
>> >> > > >>>>>> pass before submitting a PR, and would want to run that
>> >before
>> >> > > >asking
>> >> > > >>>>>> others to look at it. I think this should be our
>criteria
>> >(i.e.
>> >> > > >what
>> >> > > >>>>>> will a new but maven-savvy user run before pushing
>their
>> >code).
>> >> > > >>>>>>
>> >> > > >>>>>> As long as the pre-commit hooks still check everything
>I'm
>> >ok
>> >> > > >with
>> >> > > >>>>>>>
>> >> > > >>>>>> making
>> >> > > >>>>>
>> >> > > >>>>>> the default a little more lightweight.
>> >> > > >>>>>>>
>> >> > > >>>>>>
>> >> > > >>>>>> The fact that our pre-commit hooks take a long time to
>run
>> >does
>> >> > > >change
>> >> > > >>>>>> things. Nothing more annoying than seeing that your PR
>> >failed 3
>> >> > > >hours
>> >> > > >>>>>> later because you had some trailing whitespace...
>> >> > > >>>>>>
>> >> > > >>>>>> On Thu, 5 Jan 2017 at 21:49 Lukasz Cwik
>> >> > > ><lc...@google.com.invalid>
>> >> > > >>>>>>>
>> >> > > >>>>>> wrote:
>> >> > > >>>>>>
>> >> > > >>>>>>>
>> >> > > >>>>>>> I was hoping that the default mvn verify would be the
>> >slow
>> >> build
>> >> > > >>>>>>>>
>> >> > > >>>>>>> and a
>> >> > > >>>>
>> >> > > >>>>> profile could be enabled that would skip checks to make
>> >things
>> >> > > >>>>>>>>
>> >> > > >>>>>>> faster
>> >> > > >>>>
>> >> > > >>>>> for
>> >> > > >>>>>>
>> >> > > >>>>>>> regular contributors. This way a person doesn't need
>to
>> >have
>> >> > > >>>>>>>>
>> >> > > >>>>>>> detailed
>> >> > > >>>>
>> >> > > >>>>> knowledge of all our profiles and what they do
>(typically
>> >mvn
>> >> > > >>>>>>>>
>> >> > > >>>>>>> verify)
>> >> > > >>>>
>> >> > > >>>>> will
>> >> > > >>>>>>
>> >> > > >>>>>>> do the right thing most of the time.
>> >> > > >>>>>>>>
>> >> > > >>>>>>>> On Thu, Jan 5, 2017 at 9:30 AM, Dan Halperin
>> >> > > >>>>>>>>
>> >> > > >>>>>>> <dh...@google.com.invalid>
>> >> > > >>>>>>
>> >> > > >>>>>>> wrote:
>> >> > > >>>>>>>>
>> >> > > >>>>>>>> On Thu, Jan 5, 2017 at 9:28 AM, Jesse Anderson <
>> >> > > >>>>>>>>>
>> >> > > >>>>>>>> jesse@smokinghand.com
>> >> > > >>>>>
>> >> > > >>>>>>
>> >> > > >>>>>>> wrote:
>> >> > > >>>>>>>>>
>> >> > > >>>>>>>>> @dan are you saying that mvn verify isn't doing
>> >checkstyle
>> >> > > >>>>>>>>>>
>> >> > > >>>>>>>>> anymore?
>> >> > > >>>>>
>> >> > > >>>>>>
>> >> > > >>>>>>>>>
>> >> > > >>>>>>>>> `mvn verify` alone should not be running checkstyle,
>if
>> >> > > >modules
>> >> > > >>>>>>>>>
>> >> > > >>>>>>>> are
>> >> > > >>>>
>> >> > > >>>>> configured correctly.
>> >> > > >>>>>>>>>
>> >> > > >>>>>>>>>
>> >> > > >>>>>>>>> Some of
>> >> > > >>>>>>>>>> the checkstyles are still running for a few
>modules.
>> >Also,
>> >> > > >the
>> >> > > >>>>>>>>>>
>> >> > > >>>>>>>>> contribution
>> >> > > >>>>>>>>>
>> >> > > >>>>>>>>>> docs will need to change.
>> >> > > >>>>>>>>>>
>> >> > > >>>>>>>>>
>> >> > > >>>>>>>>>
>> >> > > >>>>>>>>> Yes. The PR includes discussion of these other
>needed
>> >> changes,
>> >> > > >>>>>>>>> unfortunately one PR can't change two repositories.
>> >> > > >>>>>>>>>
>> >> > > >>>>>>>>> Please continue the discussion on the PR, then I
>will
>> >> > > >summarize it
>> >> > > >>>>>>>>>
>> >> > > >>>>>>>> back
>> >> > > >>>>>>
>> >> > > >>>>>>> into the dev thread.
>> >> > > >>>>>>>>>
>> >> > > >>>>>>>>> Thanks,
>> >> > > >>>>>>>>> Dan
>> >> > > >>>>>>>>>
>> >> > > >>>>>>>>>
>> >> > > >>>>>>>>> They say to run mvn verify before commits.
>> >> > > >>>>>>>>>>
>> >> > > >>>>>>>>>> On Thu, Jan 5, 2017 at 9:25 AM Dan Halperin
>> >> > > >>>>>>>>>>
>> >> > > >>>>>>>>> <dhalperi@google.com.invalid
>> >> > > >>>>>>>>
>> >> > > >>>>>>>>>
>> >> > > >>>>>>>>>> wrote:
>> >> > > >>>>>>>>>>
>> >> > > >>>>>>>>>> Several folks seem to have been confused after
>> >BEAM-246,
>> >> > > >where
>> >> > > >>>>>>>>>>>
>> >> > > >>>>>>>>>> we
>> >> > > >>>>>
>> >> > > >>>>>> moved
>> >> > > >>>>>>>>
>> >> > > >>>>>>>>> the
>> >> > > >>>>>>>>>>
>> >> > > >>>>>>>>>>> "slow things" into the release profile. I've
>started
>> >a
>> >> > > >>>>>>>>>>>
>> >> > > >>>>>>>>>> discussion
>> >> > > >>>>>
>> >> > > >>>>>> with
>> >> > > >>>>>>>>
>> >> > > >>>>>>>>> https://github.com/apache/beam/pull/1740 to see if
>> >there are
>> >> > > >>>>>>>>>>>
>> >> > > >>>>>>>>>> things
>> >> > > >>>>>>
>> >> > > >>>>>>> we
>> >> > > >>>>>>>>
>> >> > > >>>>>>>>> can
>> >> > > >>>>>>>>>>
>> >> > > >>>>>>>>>>> do to fill these gaps.
>> >> > > >>>>>>>>>>>
>> >> > > >>>>>>>>>>> Would love folks to chime in with opinions.
>> >> > > >>>>>>>>>>>
>> >> > > >>>>>>>>>>> Dan
>> >> > > >>>>>>>>>>>
>> >> > > >>>>>>>>>>> On Wed, Jan 4, 2017 at 1:34 PM, Jesse Anderson <
>> >> > > >>>>>>>>>>>
>> >> > > >>>>>>>>>> jesse@smokinghand.com>
>> >> > > >>>>>>>>
>> >> > > >>>>>>>>> wrote:
>> >> > > >>>>>>>>>>>
>> >> > > >>>>>>>>>>> @Eugene, yes that failed on the checkstyle.
>> >> > > >>>>>>>>>>>>
>> >> > > >>>>>>>>>>>> On Wed, Jan 4, 2017 at 1:27 PM Eugene Kirpichov
>> >> > > >>>>>>>>>>>> <ki...@google.com.invalid> wrote:
>> >> > > >>>>>>>>>>>>
>> >> > > >>>>>>>>>>>> Try just -Prelease.
>> >> > > >>>>>>>>>>>>> On Wed, Jan 4, 2017 at 1:21 PM Jesse Anderson <
>> >> > > >>>>>>>>>>>>>
>> >> > > >>>>>>>>>>>> jesse@smokinghand.com
>> >> > > >>>>>>>>>
>> >> > > >>>>>>>>>>
>> >> > > >>>>>>>>>>> wrote:
>> >> > > >>>>>>>>>>>>>
>> >> > > >>>>>>>>>>>>> Fails because I don't have a secret key.
>> >> > > >>>>>>>>>>>>>>
>> >> > > >>>>>>>>>>>>>> On Wed, Jan 4, 2017 at 1:03 PM Jean-Baptiste
>> >Onofré <
>> >> > > >>>>>>>>>>>>>>
>> >> > > >>>>>>>>>>>>> jb@nanthrax.net
>> >> > > >>>>>>>>>>
>> >> > > >>>>>>>>>>>
>> >> > > >>>>>>>>>>>> wrote:
>> >> > > >>>>>>>>>>>>>>
>> >> > > >>>>>>>>>>>>>> Hi Jesse,
>> >> > > >>>>>>>>>>>>>>>
>> >> > > >>>>>>>>>>>>>>> Could you try the same with:
>> >> > > >>>>>>>>>>>>>>>
>> >> > > >>>>>>>>>>>>>>> mvn verify -Prelease,apache-release
>> >> > > >>>>>>>>>>>>>>>
>> >> > > >>>>>>>>>>>>>>> ?
>> >> > > >>>>>>>>>>>>>>>
>> >> > > >>>>>>>>>>>>>>> Regards
>> >> > > >>>>>>>>>>>>>>> JB
>> >> > > >>>>>>>>>>>>>>>
>> >> > > >>>>>>>>>>>>>>> On 01/04/2017 09:53 PM, Jesse Anderson wrote:
>> >> > > >>>>>>>>>>>>>>>
>> >> > > >>>>>>>>>>>>>>>> For some reason, running "mvn verify" isn't
>> >running
>> >> > > >>>>>>>>>>>>>>>>
>> >> > > >>>>>>>>>>>>>>> checkstyle
>> >> > > >>>>>>>>>
>> >> > > >>>>>>>>>> on
>> >> > > >>>>>>>>>>
>> >> > > >>>>>>>>>>> everything. I had checkstyle errors in
>> >> > > >>>>>>>>>>>>>>>>
>> >> > > >>>>>>>>>>>>>>> beam-sdks-java-core
>> >> > > >>>>>>
>> >> > > >>>>>>> that
>> >> > > >>>>>>>>>
>> >> > > >>>>>>>>>> weren't
>> >> > > >>>>>>>>>>>>>
>> >> > > >>>>>>>>>>>>>> being found.
>> >> > > >>>>>>>>>>>>>>>>
>> >> > > >>>>>>>>>>>>>>>> I thought this was due to the extra
>parameters.
>> >I
>> >> > > >>>>>>>>>>>>>>>>
>> >> > > >>>>>>>>>>>>>>> reran
>> >> > > >>>>>
>> >> > > >>>>>> with
>> >> > > >>>>>>>>
>> >> > > >>>>>>>>> the
>> >> > > >>>>>>>>>>
>> >> > > >>>>>>>>>>> plain
>> >> > > >>>>>>>>>>>>>
>> >> > > >>>>>>>>>>>>>> "mvn
>> >> > > >>>>>>>>>>>>>>>
>> >> > > >>>>>>>>>>>>>>>> verify" and it still didn't find them. From
>the
>> >> > > >>>>>>>>>>>>>>>>
>> >> > > >>>>>>>>>>>>>>> output,
>> >> > > >>>>>
>> >> > > >>>>>> it
>> >> > > >>>>>>
>> >> > > >>>>>>> doesn't
>> >> > > >>>>>>>>>>>
>> >> > > >>>>>>>>>>>> look
>> >> > > >>>>>>>>>>>>>
>> >> > > >>>>>>>>>>>>>> like they're being run at all.
>> >> > > >>>>>>>>>>>>>>>>
>> >> > > >>>>>>>>>>>>>>>> Thanks,
>> >> > > >>>>>>>>>>>>>>>>
>> >> > > >>>>>>>>>>>>>>>> Jesse
>> >> > > >>>>>>>>>>>>>>>>
>> >> > > >>>>>>>>>>>>>>>>
>> >> > > >>>>>>>>>>>>>>> --
>> >> > > >>>>>>>>>>>>>>> Jean-Baptiste Onofré
>> >> > > >>>>>>>>>>>>>>> jbonofre@apache.org
>> >> > > >>>>>>>>>>>>>>> http://blog.nanthrax.net
>> >> > > >>>>>>>>>>>>>>> Talend - http://www.talend.com
>> >> > > >>>>>>>>>>>>>>>
>> >> > > >>>>>>>>>>>>>>>
>> >> > > >>>>>>>>>>>>>>
>> >> > > >>>>>>>>>>>>>
>> >> > > >>>>>>>>>>>>
>> >> > > >>>>>>>>>>>
>> >> > > >>>>>>>>>>
>> >> > > >>>>>>>>>
>> >> > > >>>>>>>>
>> >> > > >>>>>>
>> >> > > >>>>>
>> >> > > >>>>
>> >> > > >>>
>> >> > > >> --
>> >> > > >> Jean-Baptiste Onofré
>> >> > > >> jbonofre@apache.org
>> >> > > >> http://blog.nanthrax.net
>> >> > > >> Talend - http://www.talend.com
>> >> > > >>
>> >> > >
>> >> >
>> >>
>>
Re: Better developer instructions for using Maven?
Posted by Aviem Zur <av...@gmail.com>.
This goes back to the original discussion in this thread - reduce the
amount of things pull requesters should know and keep the maven command in
the PR checklist as: 'mvn clean verify'.
So if rat and findbugs do not take that long to run I think they should be
run by 'mvn clean verify'
I ran a quick test on my laptop to see how much time they add to the build
(of the entire project):
'mvn clean install -DskipTests' => Total time: 03:51 min
'mvn clean install apache-rat:check findbugs:check -DskipTests' => Total
time: 05:29 min (Added 01:38 min)
'mvn clean install' => Total time: 09:37 min
'mvn clean install apache-rat:check findbugs:check' => Total time: 11:13
min (Added 01:36 min)
Are these times reasonable enough to add rat and findbugs to the default
build?
On Fri, Feb 10, 2017 at 1:55 PM Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:
> Hi
>
> We discussed about that at the beginning of the project. We agreed to
> execute rat and findbugs in a specific profile to reduce the build time for
> dev.
>
> That's why I do mvn clean install -Prelease before submitting a PR and
> just clean install when I'm developing.
>
> No problem to change that.
>
> Regards
> JB
>
> On Feb 10, 2017, 07:51, at 07:51, Aviem Zur <av...@gmail.com> wrote:
> >Can we consider adding rat-plugin and findbugs to the default verify
> >phase?
> >Currently they only run when the `release` profile is enabled.
> >
> >On Thu, Jan 26, 2017 at 11:42 AM Aljoscha Krettek <al...@apache.org>
> >wrote:
> >
> >> +1 to what Dan said
> >>
> >> On Wed, 25 Jan 2017 at 21:40 Kenneth Knowles <kl...@google.com.invalid>
> >> wrote:
> >>
> >> > +1
> >> >
> >> > On Jan 25, 2017 11:15, "Jean-Baptiste Onofré" <jb...@nanthrax.net>
> >wrote:
> >> >
> >> > > +1
> >> > >
> >> > > It sounds good to me.
> >> > >
> >> > > Thanks Dan !
> >> > >
> >> > > Regards
> >> > > JB
> >> > >
> >> > > On Jan 25, 2017, 19:39, at 19:39, Dan Halperin
> >> > <dh...@google.com.INVALID>
> >> > > wrote:
> >> > > >Here is my summary of the threads:
> >> > > >
> >> > > >Overwhelming agreement:
> >> > > >
> >> > > >- rename `release` to something more appropriate.
> >> > > >- add `checkstyle` to the default build (it's basically a
> >compile
> >> > > >error)
> >> > > >- add more information to contributor guide
> >> > > >
> >> > > >Reasonable agreement
> >> > > >
> >> > > >- don't update the github instructions to make passing `mvn
> >verify
> >> > > >-P<all
> >> > > >checks>` mandatory. Maybe add a hint that this is a good proxy
> >for
> >> what
> >> > > >Jenkins will run.
> >> > > >
> >> > > >Unresolved:
> >> > > >
> >> > > >- whether all checks should be in `mvn verify`
> >> > > >- whether `mvn test` is useful for most workflows
> >> > > >
> >> > > >I'll propose to proceed with the overwhelmingly agreed-upon
> >changes,
> >> > > >and as
> >> > > >we see increasingly many new contributors re-evaluate the
> >remaining
> >> > > >issues.
> >> > > >
> >> > > >Thanks,
> >> > > >Dan
> >> > > >
> >> > > >On Tue, Jan 24, 2017 at 12:51 PM, Jean-Baptiste Onofré
> >> > > ><jb...@nanthrax.net>
> >> > > >wrote:
> >> > > >
> >> > > >> +1 to at least update the contribution guide and improve the
> >profile
> >> > > >name.
> >> > > >>
> >> > > >> Regards
> >> > > >> JB
> >> > > >>
> >> > > >>
> >> > > >> On 01/24/2017 09:49 PM, Kenneth Knowles wrote:
> >> > > >>
> >> > > >>> My impression is that we don't have consensus on whether all
> >checks
> >> > > >or
> >> > > >>> minimal checks should be the default, or whether we can have
> >both
> >> > > >via `mvn
> >> > > >>> test` and `mvn verify`.
> >> > > >>>
> >> > > >>> But that doesn't prevent us from giving -P release a better
> >name
> >> and
> >> > > >>> mentioning it in the dev guide and in some manner in our PR
> >> > > >template.
> >> > > >>>
> >> > > >>> Right now we are living with the combination of the bad
> >aspects -
> >> > > >default
> >> > > >>> is not thorough but not actually fast and a thorough check is
> >> > > >>> undocumented.
> >> > > >>>
> >> > > >>> On Tue, Jan 24, 2017 at 2:22 AM, Ismaël Mejía
> ><ie...@gmail.com>
> >> > > >wrote:
> >> > > >>>
> >> > > >>> I just wanted to know if we have achieved some consensus
> >about
> >> this,
> >> > > >I
> >> > > >>>> just
> >> > > >>>> saw this PR that reminded me about this discussion.
> >> > > >>>>
> >> > > >>>> https://github.com/apache/beam/pull/1829
> >> > > >>>>
> >> > > >>>> It is important that we mention the existing profiles (and
> >the
> >> > > >intended
> >> > > >>>> checks) in the contribution guide (e.g. -Prelease (or
> >-Pall-checks
> >> > > >>>> triggers
> >> > > >>>> these validations).
> >> > > >>>>
> >> > > >>>> I can add this to the guide if you like once we define the
> >checks
> >> > > >per
> >> > > >>>> stage/profile.
> >> > > >>>>
> >> > > >>>> Ismaël
> >> > > >>>>
> >> > > >>>>
> >> > > >>>> On Wed, Jan 11, 2017 at 8:12 AM, Aviem Zur
> ><av...@gmail.com>
> >> > > >wrote:
> >> > > >>>>
> >> > > >>>> I agree with Dan and Lukasz.
> >> > > >>>>> Developers should not be expected to know beforehand which
> >> > > >specific
> >> > > >>>>> profiles to run.
> >> > > >>>>> The phase specified in the PR instructions (`verify`)
> >should run
> >> > > >all the
> >> > > >>>>> relevant verifications and be the "slower" build, while a
> >> > > >preceding
> >> > > >>>>> lifecycle, such as `test`, should run the "faster"
> >verifications.
> >> > > >>>>>
> >> > > >>>>> Aviem.
> >> > > >>>>>
> >> > > >>>>> On Mon, Jan 9, 2017 at 7:57 PM Robert Bradshaw
> >> > > >>>>>
> >> > > >>>> <robertwb@google.com.invalid
> >> > > >>>>
> >> > > >>>>>
> >> > > >>>>>> wrote:
> >> > > >>>>>
> >> > > >>>>> On Mon, Jan 9, 2017 at 3:49 AM, Aljoscha Krettek
> >> > > ><al...@apache.org>
> >> > > >>>>>> wrote:
> >> > > >>>>>>
> >> > > >>>>>>> I also usually prefer "mvn verify" to to the expected
> >thing but
> >> > > >I see
> >> > > >>>>>>>
> >> > > >>>>>> that
> >> > > >>>>>>
> >> > > >>>>>>> quick iteration times are key.
> >> > > >>>>>>>
> >> > > >>>>>>
> >> > > >>>>>> I see
> >> > > >>>>>> https://maven.apache.org/guides/introduction/
> >> > > >>>>>>
> >> > > >>>>> introduction-to-the-lifecycle.html
> >> > > >>>>>
> >> > > >>>>>>
> >> > > >>>>>> verify - run any checks on results of integration
> >tests to
> >> > > >ensure
> >> > > >>>>>> quality criteria are met
> >> > > >>>>>>
> >> > > >>>>>> Of course our integration tests are long enough that we
> >> shouldn't
> >> > > >be
> >> > > >>>>>> putting all of them here, but I too would expect
> >checkstyle.
> >> > > >>>>>>
> >> > > >>>>>> Perhaps we could introduce a verify-fast or somesuch for
> >fast
> >> > > >(but
> >> > > >>>>>> lower coverage) turnaround time. I would expect "mvn
> >verify
> >> test"
> >> > > >to
> >> > > >>>>>> pass before submitting a PR, and would want to run that
> >before
> >> > > >asking
> >> > > >>>>>> others to look at it. I think this should be our criteria
> >(i.e.
> >> > > >what
> >> > > >>>>>> will a new but maven-savvy user run before pushing their
> >code).
> >> > > >>>>>>
> >> > > >>>>>> As long as the pre-commit hooks still check everything I'm
> >ok
> >> > > >with
> >> > > >>>>>>>
> >> > > >>>>>> making
> >> > > >>>>>
> >> > > >>>>>> the default a little more lightweight.
> >> > > >>>>>>>
> >> > > >>>>>>
> >> > > >>>>>> The fact that our pre-commit hooks take a long time to run
> >does
> >> > > >change
> >> > > >>>>>> things. Nothing more annoying than seeing that your PR
> >failed 3
> >> > > >hours
> >> > > >>>>>> later because you had some trailing whitespace...
> >> > > >>>>>>
> >> > > >>>>>> On Thu, 5 Jan 2017 at 21:49 Lukasz Cwik
> >> > > ><lc...@google.com.invalid>
> >> > > >>>>>>>
> >> > > >>>>>> wrote:
> >> > > >>>>>>
> >> > > >>>>>>>
> >> > > >>>>>>> I was hoping that the default mvn verify would be the
> >slow
> >> build
> >> > > >>>>>>>>
> >> > > >>>>>>> and a
> >> > > >>>>
> >> > > >>>>> profile could be enabled that would skip checks to make
> >things
> >> > > >>>>>>>>
> >> > > >>>>>>> faster
> >> > > >>>>
> >> > > >>>>> for
> >> > > >>>>>>
> >> > > >>>>>>> regular contributors. This way a person doesn't need to
> >have
> >> > > >>>>>>>>
> >> > > >>>>>>> detailed
> >> > > >>>>
> >> > > >>>>> knowledge of all our profiles and what they do (typically
> >mvn
> >> > > >>>>>>>>
> >> > > >>>>>>> verify)
> >> > > >>>>
> >> > > >>>>> will
> >> > > >>>>>>
> >> > > >>>>>>> do the right thing most of the time.
> >> > > >>>>>>>>
> >> > > >>>>>>>> On Thu, Jan 5, 2017 at 9:30 AM, Dan Halperin
> >> > > >>>>>>>>
> >> > > >>>>>>> <dh...@google.com.invalid>
> >> > > >>>>>>
> >> > > >>>>>>> wrote:
> >> > > >>>>>>>>
> >> > > >>>>>>>> On Thu, Jan 5, 2017 at 9:28 AM, Jesse Anderson <
> >> > > >>>>>>>>>
> >> > > >>>>>>>> jesse@smokinghand.com
> >> > > >>>>>
> >> > > >>>>>>
> >> > > >>>>>>> wrote:
> >> > > >>>>>>>>>
> >> > > >>>>>>>>> @dan are you saying that mvn verify isn't doing
> >checkstyle
> >> > > >>>>>>>>>>
> >> > > >>>>>>>>> anymore?
> >> > > >>>>>
> >> > > >>>>>>
> >> > > >>>>>>>>>
> >> > > >>>>>>>>> `mvn verify` alone should not be running checkstyle, if
> >> > > >modules
> >> > > >>>>>>>>>
> >> > > >>>>>>>> are
> >> > > >>>>
> >> > > >>>>> configured correctly.
> >> > > >>>>>>>>>
> >> > > >>>>>>>>>
> >> > > >>>>>>>>> Some of
> >> > > >>>>>>>>>> the checkstyles are still running for a few modules.
> >Also,
> >> > > >the
> >> > > >>>>>>>>>>
> >> > > >>>>>>>>> contribution
> >> > > >>>>>>>>>
> >> > > >>>>>>>>>> docs will need to change.
> >> > > >>>>>>>>>>
> >> > > >>>>>>>>>
> >> > > >>>>>>>>>
> >> > > >>>>>>>>> Yes. The PR includes discussion of these other needed
> >> changes,
> >> > > >>>>>>>>> unfortunately one PR can't change two repositories.
> >> > > >>>>>>>>>
> >> > > >>>>>>>>> Please continue the discussion on the PR, then I will
> >> > > >summarize it
> >> > > >>>>>>>>>
> >> > > >>>>>>>> back
> >> > > >>>>>>
> >> > > >>>>>>> into the dev thread.
> >> > > >>>>>>>>>
> >> > > >>>>>>>>> Thanks,
> >> > > >>>>>>>>> Dan
> >> > > >>>>>>>>>
> >> > > >>>>>>>>>
> >> > > >>>>>>>>> They say to run mvn verify before commits.
> >> > > >>>>>>>>>>
> >> > > >>>>>>>>>> On Thu, Jan 5, 2017 at 9:25 AM Dan Halperin
> >> > > >>>>>>>>>>
> >> > > >>>>>>>>> <dhalperi@google.com.invalid
> >> > > >>>>>>>>
> >> > > >>>>>>>>>
> >> > > >>>>>>>>>> wrote:
> >> > > >>>>>>>>>>
> >> > > >>>>>>>>>> Several folks seem to have been confused after
> >BEAM-246,
> >> > > >where
> >> > > >>>>>>>>>>>
> >> > > >>>>>>>>>> we
> >> > > >>>>>
> >> > > >>>>>> moved
> >> > > >>>>>>>>
> >> > > >>>>>>>>> the
> >> > > >>>>>>>>>>
> >> > > >>>>>>>>>>> "slow things" into the release profile. I've started
> >a
> >> > > >>>>>>>>>>>
> >> > > >>>>>>>>>> discussion
> >> > > >>>>>
> >> > > >>>>>> with
> >> > > >>>>>>>>
> >> > > >>>>>>>>> https://github.com/apache/beam/pull/1740 to see if
> >there are
> >> > > >>>>>>>>>>>
> >> > > >>>>>>>>>> things
> >> > > >>>>>>
> >> > > >>>>>>> we
> >> > > >>>>>>>>
> >> > > >>>>>>>>> can
> >> > > >>>>>>>>>>
> >> > > >>>>>>>>>>> do to fill these gaps.
> >> > > >>>>>>>>>>>
> >> > > >>>>>>>>>>> Would love folks to chime in with opinions.
> >> > > >>>>>>>>>>>
> >> > > >>>>>>>>>>> Dan
> >> > > >>>>>>>>>>>
> >> > > >>>>>>>>>>> On Wed, Jan 4, 2017 at 1:34 PM, Jesse Anderson <
> >> > > >>>>>>>>>>>
> >> > > >>>>>>>>>> jesse@smokinghand.com>
> >> > > >>>>>>>>
> >> > > >>>>>>>>> wrote:
> >> > > >>>>>>>>>>>
> >> > > >>>>>>>>>>> @Eugene, yes that failed on the checkstyle.
> >> > > >>>>>>>>>>>>
> >> > > >>>>>>>>>>>> On Wed, Jan 4, 2017 at 1:27 PM Eugene Kirpichov
> >> > > >>>>>>>>>>>> <ki...@google.com.invalid> wrote:
> >> > > >>>>>>>>>>>>
> >> > > >>>>>>>>>>>> Try just -Prelease.
> >> > > >>>>>>>>>>>>> On Wed, Jan 4, 2017 at 1:21 PM Jesse Anderson <
> >> > > >>>>>>>>>>>>>
> >> > > >>>>>>>>>>>> jesse@smokinghand.com
> >> > > >>>>>>>>>
> >> > > >>>>>>>>>>
> >> > > >>>>>>>>>>> wrote:
> >> > > >>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>> Fails because I don't have a secret key.
> >> > > >>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>> On Wed, Jan 4, 2017 at 1:03 PM Jean-Baptiste
> >Onofré <
> >> > > >>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>> jb@nanthrax.net
> >> > > >>>>>>>>>>
> >> > > >>>>>>>>>>>
> >> > > >>>>>>>>>>>> wrote:
> >> > > >>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>> Hi Jesse,
> >> > > >>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>> Could you try the same with:
> >> > > >>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>> mvn verify -Prelease,apache-release
> >> > > >>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>> ?
> >> > > >>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>> Regards
> >> > > >>>>>>>>>>>>>>> JB
> >> > > >>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>> On 01/04/2017 09:53 PM, Jesse Anderson wrote:
> >> > > >>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>> For some reason, running "mvn verify" isn't
> >running
> >> > > >>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>> checkstyle
> >> > > >>>>>>>>>
> >> > > >>>>>>>>>> on
> >> > > >>>>>>>>>>
> >> > > >>>>>>>>>>> everything. I had checkstyle errors in
> >> > > >>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>> beam-sdks-java-core
> >> > > >>>>>>
> >> > > >>>>>>> that
> >> > > >>>>>>>>>
> >> > > >>>>>>>>>> weren't
> >> > > >>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>> being found.
> >> > > >>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>> I thought this was due to the extra parameters.
> >I
> >> > > >>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>> reran
> >> > > >>>>>
> >> > > >>>>>> with
> >> > > >>>>>>>>
> >> > > >>>>>>>>> the
> >> > > >>>>>>>>>>
> >> > > >>>>>>>>>>> plain
> >> > > >>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>> "mvn
> >> > > >>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>> verify" and it still didn't find them. From the
> >> > > >>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>> output,
> >> > > >>>>>
> >> > > >>>>>> it
> >> > > >>>>>>
> >> > > >>>>>>> doesn't
> >> > > >>>>>>>>>>>
> >> > > >>>>>>>>>>>> look
> >> > > >>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>> like they're being run at all.
> >> > > >>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>> Thanks,
> >> > > >>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>> Jesse
> >> > > >>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>> --
> >> > > >>>>>>>>>>>>>>> Jean-Baptiste Onofré
> >> > > >>>>>>>>>>>>>>> jbonofre@apache.org
> >> > > >>>>>>>>>>>>>>> http://blog.nanthrax.net
> >> > > >>>>>>>>>>>>>>> Talend - http://www.talend.com
> >> > > >>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>>
> >> > > >>>>>>>>>>>>
> >> > > >>>>>>>>>>>
> >> > > >>>>>>>>>>
> >> > > >>>>>>>>>
> >> > > >>>>>>>>
> >> > > >>>>>>
> >> > > >>>>>
> >> > > >>>>
> >> > > >>>
> >> > > >> --
> >> > > >> Jean-Baptiste Onofré
> >> > > >> jbonofre@apache.org
> >> > > >> http://blog.nanthrax.net
> >> > > >> Talend - http://www.talend.com
> >> > > >>
> >> > >
> >> >
> >>
>
Re: Better developer instructions for using Maven?
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi
We discussed about that at the beginning of the project. We agreed to execute rat and findbugs in a specific profile to reduce the build time for dev.
That's why I do mvn clean install -Prelease before submitting a PR and just clean install when I'm developing.
No problem to change that.
Regards
JB
On Feb 10, 2017, 07:51, at 07:51, Aviem Zur <av...@gmail.com> wrote:
>Can we consider adding rat-plugin and findbugs to the default verify
>phase?
>Currently they only run when the `release` profile is enabled.
>
>On Thu, Jan 26, 2017 at 11:42 AM Aljoscha Krettek <al...@apache.org>
>wrote:
>
>> +1 to what Dan said
>>
>> On Wed, 25 Jan 2017 at 21:40 Kenneth Knowles <kl...@google.com.invalid>
>> wrote:
>>
>> > +1
>> >
>> > On Jan 25, 2017 11:15, "Jean-Baptiste Onofré" <jb...@nanthrax.net>
>wrote:
>> >
>> > > +1
>> > >
>> > > It sounds good to me.
>> > >
>> > > Thanks Dan !
>> > >
>> > > Regards
>> > > JB
>> > >
>> > > On Jan 25, 2017, 19:39, at 19:39, Dan Halperin
>> > <dh...@google.com.INVALID>
>> > > wrote:
>> > > >Here is my summary of the threads:
>> > > >
>> > > >Overwhelming agreement:
>> > > >
>> > > >- rename `release` to something more appropriate.
>> > > >- add `checkstyle` to the default build (it's basically a
>compile
>> > > >error)
>> > > >- add more information to contributor guide
>> > > >
>> > > >Reasonable agreement
>> > > >
>> > > >- don't update the github instructions to make passing `mvn
>verify
>> > > >-P<all
>> > > >checks>` mandatory. Maybe add a hint that this is a good proxy
>for
>> what
>> > > >Jenkins will run.
>> > > >
>> > > >Unresolved:
>> > > >
>> > > >- whether all checks should be in `mvn verify`
>> > > >- whether `mvn test` is useful for most workflows
>> > > >
>> > > >I'll propose to proceed with the overwhelmingly agreed-upon
>changes,
>> > > >and as
>> > > >we see increasingly many new contributors re-evaluate the
>remaining
>> > > >issues.
>> > > >
>> > > >Thanks,
>> > > >Dan
>> > > >
>> > > >On Tue, Jan 24, 2017 at 12:51 PM, Jean-Baptiste Onofré
>> > > ><jb...@nanthrax.net>
>> > > >wrote:
>> > > >
>> > > >> +1 to at least update the contribution guide and improve the
>profile
>> > > >name.
>> > > >>
>> > > >> Regards
>> > > >> JB
>> > > >>
>> > > >>
>> > > >> On 01/24/2017 09:49 PM, Kenneth Knowles wrote:
>> > > >>
>> > > >>> My impression is that we don't have consensus on whether all
>checks
>> > > >or
>> > > >>> minimal checks should be the default, or whether we can have
>both
>> > > >via `mvn
>> > > >>> test` and `mvn verify`.
>> > > >>>
>> > > >>> But that doesn't prevent us from giving -P release a better
>name
>> and
>> > > >>> mentioning it in the dev guide and in some manner in our PR
>> > > >template.
>> > > >>>
>> > > >>> Right now we are living with the combination of the bad
>aspects -
>> > > >default
>> > > >>> is not thorough but not actually fast and a thorough check is
>> > > >>> undocumented.
>> > > >>>
>> > > >>> On Tue, Jan 24, 2017 at 2:22 AM, Ismaël Mejía
><ie...@gmail.com>
>> > > >wrote:
>> > > >>>
>> > > >>> I just wanted to know if we have achieved some consensus
>about
>> this,
>> > > >I
>> > > >>>> just
>> > > >>>> saw this PR that reminded me about this discussion.
>> > > >>>>
>> > > >>>> https://github.com/apache/beam/pull/1829
>> > > >>>>
>> > > >>>> It is important that we mention the existing profiles (and
>the
>> > > >intended
>> > > >>>> checks) in the contribution guide (e.g. -Prelease (or
>-Pall-checks
>> > > >>>> triggers
>> > > >>>> these validations).
>> > > >>>>
>> > > >>>> I can add this to the guide if you like once we define the
>checks
>> > > >per
>> > > >>>> stage/profile.
>> > > >>>>
>> > > >>>> Ismaël
>> > > >>>>
>> > > >>>>
>> > > >>>> On Wed, Jan 11, 2017 at 8:12 AM, Aviem Zur
><av...@gmail.com>
>> > > >wrote:
>> > > >>>>
>> > > >>>> I agree with Dan and Lukasz.
>> > > >>>>> Developers should not be expected to know beforehand which
>> > > >specific
>> > > >>>>> profiles to run.
>> > > >>>>> The phase specified in the PR instructions (`verify`)
>should run
>> > > >all the
>> > > >>>>> relevant verifications and be the "slower" build, while a
>> > > >preceding
>> > > >>>>> lifecycle, such as `test`, should run the "faster"
>verifications.
>> > > >>>>>
>> > > >>>>> Aviem.
>> > > >>>>>
>> > > >>>>> On Mon, Jan 9, 2017 at 7:57 PM Robert Bradshaw
>> > > >>>>>
>> > > >>>> <robertwb@google.com.invalid
>> > > >>>>
>> > > >>>>>
>> > > >>>>>> wrote:
>> > > >>>>>
>> > > >>>>> On Mon, Jan 9, 2017 at 3:49 AM, Aljoscha Krettek
>> > > ><al...@apache.org>
>> > > >>>>>> wrote:
>> > > >>>>>>
>> > > >>>>>>> I also usually prefer "mvn verify" to to the expected
>thing but
>> > > >I see
>> > > >>>>>>>
>> > > >>>>>> that
>> > > >>>>>>
>> > > >>>>>>> quick iteration times are key.
>> > > >>>>>>>
>> > > >>>>>>
>> > > >>>>>> I see
>> > > >>>>>> https://maven.apache.org/guides/introduction/
>> > > >>>>>>
>> > > >>>>> introduction-to-the-lifecycle.html
>> > > >>>>>
>> > > >>>>>>
>> > > >>>>>> verify - run any checks on results of integration
>tests to
>> > > >ensure
>> > > >>>>>> quality criteria are met
>> > > >>>>>>
>> > > >>>>>> Of course our integration tests are long enough that we
>> shouldn't
>> > > >be
>> > > >>>>>> putting all of them here, but I too would expect
>checkstyle.
>> > > >>>>>>
>> > > >>>>>> Perhaps we could introduce a verify-fast or somesuch for
>fast
>> > > >(but
>> > > >>>>>> lower coverage) turnaround time. I would expect "mvn
>verify
>> test"
>> > > >to
>> > > >>>>>> pass before submitting a PR, and would want to run that
>before
>> > > >asking
>> > > >>>>>> others to look at it. I think this should be our criteria
>(i.e.
>> > > >what
>> > > >>>>>> will a new but maven-savvy user run before pushing their
>code).
>> > > >>>>>>
>> > > >>>>>> As long as the pre-commit hooks still check everything I'm
>ok
>> > > >with
>> > > >>>>>>>
>> > > >>>>>> making
>> > > >>>>>
>> > > >>>>>> the default a little more lightweight.
>> > > >>>>>>>
>> > > >>>>>>
>> > > >>>>>> The fact that our pre-commit hooks take a long time to run
>does
>> > > >change
>> > > >>>>>> things. Nothing more annoying than seeing that your PR
>failed 3
>> > > >hours
>> > > >>>>>> later because you had some trailing whitespace...
>> > > >>>>>>
>> > > >>>>>> On Thu, 5 Jan 2017 at 21:49 Lukasz Cwik
>> > > ><lc...@google.com.invalid>
>> > > >>>>>>>
>> > > >>>>>> wrote:
>> > > >>>>>>
>> > > >>>>>>>
>> > > >>>>>>> I was hoping that the default mvn verify would be the
>slow
>> build
>> > > >>>>>>>>
>> > > >>>>>>> and a
>> > > >>>>
>> > > >>>>> profile could be enabled that would skip checks to make
>things
>> > > >>>>>>>>
>> > > >>>>>>> faster
>> > > >>>>
>> > > >>>>> for
>> > > >>>>>>
>> > > >>>>>>> regular contributors. This way a person doesn't need to
>have
>> > > >>>>>>>>
>> > > >>>>>>> detailed
>> > > >>>>
>> > > >>>>> knowledge of all our profiles and what they do (typically
>mvn
>> > > >>>>>>>>
>> > > >>>>>>> verify)
>> > > >>>>
>> > > >>>>> will
>> > > >>>>>>
>> > > >>>>>>> do the right thing most of the time.
>> > > >>>>>>>>
>> > > >>>>>>>> On Thu, Jan 5, 2017 at 9:30 AM, Dan Halperin
>> > > >>>>>>>>
>> > > >>>>>>> <dh...@google.com.invalid>
>> > > >>>>>>
>> > > >>>>>>> wrote:
>> > > >>>>>>>>
>> > > >>>>>>>> On Thu, Jan 5, 2017 at 9:28 AM, Jesse Anderson <
>> > > >>>>>>>>>
>> > > >>>>>>>> jesse@smokinghand.com
>> > > >>>>>
>> > > >>>>>>
>> > > >>>>>>> wrote:
>> > > >>>>>>>>>
>> > > >>>>>>>>> @dan are you saying that mvn verify isn't doing
>checkstyle
>> > > >>>>>>>>>>
>> > > >>>>>>>>> anymore?
>> > > >>>>>
>> > > >>>>>>
>> > > >>>>>>>>>
>> > > >>>>>>>>> `mvn verify` alone should not be running checkstyle, if
>> > > >modules
>> > > >>>>>>>>>
>> > > >>>>>>>> are
>> > > >>>>
>> > > >>>>> configured correctly.
>> > > >>>>>>>>>
>> > > >>>>>>>>>
>> > > >>>>>>>>> Some of
>> > > >>>>>>>>>> the checkstyles are still running for a few modules.
>Also,
>> > > >the
>> > > >>>>>>>>>>
>> > > >>>>>>>>> contribution
>> > > >>>>>>>>>
>> > > >>>>>>>>>> docs will need to change.
>> > > >>>>>>>>>>
>> > > >>>>>>>>>
>> > > >>>>>>>>>
>> > > >>>>>>>>> Yes. The PR includes discussion of these other needed
>> changes,
>> > > >>>>>>>>> unfortunately one PR can't change two repositories.
>> > > >>>>>>>>>
>> > > >>>>>>>>> Please continue the discussion on the PR, then I will
>> > > >summarize it
>> > > >>>>>>>>>
>> > > >>>>>>>> back
>> > > >>>>>>
>> > > >>>>>>> into the dev thread.
>> > > >>>>>>>>>
>> > > >>>>>>>>> Thanks,
>> > > >>>>>>>>> Dan
>> > > >>>>>>>>>
>> > > >>>>>>>>>
>> > > >>>>>>>>> They say to run mvn verify before commits.
>> > > >>>>>>>>>>
>> > > >>>>>>>>>> On Thu, Jan 5, 2017 at 9:25 AM Dan Halperin
>> > > >>>>>>>>>>
>> > > >>>>>>>>> <dhalperi@google.com.invalid
>> > > >>>>>>>>
>> > > >>>>>>>>>
>> > > >>>>>>>>>> wrote:
>> > > >>>>>>>>>>
>> > > >>>>>>>>>> Several folks seem to have been confused after
>BEAM-246,
>> > > >where
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>> we
>> > > >>>>>
>> > > >>>>>> moved
>> > > >>>>>>>>
>> > > >>>>>>>>> the
>> > > >>>>>>>>>>
>> > > >>>>>>>>>>> "slow things" into the release profile. I've started
>a
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>> discussion
>> > > >>>>>
>> > > >>>>>> with
>> > > >>>>>>>>
>> > > >>>>>>>>> https://github.com/apache/beam/pull/1740 to see if
>there are
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>> things
>> > > >>>>>>
>> > > >>>>>>> we
>> > > >>>>>>>>
>> > > >>>>>>>>> can
>> > > >>>>>>>>>>
>> > > >>>>>>>>>>> do to fill these gaps.
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>> Would love folks to chime in with opinions.
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>> Dan
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>> On Wed, Jan 4, 2017 at 1:34 PM, Jesse Anderson <
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>> jesse@smokinghand.com>
>> > > >>>>>>>>
>> > > >>>>>>>>> wrote:
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>> @Eugene, yes that failed on the checkstyle.
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> On Wed, Jan 4, 2017 at 1:27 PM Eugene Kirpichov
>> > > >>>>>>>>>>>> <ki...@google.com.invalid> wrote:
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>> Try just -Prelease.
>> > > >>>>>>>>>>>>> On Wed, Jan 4, 2017 at 1:21 PM Jesse Anderson <
>> > > >>>>>>>>>>>>>
>> > > >>>>>>>>>>>> jesse@smokinghand.com
>> > > >>>>>>>>>
>> > > >>>>>>>>>>
>> > > >>>>>>>>>>> wrote:
>> > > >>>>>>>>>>>>>
>> > > >>>>>>>>>>>>> Fails because I don't have a secret key.
>> > > >>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>> On Wed, Jan 4, 2017 at 1:03 PM Jean-Baptiste
>Onofré <
>> > > >>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>> jb@nanthrax.net
>> > > >>>>>>>>>>
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>>> wrote:
>> > > >>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>> Hi Jesse,
>> > > >>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>> Could you try the same with:
>> > > >>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>> mvn verify -Prelease,apache-release
>> > > >>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>> ?
>> > > >>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>> Regards
>> > > >>>>>>>>>>>>>>> JB
>> > > >>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>> On 01/04/2017 09:53 PM, Jesse Anderson wrote:
>> > > >>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>> For some reason, running "mvn verify" isn't
>running
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>> checkstyle
>> > > >>>>>>>>>
>> > > >>>>>>>>>> on
>> > > >>>>>>>>>>
>> > > >>>>>>>>>>> everything. I had checkstyle errors in
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>> beam-sdks-java-core
>> > > >>>>>>
>> > > >>>>>>> that
>> > > >>>>>>>>>
>> > > >>>>>>>>>> weren't
>> > > >>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>> being found.
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>> I thought this was due to the extra parameters.
>I
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>> reran
>> > > >>>>>
>> > > >>>>>> with
>> > > >>>>>>>>
>> > > >>>>>>>>> the
>> > > >>>>>>>>>>
>> > > >>>>>>>>>>> plain
>> > > >>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>> "mvn
>> > > >>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>> verify" and it still didn't find them. From the
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>> output,
>> > > >>>>>
>> > > >>>>>> it
>> > > >>>>>>
>> > > >>>>>>> doesn't
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>>> look
>> > > >>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>> like they're being run at all.
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>> Thanks,
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>> Jesse
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>> --
>> > > >>>>>>>>>>>>>>> Jean-Baptiste Onofré
>> > > >>>>>>>>>>>>>>> jbonofre@apache.org
>> > > >>>>>>>>>>>>>>> http://blog.nanthrax.net
>> > > >>>>>>>>>>>>>>> Talend - http://www.talend.com
>> > > >>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>>
>> > > >>>>>>>>>>>>>
>> > > >>>>>>>>>>>>
>> > > >>>>>>>>>>>
>> > > >>>>>>>>>>
>> > > >>>>>>>>>
>> > > >>>>>>>>
>> > > >>>>>>
>> > > >>>>>
>> > > >>>>
>> > > >>>
>> > > >> --
>> > > >> Jean-Baptiste Onofré
>> > > >> jbonofre@apache.org
>> > > >> http://blog.nanthrax.net
>> > > >> Talend - http://www.talend.com
>> > > >>
>> > >
>> >
>>
Re: Better developer instructions for using Maven?
Posted by Aviem Zur <av...@gmail.com>.
Can we consider adding rat-plugin and findbugs to the default verify phase?
Currently they only run when the `release` profile is enabled.
On Thu, Jan 26, 2017 at 11:42 AM Aljoscha Krettek <al...@apache.org>
wrote:
> +1 to what Dan said
>
> On Wed, 25 Jan 2017 at 21:40 Kenneth Knowles <kl...@google.com.invalid>
> wrote:
>
> > +1
> >
> > On Jan 25, 2017 11:15, "Jean-Baptiste Onofré" <jb...@nanthrax.net> wrote:
> >
> > > +1
> > >
> > > It sounds good to me.
> > >
> > > Thanks Dan !
> > >
> > > Regards
> > > JB
> > >
> > > On Jan 25, 2017, 19:39, at 19:39, Dan Halperin
> > <dh...@google.com.INVALID>
> > > wrote:
> > > >Here is my summary of the threads:
> > > >
> > > >Overwhelming agreement:
> > > >
> > > >- rename `release` to something more appropriate.
> > > >- add `checkstyle` to the default build (it's basically a compile
> > > >error)
> > > >- add more information to contributor guide
> > > >
> > > >Reasonable agreement
> > > >
> > > >- don't update the github instructions to make passing `mvn verify
> > > >-P<all
> > > >checks>` mandatory. Maybe add a hint that this is a good proxy for
> what
> > > >Jenkins will run.
> > > >
> > > >Unresolved:
> > > >
> > > >- whether all checks should be in `mvn verify`
> > > >- whether `mvn test` is useful for most workflows
> > > >
> > > >I'll propose to proceed with the overwhelmingly agreed-upon changes,
> > > >and as
> > > >we see increasingly many new contributors re-evaluate the remaining
> > > >issues.
> > > >
> > > >Thanks,
> > > >Dan
> > > >
> > > >On Tue, Jan 24, 2017 at 12:51 PM, Jean-Baptiste Onofré
> > > ><jb...@nanthrax.net>
> > > >wrote:
> > > >
> > > >> +1 to at least update the contribution guide and improve the profile
> > > >name.
> > > >>
> > > >> Regards
> > > >> JB
> > > >>
> > > >>
> > > >> On 01/24/2017 09:49 PM, Kenneth Knowles wrote:
> > > >>
> > > >>> My impression is that we don't have consensus on whether all checks
> > > >or
> > > >>> minimal checks should be the default, or whether we can have both
> > > >via `mvn
> > > >>> test` and `mvn verify`.
> > > >>>
> > > >>> But that doesn't prevent us from giving -P release a better name
> and
> > > >>> mentioning it in the dev guide and in some manner in our PR
> > > >template.
> > > >>>
> > > >>> Right now we are living with the combination of the bad aspects -
> > > >default
> > > >>> is not thorough but not actually fast and a thorough check is
> > > >>> undocumented.
> > > >>>
> > > >>> On Tue, Jan 24, 2017 at 2:22 AM, Ismaël Mejía <ie...@gmail.com>
> > > >wrote:
> > > >>>
> > > >>> I just wanted to know if we have achieved some consensus about
> this,
> > > >I
> > > >>>> just
> > > >>>> saw this PR that reminded me about this discussion.
> > > >>>>
> > > >>>> https://github.com/apache/beam/pull/1829
> > > >>>>
> > > >>>> It is important that we mention the existing profiles (and the
> > > >intended
> > > >>>> checks) in the contribution guide (e.g. -Prelease (or -Pall-checks
> > > >>>> triggers
> > > >>>> these validations).
> > > >>>>
> > > >>>> I can add this to the guide if you like once we define the checks
> > > >per
> > > >>>> stage/profile.
> > > >>>>
> > > >>>> Ismaël
> > > >>>>
> > > >>>>
> > > >>>> On Wed, Jan 11, 2017 at 8:12 AM, Aviem Zur <av...@gmail.com>
> > > >wrote:
> > > >>>>
> > > >>>> I agree with Dan and Lukasz.
> > > >>>>> Developers should not be expected to know beforehand which
> > > >specific
> > > >>>>> profiles to run.
> > > >>>>> The phase specified in the PR instructions (`verify`) should run
> > > >all the
> > > >>>>> relevant verifications and be the "slower" build, while a
> > > >preceding
> > > >>>>> lifecycle, such as `test`, should run the "faster" verifications.
> > > >>>>>
> > > >>>>> Aviem.
> > > >>>>>
> > > >>>>> On Mon, Jan 9, 2017 at 7:57 PM Robert Bradshaw
> > > >>>>>
> > > >>>> <robertwb@google.com.invalid
> > > >>>>
> > > >>>>>
> > > >>>>>> wrote:
> > > >>>>>
> > > >>>>> On Mon, Jan 9, 2017 at 3:49 AM, Aljoscha Krettek
> > > ><al...@apache.org>
> > > >>>>>> wrote:
> > > >>>>>>
> > > >>>>>>> I also usually prefer "mvn verify" to to the expected thing but
> > > >I see
> > > >>>>>>>
> > > >>>>>> that
> > > >>>>>>
> > > >>>>>>> quick iteration times are key.
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>> I see
> > > >>>>>> https://maven.apache.org/guides/introduction/
> > > >>>>>>
> > > >>>>> introduction-to-the-lifecycle.html
> > > >>>>>
> > > >>>>>>
> > > >>>>>> verify - run any checks on results of integration tests to
> > > >ensure
> > > >>>>>> quality criteria are met
> > > >>>>>>
> > > >>>>>> Of course our integration tests are long enough that we
> shouldn't
> > > >be
> > > >>>>>> putting all of them here, but I too would expect checkstyle.
> > > >>>>>>
> > > >>>>>> Perhaps we could introduce a verify-fast or somesuch for fast
> > > >(but
> > > >>>>>> lower coverage) turnaround time. I would expect "mvn verify
> test"
> > > >to
> > > >>>>>> pass before submitting a PR, and would want to run that before
> > > >asking
> > > >>>>>> others to look at it. I think this should be our criteria (i.e.
> > > >what
> > > >>>>>> will a new but maven-savvy user run before pushing their code).
> > > >>>>>>
> > > >>>>>> As long as the pre-commit hooks still check everything I'm ok
> > > >with
> > > >>>>>>>
> > > >>>>>> making
> > > >>>>>
> > > >>>>>> the default a little more lightweight.
> > > >>>>>>>
> > > >>>>>>
> > > >>>>>> The fact that our pre-commit hooks take a long time to run does
> > > >change
> > > >>>>>> things. Nothing more annoying than seeing that your PR failed 3
> > > >hours
> > > >>>>>> later because you had some trailing whitespace...
> > > >>>>>>
> > > >>>>>> On Thu, 5 Jan 2017 at 21:49 Lukasz Cwik
> > > ><lc...@google.com.invalid>
> > > >>>>>>>
> > > >>>>>> wrote:
> > > >>>>>>
> > > >>>>>>>
> > > >>>>>>> I was hoping that the default mvn verify would be the slow
> build
> > > >>>>>>>>
> > > >>>>>>> and a
> > > >>>>
> > > >>>>> profile could be enabled that would skip checks to make things
> > > >>>>>>>>
> > > >>>>>>> faster
> > > >>>>
> > > >>>>> for
> > > >>>>>>
> > > >>>>>>> regular contributors. This way a person doesn't need to have
> > > >>>>>>>>
> > > >>>>>>> detailed
> > > >>>>
> > > >>>>> knowledge of all our profiles and what they do (typically mvn
> > > >>>>>>>>
> > > >>>>>>> verify)
> > > >>>>
> > > >>>>> will
> > > >>>>>>
> > > >>>>>>> do the right thing most of the time.
> > > >>>>>>>>
> > > >>>>>>>> On Thu, Jan 5, 2017 at 9:30 AM, Dan Halperin
> > > >>>>>>>>
> > > >>>>>>> <dh...@google.com.invalid>
> > > >>>>>>
> > > >>>>>>> wrote:
> > > >>>>>>>>
> > > >>>>>>>> On Thu, Jan 5, 2017 at 9:28 AM, Jesse Anderson <
> > > >>>>>>>>>
> > > >>>>>>>> jesse@smokinghand.com
> > > >>>>>
> > > >>>>>>
> > > >>>>>>> wrote:
> > > >>>>>>>>>
> > > >>>>>>>>> @dan are you saying that mvn verify isn't doing checkstyle
> > > >>>>>>>>>>
> > > >>>>>>>>> anymore?
> > > >>>>>
> > > >>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>> `mvn verify` alone should not be running checkstyle, if
> > > >modules
> > > >>>>>>>>>
> > > >>>>>>>> are
> > > >>>>
> > > >>>>> configured correctly.
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>> Some of
> > > >>>>>>>>>> the checkstyles are still running for a few modules. Also,
> > > >the
> > > >>>>>>>>>>
> > > >>>>>>>>> contribution
> > > >>>>>>>>>
> > > >>>>>>>>>> docs will need to change.
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>> Yes. The PR includes discussion of these other needed
> changes,
> > > >>>>>>>>> unfortunately one PR can't change two repositories.
> > > >>>>>>>>>
> > > >>>>>>>>> Please continue the discussion on the PR, then I will
> > > >summarize it
> > > >>>>>>>>>
> > > >>>>>>>> back
> > > >>>>>>
> > > >>>>>>> into the dev thread.
> > > >>>>>>>>>
> > > >>>>>>>>> Thanks,
> > > >>>>>>>>> Dan
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>> They say to run mvn verify before commits.
> > > >>>>>>>>>>
> > > >>>>>>>>>> On Thu, Jan 5, 2017 at 9:25 AM Dan Halperin
> > > >>>>>>>>>>
> > > >>>>>>>>> <dhalperi@google.com.invalid
> > > >>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>>> wrote:
> > > >>>>>>>>>>
> > > >>>>>>>>>> Several folks seem to have been confused after BEAM-246,
> > > >where
> > > >>>>>>>>>>>
> > > >>>>>>>>>> we
> > > >>>>>
> > > >>>>>> moved
> > > >>>>>>>>
> > > >>>>>>>>> the
> > > >>>>>>>>>>
> > > >>>>>>>>>>> "slow things" into the release profile. I've started a
> > > >>>>>>>>>>>
> > > >>>>>>>>>> discussion
> > > >>>>>
> > > >>>>>> with
> > > >>>>>>>>
> > > >>>>>>>>> https://github.com/apache/beam/pull/1740 to see if there are
> > > >>>>>>>>>>>
> > > >>>>>>>>>> things
> > > >>>>>>
> > > >>>>>>> we
> > > >>>>>>>>
> > > >>>>>>>>> can
> > > >>>>>>>>>>
> > > >>>>>>>>>>> do to fill these gaps.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Would love folks to chime in with opinions.
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> Dan
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> On Wed, Jan 4, 2017 at 1:34 PM, Jesse Anderson <
> > > >>>>>>>>>>>
> > > >>>>>>>>>> jesse@smokinghand.com>
> > > >>>>>>>>
> > > >>>>>>>>> wrote:
> > > >>>>>>>>>>>
> > > >>>>>>>>>>> @Eugene, yes that failed on the checkstyle.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> On Wed, Jan 4, 2017 at 1:27 PM Eugene Kirpichov
> > > >>>>>>>>>>>> <ki...@google.com.invalid> wrote:
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> Try just -Prelease.
> > > >>>>>>>>>>>>> On Wed, Jan 4, 2017 at 1:21 PM Jesse Anderson <
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>> jesse@smokinghand.com
> > > >>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>> Fails because I don't have a secret key.
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> On Wed, Jan 4, 2017 at 1:03 PM Jean-Baptiste Onofré <
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>> jb@nanthrax.net
> > > >>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> wrote:
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> Hi Jesse,
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Could you try the same with:
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> mvn verify -Prelease,apache-release
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> ?
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> Regards
> > > >>>>>>>>>>>>>>> JB
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> On 01/04/2017 09:53 PM, Jesse Anderson wrote:
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> For some reason, running "mvn verify" isn't running
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> checkstyle
> > > >>>>>>>>>
> > > >>>>>>>>>> on
> > > >>>>>>>>>>
> > > >>>>>>>>>>> everything. I had checkstyle errors in
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> beam-sdks-java-core
> > > >>>>>>
> > > >>>>>>> that
> > > >>>>>>>>>
> > > >>>>>>>>>> weren't
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> being found.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> I thought this was due to the extra parameters. I
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> reran
> > > >>>>>
> > > >>>>>> with
> > > >>>>>>>>
> > > >>>>>>>>> the
> > > >>>>>>>>>>
> > > >>>>>>>>>>> plain
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> "mvn
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> verify" and it still didn't find them. From the
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> output,
> > > >>>>>
> > > >>>>>> it
> > > >>>>>>
> > > >>>>>>> doesn't
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>> look
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>>> like they're being run at all.
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Thanks,
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>> Jesse
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>> --
> > > >>>>>>>>>>>>>>> Jean-Baptiste Onofré
> > > >>>>>>>>>>>>>>> jbonofre@apache.org
> > > >>>>>>>>>>>>>>> http://blog.nanthrax.net
> > > >>>>>>>>>>>>>>> Talend - http://www.talend.com
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>>
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>
> > > >>>>>
> > > >>>>
> > > >>>
> > > >> --
> > > >> Jean-Baptiste Onofré
> > > >> jbonofre@apache.org
> > > >> http://blog.nanthrax.net
> > > >> Talend - http://www.talend.com
> > > >>
> > >
> >
>
Re: Better developer instructions for using Maven?
Posted by Aljoscha Krettek <al...@apache.org>.
+1 to what Dan said
On Wed, 25 Jan 2017 at 21:40 Kenneth Knowles <kl...@google.com.invalid> wrote:
> +1
>
> On Jan 25, 2017 11:15, "Jean-Baptiste Onofré" <jb...@nanthrax.net> wrote:
>
> > +1
> >
> > It sounds good to me.
> >
> > Thanks Dan !
> >
> > Regards
> > JB
> >
> > On Jan 25, 2017, 19:39, at 19:39, Dan Halperin
> <dh...@google.com.INVALID>
> > wrote:
> > >Here is my summary of the threads:
> > >
> > >Overwhelming agreement:
> > >
> > >- rename `release` to something more appropriate.
> > >- add `checkstyle` to the default build (it's basically a compile
> > >error)
> > >- add more information to contributor guide
> > >
> > >Reasonable agreement
> > >
> > >- don't update the github instructions to make passing `mvn verify
> > >-P<all
> > >checks>` mandatory. Maybe add a hint that this is a good proxy for what
> > >Jenkins will run.
> > >
> > >Unresolved:
> > >
> > >- whether all checks should be in `mvn verify`
> > >- whether `mvn test` is useful for most workflows
> > >
> > >I'll propose to proceed with the overwhelmingly agreed-upon changes,
> > >and as
> > >we see increasingly many new contributors re-evaluate the remaining
> > >issues.
> > >
> > >Thanks,
> > >Dan
> > >
> > >On Tue, Jan 24, 2017 at 12:51 PM, Jean-Baptiste Onofré
> > ><jb...@nanthrax.net>
> > >wrote:
> > >
> > >> +1 to at least update the contribution guide and improve the profile
> > >name.
> > >>
> > >> Regards
> > >> JB
> > >>
> > >>
> > >> On 01/24/2017 09:49 PM, Kenneth Knowles wrote:
> > >>
> > >>> My impression is that we don't have consensus on whether all checks
> > >or
> > >>> minimal checks should be the default, or whether we can have both
> > >via `mvn
> > >>> test` and `mvn verify`.
> > >>>
> > >>> But that doesn't prevent us from giving -P release a better name and
> > >>> mentioning it in the dev guide and in some manner in our PR
> > >template.
> > >>>
> > >>> Right now we are living with the combination of the bad aspects -
> > >default
> > >>> is not thorough but not actually fast and a thorough check is
> > >>> undocumented.
> > >>>
> > >>> On Tue, Jan 24, 2017 at 2:22 AM, Ismaël Mejía <ie...@gmail.com>
> > >wrote:
> > >>>
> > >>> I just wanted to know if we have achieved some consensus about this,
> > >I
> > >>>> just
> > >>>> saw this PR that reminded me about this discussion.
> > >>>>
> > >>>> https://github.com/apache/beam/pull/1829
> > >>>>
> > >>>> It is important that we mention the existing profiles (and the
> > >intended
> > >>>> checks) in the contribution guide (e.g. -Prelease (or -Pall-checks
> > >>>> triggers
> > >>>> these validations).
> > >>>>
> > >>>> I can add this to the guide if you like once we define the checks
> > >per
> > >>>> stage/profile.
> > >>>>
> > >>>> Ismaël
> > >>>>
> > >>>>
> > >>>> On Wed, Jan 11, 2017 at 8:12 AM, Aviem Zur <av...@gmail.com>
> > >wrote:
> > >>>>
> > >>>> I agree with Dan and Lukasz.
> > >>>>> Developers should not be expected to know beforehand which
> > >specific
> > >>>>> profiles to run.
> > >>>>> The phase specified in the PR instructions (`verify`) should run
> > >all the
> > >>>>> relevant verifications and be the "slower" build, while a
> > >preceding
> > >>>>> lifecycle, such as `test`, should run the "faster" verifications.
> > >>>>>
> > >>>>> Aviem.
> > >>>>>
> > >>>>> On Mon, Jan 9, 2017 at 7:57 PM Robert Bradshaw
> > >>>>>
> > >>>> <robertwb@google.com.invalid
> > >>>>
> > >>>>>
> > >>>>>> wrote:
> > >>>>>
> > >>>>> On Mon, Jan 9, 2017 at 3:49 AM, Aljoscha Krettek
> > ><al...@apache.org>
> > >>>>>> wrote:
> > >>>>>>
> > >>>>>>> I also usually prefer "mvn verify" to to the expected thing but
> > >I see
> > >>>>>>>
> > >>>>>> that
> > >>>>>>
> > >>>>>>> quick iteration times are key.
> > >>>>>>>
> > >>>>>>
> > >>>>>> I see
> > >>>>>> https://maven.apache.org/guides/introduction/
> > >>>>>>
> > >>>>> introduction-to-the-lifecycle.html
> > >>>>>
> > >>>>>>
> > >>>>>> verify - run any checks on results of integration tests to
> > >ensure
> > >>>>>> quality criteria are met
> > >>>>>>
> > >>>>>> Of course our integration tests are long enough that we shouldn't
> > >be
> > >>>>>> putting all of them here, but I too would expect checkstyle.
> > >>>>>>
> > >>>>>> Perhaps we could introduce a verify-fast or somesuch for fast
> > >(but
> > >>>>>> lower coverage) turnaround time. I would expect "mvn verify test"
> > >to
> > >>>>>> pass before submitting a PR, and would want to run that before
> > >asking
> > >>>>>> others to look at it. I think this should be our criteria (i.e.
> > >what
> > >>>>>> will a new but maven-savvy user run before pushing their code).
> > >>>>>>
> > >>>>>> As long as the pre-commit hooks still check everything I'm ok
> > >with
> > >>>>>>>
> > >>>>>> making
> > >>>>>
> > >>>>>> the default a little more lightweight.
> > >>>>>>>
> > >>>>>>
> > >>>>>> The fact that our pre-commit hooks take a long time to run does
> > >change
> > >>>>>> things. Nothing more annoying than seeing that your PR failed 3
> > >hours
> > >>>>>> later because you had some trailing whitespace...
> > >>>>>>
> > >>>>>> On Thu, 5 Jan 2017 at 21:49 Lukasz Cwik
> > ><lc...@google.com.invalid>
> > >>>>>>>
> > >>>>>> wrote:
> > >>>>>>
> > >>>>>>>
> > >>>>>>> I was hoping that the default mvn verify would be the slow build
> > >>>>>>>>
> > >>>>>>> and a
> > >>>>
> > >>>>> profile could be enabled that would skip checks to make things
> > >>>>>>>>
> > >>>>>>> faster
> > >>>>
> > >>>>> for
> > >>>>>>
> > >>>>>>> regular contributors. This way a person doesn't need to have
> > >>>>>>>>
> > >>>>>>> detailed
> > >>>>
> > >>>>> knowledge of all our profiles and what they do (typically mvn
> > >>>>>>>>
> > >>>>>>> verify)
> > >>>>
> > >>>>> will
> > >>>>>>
> > >>>>>>> do the right thing most of the time.
> > >>>>>>>>
> > >>>>>>>> On Thu, Jan 5, 2017 at 9:30 AM, Dan Halperin
> > >>>>>>>>
> > >>>>>>> <dh...@google.com.invalid>
> > >>>>>>
> > >>>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>> On Thu, Jan 5, 2017 at 9:28 AM, Jesse Anderson <
> > >>>>>>>>>
> > >>>>>>>> jesse@smokinghand.com
> > >>>>>
> > >>>>>>
> > >>>>>>> wrote:
> > >>>>>>>>>
> > >>>>>>>>> @dan are you saying that mvn verify isn't doing checkstyle
> > >>>>>>>>>>
> > >>>>>>>>> anymore?
> > >>>>>
> > >>>>>>
> > >>>>>>>>>
> > >>>>>>>>> `mvn verify` alone should not be running checkstyle, if
> > >modules
> > >>>>>>>>>
> > >>>>>>>> are
> > >>>>
> > >>>>> configured correctly.
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> Some of
> > >>>>>>>>>> the checkstyles are still running for a few modules. Also,
> > >the
> > >>>>>>>>>>
> > >>>>>>>>> contribution
> > >>>>>>>>>
> > >>>>>>>>>> docs will need to change.
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> Yes. The PR includes discussion of these other needed changes,
> > >>>>>>>>> unfortunately one PR can't change two repositories.
> > >>>>>>>>>
> > >>>>>>>>> Please continue the discussion on the PR, then I will
> > >summarize it
> > >>>>>>>>>
> > >>>>>>>> back
> > >>>>>>
> > >>>>>>> into the dev thread.
> > >>>>>>>>>
> > >>>>>>>>> Thanks,
> > >>>>>>>>> Dan
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> They say to run mvn verify before commits.
> > >>>>>>>>>>
> > >>>>>>>>>> On Thu, Jan 5, 2017 at 9:25 AM Dan Halperin
> > >>>>>>>>>>
> > >>>>>>>>> <dhalperi@google.com.invalid
> > >>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>> Several folks seem to have been confused after BEAM-246,
> > >where
> > >>>>>>>>>>>
> > >>>>>>>>>> we
> > >>>>>
> > >>>>>> moved
> > >>>>>>>>
> > >>>>>>>>> the
> > >>>>>>>>>>
> > >>>>>>>>>>> "slow things" into the release profile. I've started a
> > >>>>>>>>>>>
> > >>>>>>>>>> discussion
> > >>>>>
> > >>>>>> with
> > >>>>>>>>
> > >>>>>>>>> https://github.com/apache/beam/pull/1740 to see if there are
> > >>>>>>>>>>>
> > >>>>>>>>>> things
> > >>>>>>
> > >>>>>>> we
> > >>>>>>>>
> > >>>>>>>>> can
> > >>>>>>>>>>
> > >>>>>>>>>>> do to fill these gaps.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Would love folks to chime in with opinions.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Dan
> > >>>>>>>>>>>
> > >>>>>>>>>>> On Wed, Jan 4, 2017 at 1:34 PM, Jesse Anderson <
> > >>>>>>>>>>>
> > >>>>>>>>>> jesse@smokinghand.com>
> > >>>>>>>>
> > >>>>>>>>> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>> @Eugene, yes that failed on the checkstyle.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> On Wed, Jan 4, 2017 at 1:27 PM Eugene Kirpichov
> > >>>>>>>>>>>> <ki...@google.com.invalid> wrote:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Try just -Prelease.
> > >>>>>>>>>>>>> On Wed, Jan 4, 2017 at 1:21 PM Jesse Anderson <
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>> jesse@smokinghand.com
> > >>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Fails because I don't have a secret key.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> On Wed, Jan 4, 2017 at 1:03 PM Jean-Baptiste Onofré <
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>> jb@nanthrax.net
> > >>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Hi Jesse,
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Could you try the same with:
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> mvn verify -Prelease,apache-release
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> ?
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Regards
> > >>>>>>>>>>>>>>> JB
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> On 01/04/2017 09:53 PM, Jesse Anderson wrote:
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> For some reason, running "mvn verify" isn't running
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> checkstyle
> > >>>>>>>>>
> > >>>>>>>>>> on
> > >>>>>>>>>>
> > >>>>>>>>>>> everything. I had checkstyle errors in
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> beam-sdks-java-core
> > >>>>>>
> > >>>>>>> that
> > >>>>>>>>>
> > >>>>>>>>>> weren't
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> being found.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> I thought this was due to the extra parameters. I
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> reran
> > >>>>>
> > >>>>>> with
> > >>>>>>>>
> > >>>>>>>>> the
> > >>>>>>>>>>
> > >>>>>>>>>>> plain
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> "mvn
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> verify" and it still didn't find them. From the
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> output,
> > >>>>>
> > >>>>>> it
> > >>>>>>
> > >>>>>>> doesn't
> > >>>>>>>>>>>
> > >>>>>>>>>>>> look
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> like they're being run at all.
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Thanks,
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>> Jesse
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> --
> > >>>>>>>>>>>>>>> Jean-Baptiste Onofré
> > >>>>>>>>>>>>>>> jbonofre@apache.org
> > >>>>>>>>>>>>>>> http://blog.nanthrax.net
> > >>>>>>>>>>>>>>> Talend - http://www.talend.com
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>
> > >> --
> > >> Jean-Baptiste Onofré
> > >> jbonofre@apache.org
> > >> http://blog.nanthrax.net
> > >> Talend - http://www.talend.com
> > >>
> >
>
Re: Better developer instructions for using Maven?
Posted by Kenneth Knowles <kl...@google.com.INVALID>.
+1
On Jan 25, 2017 11:15, "Jean-Baptiste Onofré" <jb...@nanthrax.net> wrote:
> +1
>
> It sounds good to me.
>
> Thanks Dan !
>
> Regards
> JB
>
> On Jan 25, 2017, 19:39, at 19:39, Dan Halperin <dh...@google.com.INVALID>
> wrote:
> >Here is my summary of the threads:
> >
> >Overwhelming agreement:
> >
> >- rename `release` to something more appropriate.
> >- add `checkstyle` to the default build (it's basically a compile
> >error)
> >- add more information to contributor guide
> >
> >Reasonable agreement
> >
> >- don't update the github instructions to make passing `mvn verify
> >-P<all
> >checks>` mandatory. Maybe add a hint that this is a good proxy for what
> >Jenkins will run.
> >
> >Unresolved:
> >
> >- whether all checks should be in `mvn verify`
> >- whether `mvn test` is useful for most workflows
> >
> >I'll propose to proceed with the overwhelmingly agreed-upon changes,
> >and as
> >we see increasingly many new contributors re-evaluate the remaining
> >issues.
> >
> >Thanks,
> >Dan
> >
> >On Tue, Jan 24, 2017 at 12:51 PM, Jean-Baptiste Onofré
> ><jb...@nanthrax.net>
> >wrote:
> >
> >> +1 to at least update the contribution guide and improve the profile
> >name.
> >>
> >> Regards
> >> JB
> >>
> >>
> >> On 01/24/2017 09:49 PM, Kenneth Knowles wrote:
> >>
> >>> My impression is that we don't have consensus on whether all checks
> >or
> >>> minimal checks should be the default, or whether we can have both
> >via `mvn
> >>> test` and `mvn verify`.
> >>>
> >>> But that doesn't prevent us from giving -P release a better name and
> >>> mentioning it in the dev guide and in some manner in our PR
> >template.
> >>>
> >>> Right now we are living with the combination of the bad aspects -
> >default
> >>> is not thorough but not actually fast and a thorough check is
> >>> undocumented.
> >>>
> >>> On Tue, Jan 24, 2017 at 2:22 AM, Ismaël Mejía <ie...@gmail.com>
> >wrote:
> >>>
> >>> I just wanted to know if we have achieved some consensus about this,
> >I
> >>>> just
> >>>> saw this PR that reminded me about this discussion.
> >>>>
> >>>> https://github.com/apache/beam/pull/1829
> >>>>
> >>>> It is important that we mention the existing profiles (and the
> >intended
> >>>> checks) in the contribution guide (e.g. -Prelease (or -Pall-checks
> >>>> triggers
> >>>> these validations).
> >>>>
> >>>> I can add this to the guide if you like once we define the checks
> >per
> >>>> stage/profile.
> >>>>
> >>>> Ismaël
> >>>>
> >>>>
> >>>> On Wed, Jan 11, 2017 at 8:12 AM, Aviem Zur <av...@gmail.com>
> >wrote:
> >>>>
> >>>> I agree with Dan and Lukasz.
> >>>>> Developers should not be expected to know beforehand which
> >specific
> >>>>> profiles to run.
> >>>>> The phase specified in the PR instructions (`verify`) should run
> >all the
> >>>>> relevant verifications and be the "slower" build, while a
> >preceding
> >>>>> lifecycle, such as `test`, should run the "faster" verifications.
> >>>>>
> >>>>> Aviem.
> >>>>>
> >>>>> On Mon, Jan 9, 2017 at 7:57 PM Robert Bradshaw
> >>>>>
> >>>> <robertwb@google.com.invalid
> >>>>
> >>>>>
> >>>>>> wrote:
> >>>>>
> >>>>> On Mon, Jan 9, 2017 at 3:49 AM, Aljoscha Krettek
> ><al...@apache.org>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> I also usually prefer "mvn verify" to to the expected thing but
> >I see
> >>>>>>>
> >>>>>> that
> >>>>>>
> >>>>>>> quick iteration times are key.
> >>>>>>>
> >>>>>>
> >>>>>> I see
> >>>>>> https://maven.apache.org/guides/introduction/
> >>>>>>
> >>>>> introduction-to-the-lifecycle.html
> >>>>>
> >>>>>>
> >>>>>> verify - run any checks on results of integration tests to
> >ensure
> >>>>>> quality criteria are met
> >>>>>>
> >>>>>> Of course our integration tests are long enough that we shouldn't
> >be
> >>>>>> putting all of them here, but I too would expect checkstyle.
> >>>>>>
> >>>>>> Perhaps we could introduce a verify-fast or somesuch for fast
> >(but
> >>>>>> lower coverage) turnaround time. I would expect "mvn verify test"
> >to
> >>>>>> pass before submitting a PR, and would want to run that before
> >asking
> >>>>>> others to look at it. I think this should be our criteria (i.e.
> >what
> >>>>>> will a new but maven-savvy user run before pushing their code).
> >>>>>>
> >>>>>> As long as the pre-commit hooks still check everything I'm ok
> >with
> >>>>>>>
> >>>>>> making
> >>>>>
> >>>>>> the default a little more lightweight.
> >>>>>>>
> >>>>>>
> >>>>>> The fact that our pre-commit hooks take a long time to run does
> >change
> >>>>>> things. Nothing more annoying than seeing that your PR failed 3
> >hours
> >>>>>> later because you had some trailing whitespace...
> >>>>>>
> >>>>>> On Thu, 5 Jan 2017 at 21:49 Lukasz Cwik
> ><lc...@google.com.invalid>
> >>>>>>>
> >>>>>> wrote:
> >>>>>>
> >>>>>>>
> >>>>>>> I was hoping that the default mvn verify would be the slow build
> >>>>>>>>
> >>>>>>> and a
> >>>>
> >>>>> profile could be enabled that would skip checks to make things
> >>>>>>>>
> >>>>>>> faster
> >>>>
> >>>>> for
> >>>>>>
> >>>>>>> regular contributors. This way a person doesn't need to have
> >>>>>>>>
> >>>>>>> detailed
> >>>>
> >>>>> knowledge of all our profiles and what they do (typically mvn
> >>>>>>>>
> >>>>>>> verify)
> >>>>
> >>>>> will
> >>>>>>
> >>>>>>> do the right thing most of the time.
> >>>>>>>>
> >>>>>>>> On Thu, Jan 5, 2017 at 9:30 AM, Dan Halperin
> >>>>>>>>
> >>>>>>> <dh...@google.com.invalid>
> >>>>>>
> >>>>>>> wrote:
> >>>>>>>>
> >>>>>>>> On Thu, Jan 5, 2017 at 9:28 AM, Jesse Anderson <
> >>>>>>>>>
> >>>>>>>> jesse@smokinghand.com
> >>>>>
> >>>>>>
> >>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>> @dan are you saying that mvn verify isn't doing checkstyle
> >>>>>>>>>>
> >>>>>>>>> anymore?
> >>>>>
> >>>>>>
> >>>>>>>>>
> >>>>>>>>> `mvn verify` alone should not be running checkstyle, if
> >modules
> >>>>>>>>>
> >>>>>>>> are
> >>>>
> >>>>> configured correctly.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Some of
> >>>>>>>>>> the checkstyles are still running for a few modules. Also,
> >the
> >>>>>>>>>>
> >>>>>>>>> contribution
> >>>>>>>>>
> >>>>>>>>>> docs will need to change.
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Yes. The PR includes discussion of these other needed changes,
> >>>>>>>>> unfortunately one PR can't change two repositories.
> >>>>>>>>>
> >>>>>>>>> Please continue the discussion on the PR, then I will
> >summarize it
> >>>>>>>>>
> >>>>>>>> back
> >>>>>>
> >>>>>>> into the dev thread.
> >>>>>>>>>
> >>>>>>>>> Thanks,
> >>>>>>>>> Dan
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> They say to run mvn verify before commits.
> >>>>>>>>>>
> >>>>>>>>>> On Thu, Jan 5, 2017 at 9:25 AM Dan Halperin
> >>>>>>>>>>
> >>>>>>>>> <dhalperi@google.com.invalid
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>> Several folks seem to have been confused after BEAM-246,
> >where
> >>>>>>>>>>>
> >>>>>>>>>> we
> >>>>>
> >>>>>> moved
> >>>>>>>>
> >>>>>>>>> the
> >>>>>>>>>>
> >>>>>>>>>>> "slow things" into the release profile. I've started a
> >>>>>>>>>>>
> >>>>>>>>>> discussion
> >>>>>
> >>>>>> with
> >>>>>>>>
> >>>>>>>>> https://github.com/apache/beam/pull/1740 to see if there are
> >>>>>>>>>>>
> >>>>>>>>>> things
> >>>>>>
> >>>>>>> we
> >>>>>>>>
> >>>>>>>>> can
> >>>>>>>>>>
> >>>>>>>>>>> do to fill these gaps.
> >>>>>>>>>>>
> >>>>>>>>>>> Would love folks to chime in with opinions.
> >>>>>>>>>>>
> >>>>>>>>>>> Dan
> >>>>>>>>>>>
> >>>>>>>>>>> On Wed, Jan 4, 2017 at 1:34 PM, Jesse Anderson <
> >>>>>>>>>>>
> >>>>>>>>>> jesse@smokinghand.com>
> >>>>>>>>
> >>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> @Eugene, yes that failed on the checkstyle.
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Wed, Jan 4, 2017 at 1:27 PM Eugene Kirpichov
> >>>>>>>>>>>> <ki...@google.com.invalid> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>> Try just -Prelease.
> >>>>>>>>>>>>> On Wed, Jan 4, 2017 at 1:21 PM Jesse Anderson <
> >>>>>>>>>>>>>
> >>>>>>>>>>>> jesse@smokinghand.com
> >>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Fails because I don't have a secret key.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Wed, Jan 4, 2017 at 1:03 PM Jean-Baptiste Onofré <
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>> jb@nanthrax.net
> >>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Hi Jesse,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Could you try the same with:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> mvn verify -Prelease,apache-release
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> ?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Regards
> >>>>>>>>>>>>>>> JB
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On 01/04/2017 09:53 PM, Jesse Anderson wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> For some reason, running "mvn verify" isn't running
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> checkstyle
> >>>>>>>>>
> >>>>>>>>>> on
> >>>>>>>>>>
> >>>>>>>>>>> everything. I had checkstyle errors in
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> beam-sdks-java-core
> >>>>>>
> >>>>>>> that
> >>>>>>>>>
> >>>>>>>>>> weren't
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> being found.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> I thought this was due to the extra parameters. I
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> reran
> >>>>>
> >>>>>> with
> >>>>>>>>
> >>>>>>>>> the
> >>>>>>>>>>
> >>>>>>>>>>> plain
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> "mvn
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> verify" and it still didn't find them. From the
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> output,
> >>>>>
> >>>>>> it
> >>>>>>
> >>>>>>> doesn't
> >>>>>>>>>>>
> >>>>>>>>>>>> look
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> like they're being run at all.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Jesse
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>> Jean-Baptiste Onofré
> >>>>>>>>>>>>>>> jbonofre@apache.org
> >>>>>>>>>>>>>>> http://blog.nanthrax.net
> >>>>>>>>>>>>>>> Talend - http://www.talend.com
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >> --
> >> Jean-Baptiste Onofré
> >> jbonofre@apache.org
> >> http://blog.nanthrax.net
> >> Talend - http://www.talend.com
> >>
>
Re: Better developer instructions for using Maven?
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
+1
It sounds good to me.
Thanks Dan !
Regards
JB\u2063\u200b
On Jan 25, 2017, 19:39, at 19:39, Dan Halperin <dh...@google.com.INVALID> wrote:
>Here is my summary of the threads:
>
>Overwhelming agreement:
>
>- rename `release` to something more appropriate.
>- add `checkstyle` to the default build (it's basically a compile
>error)
>- add more information to contributor guide
>
>Reasonable agreement
>
>- don't update the github instructions to make passing `mvn verify
>-P<all
>checks>` mandatory. Maybe add a hint that this is a good proxy for what
>Jenkins will run.
>
>Unresolved:
>
>- whether all checks should be in `mvn verify`
>- whether `mvn test` is useful for most workflows
>
>I'll propose to proceed with the overwhelmingly agreed-upon changes,
>and as
>we see increasingly many new contributors re-evaluate the remaining
>issues.
>
>Thanks,
>Dan
>
>On Tue, Jan 24, 2017 at 12:51 PM, Jean-Baptiste Onofr�
><jb...@nanthrax.net>
>wrote:
>
>> +1 to at least update the contribution guide and improve the profile
>name.
>>
>> Regards
>> JB
>>
>>
>> On 01/24/2017 09:49 PM, Kenneth Knowles wrote:
>>
>>> My impression is that we don't have consensus on whether all checks
>or
>>> minimal checks should be the default, or whether we can have both
>via `mvn
>>> test` and `mvn verify`.
>>>
>>> But that doesn't prevent us from giving -P release a better name and
>>> mentioning it in the dev guide and in some manner in our PR
>template.
>>>
>>> Right now we are living with the combination of the bad aspects -
>default
>>> is not thorough but not actually fast and a thorough check is
>>> undocumented.
>>>
>>> On Tue, Jan 24, 2017 at 2:22 AM, Isma�l Mej�a <ie...@gmail.com>
>wrote:
>>>
>>> I just wanted to know if we have achieved some consensus about this,
>I
>>>> just
>>>> saw this PR that reminded me about this discussion.
>>>>
>>>> \u200bhttps://github.com/apache/beam/pull/1829\u200b
>>>>
>>>> It is important that we mention the existing profiles (and the
>intended
>>>> checks) in the contribution guide (e.g. -Prelease (or -Pall-checks
>>>> triggers
>>>> these validations).
>>>>
>>>> I can add this to the guide if you like once we define the checks
>per
>>>> stage/profile.
>>>>
>>>> Isma�l
>>>>
>>>>
>>>> On Wed, Jan 11, 2017 at 8:12 AM, Aviem Zur <av...@gmail.com>
>wrote:
>>>>
>>>> I agree with Dan and Lukasz.
>>>>> Developers should not be expected to know beforehand which
>specific
>>>>> profiles to run.
>>>>> The phase specified in the PR instructions (`verify`) should run
>all the
>>>>> relevant verifications and be the "slower" build, while a
>preceding
>>>>> lifecycle, such as `test`, should run the "faster" verifications.
>>>>>
>>>>> Aviem.
>>>>>
>>>>> On Mon, Jan 9, 2017 at 7:57 PM Robert Bradshaw
>>>>>
>>>> <robertwb@google.com.invalid
>>>>
>>>>>
>>>>>> wrote:
>>>>>
>>>>> On Mon, Jan 9, 2017 at 3:49 AM, Aljoscha Krettek
><al...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>>> I also usually prefer "mvn verify" to to the expected thing but
>I see
>>>>>>>
>>>>>> that
>>>>>>
>>>>>>> quick iteration times are key.
>>>>>>>
>>>>>>
>>>>>> I see
>>>>>> https://maven.apache.org/guides/introduction/
>>>>>>
>>>>> introduction-to-the-lifecycle.html
>>>>>
>>>>>>
>>>>>> verify - run any checks on results of integration tests to
>ensure
>>>>>> quality criteria are met
>>>>>>
>>>>>> Of course our integration tests are long enough that we shouldn't
>be
>>>>>> putting all of them here, but I too would expect checkstyle.
>>>>>>
>>>>>> Perhaps we could introduce a verify-fast or somesuch for fast
>(but
>>>>>> lower coverage) turnaround time. I would expect "mvn verify test"
>to
>>>>>> pass before submitting a PR, and would want to run that before
>asking
>>>>>> others to look at it. I think this should be our criteria (i.e.
>what
>>>>>> will a new but maven-savvy user run before pushing their code).
>>>>>>
>>>>>> As long as the pre-commit hooks still check everything I'm ok
>with
>>>>>>>
>>>>>> making
>>>>>
>>>>>> the default a little more lightweight.
>>>>>>>
>>>>>>
>>>>>> The fact that our pre-commit hooks take a long time to run does
>change
>>>>>> things. Nothing more annoying than seeing that your PR failed 3
>hours
>>>>>> later because you had some trailing whitespace...
>>>>>>
>>>>>> On Thu, 5 Jan 2017 at 21:49 Lukasz Cwik
><lc...@google.com.invalid>
>>>>>>>
>>>>>> wrote:
>>>>>>
>>>>>>>
>>>>>>> I was hoping that the default mvn verify would be the slow build
>>>>>>>>
>>>>>>> and a
>>>>
>>>>> profile could be enabled that would skip checks to make things
>>>>>>>>
>>>>>>> faster
>>>>
>>>>> for
>>>>>>
>>>>>>> regular contributors. This way a person doesn't need to have
>>>>>>>>
>>>>>>> detailed
>>>>
>>>>> knowledge of all our profiles and what they do (typically mvn
>>>>>>>>
>>>>>>> verify)
>>>>
>>>>> will
>>>>>>
>>>>>>> do the right thing most of the time.
>>>>>>>>
>>>>>>>> On Thu, Jan 5, 2017 at 9:30 AM, Dan Halperin
>>>>>>>>
>>>>>>> <dh...@google.com.invalid>
>>>>>>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> On Thu, Jan 5, 2017 at 9:28 AM, Jesse Anderson <
>>>>>>>>>
>>>>>>>> jesse@smokinghand.com
>>>>>
>>>>>>
>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> @dan are you saying that mvn verify isn't doing checkstyle
>>>>>>>>>>
>>>>>>>>> anymore?
>>>>>
>>>>>>
>>>>>>>>>
>>>>>>>>> `mvn verify` alone should not be running checkstyle, if
>modules
>>>>>>>>>
>>>>>>>> are
>>>>
>>>>> configured correctly.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Some of
>>>>>>>>>> the checkstyles are still running for a few modules. Also,
>the
>>>>>>>>>>
>>>>>>>>> contribution
>>>>>>>>>
>>>>>>>>>> docs will need to change.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Yes. The PR includes discussion of these other needed changes,
>>>>>>>>> unfortunately one PR can't change two repositories.
>>>>>>>>>
>>>>>>>>> Please continue the discussion on the PR, then I will
>summarize it
>>>>>>>>>
>>>>>>>> back
>>>>>>
>>>>>>> into the dev thread.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Dan
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> They say to run mvn verify before commits.
>>>>>>>>>>
>>>>>>>>>> On Thu, Jan 5, 2017 at 9:25 AM Dan Halperin
>>>>>>>>>>
>>>>>>>>> <dhalperi@google.com.invalid
>>>>>>>>
>>>>>>>>>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> Several folks seem to have been confused after BEAM-246,
>where
>>>>>>>>>>>
>>>>>>>>>> we
>>>>>
>>>>>> moved
>>>>>>>>
>>>>>>>>> the
>>>>>>>>>>
>>>>>>>>>>> "slow things" into the release profile. I've started a
>>>>>>>>>>>
>>>>>>>>>> discussion
>>>>>
>>>>>> with
>>>>>>>>
>>>>>>>>> https://github.com/apache/beam/pull/1740 to see if there are
>>>>>>>>>>>
>>>>>>>>>> things
>>>>>>
>>>>>>> we
>>>>>>>>
>>>>>>>>> can
>>>>>>>>>>
>>>>>>>>>>> do to fill these gaps.
>>>>>>>>>>>
>>>>>>>>>>> Would love folks to chime in with opinions.
>>>>>>>>>>>
>>>>>>>>>>> Dan
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Jan 4, 2017 at 1:34 PM, Jesse Anderson <
>>>>>>>>>>>
>>>>>>>>>> jesse@smokinghand.com>
>>>>>>>>
>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> @Eugene, yes that failed on the checkstyle.
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Jan 4, 2017 at 1:27 PM Eugene Kirpichov
>>>>>>>>>>>> <ki...@google.com.invalid> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Try just -Prelease.
>>>>>>>>>>>>> On Wed, Jan 4, 2017 at 1:21 PM Jesse Anderson <
>>>>>>>>>>>>>
>>>>>>>>>>>> jesse@smokinghand.com
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Fails because I don't have a secret key.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Wed, Jan 4, 2017 at 1:03 PM Jean-Baptiste Onofr� <
>>>>>>>>>>>>>>
>>>>>>>>>>>>> jb@nanthrax.net
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Jesse,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Could you try the same with:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> mvn verify -Prelease,apache-release
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> ?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Regards
>>>>>>>>>>>>>>> JB
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 01/04/2017 09:53 PM, Jesse Anderson wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> For some reason, running "mvn verify" isn't running
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> checkstyle
>>>>>>>>>
>>>>>>>>>> on
>>>>>>>>>>
>>>>>>>>>>> everything. I had checkstyle errors in
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> beam-sdks-java-core
>>>>>>
>>>>>>> that
>>>>>>>>>
>>>>>>>>>> weren't
>>>>>>>>>>>>>
>>>>>>>>>>>>>> being found.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I thought this was due to the extra parameters. I
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> reran
>>>>>
>>>>>> with
>>>>>>>>
>>>>>>>>> the
>>>>>>>>>>
>>>>>>>>>>> plain
>>>>>>>>>>>>>
>>>>>>>>>>>>>> "mvn
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> verify" and it still didn't find them. From the
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> output,
>>>>>
>>>>>> it
>>>>>>
>>>>>>> doesn't
>>>>>>>>>>>
>>>>>>>>>>>> look
>>>>>>>>>>>>>
>>>>>>>>>>>>>> like they're being run at all.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Jesse
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> Jean-Baptiste Onofr�
>>>>>>>>>>>>>>> jbonofre@apache.org
>>>>>>>>>>>>>>> http://blog.nanthrax.net
>>>>>>>>>>>>>>> Talend - http://www.talend.com
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>> --
>> Jean-Baptiste Onofr�
>> jbonofre@apache.org
>> http://blog.nanthrax.net
>> Talend - http://www.talend.com
>>
Re: Better developer instructions for using Maven?
Posted by Jason Kuster <ja...@google.com.INVALID>.
+1
On Wed, Jan 25, 2017 at 10:38 AM, Dan Halperin <dh...@google.com.invalid>
wrote:
> Here is my summary of the threads:
>
> Overwhelming agreement:
>
> - rename `release` to something more appropriate.
> - add `checkstyle` to the default build (it's basically a compile error)
> - add more information to contributor guide
>
> Reasonable agreement
>
> - don't update the github instructions to make passing `mvn verify -P<all
> checks>` mandatory. Maybe add a hint that this is a good proxy for what
> Jenkins will run.
>
> Unresolved:
>
> - whether all checks should be in `mvn verify`
> - whether `mvn test` is useful for most workflows
>
> I'll propose to proceed with the overwhelmingly agreed-upon changes, and as
> we see increasingly many new contributors re-evaluate the remaining issues.
>
> Thanks,
> Dan
>
> On Tue, Jan 24, 2017 at 12:51 PM, Jean-Baptiste Onofré <jb...@nanthrax.net>
> wrote:
>
> > +1 to at least update the contribution guide and improve the profile
> name.
> >
> > Regards
> > JB
> >
> >
> > On 01/24/2017 09:49 PM, Kenneth Knowles wrote:
> >
> >> My impression is that we don't have consensus on whether all checks or
> >> minimal checks should be the default, or whether we can have both via
> `mvn
> >> test` and `mvn verify`.
> >>
> >> But that doesn't prevent us from giving -P release a better name and
> >> mentioning it in the dev guide and in some manner in our PR template.
> >>
> >> Right now we are living with the combination of the bad aspects -
> default
> >> is not thorough but not actually fast and a thorough check is
> >> undocumented.
> >>
> >> On Tue, Jan 24, 2017 at 2:22 AM, Ismaël Mejía <ie...@gmail.com>
> wrote:
> >>
> >> I just wanted to know if we have achieved some consensus about this, I
> >>> just
> >>> saw this PR that reminded me about this discussion.
> >>>
> >>> https://github.com/apache/beam/pull/1829
> >>>
> >>> It is important that we mention the existing profiles (and the intended
> >>> checks) in the contribution guide (e.g. -Prelease (or -Pall-checks
> >>> triggers
> >>> these validations).
> >>>
> >>> I can add this to the guide if you like once we define the checks per
> >>> stage/profile.
> >>>
> >>> Ismaël
> >>>
> >>>
> >>> On Wed, Jan 11, 2017 at 8:12 AM, Aviem Zur <av...@gmail.com> wrote:
> >>>
> >>> I agree with Dan and Lukasz.
> >>>> Developers should not be expected to know beforehand which specific
> >>>> profiles to run.
> >>>> The phase specified in the PR instructions (`verify`) should run all
> the
> >>>> relevant verifications and be the "slower" build, while a preceding
> >>>> lifecycle, such as `test`, should run the "faster" verifications.
> >>>>
> >>>> Aviem.
> >>>>
> >>>> On Mon, Jan 9, 2017 at 7:57 PM Robert Bradshaw
> >>>>
> >>> <robertwb@google.com.invalid
> >>>
> >>>>
> >>>>> wrote:
> >>>>
> >>>> On Mon, Jan 9, 2017 at 3:49 AM, Aljoscha Krettek <aljoscha@apache.org
> >
> >>>>> wrote:
> >>>>>
> >>>>>> I also usually prefer "mvn verify" to to the expected thing but I
> see
> >>>>>>
> >>>>> that
> >>>>>
> >>>>>> quick iteration times are key.
> >>>>>>
> >>>>>
> >>>>> I see
> >>>>> https://maven.apache.org/guides/introduction/
> >>>>>
> >>>> introduction-to-the-lifecycle.html
> >>>>
> >>>>>
> >>>>> verify - run any checks on results of integration tests to ensure
> >>>>> quality criteria are met
> >>>>>
> >>>>> Of course our integration tests are long enough that we shouldn't be
> >>>>> putting all of them here, but I too would expect checkstyle.
> >>>>>
> >>>>> Perhaps we could introduce a verify-fast or somesuch for fast (but
> >>>>> lower coverage) turnaround time. I would expect "mvn verify test" to
> >>>>> pass before submitting a PR, and would want to run that before asking
> >>>>> others to look at it. I think this should be our criteria (i.e. what
> >>>>> will a new but maven-savvy user run before pushing their code).
> >>>>>
> >>>>> As long as the pre-commit hooks still check everything I'm ok with
> >>>>>>
> >>>>> making
> >>>>
> >>>>> the default a little more lightweight.
> >>>>>>
> >>>>>
> >>>>> The fact that our pre-commit hooks take a long time to run does
> change
> >>>>> things. Nothing more annoying than seeing that your PR failed 3 hours
> >>>>> later because you had some trailing whitespace...
> >>>>>
> >>>>> On Thu, 5 Jan 2017 at 21:49 Lukasz Cwik <lc...@google.com.invalid>
> >>>>>>
> >>>>> wrote:
> >>>>>
> >>>>>>
> >>>>>> I was hoping that the default mvn verify would be the slow build
> >>>>>>>
> >>>>>> and a
> >>>
> >>>> profile could be enabled that would skip checks to make things
> >>>>>>>
> >>>>>> faster
> >>>
> >>>> for
> >>>>>
> >>>>>> regular contributors. This way a person doesn't need to have
> >>>>>>>
> >>>>>> detailed
> >>>
> >>>> knowledge of all our profiles and what they do (typically mvn
> >>>>>>>
> >>>>>> verify)
> >>>
> >>>> will
> >>>>>
> >>>>>> do the right thing most of the time.
> >>>>>>>
> >>>>>>> On Thu, Jan 5, 2017 at 9:30 AM, Dan Halperin
> >>>>>>>
> >>>>>> <dh...@google.com.invalid>
> >>>>>
> >>>>>> wrote:
> >>>>>>>
> >>>>>>> On Thu, Jan 5, 2017 at 9:28 AM, Jesse Anderson <
> >>>>>>>>
> >>>>>>> jesse@smokinghand.com
> >>>>
> >>>>>
> >>>>>> wrote:
> >>>>>>>>
> >>>>>>>> @dan are you saying that mvn verify isn't doing checkstyle
> >>>>>>>>>
> >>>>>>>> anymore?
> >>>>
> >>>>>
> >>>>>>>>
> >>>>>>>> `mvn verify` alone should not be running checkstyle, if modules
> >>>>>>>>
> >>>>>>> are
> >>>
> >>>> configured correctly.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Some of
> >>>>>>>>> the checkstyles are still running for a few modules. Also, the
> >>>>>>>>>
> >>>>>>>> contribution
> >>>>>>>>
> >>>>>>>>> docs will need to change.
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Yes. The PR includes discussion of these other needed changes,
> >>>>>>>> unfortunately one PR can't change two repositories.
> >>>>>>>>
> >>>>>>>> Please continue the discussion on the PR, then I will summarize it
> >>>>>>>>
> >>>>>>> back
> >>>>>
> >>>>>> into the dev thread.
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>> Dan
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> They say to run mvn verify before commits.
> >>>>>>>>>
> >>>>>>>>> On Thu, Jan 5, 2017 at 9:25 AM Dan Halperin
> >>>>>>>>>
> >>>>>>>> <dhalperi@google.com.invalid
> >>>>>>>
> >>>>>>>>
> >>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>> Several folks seem to have been confused after BEAM-246, where
> >>>>>>>>>>
> >>>>>>>>> we
> >>>>
> >>>>> moved
> >>>>>>>
> >>>>>>>> the
> >>>>>>>>>
> >>>>>>>>>> "slow things" into the release profile. I've started a
> >>>>>>>>>>
> >>>>>>>>> discussion
> >>>>
> >>>>> with
> >>>>>>>
> >>>>>>>> https://github.com/apache/beam/pull/1740 to see if there are
> >>>>>>>>>>
> >>>>>>>>> things
> >>>>>
> >>>>>> we
> >>>>>>>
> >>>>>>>> can
> >>>>>>>>>
> >>>>>>>>>> do to fill these gaps.
> >>>>>>>>>>
> >>>>>>>>>> Would love folks to chime in with opinions.
> >>>>>>>>>>
> >>>>>>>>>> Dan
> >>>>>>>>>>
> >>>>>>>>>> On Wed, Jan 4, 2017 at 1:34 PM, Jesse Anderson <
> >>>>>>>>>>
> >>>>>>>>> jesse@smokinghand.com>
> >>>>>>>
> >>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>> @Eugene, yes that failed on the checkstyle.
> >>>>>>>>>>>
> >>>>>>>>>>> On Wed, Jan 4, 2017 at 1:27 PM Eugene Kirpichov
> >>>>>>>>>>> <ki...@google.com.invalid> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> Try just -Prelease.
> >>>>>>>>>>>> On Wed, Jan 4, 2017 at 1:21 PM Jesse Anderson <
> >>>>>>>>>>>>
> >>>>>>>>>>> jesse@smokinghand.com
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>> Fails because I don't have a secret key.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Wed, Jan 4, 2017 at 1:03 PM Jean-Baptiste Onofré <
> >>>>>>>>>>>>>
> >>>>>>>>>>>> jb@nanthrax.net
> >>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Hi Jesse,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Could you try the same with:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> mvn verify -Prelease,apache-release
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> ?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Regards
> >>>>>>>>>>>>>> JB
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On 01/04/2017 09:53 PM, Jesse Anderson wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> For some reason, running "mvn verify" isn't running
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>> checkstyle
> >>>>>>>>
> >>>>>>>>> on
> >>>>>>>>>
> >>>>>>>>>> everything. I had checkstyle errors in
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>> beam-sdks-java-core
> >>>>>
> >>>>>> that
> >>>>>>>>
> >>>>>>>>> weren't
> >>>>>>>>>>>>
> >>>>>>>>>>>>> being found.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I thought this was due to the extra parameters. I
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>> reran
> >>>>
> >>>>> with
> >>>>>>>
> >>>>>>>> the
> >>>>>>>>>
> >>>>>>>>>> plain
> >>>>>>>>>>>>
> >>>>>>>>>>>>> "mvn
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> verify" and it still didn't find them. From the
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>> output,
> >>>>
> >>>>> it
> >>>>>
> >>>>>> doesn't
> >>>>>>>>>>
> >>>>>>>>>>> look
> >>>>>>>>>>>>
> >>>>>>>>>>>>> like they're being run at all.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Jesse
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>> --
> >>>>>>>>>>>>>> Jean-Baptiste Onofré
> >>>>>>>>>>>>>> jbonofre@apache.org
> >>>>>>>>>>>>>> http://blog.nanthrax.net
> >>>>>>>>>>>>>> Talend - http://www.talend.com
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>>
> >>>
> >>
> > --
> > Jean-Baptiste Onofré
> > jbonofre@apache.org
> > http://blog.nanthrax.net
> > Talend - http://www.talend.com
> >
>
--
-------
Jason Kuster
Apache Beam (Incubating) / Google Cloud Dataflow
Re: Better developer instructions for using Maven?
Posted by Dan Halperin <dh...@google.com.INVALID>.
Here is my summary of the threads:
Overwhelming agreement:
- rename `release` to something more appropriate.
- add `checkstyle` to the default build (it's basically a compile error)
- add more information to contributor guide
Reasonable agreement
- don't update the github instructions to make passing `mvn verify -P<all
checks>` mandatory. Maybe add a hint that this is a good proxy for what
Jenkins will run.
Unresolved:
- whether all checks should be in `mvn verify`
- whether `mvn test` is useful for most workflows
I'll propose to proceed with the overwhelmingly agreed-upon changes, and as
we see increasingly many new contributors re-evaluate the remaining issues.
Thanks,
Dan
On Tue, Jan 24, 2017 at 12:51 PM, Jean-Baptiste Onofré <jb...@nanthrax.net>
wrote:
> +1 to at least update the contribution guide and improve the profile name.
>
> Regards
> JB
>
>
> On 01/24/2017 09:49 PM, Kenneth Knowles wrote:
>
>> My impression is that we don't have consensus on whether all checks or
>> minimal checks should be the default, or whether we can have both via `mvn
>> test` and `mvn verify`.
>>
>> But that doesn't prevent us from giving -P release a better name and
>> mentioning it in the dev guide and in some manner in our PR template.
>>
>> Right now we are living with the combination of the bad aspects - default
>> is not thorough but not actually fast and a thorough check is
>> undocumented.
>>
>> On Tue, Jan 24, 2017 at 2:22 AM, Ismaël Mejía <ie...@gmail.com> wrote:
>>
>> I just wanted to know if we have achieved some consensus about this, I
>>> just
>>> saw this PR that reminded me about this discussion.
>>>
>>> https://github.com/apache/beam/pull/1829
>>>
>>> It is important that we mention the existing profiles (and the intended
>>> checks) in the contribution guide (e.g. -Prelease (or -Pall-checks
>>> triggers
>>> these validations).
>>>
>>> I can add this to the guide if you like once we define the checks per
>>> stage/profile.
>>>
>>> Ismaël
>>>
>>>
>>> On Wed, Jan 11, 2017 at 8:12 AM, Aviem Zur <av...@gmail.com> wrote:
>>>
>>> I agree with Dan and Lukasz.
>>>> Developers should not be expected to know beforehand which specific
>>>> profiles to run.
>>>> The phase specified in the PR instructions (`verify`) should run all the
>>>> relevant verifications and be the "slower" build, while a preceding
>>>> lifecycle, such as `test`, should run the "faster" verifications.
>>>>
>>>> Aviem.
>>>>
>>>> On Mon, Jan 9, 2017 at 7:57 PM Robert Bradshaw
>>>>
>>> <robertwb@google.com.invalid
>>>
>>>>
>>>>> wrote:
>>>>
>>>> On Mon, Jan 9, 2017 at 3:49 AM, Aljoscha Krettek <al...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> I also usually prefer "mvn verify" to to the expected thing but I see
>>>>>>
>>>>> that
>>>>>
>>>>>> quick iteration times are key.
>>>>>>
>>>>>
>>>>> I see
>>>>> https://maven.apache.org/guides/introduction/
>>>>>
>>>> introduction-to-the-lifecycle.html
>>>>
>>>>>
>>>>> verify - run any checks on results of integration tests to ensure
>>>>> quality criteria are met
>>>>>
>>>>> Of course our integration tests are long enough that we shouldn't be
>>>>> putting all of them here, but I too would expect checkstyle.
>>>>>
>>>>> Perhaps we could introduce a verify-fast or somesuch for fast (but
>>>>> lower coverage) turnaround time. I would expect "mvn verify test" to
>>>>> pass before submitting a PR, and would want to run that before asking
>>>>> others to look at it. I think this should be our criteria (i.e. what
>>>>> will a new but maven-savvy user run before pushing their code).
>>>>>
>>>>> As long as the pre-commit hooks still check everything I'm ok with
>>>>>>
>>>>> making
>>>>
>>>>> the default a little more lightweight.
>>>>>>
>>>>>
>>>>> The fact that our pre-commit hooks take a long time to run does change
>>>>> things. Nothing more annoying than seeing that your PR failed 3 hours
>>>>> later because you had some trailing whitespace...
>>>>>
>>>>> On Thu, 5 Jan 2017 at 21:49 Lukasz Cwik <lc...@google.com.invalid>
>>>>>>
>>>>> wrote:
>>>>>
>>>>>>
>>>>>> I was hoping that the default mvn verify would be the slow build
>>>>>>>
>>>>>> and a
>>>
>>>> profile could be enabled that would skip checks to make things
>>>>>>>
>>>>>> faster
>>>
>>>> for
>>>>>
>>>>>> regular contributors. This way a person doesn't need to have
>>>>>>>
>>>>>> detailed
>>>
>>>> knowledge of all our profiles and what they do (typically mvn
>>>>>>>
>>>>>> verify)
>>>
>>>> will
>>>>>
>>>>>> do the right thing most of the time.
>>>>>>>
>>>>>>> On Thu, Jan 5, 2017 at 9:30 AM, Dan Halperin
>>>>>>>
>>>>>> <dh...@google.com.invalid>
>>>>>
>>>>>> wrote:
>>>>>>>
>>>>>>> On Thu, Jan 5, 2017 at 9:28 AM, Jesse Anderson <
>>>>>>>>
>>>>>>> jesse@smokinghand.com
>>>>
>>>>>
>>>>>> wrote:
>>>>>>>>
>>>>>>>> @dan are you saying that mvn verify isn't doing checkstyle
>>>>>>>>>
>>>>>>>> anymore?
>>>>
>>>>>
>>>>>>>>
>>>>>>>> `mvn verify` alone should not be running checkstyle, if modules
>>>>>>>>
>>>>>>> are
>>>
>>>> configured correctly.
>>>>>>>>
>>>>>>>>
>>>>>>>> Some of
>>>>>>>>> the checkstyles are still running for a few modules. Also, the
>>>>>>>>>
>>>>>>>> contribution
>>>>>>>>
>>>>>>>>> docs will need to change.
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Yes. The PR includes discussion of these other needed changes,
>>>>>>>> unfortunately one PR can't change two repositories.
>>>>>>>>
>>>>>>>> Please continue the discussion on the PR, then I will summarize it
>>>>>>>>
>>>>>>> back
>>>>>
>>>>>> into the dev thread.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Dan
>>>>>>>>
>>>>>>>>
>>>>>>>> They say to run mvn verify before commits.
>>>>>>>>>
>>>>>>>>> On Thu, Jan 5, 2017 at 9:25 AM Dan Halperin
>>>>>>>>>
>>>>>>>> <dhalperi@google.com.invalid
>>>>>>>
>>>>>>>>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Several folks seem to have been confused after BEAM-246, where
>>>>>>>>>>
>>>>>>>>> we
>>>>
>>>>> moved
>>>>>>>
>>>>>>>> the
>>>>>>>>>
>>>>>>>>>> "slow things" into the release profile. I've started a
>>>>>>>>>>
>>>>>>>>> discussion
>>>>
>>>>> with
>>>>>>>
>>>>>>>> https://github.com/apache/beam/pull/1740 to see if there are
>>>>>>>>>>
>>>>>>>>> things
>>>>>
>>>>>> we
>>>>>>>
>>>>>>>> can
>>>>>>>>>
>>>>>>>>>> do to fill these gaps.
>>>>>>>>>>
>>>>>>>>>> Would love folks to chime in with opinions.
>>>>>>>>>>
>>>>>>>>>> Dan
>>>>>>>>>>
>>>>>>>>>> On Wed, Jan 4, 2017 at 1:34 PM, Jesse Anderson <
>>>>>>>>>>
>>>>>>>>> jesse@smokinghand.com>
>>>>>>>
>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> @Eugene, yes that failed on the checkstyle.
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Jan 4, 2017 at 1:27 PM Eugene Kirpichov
>>>>>>>>>>> <ki...@google.com.invalid> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Try just -Prelease.
>>>>>>>>>>>> On Wed, Jan 4, 2017 at 1:21 PM Jesse Anderson <
>>>>>>>>>>>>
>>>>>>>>>>> jesse@smokinghand.com
>>>>>>>>
>>>>>>>>>
>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Fails because I don't have a secret key.
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, Jan 4, 2017 at 1:03 PM Jean-Baptiste Onofré <
>>>>>>>>>>>>>
>>>>>>>>>>>> jb@nanthrax.net
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Jesse,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Could you try the same with:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> mvn verify -Prelease,apache-release
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Regards
>>>>>>>>>>>>>> JB
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 01/04/2017 09:53 PM, Jesse Anderson wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> For some reason, running "mvn verify" isn't running
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> checkstyle
>>>>>>>>
>>>>>>>>> on
>>>>>>>>>
>>>>>>>>>> everything. I had checkstyle errors in
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> beam-sdks-java-core
>>>>>
>>>>>> that
>>>>>>>>
>>>>>>>>> weren't
>>>>>>>>>>>>
>>>>>>>>>>>>> being found.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I thought this was due to the extra parameters. I
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> reran
>>>>
>>>>> with
>>>>>>>
>>>>>>>> the
>>>>>>>>>
>>>>>>>>>> plain
>>>>>>>>>>>>
>>>>>>>>>>>>> "mvn
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> verify" and it still didn't find them. From the
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> output,
>>>>
>>>>> it
>>>>>
>>>>>> doesn't
>>>>>>>>>>
>>>>>>>>>>> look
>>>>>>>>>>>>
>>>>>>>>>>>>> like they're being run at all.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Jesse
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Jean-Baptiste Onofré
>>>>>>>>>>>>>> jbonofre@apache.org
>>>>>>>>>>>>>> http://blog.nanthrax.net
>>>>>>>>>>>>>> Talend - http://www.talend.com
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>>
>>>
>>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
Re: Better developer instructions for using Maven?
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
+1 to at least update the contribution guide and improve the profile name.
Regards
JB
On 01/24/2017 09:49 PM, Kenneth Knowles wrote:
> My impression is that we don't have consensus on whether all checks or
> minimal checks should be the default, or whether we can have both via `mvn
> test` and `mvn verify`.
>
> But that doesn't prevent us from giving -P release a better name and
> mentioning it in the dev guide and in some manner in our PR template.
>
> Right now we are living with the combination of the bad aspects - default
> is not thorough but not actually fast and a thorough check is undocumented.
>
> On Tue, Jan 24, 2017 at 2:22 AM, Isma�l Mej�a <ie...@gmail.com> wrote:
>
>> I just wanted to know if we have achieved some consensus about this, I just
>> saw this PR that reminded me about this discussion.
>>
>> \u200bhttps://github.com/apache/beam/pull/1829\u200b
>>
>> It is important that we mention the existing profiles (and the intended
>> checks) in the contribution guide (e.g. -Prelease (or -Pall-checks triggers
>> these validations).
>>
>> I can add this to the guide if you like once we define the checks per
>> stage/profile.
>>
>> Isma�l
>>
>>
>> On Wed, Jan 11, 2017 at 8:12 AM, Aviem Zur <av...@gmail.com> wrote:
>>
>>> I agree with Dan and Lukasz.
>>> Developers should not be expected to know beforehand which specific
>>> profiles to run.
>>> The phase specified in the PR instructions (`verify`) should run all the
>>> relevant verifications and be the "slower" build, while a preceding
>>> lifecycle, such as `test`, should run the "faster" verifications.
>>>
>>> Aviem.
>>>
>>> On Mon, Jan 9, 2017 at 7:57 PM Robert Bradshaw
>> <robertwb@google.com.invalid
>>>>
>>> wrote:
>>>
>>>> On Mon, Jan 9, 2017 at 3:49 AM, Aljoscha Krettek <al...@apache.org>
>>>> wrote:
>>>>> I also usually prefer "mvn verify" to to the expected thing but I see
>>>> that
>>>>> quick iteration times are key.
>>>>
>>>> I see
>>>> https://maven.apache.org/guides/introduction/
>>> introduction-to-the-lifecycle.html
>>>>
>>>> verify - run any checks on results of integration tests to ensure
>>>> quality criteria are met
>>>>
>>>> Of course our integration tests are long enough that we shouldn't be
>>>> putting all of them here, but I too would expect checkstyle.
>>>>
>>>> Perhaps we could introduce a verify-fast or somesuch for fast (but
>>>> lower coverage) turnaround time. I would expect "mvn verify test" to
>>>> pass before submitting a PR, and would want to run that before asking
>>>> others to look at it. I think this should be our criteria (i.e. what
>>>> will a new but maven-savvy user run before pushing their code).
>>>>
>>>>> As long as the pre-commit hooks still check everything I'm ok with
>>> making
>>>>> the default a little more lightweight.
>>>>
>>>> The fact that our pre-commit hooks take a long time to run does change
>>>> things. Nothing more annoying than seeing that your PR failed 3 hours
>>>> later because you had some trailing whitespace...
>>>>
>>>>> On Thu, 5 Jan 2017 at 21:49 Lukasz Cwik <lc...@google.com.invalid>
>>>> wrote:
>>>>>
>>>>>> I was hoping that the default mvn verify would be the slow build
>> and a
>>>>>> profile could be enabled that would skip checks to make things
>> faster
>>>> for
>>>>>> regular contributors. This way a person doesn't need to have
>> detailed
>>>>>> knowledge of all our profiles and what they do (typically mvn
>> verify)
>>>> will
>>>>>> do the right thing most of the time.
>>>>>>
>>>>>> On Thu, Jan 5, 2017 at 9:30 AM, Dan Halperin
>>>> <dh...@google.com.invalid>
>>>>>> wrote:
>>>>>>
>>>>>>> On Thu, Jan 5, 2017 at 9:28 AM, Jesse Anderson <
>>> jesse@smokinghand.com
>>>>>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> @dan are you saying that mvn verify isn't doing checkstyle
>>> anymore?
>>>>>>>
>>>>>>>
>>>>>>> `mvn verify` alone should not be running checkstyle, if modules
>> are
>>>>>>> configured correctly.
>>>>>>>
>>>>>>>
>>>>>>>> Some of
>>>>>>>> the checkstyles are still running for a few modules. Also, the
>>>>>>> contribution
>>>>>>>> docs will need to change.
>>>>>>>
>>>>>>>
>>>>>>> Yes. The PR includes discussion of these other needed changes,
>>>>>>> unfortunately one PR can't change two repositories.
>>>>>>>
>>>>>>> Please continue the discussion on the PR, then I will summarize it
>>>> back
>>>>>>> into the dev thread.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Dan
>>>>>>>
>>>>>>>
>>>>>>>> They say to run mvn verify before commits.
>>>>>>>>
>>>>>>>> On Thu, Jan 5, 2017 at 9:25 AM Dan Halperin
>>>>>> <dhalperi@google.com.invalid
>>>>>>>>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Several folks seem to have been confused after BEAM-246, where
>>> we
>>>>>> moved
>>>>>>>> the
>>>>>>>>> "slow things" into the release profile. I've started a
>>> discussion
>>>>>> with
>>>>>>>>> https://github.com/apache/beam/pull/1740 to see if there are
>>>> things
>>>>>> we
>>>>>>>> can
>>>>>>>>> do to fill these gaps.
>>>>>>>>>
>>>>>>>>> Would love folks to chime in with opinions.
>>>>>>>>>
>>>>>>>>> Dan
>>>>>>>>>
>>>>>>>>> On Wed, Jan 4, 2017 at 1:34 PM, Jesse Anderson <
>>>>>> jesse@smokinghand.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> @Eugene, yes that failed on the checkstyle.
>>>>>>>>>>
>>>>>>>>>> On Wed, Jan 4, 2017 at 1:27 PM Eugene Kirpichov
>>>>>>>>>> <ki...@google.com.invalid> wrote:
>>>>>>>>>>
>>>>>>>>>>> Try just -Prelease.
>>>>>>>>>>> On Wed, Jan 4, 2017 at 1:21 PM Jesse Anderson <
>>>>>>> jesse@smokinghand.com
>>>>>>>>>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Fails because I don't have a secret key.
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Jan 4, 2017 at 1:03 PM Jean-Baptiste Onofr� <
>>>>>>>> jb@nanthrax.net
>>>>>>>>>>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Jesse,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Could you try the same with:
>>>>>>>>>>>>>
>>>>>>>>>>>>> mvn verify -Prelease,apache-release
>>>>>>>>>>>>>
>>>>>>>>>>>>> ?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Regards
>>>>>>>>>>>>> JB
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 01/04/2017 09:53 PM, Jesse Anderson wrote:
>>>>>>>>>>>>>> For some reason, running "mvn verify" isn't running
>>>>>>> checkstyle
>>>>>>>> on
>>>>>>>>>>>>>> everything. I had checkstyle errors in
>>>> beam-sdks-java-core
>>>>>>> that
>>>>>>>>>>> weren't
>>>>>>>>>>>>>> being found.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I thought this was due to the extra parameters. I
>>> reran
>>>>>> with
>>>>>>>> the
>>>>>>>>>>> plain
>>>>>>>>>>>>> "mvn
>>>>>>>>>>>>>> verify" and it still didn't find them. From the
>>> output,
>>>> it
>>>>>>>>> doesn't
>>>>>>>>>>> look
>>>>>>>>>>>>>> like they're being run at all.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Jesse
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Jean-Baptiste Onofr�
>>>>>>>>>>>>> jbonofre@apache.org
>>>>>>>>>>>>> http://blog.nanthrax.net
>>>>>>>>>>>>> Talend - http://www.talend.com
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>>
>>
>
--
Jean-Baptiste Onofr�
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: Better developer instructions for using Maven?
Posted by Kenneth Knowles <kl...@google.com.INVALID>.
My impression is that we don't have consensus on whether all checks or
minimal checks should be the default, or whether we can have both via `mvn
test` and `mvn verify`.
But that doesn't prevent us from giving -P release a better name and
mentioning it in the dev guide and in some manner in our PR template.
Right now we are living with the combination of the bad aspects - default
is not thorough but not actually fast and a thorough check is undocumented.
On Tue, Jan 24, 2017 at 2:22 AM, Ismaël Mejía <ie...@gmail.com> wrote:
> I just wanted to know if we have achieved some consensus about this, I just
> saw this PR that reminded me about this discussion.
>
> https://github.com/apache/beam/pull/1829
>
> It is important that we mention the existing profiles (and the intended
> checks) in the contribution guide (e.g. -Prelease (or -Pall-checks triggers
> these validations).
>
> I can add this to the guide if you like once we define the checks per
> stage/profile.
>
> Ismaël
>
>
> On Wed, Jan 11, 2017 at 8:12 AM, Aviem Zur <av...@gmail.com> wrote:
>
> > I agree with Dan and Lukasz.
> > Developers should not be expected to know beforehand which specific
> > profiles to run.
> > The phase specified in the PR instructions (`verify`) should run all the
> > relevant verifications and be the "slower" build, while a preceding
> > lifecycle, such as `test`, should run the "faster" verifications.
> >
> > Aviem.
> >
> > On Mon, Jan 9, 2017 at 7:57 PM Robert Bradshaw
> <robertwb@google.com.invalid
> > >
> > wrote:
> >
> > > On Mon, Jan 9, 2017 at 3:49 AM, Aljoscha Krettek <al...@apache.org>
> > > wrote:
> > > > I also usually prefer "mvn verify" to to the expected thing but I see
> > > that
> > > > quick iteration times are key.
> > >
> > > I see
> > > https://maven.apache.org/guides/introduction/
> > introduction-to-the-lifecycle.html
> > >
> > > verify - run any checks on results of integration tests to ensure
> > > quality criteria are met
> > >
> > > Of course our integration tests are long enough that we shouldn't be
> > > putting all of them here, but I too would expect checkstyle.
> > >
> > > Perhaps we could introduce a verify-fast or somesuch for fast (but
> > > lower coverage) turnaround time. I would expect "mvn verify test" to
> > > pass before submitting a PR, and would want to run that before asking
> > > others to look at it. I think this should be our criteria (i.e. what
> > > will a new but maven-savvy user run before pushing their code).
> > >
> > > > As long as the pre-commit hooks still check everything I'm ok with
> > making
> > > > the default a little more lightweight.
> > >
> > > The fact that our pre-commit hooks take a long time to run does change
> > > things. Nothing more annoying than seeing that your PR failed 3 hours
> > > later because you had some trailing whitespace...
> > >
> > > > On Thu, 5 Jan 2017 at 21:49 Lukasz Cwik <lc...@google.com.invalid>
> > > wrote:
> > > >
> > > >> I was hoping that the default mvn verify would be the slow build
> and a
> > > >> profile could be enabled that would skip checks to make things
> faster
> > > for
> > > >> regular contributors. This way a person doesn't need to have
> detailed
> > > >> knowledge of all our profiles and what they do (typically mvn
> verify)
> > > will
> > > >> do the right thing most of the time.
> > > >>
> > > >> On Thu, Jan 5, 2017 at 9:30 AM, Dan Halperin
> > > <dh...@google.com.invalid>
> > > >> wrote:
> > > >>
> > > >> > On Thu, Jan 5, 2017 at 9:28 AM, Jesse Anderson <
> > jesse@smokinghand.com
> > > >
> > > >> > wrote:
> > > >> >
> > > >> > > @dan are you saying that mvn verify isn't doing checkstyle
> > anymore?
> > > >> >
> > > >> >
> > > >> > `mvn verify` alone should not be running checkstyle, if modules
> are
> > > >> > configured correctly.
> > > >> >
> > > >> >
> > > >> > > Some of
> > > >> > > the checkstyles are still running for a few modules. Also, the
> > > >> > contribution
> > > >> > > docs will need to change.
> > > >> >
> > > >> >
> > > >> > Yes. The PR includes discussion of these other needed changes,
> > > >> > unfortunately one PR can't change two repositories.
> > > >> >
> > > >> > Please continue the discussion on the PR, then I will summarize it
> > > back
> > > >> > into the dev thread.
> > > >> >
> > > >> > Thanks,
> > > >> > Dan
> > > >> >
> > > >> >
> > > >> > > They say to run mvn verify before commits.
> > > >> > >
> > > >> > > On Thu, Jan 5, 2017 at 9:25 AM Dan Halperin
> > > >> <dhalperi@google.com.invalid
> > > >> > >
> > > >> > > wrote:
> > > >> > >
> > > >> > > > Several folks seem to have been confused after BEAM-246, where
> > we
> > > >> moved
> > > >> > > the
> > > >> > > > "slow things" into the release profile. I've started a
> > discussion
> > > >> with
> > > >> > > > https://github.com/apache/beam/pull/1740 to see if there are
> > > things
> > > >> we
> > > >> > > can
> > > >> > > > do to fill these gaps.
> > > >> > > >
> > > >> > > > Would love folks to chime in with opinions.
> > > >> > > >
> > > >> > > > Dan
> > > >> > > >
> > > >> > > > On Wed, Jan 4, 2017 at 1:34 PM, Jesse Anderson <
> > > >> jesse@smokinghand.com>
> > > >> > > > wrote:
> > > >> > > >
> > > >> > > > > @Eugene, yes that failed on the checkstyle.
> > > >> > > > >
> > > >> > > > > On Wed, Jan 4, 2017 at 1:27 PM Eugene Kirpichov
> > > >> > > > > <ki...@google.com.invalid> wrote:
> > > >> > > > >
> > > >> > > > > > Try just -Prelease.
> > > >> > > > > > On Wed, Jan 4, 2017 at 1:21 PM Jesse Anderson <
> > > >> > jesse@smokinghand.com
> > > >> > > >
> > > >> > > > > > wrote:
> > > >> > > > > >
> > > >> > > > > > > Fails because I don't have a secret key.
> > > >> > > > > > >
> > > >> > > > > > > On Wed, Jan 4, 2017 at 1:03 PM Jean-Baptiste Onofré <
> > > >> > > jb@nanthrax.net
> > > >> > > > >
> > > >> > > > > > > wrote:
> > > >> > > > > > >
> > > >> > > > > > > > Hi Jesse,
> > > >> > > > > > > >
> > > >> > > > > > > > Could you try the same with:
> > > >> > > > > > > >
> > > >> > > > > > > > mvn verify -Prelease,apache-release
> > > >> > > > > > > >
> > > >> > > > > > > > ?
> > > >> > > > > > > >
> > > >> > > > > > > > Regards
> > > >> > > > > > > > JB
> > > >> > > > > > > >
> > > >> > > > > > > > On 01/04/2017 09:53 PM, Jesse Anderson wrote:
> > > >> > > > > > > > > For some reason, running "mvn verify" isn't running
> > > >> > checkstyle
> > > >> > > on
> > > >> > > > > > > > > everything. I had checkstyle errors in
> > > beam-sdks-java-core
> > > >> > that
> > > >> > > > > > weren't
> > > >> > > > > > > > > being found.
> > > >> > > > > > > > >
> > > >> > > > > > > > > I thought this was due to the extra parameters. I
> > reran
> > > >> with
> > > >> > > the
> > > >> > > > > > plain
> > > >> > > > > > > > "mvn
> > > >> > > > > > > > > verify" and it still didn't find them. From the
> > output,
> > > it
> > > >> > > > doesn't
> > > >> > > > > > look
> > > >> > > > > > > > > like they're being run at all.
> > > >> > > > > > > > >
> > > >> > > > > > > > > Thanks,
> > > >> > > > > > > > >
> > > >> > > > > > > > > Jesse
> > > >> > > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > > > --
> > > >> > > > > > > > Jean-Baptiste Onofré
> > > >> > > > > > > > jbonofre@apache.org
> > > >> > > > > > > > http://blog.nanthrax.net
> > > >> > > > > > > > Talend - http://www.talend.com
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
>
Re: Better developer instructions for using Maven?
Posted by Ismaël Mejía <ie...@gmail.com>.
I just wanted to know if we have achieved some consensus about this, I just
saw this PR that reminded me about this discussion.
https://github.com/apache/beam/pull/1829
It is important that we mention the existing profiles (and the intended
checks) in the contribution guide (e.g. -Prelease (or -Pall-checks triggers
these validations).
I can add this to the guide if you like once we define the checks per
stage/profile.
Ismaël
On Wed, Jan 11, 2017 at 8:12 AM, Aviem Zur <av...@gmail.com> wrote:
> I agree with Dan and Lukasz.
> Developers should not be expected to know beforehand which specific
> profiles to run.
> The phase specified in the PR instructions (`verify`) should run all the
> relevant verifications and be the "slower" build, while a preceding
> lifecycle, such as `test`, should run the "faster" verifications.
>
> Aviem.
>
> On Mon, Jan 9, 2017 at 7:57 PM Robert Bradshaw <robertwb@google.com.invalid
> >
> wrote:
>
> > On Mon, Jan 9, 2017 at 3:49 AM, Aljoscha Krettek <al...@apache.org>
> > wrote:
> > > I also usually prefer "mvn verify" to to the expected thing but I see
> > that
> > > quick iteration times are key.
> >
> > I see
> > https://maven.apache.org/guides/introduction/
> introduction-to-the-lifecycle.html
> >
> > verify - run any checks on results of integration tests to ensure
> > quality criteria are met
> >
> > Of course our integration tests are long enough that we shouldn't be
> > putting all of them here, but I too would expect checkstyle.
> >
> > Perhaps we could introduce a verify-fast or somesuch for fast (but
> > lower coverage) turnaround time. I would expect "mvn verify test" to
> > pass before submitting a PR, and would want to run that before asking
> > others to look at it. I think this should be our criteria (i.e. what
> > will a new but maven-savvy user run before pushing their code).
> >
> > > As long as the pre-commit hooks still check everything I'm ok with
> making
> > > the default a little more lightweight.
> >
> > The fact that our pre-commit hooks take a long time to run does change
> > things. Nothing more annoying than seeing that your PR failed 3 hours
> > later because you had some trailing whitespace...
> >
> > > On Thu, 5 Jan 2017 at 21:49 Lukasz Cwik <lc...@google.com.invalid>
> > wrote:
> > >
> > >> I was hoping that the default mvn verify would be the slow build and a
> > >> profile could be enabled that would skip checks to make things faster
> > for
> > >> regular contributors. This way a person doesn't need to have detailed
> > >> knowledge of all our profiles and what they do (typically mvn verify)
> > will
> > >> do the right thing most of the time.
> > >>
> > >> On Thu, Jan 5, 2017 at 9:30 AM, Dan Halperin
> > <dh...@google.com.invalid>
> > >> wrote:
> > >>
> > >> > On Thu, Jan 5, 2017 at 9:28 AM, Jesse Anderson <
> jesse@smokinghand.com
> > >
> > >> > wrote:
> > >> >
> > >> > > @dan are you saying that mvn verify isn't doing checkstyle
> anymore?
> > >> >
> > >> >
> > >> > `mvn verify` alone should not be running checkstyle, if modules are
> > >> > configured correctly.
> > >> >
> > >> >
> > >> > > Some of
> > >> > > the checkstyles are still running for a few modules. Also, the
> > >> > contribution
> > >> > > docs will need to change.
> > >> >
> > >> >
> > >> > Yes. The PR includes discussion of these other needed changes,
> > >> > unfortunately one PR can't change two repositories.
> > >> >
> > >> > Please continue the discussion on the PR, then I will summarize it
> > back
> > >> > into the dev thread.
> > >> >
> > >> > Thanks,
> > >> > Dan
> > >> >
> > >> >
> > >> > > They say to run mvn verify before commits.
> > >> > >
> > >> > > On Thu, Jan 5, 2017 at 9:25 AM Dan Halperin
> > >> <dhalperi@google.com.invalid
> > >> > >
> > >> > > wrote:
> > >> > >
> > >> > > > Several folks seem to have been confused after BEAM-246, where
> we
> > >> moved
> > >> > > the
> > >> > > > "slow things" into the release profile. I've started a
> discussion
> > >> with
> > >> > > > https://github.com/apache/beam/pull/1740 to see if there are
> > things
> > >> we
> > >> > > can
> > >> > > > do to fill these gaps.
> > >> > > >
> > >> > > > Would love folks to chime in with opinions.
> > >> > > >
> > >> > > > Dan
> > >> > > >
> > >> > > > On Wed, Jan 4, 2017 at 1:34 PM, Jesse Anderson <
> > >> jesse@smokinghand.com>
> > >> > > > wrote:
> > >> > > >
> > >> > > > > @Eugene, yes that failed on the checkstyle.
> > >> > > > >
> > >> > > > > On Wed, Jan 4, 2017 at 1:27 PM Eugene Kirpichov
> > >> > > > > <ki...@google.com.invalid> wrote:
> > >> > > > >
> > >> > > > > > Try just -Prelease.
> > >> > > > > > On Wed, Jan 4, 2017 at 1:21 PM Jesse Anderson <
> > >> > jesse@smokinghand.com
> > >> > > >
> > >> > > > > > wrote:
> > >> > > > > >
> > >> > > > > > > Fails because I don't have a secret key.
> > >> > > > > > >
> > >> > > > > > > On Wed, Jan 4, 2017 at 1:03 PM Jean-Baptiste Onofré <
> > >> > > jb@nanthrax.net
> > >> > > > >
> > >> > > > > > > wrote:
> > >> > > > > > >
> > >> > > > > > > > Hi Jesse,
> > >> > > > > > > >
> > >> > > > > > > > Could you try the same with:
> > >> > > > > > > >
> > >> > > > > > > > mvn verify -Prelease,apache-release
> > >> > > > > > > >
> > >> > > > > > > > ?
> > >> > > > > > > >
> > >> > > > > > > > Regards
> > >> > > > > > > > JB
> > >> > > > > > > >
> > >> > > > > > > > On 01/04/2017 09:53 PM, Jesse Anderson wrote:
> > >> > > > > > > > > For some reason, running "mvn verify" isn't running
> > >> > checkstyle
> > >> > > on
> > >> > > > > > > > > everything. I had checkstyle errors in
> > beam-sdks-java-core
> > >> > that
> > >> > > > > > weren't
> > >> > > > > > > > > being found.
> > >> > > > > > > > >
> > >> > > > > > > > > I thought this was due to the extra parameters. I
> reran
> > >> with
> > >> > > the
> > >> > > > > > plain
> > >> > > > > > > > "mvn
> > >> > > > > > > > > verify" and it still didn't find them. From the
> output,
> > it
> > >> > > > doesn't
> > >> > > > > > look
> > >> > > > > > > > > like they're being run at all.
> > >> > > > > > > > >
> > >> > > > > > > > > Thanks,
> > >> > > > > > > > >
> > >> > > > > > > > > Jesse
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > > --
> > >> > > > > > > > Jean-Baptiste Onofré
> > >> > > > > > > > jbonofre@apache.org
> > >> > > > > > > > http://blog.nanthrax.net
> > >> > > > > > > > Talend - http://www.talend.com
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
>
Re: Better developer instructions for using Maven?
Posted by Aviem Zur <av...@gmail.com>.
I agree with Dan and Lukasz.
Developers should not be expected to know beforehand which specific
profiles to run.
The phase specified in the PR instructions (`verify`) should run all the
relevant verifications and be the "slower" build, while a preceding
lifecycle, such as `test`, should run the "faster" verifications.
Aviem.
On Mon, Jan 9, 2017 at 7:57 PM Robert Bradshaw <ro...@google.com.invalid>
wrote:
> On Mon, Jan 9, 2017 at 3:49 AM, Aljoscha Krettek <al...@apache.org>
> wrote:
> > I also usually prefer "mvn verify" to to the expected thing but I see
> that
> > quick iteration times are key.
>
> I see
> https://maven.apache.org/guides/introduction/introduction-to-the-lifecycle.html
>
> verify - run any checks on results of integration tests to ensure
> quality criteria are met
>
> Of course our integration tests are long enough that we shouldn't be
> putting all of them here, but I too would expect checkstyle.
>
> Perhaps we could introduce a verify-fast or somesuch for fast (but
> lower coverage) turnaround time. I would expect "mvn verify test" to
> pass before submitting a PR, and would want to run that before asking
> others to look at it. I think this should be our criteria (i.e. what
> will a new but maven-savvy user run before pushing their code).
>
> > As long as the pre-commit hooks still check everything I'm ok with making
> > the default a little more lightweight.
>
> The fact that our pre-commit hooks take a long time to run does change
> things. Nothing more annoying than seeing that your PR failed 3 hours
> later because you had some trailing whitespace...
>
> > On Thu, 5 Jan 2017 at 21:49 Lukasz Cwik <lc...@google.com.invalid>
> wrote:
> >
> >> I was hoping that the default mvn verify would be the slow build and a
> >> profile could be enabled that would skip checks to make things faster
> for
> >> regular contributors. This way a person doesn't need to have detailed
> >> knowledge of all our profiles and what they do (typically mvn verify)
> will
> >> do the right thing most of the time.
> >>
> >> On Thu, Jan 5, 2017 at 9:30 AM, Dan Halperin
> <dh...@google.com.invalid>
> >> wrote:
> >>
> >> > On Thu, Jan 5, 2017 at 9:28 AM, Jesse Anderson <jesse@smokinghand.com
> >
> >> > wrote:
> >> >
> >> > > @dan are you saying that mvn verify isn't doing checkstyle anymore?
> >> >
> >> >
> >> > `mvn verify` alone should not be running checkstyle, if modules are
> >> > configured correctly.
> >> >
> >> >
> >> > > Some of
> >> > > the checkstyles are still running for a few modules. Also, the
> >> > contribution
> >> > > docs will need to change.
> >> >
> >> >
> >> > Yes. The PR includes discussion of these other needed changes,
> >> > unfortunately one PR can't change two repositories.
> >> >
> >> > Please continue the discussion on the PR, then I will summarize it
> back
> >> > into the dev thread.
> >> >
> >> > Thanks,
> >> > Dan
> >> >
> >> >
> >> > > They say to run mvn verify before commits.
> >> > >
> >> > > On Thu, Jan 5, 2017 at 9:25 AM Dan Halperin
> >> <dhalperi@google.com.invalid
> >> > >
> >> > > wrote:
> >> > >
> >> > > > Several folks seem to have been confused after BEAM-246, where we
> >> moved
> >> > > the
> >> > > > "slow things" into the release profile. I've started a discussion
> >> with
> >> > > > https://github.com/apache/beam/pull/1740 to see if there are
> things
> >> we
> >> > > can
> >> > > > do to fill these gaps.
> >> > > >
> >> > > > Would love folks to chime in with opinions.
> >> > > >
> >> > > > Dan
> >> > > >
> >> > > > On Wed, Jan 4, 2017 at 1:34 PM, Jesse Anderson <
> >> jesse@smokinghand.com>
> >> > > > wrote:
> >> > > >
> >> > > > > @Eugene, yes that failed on the checkstyle.
> >> > > > >
> >> > > > > On Wed, Jan 4, 2017 at 1:27 PM Eugene Kirpichov
> >> > > > > <ki...@google.com.invalid> wrote:
> >> > > > >
> >> > > > > > Try just -Prelease.
> >> > > > > > On Wed, Jan 4, 2017 at 1:21 PM Jesse Anderson <
> >> > jesse@smokinghand.com
> >> > > >
> >> > > > > > wrote:
> >> > > > > >
> >> > > > > > > Fails because I don't have a secret key.
> >> > > > > > >
> >> > > > > > > On Wed, Jan 4, 2017 at 1:03 PM Jean-Baptiste Onofré <
> >> > > jb@nanthrax.net
> >> > > > >
> >> > > > > > > wrote:
> >> > > > > > >
> >> > > > > > > > Hi Jesse,
> >> > > > > > > >
> >> > > > > > > > Could you try the same with:
> >> > > > > > > >
> >> > > > > > > > mvn verify -Prelease,apache-release
> >> > > > > > > >
> >> > > > > > > > ?
> >> > > > > > > >
> >> > > > > > > > Regards
> >> > > > > > > > JB
> >> > > > > > > >
> >> > > > > > > > On 01/04/2017 09:53 PM, Jesse Anderson wrote:
> >> > > > > > > > > For some reason, running "mvn verify" isn't running
> >> > checkstyle
> >> > > on
> >> > > > > > > > > everything. I had checkstyle errors in
> beam-sdks-java-core
> >> > that
> >> > > > > > weren't
> >> > > > > > > > > being found.
> >> > > > > > > > >
> >> > > > > > > > > I thought this was due to the extra parameters. I reran
> >> with
> >> > > the
> >> > > > > > plain
> >> > > > > > > > "mvn
> >> > > > > > > > > verify" and it still didn't find them. From the output,
> it
> >> > > > doesn't
> >> > > > > > look
> >> > > > > > > > > like they're being run at all.
> >> > > > > > > > >
> >> > > > > > > > > Thanks,
> >> > > > > > > > >
> >> > > > > > > > > Jesse
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > > > --
> >> > > > > > > > Jean-Baptiste Onofré
> >> > > > > > > > jbonofre@apache.org
> >> > > > > > > > http://blog.nanthrax.net
> >> > > > > > > > Talend - http://www.talend.com
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
>
Re: Better developer instructions for using Maven?
Posted by Robert Bradshaw <ro...@google.com.INVALID>.
On Mon, Jan 9, 2017 at 3:49 AM, Aljoscha Krettek <al...@apache.org> wrote:
> I also usually prefer "mvn verify" to to the expected thing but I see that
> quick iteration times are key.
I see https://maven.apache.org/guides/introduction/introduction-to-the-lifecycle.html
verify - run any checks on results of integration tests to ensure
quality criteria are met
Of course our integration tests are long enough that we shouldn't be
putting all of them here, but I too would expect checkstyle.
Perhaps we could introduce a verify-fast or somesuch for fast (but
lower coverage) turnaround time. I would expect "mvn verify test" to
pass before submitting a PR, and would want to run that before asking
others to look at it. I think this should be our criteria (i.e. what
will a new but maven-savvy user run before pushing their code).
> As long as the pre-commit hooks still check everything I'm ok with making
> the default a little more lightweight.
The fact that our pre-commit hooks take a long time to run does change
things. Nothing more annoying than seeing that your PR failed 3 hours
later because you had some trailing whitespace...
> On Thu, 5 Jan 2017 at 21:49 Lukasz Cwik <lc...@google.com.invalid> wrote:
>
>> I was hoping that the default mvn verify would be the slow build and a
>> profile could be enabled that would skip checks to make things faster for
>> regular contributors. This way a person doesn't need to have detailed
>> knowledge of all our profiles and what they do (typically mvn verify) will
>> do the right thing most of the time.
>>
>> On Thu, Jan 5, 2017 at 9:30 AM, Dan Halperin <dh...@google.com.invalid>
>> wrote:
>>
>> > On Thu, Jan 5, 2017 at 9:28 AM, Jesse Anderson <je...@smokinghand.com>
>> > wrote:
>> >
>> > > @dan are you saying that mvn verify isn't doing checkstyle anymore?
>> >
>> >
>> > `mvn verify` alone should not be running checkstyle, if modules are
>> > configured correctly.
>> >
>> >
>> > > Some of
>> > > the checkstyles are still running for a few modules. Also, the
>> > contribution
>> > > docs will need to change.
>> >
>> >
>> > Yes. The PR includes discussion of these other needed changes,
>> > unfortunately one PR can't change two repositories.
>> >
>> > Please continue the discussion on the PR, then I will summarize it back
>> > into the dev thread.
>> >
>> > Thanks,
>> > Dan
>> >
>> >
>> > > They say to run mvn verify before commits.
>> > >
>> > > On Thu, Jan 5, 2017 at 9:25 AM Dan Halperin
>> <dhalperi@google.com.invalid
>> > >
>> > > wrote:
>> > >
>> > > > Several folks seem to have been confused after BEAM-246, where we
>> moved
>> > > the
>> > > > "slow things" into the release profile. I've started a discussion
>> with
>> > > > https://github.com/apache/beam/pull/1740 to see if there are things
>> we
>> > > can
>> > > > do to fill these gaps.
>> > > >
>> > > > Would love folks to chime in with opinions.
>> > > >
>> > > > Dan
>> > > >
>> > > > On Wed, Jan 4, 2017 at 1:34 PM, Jesse Anderson <
>> jesse@smokinghand.com>
>> > > > wrote:
>> > > >
>> > > > > @Eugene, yes that failed on the checkstyle.
>> > > > >
>> > > > > On Wed, Jan 4, 2017 at 1:27 PM Eugene Kirpichov
>> > > > > <ki...@google.com.invalid> wrote:
>> > > > >
>> > > > > > Try just -Prelease.
>> > > > > > On Wed, Jan 4, 2017 at 1:21 PM Jesse Anderson <
>> > jesse@smokinghand.com
>> > > >
>> > > > > > wrote:
>> > > > > >
>> > > > > > > Fails because I don't have a secret key.
>> > > > > > >
>> > > > > > > On Wed, Jan 4, 2017 at 1:03 PM Jean-Baptiste Onofré <
>> > > jb@nanthrax.net
>> > > > >
>> > > > > > > wrote:
>> > > > > > >
>> > > > > > > > Hi Jesse,
>> > > > > > > >
>> > > > > > > > Could you try the same with:
>> > > > > > > >
>> > > > > > > > mvn verify -Prelease,apache-release
>> > > > > > > >
>> > > > > > > > ?
>> > > > > > > >
>> > > > > > > > Regards
>> > > > > > > > JB
>> > > > > > > >
>> > > > > > > > On 01/04/2017 09:53 PM, Jesse Anderson wrote:
>> > > > > > > > > For some reason, running "mvn verify" isn't running
>> > checkstyle
>> > > on
>> > > > > > > > > everything. I had checkstyle errors in beam-sdks-java-core
>> > that
>> > > > > > weren't
>> > > > > > > > > being found.
>> > > > > > > > >
>> > > > > > > > > I thought this was due to the extra parameters. I reran
>> with
>> > > the
>> > > > > > plain
>> > > > > > > > "mvn
>> > > > > > > > > verify" and it still didn't find them. From the output, it
>> > > > doesn't
>> > > > > > look
>> > > > > > > > > like they're being run at all.
>> > > > > > > > >
>> > > > > > > > > Thanks,
>> > > > > > > > >
>> > > > > > > > > Jesse
>> > > > > > > > >
>> > > > > > > >
>> > > > > > > > --
>> > > > > > > > Jean-Baptiste Onofré
>> > > > > > > > jbonofre@apache.org
>> > > > > > > > http://blog.nanthrax.net
>> > > > > > > > Talend - http://www.talend.com
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
Re: Better developer instructions for using Maven?
Posted by Aljoscha Krettek <al...@apache.org>.
I also usually prefer "mvn verify" to to the expected thing but I see that
quick iteration times are key.
As long as the pre-commit hooks still check everything I'm ok with making
the default a little more lightweight.
On Thu, 5 Jan 2017 at 21:49 Lukasz Cwik <lc...@google.com.invalid> wrote:
> I was hoping that the default mvn verify would be the slow build and a
> profile could be enabled that would skip checks to make things faster for
> regular contributors. This way a person doesn't need to have detailed
> knowledge of all our profiles and what they do (typically mvn verify) will
> do the right thing most of the time.
>
> On Thu, Jan 5, 2017 at 9:30 AM, Dan Halperin <dh...@google.com.invalid>
> wrote:
>
> > On Thu, Jan 5, 2017 at 9:28 AM, Jesse Anderson <je...@smokinghand.com>
> > wrote:
> >
> > > @dan are you saying that mvn verify isn't doing checkstyle anymore?
> >
> >
> > `mvn verify` alone should not be running checkstyle, if modules are
> > configured correctly.
> >
> >
> > > Some of
> > > the checkstyles are still running for a few modules. Also, the
> > contribution
> > > docs will need to change.
> >
> >
> > Yes. The PR includes discussion of these other needed changes,
> > unfortunately one PR can't change two repositories.
> >
> > Please continue the discussion on the PR, then I will summarize it back
> > into the dev thread.
> >
> > Thanks,
> > Dan
> >
> >
> > > They say to run mvn verify before commits.
> > >
> > > On Thu, Jan 5, 2017 at 9:25 AM Dan Halperin
> <dhalperi@google.com.invalid
> > >
> > > wrote:
> > >
> > > > Several folks seem to have been confused after BEAM-246, where we
> moved
> > > the
> > > > "slow things" into the release profile. I've started a discussion
> with
> > > > https://github.com/apache/beam/pull/1740 to see if there are things
> we
> > > can
> > > > do to fill these gaps.
> > > >
> > > > Would love folks to chime in with opinions.
> > > >
> > > > Dan
> > > >
> > > > On Wed, Jan 4, 2017 at 1:34 PM, Jesse Anderson <
> jesse@smokinghand.com>
> > > > wrote:
> > > >
> > > > > @Eugene, yes that failed on the checkstyle.
> > > > >
> > > > > On Wed, Jan 4, 2017 at 1:27 PM Eugene Kirpichov
> > > > > <ki...@google.com.invalid> wrote:
> > > > >
> > > > > > Try just -Prelease.
> > > > > > On Wed, Jan 4, 2017 at 1:21 PM Jesse Anderson <
> > jesse@smokinghand.com
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Fails because I don't have a secret key.
> > > > > > >
> > > > > > > On Wed, Jan 4, 2017 at 1:03 PM Jean-Baptiste Onofré <
> > > jb@nanthrax.net
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Jesse,
> > > > > > > >
> > > > > > > > Could you try the same with:
> > > > > > > >
> > > > > > > > mvn verify -Prelease,apache-release
> > > > > > > >
> > > > > > > > ?
> > > > > > > >
> > > > > > > > Regards
> > > > > > > > JB
> > > > > > > >
> > > > > > > > On 01/04/2017 09:53 PM, Jesse Anderson wrote:
> > > > > > > > > For some reason, running "mvn verify" isn't running
> > checkstyle
> > > on
> > > > > > > > > everything. I had checkstyle errors in beam-sdks-java-core
> > that
> > > > > > weren't
> > > > > > > > > being found.
> > > > > > > > >
> > > > > > > > > I thought this was due to the extra parameters. I reran
> with
> > > the
> > > > > > plain
> > > > > > > > "mvn
> > > > > > > > > verify" and it still didn't find them. From the output, it
> > > > doesn't
> > > > > > look
> > > > > > > > > like they're being run at all.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > >
> > > > > > > > > Jesse
> > > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Jean-Baptiste Onofré
> > > > > > > > jbonofre@apache.org
> > > > > > > > http://blog.nanthrax.net
> > > > > > > > Talend - http://www.talend.com
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Better developer instructions for using Maven?
Posted by Lukasz Cwik <lc...@google.com.INVALID>.
I was hoping that the default mvn verify would be the slow build and a
profile could be enabled that would skip checks to make things faster for
regular contributors. This way a person doesn't need to have detailed
knowledge of all our profiles and what they do (typically mvn verify) will
do the right thing most of the time.
On Thu, Jan 5, 2017 at 9:30 AM, Dan Halperin <dh...@google.com.invalid>
wrote:
> On Thu, Jan 5, 2017 at 9:28 AM, Jesse Anderson <je...@smokinghand.com>
> wrote:
>
> > @dan are you saying that mvn verify isn't doing checkstyle anymore?
>
>
> `mvn verify` alone should not be running checkstyle, if modules are
> configured correctly.
>
>
> > Some of
> > the checkstyles are still running for a few modules. Also, the
> contribution
> > docs will need to change.
>
>
> Yes. The PR includes discussion of these other needed changes,
> unfortunately one PR can't change two repositories.
>
> Please continue the discussion on the PR, then I will summarize it back
> into the dev thread.
>
> Thanks,
> Dan
>
>
> > They say to run mvn verify before commits.
> >
> > On Thu, Jan 5, 2017 at 9:25 AM Dan Halperin <dhalperi@google.com.invalid
> >
> > wrote:
> >
> > > Several folks seem to have been confused after BEAM-246, where we moved
> > the
> > > "slow things" into the release profile. I've started a discussion with
> > > https://github.com/apache/beam/pull/1740 to see if there are things we
> > can
> > > do to fill these gaps.
> > >
> > > Would love folks to chime in with opinions.
> > >
> > > Dan
> > >
> > > On Wed, Jan 4, 2017 at 1:34 PM, Jesse Anderson <je...@smokinghand.com>
> > > wrote:
> > >
> > > > @Eugene, yes that failed on the checkstyle.
> > > >
> > > > On Wed, Jan 4, 2017 at 1:27 PM Eugene Kirpichov
> > > > <ki...@google.com.invalid> wrote:
> > > >
> > > > > Try just -Prelease.
> > > > > On Wed, Jan 4, 2017 at 1:21 PM Jesse Anderson <
> jesse@smokinghand.com
> > >
> > > > > wrote:
> > > > >
> > > > > > Fails because I don't have a secret key.
> > > > > >
> > > > > > On Wed, Jan 4, 2017 at 1:03 PM Jean-Baptiste Onofré <
> > jb@nanthrax.net
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Jesse,
> > > > > > >
> > > > > > > Could you try the same with:
> > > > > > >
> > > > > > > mvn verify -Prelease,apache-release
> > > > > > >
> > > > > > > ?
> > > > > > >
> > > > > > > Regards
> > > > > > > JB
> > > > > > >
> > > > > > > On 01/04/2017 09:53 PM, Jesse Anderson wrote:
> > > > > > > > For some reason, running "mvn verify" isn't running
> checkstyle
> > on
> > > > > > > > everything. I had checkstyle errors in beam-sdks-java-core
> that
> > > > > weren't
> > > > > > > > being found.
> > > > > > > >
> > > > > > > > I thought this was due to the extra parameters. I reran with
> > the
> > > > > plain
> > > > > > > "mvn
> > > > > > > > verify" and it still didn't find them. From the output, it
> > > doesn't
> > > > > look
> > > > > > > > like they're being run at all.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > Jesse
> > > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Jean-Baptiste Onofré
> > > > > > > jbonofre@apache.org
> > > > > > > http://blog.nanthrax.net
> > > > > > > Talend - http://www.talend.com
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Better developer instructions for using Maven?
Posted by Dan Halperin <dh...@google.com.INVALID>.
On Thu, Jan 5, 2017 at 9:28 AM, Jesse Anderson <je...@smokinghand.com>
wrote:
> @dan are you saying that mvn verify isn't doing checkstyle anymore?
`mvn verify` alone should not be running checkstyle, if modules are
configured correctly.
> Some of
> the checkstyles are still running for a few modules. Also, the contribution
> docs will need to change.
Yes. The PR includes discussion of these other needed changes,
unfortunately one PR can't change two repositories.
Please continue the discussion on the PR, then I will summarize it back
into the dev thread.
Thanks,
Dan
> They say to run mvn verify before commits.
>
> On Thu, Jan 5, 2017 at 9:25 AM Dan Halperin <dh...@google.com.invalid>
> wrote:
>
> > Several folks seem to have been confused after BEAM-246, where we moved
> the
> > "slow things" into the release profile. I've started a discussion with
> > https://github.com/apache/beam/pull/1740 to see if there are things we
> can
> > do to fill these gaps.
> >
> > Would love folks to chime in with opinions.
> >
> > Dan
> >
> > On Wed, Jan 4, 2017 at 1:34 PM, Jesse Anderson <je...@smokinghand.com>
> > wrote:
> >
> > > @Eugene, yes that failed on the checkstyle.
> > >
> > > On Wed, Jan 4, 2017 at 1:27 PM Eugene Kirpichov
> > > <ki...@google.com.invalid> wrote:
> > >
> > > > Try just -Prelease.
> > > > On Wed, Jan 4, 2017 at 1:21 PM Jesse Anderson <jesse@smokinghand.com
> >
> > > > wrote:
> > > >
> > > > > Fails because I don't have a secret key.
> > > > >
> > > > > On Wed, Jan 4, 2017 at 1:03 PM Jean-Baptiste Onofré <
> jb@nanthrax.net
> > >
> > > > > wrote:
> > > > >
> > > > > > Hi Jesse,
> > > > > >
> > > > > > Could you try the same with:
> > > > > >
> > > > > > mvn verify -Prelease,apache-release
> > > > > >
> > > > > > ?
> > > > > >
> > > > > > Regards
> > > > > > JB
> > > > > >
> > > > > > On 01/04/2017 09:53 PM, Jesse Anderson wrote:
> > > > > > > For some reason, running "mvn verify" isn't running checkstyle
> on
> > > > > > > everything. I had checkstyle errors in beam-sdks-java-core that
> > > > weren't
> > > > > > > being found.
> > > > > > >
> > > > > > > I thought this was due to the extra parameters. I reran with
> the
> > > > plain
> > > > > > "mvn
> > > > > > > verify" and it still didn't find them. From the output, it
> > doesn't
> > > > look
> > > > > > > like they're being run at all.
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Jesse
> > > > > > >
> > > > > >
> > > > > > --
> > > > > > Jean-Baptiste Onofré
> > > > > > jbonofre@apache.org
> > > > > > http://blog.nanthrax.net
> > > > > > Talend - http://www.talend.com
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Better developer instructions for using Maven?
Posted by Jesse Anderson <je...@smokinghand.com>.
@dan are you saying that mvn verify isn't doing checkstyle anymore? Some of
the checkstyles are still running for a few modules. Also, the contribution
docs will need to change. They say to run mvn verify before commits.
On Thu, Jan 5, 2017 at 9:25 AM Dan Halperin <dh...@google.com.invalid>
wrote:
> Several folks seem to have been confused after BEAM-246, where we moved the
> "slow things" into the release profile. I've started a discussion with
> https://github.com/apache/beam/pull/1740 to see if there are things we can
> do to fill these gaps.
>
> Would love folks to chime in with opinions.
>
> Dan
>
> On Wed, Jan 4, 2017 at 1:34 PM, Jesse Anderson <je...@smokinghand.com>
> wrote:
>
> > @Eugene, yes that failed on the checkstyle.
> >
> > On Wed, Jan 4, 2017 at 1:27 PM Eugene Kirpichov
> > <ki...@google.com.invalid> wrote:
> >
> > > Try just -Prelease.
> > > On Wed, Jan 4, 2017 at 1:21 PM Jesse Anderson <je...@smokinghand.com>
> > > wrote:
> > >
> > > > Fails because I don't have a secret key.
> > > >
> > > > On Wed, Jan 4, 2017 at 1:03 PM Jean-Baptiste Onofré <jb@nanthrax.net
> >
> > > > wrote:
> > > >
> > > > > Hi Jesse,
> > > > >
> > > > > Could you try the same with:
> > > > >
> > > > > mvn verify -Prelease,apache-release
> > > > >
> > > > > ?
> > > > >
> > > > > Regards
> > > > > JB
> > > > >
> > > > > On 01/04/2017 09:53 PM, Jesse Anderson wrote:
> > > > > > For some reason, running "mvn verify" isn't running checkstyle on
> > > > > > everything. I had checkstyle errors in beam-sdks-java-core that
> > > weren't
> > > > > > being found.
> > > > > >
> > > > > > I thought this was due to the extra parameters. I reran with the
> > > plain
> > > > > "mvn
> > > > > > verify" and it still didn't find them. From the output, it
> doesn't
> > > look
> > > > > > like they're being run at all.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jesse
> > > > > >
> > > > >
> > > > > --
> > > > > Jean-Baptiste Onofré
> > > > > jbonofre@apache.org
> > > > > http://blog.nanthrax.net
> > > > > Talend - http://www.talend.com
> > > > >
> > > >
> > >
> >
>