You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Romain Manni-Bucau <rm...@gmail.com> on 2018/03/07 10:24:23 UTC
Re: Gradle status
Up,
We discussed to have a strong switch to gradle or rollback to maven around
april to not be blocked by the build tool. I noticed gradle build rarely
passes on PR and kind of blurry our vision - not sure why exactly. Also, PR
don't always contain the gradle updates - generally dependencies+plugins
are added in pom.xml AFAIK, so it seems the adoption is very slow to not
say rejected.
What do we do about that? When do we drop the double build maintenance -
whatever is picked?
Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> | Blog
<https://rmannibucau.metawerx.net/> | Old Blog
<http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
<https://www.packtpub.com/application-development/java-ee-8-high-performance>
2018-01-12 6:30 GMT+01:00 Romain Manni-Bucau <rm...@gmail.com>:
>
>
> Le 11 janv. 2018 23:13, "Kenneth Knowles" <kl...@google.com> a écrit :
>
> On Thu, Jan 11, 2018 at 8:43 AM, Romain Manni-Bucau <rmannibucau@gmail.com
> > wrote:
>
>> 2. gradle build doesn't use the same output directory than maven so it is
>> not really smooth to have both and have to maintain both
>>
>
> I also have an opinion on this. It is useful and reasonable to be able to
> build even when the source is on a read-only filesystem. Maven's defaults
> are undesirable and require workarounds. We shouldn't mimic the behavior,
> but actually should set gradle up to build to a directory outside the
> source tree always, if we can.
>
>
> Hmm, which is something you can do with maven as well so not sure I get
> it.
>
> Also note the thread is no more about the technical points but more the
> sources maintenance and consistency.
>
>
> Kenn
>
>
>
Re: Gradle status
Posted by Kenneth Knowles <kl...@google.com>.
Now that the NeedsRunner tests are running via Gradle I support removing
the Java maven precommit.
Lower latency, simpler config, and more available Jenkins workers FTW.
Now let's fix the --rerun-tasks bugs so we can do even better.
On Wed, Mar 7, 2018 at 11:21 AM Lukasz Cwik <lc...@google.com> wrote:
> Note that Alan Myrvold has been making steady progress making the release
> process via Gradle a reality:
> 1) Creating a jenkins job which can run the quickstart validation against
> nightly snapshots and also can be used for release candidates (
> https://github.com/apache/beam/pull/4252)
> 2) Building a release candidate using Gradle (
> https://github.com/apache/beam/pull/4812)
>
> Also, Gradle is the tool that has been selected already and there was a
> community discussion about what was needed for the migration to occur with
> a clear set of criteria. Romain, it doesn't seem like we should ignore that
> criteria or are you suggesting we change that criteria (if yes, how)?
>
>
> On Wed, Mar 7, 2018 at 10:39 AM, Romain Manni-Bucau <rmannibucau@gmail.com
> > wrote:
>
>>
>>
>> Le 7 mars 2018 17:34, "Lukasz Cwik" <lc...@google.com> a écrit :
>>
>> Thanks for bringing this up Romain but I believe your data points on pass
>> rates are only partially correct.
>>
>>
>> Sure sure, it is mainly about my own PR which a very small % of the whole
>> project ;).
>>
>>
>>
>> In the past week the Java Gradle precommit passed 46.34% of the time
>> compared to the Java Maven precommit which passed 46.15% of the time. When
>> I looked at these numbers in mid January they were around 37% so there has
>> been some improvement. Regardless of the build tool it seems that our pass
>> rates aren't stellar for the Java build and are causing the community to
>> not follow best practices (wait for precommits to be green before merging).
>> I know that on the website we used the mergebot to ensure that things
>> passed before they were merged, should we institute this on the master
>> branch or are their any other ideas?
>>
>> As a side note we had achieved the goals we set out to not need to
>> maintain the Maven precommit and have authored the first PR to drop the
>> Maven precommit: https://github.com/apache/beam/pull/4814
>>
>>
>> Well, I'd be for a strong switch otherwise PR will keep using maven,
>> jenkins will not test the code and at the end we fail to deliver something
>> consistent. So whatever tool is selected I'm tempted to say drop other
>> build files and jenkins hooks references.
>>
>> What about doing it after 2.4 vote?
>>
>>
>>
>> On Wed, Mar 7, 2018 at 2:24 AM, Romain Manni-Bucau <rmannibucau@gmail.com
>> > wrote:
>>
>>> Up,
>>>
>>> We discussed to have a strong switch to gradle or rollback to maven
>>> around april to not be blocked by the build tool. I noticed gradle build
>>> rarely passes on PR and kind of blurry our vision - not sure why exactly.
>>> Also, PR don't always contain the gradle updates - generally
>>> dependencies+plugins are added in pom.xml AFAIK, so it seems the adoption
>>> is very slow to not say rejected.
>>>
>>> What do we do about that? When do we drop the double build maintenance -
>>> whatever is picked?
>>>
>>>
>>>
>>> Romain Manni-Bucau
>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
>>> <https://rmannibucau.metawerx.net/> | Old Blog
>>> <http://rmannibucau.wordpress.com> | Github
>>> <https://github.com/rmannibucau> | LinkedIn
>>> <https://www.linkedin.com/in/rmannibucau> | Book
>>> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>>>
>>> 2018-01-12 6:30 GMT+01:00 Romain Manni-Bucau <rm...@gmail.com>:
>>>
>>>>
>>>>
>>>> Le 11 janv. 2018 23:13, "Kenneth Knowles" <kl...@google.com> a écrit :
>>>>
>>>> On Thu, Jan 11, 2018 at 8:43 AM, Romain Manni-Bucau <
>>>> rmannibucau@gmail.com> wrote:
>>>>
>>>>> 2. gradle build doesn't use the same output directory than maven so it
>>>>> is not really smooth to have both and have to maintain both
>>>>>
>>>>
>>>> I also have an opinion on this. It is useful and reasonable to be able
>>>> to build even when the source is on a read-only filesystem. Maven's
>>>> defaults are undesirable and require workarounds. We shouldn't mimic the
>>>> behavior, but actually should set gradle up to build to a directory outside
>>>> the source tree always, if we can.
>>>>
>>>>
>>>> Hmm, which is something you can do with maven as well so not sure I get
>>>> it.
>>>>
>>>> Also note the thread is no more about the technical points but more the
>>>> sources maintenance and consistency.
>>>>
>>>>
>>>> Kenn
>>>>
>>>>
>>>>
>>>
>>
>>
>
Re: Gradle status
Posted by Reuven Lax <re...@google.com>.
Are there instructions for how to do this? I would like too switch my
IntelliJ over to Gradle (it's still setup using Maven)
On Wed, Mar 7, 2018 at 1:29 PM Kenneth Knowles <kl...@google.com> wrote:
> SGTM. I should also say that every day is Gradle fixit day for me, as I
> have been using only Gradle (with IntelliJ) for a while :-). If anyone is
> hesitant, definitely it is ready to be used for normal dev.
>
> Seems like changing the messaging in onboarding docs is the main thing to
> fixit.
>
> Based on https://builds.apache.org/view/A-D/view/Beam/ and our failure
> spam level the performance tests are mostly not healthy anyhow. So is there
> any high level blocker to switching them or is it just someone sitting down
> with each one?
>
> Kenn
>
>
> On Wed, Mar 7, 2018 at 1:22 PM Lukasz Cwik <lc...@google.com> wrote:
>
>> Largest outstanding areas are:
>> * Documentation relevant to the contributors guide/release process/testing
>> * Performance tests
>>
>> There has been good progress towards:
>> * Release artifact validations and generation
>> * ValidatesRunner post commits
>> * Pre commits
>> * Container builds
>>
>>
>> On Wed, Mar 7, 2018 at 1:19 PM, Reuven Lax <re...@google.com> wrote:
>>
>>> I think Alan was making progress on the Gradle build.
>>>
>>> What do people think of a "fixit" day for Gradle work? (or given that
>>> people are distributed, maybe a fixit week, where everyone takes one day
>>> from the week).
>>>
>>>
>>> On Wed, Mar 7, 2018 at 1:17 PM Kenneth Knowles <kl...@google.com> wrote:
>>>
>>>> I also cannot drop everything to work on Gradle build, but maybe it
>>>> isn't that drastic anyhow. Now that we have ValidatesRunner and NeedsRunner
>>>> tests and some progress on the release, is there any other known missing
>>>> functionality in the Gradle builds? Archetypes? Docker container images?
>>>>
>>>>
>>>> On Wed, Mar 7, 2018 at 1:12 PM Lukasz Cwik <lc...@google.com> wrote:
>>>>
>>>>> I am working on various projects and may not be able to pause my work
>>>>> for a couple of weeks while the build/test process is migrated.
>>>>>
>>>>> What is everyone thinking about Romain's suggestion because If I'm the
>>>>> only person in such a situation, I would be willing to go along with the
>>>>> plan.
>>>>>
>>>>> On Wed, Mar 7, 2018 at 12:59 PM, Romain Manni-Bucau <
>>>>> rmannibucau@gmail.com> wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> Le 7 mars 2018 20:21, "Lukasz Cwik" <lc...@google.com> a écrit :
>>>>>>
>>>>>> Note that Alan Myrvold has been making steady progress making the
>>>>>> release process via Gradle a reality:
>>>>>> 1) Creating a jenkins job which can run the quickstart validation
>>>>>> against nightly snapshots and also can be used for release candidates (
>>>>>> https://github.com/apache/beam/pull/4252)
>>>>>> 2) Building a release candidate using Gradle (
>>>>>> https://github.com/apache/beam/pull/4812)
>>>>>>
>>>>>> Also, Gradle is the tool that has been selected already and there was
>>>>>> a community discussion about what was needed for the migration to occur
>>>>>> with a clear set of criteria. Romain, it doesn't seem like we should ignore
>>>>>> that criteria or are you suggesting we change that criteria (if yes, how)?
>>>>>>
>>>>>>
>>>>>> No, no. My goal is just to quit this state.
>>>>>>
>>>>>> Let s draft a plan:
>>>>>>
>>>>>> 1. 2.4 is released - i assume it is done with mvn here
>>>>>> 2. We drop all poms and jenkins mvn config
>>>>>> 3. We fix all build issues if so (let say in a week)
>>>>>> 4. Pr can nees updates but no more mvn merge
>>>>>>
>>>>>> April is gradle month :)
>>>>>>
>>>>>> Wdyt?
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Mar 7, 2018 at 10:39 AM, Romain Manni-Bucau <
>>>>>> rmannibucau@gmail.com> wrote:
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Le 7 mars 2018 17:34, "Lukasz Cwik" <lc...@google.com> a écrit :
>>>>>>>
>>>>>>> Thanks for bringing this up Romain but I believe your data points on
>>>>>>> pass rates are only partially correct.
>>>>>>>
>>>>>>>
>>>>>>> Sure sure, it is mainly about my own PR which a very small % of the
>>>>>>> whole project ;).
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> In the past week the Java Gradle precommit passed 46.34% of the time
>>>>>>> compared to the Java Maven precommit which passed 46.15% of the time. When
>>>>>>> I looked at these numbers in mid January they were around 37% so there has
>>>>>>> been some improvement. Regardless of the build tool it seems that our pass
>>>>>>> rates aren't stellar for the Java build and are causing the community to
>>>>>>> not follow best practices (wait for precommits to be green before merging).
>>>>>>> I know that on the website we used the mergebot to ensure that things
>>>>>>> passed before they were merged, should we institute this on the master
>>>>>>> branch or are their any other ideas?
>>>>>>>
>>>>>>> As a side note we had achieved the goals we set out to not need to
>>>>>>> maintain the Maven precommit and have authored the first PR to drop the
>>>>>>> Maven precommit: https://github.com/apache/beam/pull/4814
>>>>>>>
>>>>>>>
>>>>>>> Well, I'd be for a strong switch otherwise PR will keep using maven,
>>>>>>> jenkins will not test the code and at the end we fail to deliver something
>>>>>>> consistent. So whatever tool is selected I'm tempted to say drop other
>>>>>>> build files and jenkins hooks references.
>>>>>>>
>>>>>>> What about doing it after 2.4 vote?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Mar 7, 2018 at 2:24 AM, Romain Manni-Bucau <
>>>>>>> rmannibucau@gmail.com> wrote:
>>>>>>>
>>>>>>>> Up,
>>>>>>>>
>>>>>>>> We discussed to have a strong switch to gradle or rollback to maven
>>>>>>>> around april to not be blocked by the build tool. I noticed gradle build
>>>>>>>> rarely passes on PR and kind of blurry our vision - not sure why exactly.
>>>>>>>> Also, PR don't always contain the gradle updates - generally
>>>>>>>> dependencies+plugins are added in pom.xml AFAIK, so it seems the adoption
>>>>>>>> is very slow to not say rejected.
>>>>>>>>
>>>>>>>> What do we do about that? When do we drop the double build
>>>>>>>> maintenance - whatever is picked?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Romain Manni-Bucau
>>>>>>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
>>>>>>>> <https://rmannibucau.metawerx.net/> | Old Blog
>>>>>>>> <http://rmannibucau.wordpress.com> | Github
>>>>>>>> <https://github.com/rmannibucau> | LinkedIn
>>>>>>>> <https://www.linkedin.com/in/rmannibucau> | Book
>>>>>>>> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>>>>>>>>
>>>>>>>> 2018-01-12 6:30 GMT+01:00 Romain Manni-Bucau <rmannibucau@gmail.com
>>>>>>>> >:
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Le 11 janv. 2018 23:13, "Kenneth Knowles" <kl...@google.com> a
>>>>>>>>> écrit :
>>>>>>>>>
>>>>>>>>> On Thu, Jan 11, 2018 at 8:43 AM, Romain Manni-Bucau <
>>>>>>>>> rmannibucau@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> 2. gradle build doesn't use the same output directory than maven
>>>>>>>>>> so it is not really smooth to have both and have to maintain both
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I also have an opinion on this. It is useful and reasonable to be
>>>>>>>>> able to build even when the source is on a read-only filesystem. Maven's
>>>>>>>>> defaults are undesirable and require workarounds. We shouldn't mimic the
>>>>>>>>> behavior, but actually should set gradle up to build to a directory outside
>>>>>>>>> the source tree always, if we can.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hmm, which is something you can do with maven as well so not sure
>>>>>>>>> I get it.
>>>>>>>>>
>>>>>>>>> Also note the thread is no more about the technical points but
>>>>>>>>> more the sources maintenance and consistency.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Kenn
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>
Re: Gradle status
Posted by Kenneth Knowles <kl...@google.com>.
I'll write up the steps as part of the fixit :-)
I put IntelliJ hints into the build.gradle enough that you don't do any
specific tweaking. Here's short version:
1. Create new empty project and set up JDK or create new Java project. Put
it outside the source tree so you can `git clean -d -f -x`
2. Create new module from existing sources -> import module from external
model -> Gradle
- Use auto-import because our modules can be churny
- Create separate module per source set, necessary for smaller build
times IIRC
- Store generated project files externally or it is all screwy
- Use default gradle wrapper
3. In settings, delegate build actions to Gradle
4. Build project. This populates generated source folders so IntelliJ will
grok references to it.
On Wed, Mar 7, 2018 at 1:49 PM Henning Rohde <he...@google.com> wrote:
> +1 to a Gradle fixit day/week. I agree with Romain that we should make an
> effort quit this dual state. The Go, Go PreCommit and various container
> image Gradle builds, I think, are in a reasonable state (modulo some
> documentation updates).
>
> On Wed, Mar 7, 2018 at 1:29 PM, Kenneth Knowles <kl...@google.com> wrote:
>
>> SGTM. I should also say that every day is Gradle fixit day for me, as I
>> have been using only Gradle (with IntelliJ) for a while :-). If anyone is
>> hesitant, definitely it is ready to be used for normal dev.
>>
>> Seems like changing the messaging in onboarding docs is the main thing to
>> fixit.
>>
>> Based on https://builds.apache.org/view/A-D/view/Beam/ and our failure
>> spam level the performance tests are mostly not healthy anyhow. So is there
>> any high level blocker to switching them or is it just someone sitting down
>> with each one?
>>
>> Kenn
>>
>>
>> On Wed, Mar 7, 2018 at 1:22 PM Lukasz Cwik <lc...@google.com> wrote:
>>
>>> Largest outstanding areas are:
>>> * Documentation relevant to the contributors guide/release
>>> process/testing
>>> * Performance tests
>>>
>>> There has been good progress towards:
>>> * Release artifact validations and generation
>>> * ValidatesRunner post commits
>>> * Pre commits
>>> * Container builds
>>>
>>>
>>> On Wed, Mar 7, 2018 at 1:19 PM, Reuven Lax <re...@google.com> wrote:
>>>
>>>> I think Alan was making progress on the Gradle build.
>>>>
>>>> What do people think of a "fixit" day for Gradle work? (or given that
>>>> people are distributed, maybe a fixit week, where everyone takes one day
>>>> from the week).
>>>>
>>>>
>>>> On Wed, Mar 7, 2018 at 1:17 PM Kenneth Knowles <kl...@google.com> wrote:
>>>>
>>>>> I also cannot drop everything to work on Gradle build, but maybe it
>>>>> isn't that drastic anyhow. Now that we have ValidatesRunner and NeedsRunner
>>>>> tests and some progress on the release, is there any other known missing
>>>>> functionality in the Gradle builds? Archetypes? Docker container images?
>>>>>
>>>>>
>>>>> On Wed, Mar 7, 2018 at 1:12 PM Lukasz Cwik <lc...@google.com> wrote:
>>>>>
>>>>>> I am working on various projects and may not be able to pause my work
>>>>>> for a couple of weeks while the build/test process is migrated.
>>>>>>
>>>>>> What is everyone thinking about Romain's suggestion because If I'm
>>>>>> the only person in such a situation, I would be willing to go along with
>>>>>> the plan.
>>>>>>
>>>>>> On Wed, Mar 7, 2018 at 12:59 PM, Romain Manni-Bucau <
>>>>>> rmannibucau@gmail.com> wrote:
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Le 7 mars 2018 20:21, "Lukasz Cwik" <lc...@google.com> a écrit :
>>>>>>>
>>>>>>> Note that Alan Myrvold has been making steady progress making the
>>>>>>> release process via Gradle a reality:
>>>>>>> 1) Creating a jenkins job which can run the quickstart validation
>>>>>>> against nightly snapshots and also can be used for release candidates (
>>>>>>> https://github.com/apache/beam/pull/4252)
>>>>>>> 2) Building a release candidate using Gradle (
>>>>>>> https://github.com/apache/beam/pull/4812)
>>>>>>>
>>>>>>> Also, Gradle is the tool that has been selected already and there
>>>>>>> was a community discussion about what was needed for the migration to occur
>>>>>>> with a clear set of criteria. Romain, it doesn't seem like we should ignore
>>>>>>> that criteria or are you suggesting we change that criteria (if yes, how)?
>>>>>>>
>>>>>>>
>>>>>>> No, no. My goal is just to quit this state.
>>>>>>>
>>>>>>> Let s draft a plan:
>>>>>>>
>>>>>>> 1. 2.4 is released - i assume it is done with mvn here
>>>>>>> 2. We drop all poms and jenkins mvn config
>>>>>>> 3. We fix all build issues if so (let say in a week)
>>>>>>> 4. Pr can nees updates but no more mvn merge
>>>>>>>
>>>>>>> April is gradle month :)
>>>>>>>
>>>>>>> Wdyt?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Mar 7, 2018 at 10:39 AM, Romain Manni-Bucau <
>>>>>>> rmannibucau@gmail.com> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Le 7 mars 2018 17:34, "Lukasz Cwik" <lc...@google.com> a écrit :
>>>>>>>>
>>>>>>>> Thanks for bringing this up Romain but I believe your data points
>>>>>>>> on pass rates are only partially correct.
>>>>>>>>
>>>>>>>>
>>>>>>>> Sure sure, it is mainly about my own PR which a very small % of the
>>>>>>>> whole project ;).
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> In the past week the Java Gradle precommit passed 46.34% of the
>>>>>>>> time compared to the Java Maven precommit which passed 46.15% of the time.
>>>>>>>> When I looked at these numbers in mid January they were around 37% so there
>>>>>>>> has been some improvement. Regardless of the build tool it seems that our
>>>>>>>> pass rates aren't stellar for the Java build and are causing the community
>>>>>>>> to not follow best practices (wait for precommits to be green before
>>>>>>>> merging). I know that on the website we used the mergebot to ensure that
>>>>>>>> things passed before they were merged, should we institute this on the
>>>>>>>> master branch or are their any other ideas?
>>>>>>>>
>>>>>>>> As a side note we had achieved the goals we set out to not need to
>>>>>>>> maintain the Maven precommit and have authored the first PR to drop the
>>>>>>>> Maven precommit: https://github.com/apache/beam/pull/4814
>>>>>>>>
>>>>>>>>
>>>>>>>> Well, I'd be for a strong switch otherwise PR will keep using
>>>>>>>> maven, jenkins will not test the code and at the end we fail to deliver
>>>>>>>> something consistent. So whatever tool is selected I'm tempted to say drop
>>>>>>>> other build files and jenkins hooks references.
>>>>>>>>
>>>>>>>> What about doing it after 2.4 vote?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Mar 7, 2018 at 2:24 AM, Romain Manni-Bucau <
>>>>>>>> rmannibucau@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Up,
>>>>>>>>>
>>>>>>>>> We discussed to have a strong switch to gradle or rollback to
>>>>>>>>> maven around april to not be blocked by the build tool. I noticed gradle
>>>>>>>>> build rarely passes on PR and kind of blurry our vision - not sure why
>>>>>>>>> exactly. Also, PR don't always contain the gradle updates - generally
>>>>>>>>> dependencies+plugins are added in pom.xml AFAIK, so it seems the adoption
>>>>>>>>> is very slow to not say rejected.
>>>>>>>>>
>>>>>>>>> What do we do about that? When do we drop the double build
>>>>>>>>> maintenance - whatever is picked?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Romain Manni-Bucau
>>>>>>>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
>>>>>>>>> <https://rmannibucau.metawerx.net/> | Old Blog
>>>>>>>>> <http://rmannibucau.wordpress.com> | Github
>>>>>>>>> <https://github.com/rmannibucau> | LinkedIn
>>>>>>>>> <https://www.linkedin.com/in/rmannibucau> | Book
>>>>>>>>> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>>>>>>>>>
>>>>>>>>> 2018-01-12 6:30 GMT+01:00 Romain Manni-Bucau <
>>>>>>>>> rmannibucau@gmail.com>:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Le 11 janv. 2018 23:13, "Kenneth Knowles" <kl...@google.com> a
>>>>>>>>>> écrit :
>>>>>>>>>>
>>>>>>>>>> On Thu, Jan 11, 2018 at 8:43 AM, Romain Manni-Bucau <
>>>>>>>>>> rmannibucau@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> 2. gradle build doesn't use the same output directory than maven
>>>>>>>>>>> so it is not really smooth to have both and have to maintain both
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I also have an opinion on this. It is useful and reasonable to be
>>>>>>>>>> able to build even when the source is on a read-only filesystem. Maven's
>>>>>>>>>> defaults are undesirable and require workarounds. We shouldn't mimic the
>>>>>>>>>> behavior, but actually should set gradle up to build to a directory outside
>>>>>>>>>> the source tree always, if we can.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Hmm, which is something you can do with maven as well so not sure
>>>>>>>>>> I get it.
>>>>>>>>>>
>>>>>>>>>> Also note the thread is no more about the technical points but
>>>>>>>>>> more the sources maintenance and consistency.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Kenn
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>
>
Re: Gradle status
Posted by Romain Manni-Bucau <rm...@gmail.com>.
@Kenneth: this looks like my experience as well but is Idea - not even
speaking of eclipse - that bad with gradle? Thought that it was not since
android picked it up but never managed to make it scale on bigger projects
due to the prebuild phases which are super slow, require a full passing
build (which can be tricky with the native expectations of beam for pyyhon
for instance :() and regular updates when working on new modules in PR.
Anyone knows a way to make it better, and more important, working OOTB
whatever the machine you are using (with gradle/java but not go and with a
bad version of python to take a random example ;))? Does it need a fast
build import or idea meta generation?
Le 7 mars 2018 23:22, "Jason Kuster" <ja...@google.com> a écrit :
> +1
>
>
> On Wed, Mar 7, 2018 at 2:21 PM Robert Bradshaw <ro...@google.com>
> wrote:
>
>> +1 to a fixit day. I'd be happy to help out myself.
>>
>>
>> On Wed, Mar 7, 2018 at 1:49 PM Henning Rohde <he...@google.com> wrote:
>>
>>> +1 to a Gradle fixit day/week. I agree with Romain that we should make
>>> an effort quit this dual state. The Go, Go PreCommit and various container
>>> image Gradle builds, I think, are in a reasonable state (modulo some
>>> documentation updates).
>>>
>>> On Wed, Mar 7, 2018 at 1:29 PM, Kenneth Knowles <kl...@google.com> wrote:
>>>
>>>> SGTM. I should also say that every day is Gradle fixit day for me, as I
>>>> have been using only Gradle (with IntelliJ) for a while :-). If anyone is
>>>> hesitant, definitely it is ready to be used for normal dev.
>>>>
>>>> Seems like changing the messaging in onboarding docs is the main thing
>>>> to fixit.
>>>>
>>>> Based on https://builds.apache.org/view/A-D/view/Beam/ and our failure
>>>> spam level the performance tests are mostly not healthy anyhow. So is there
>>>> any high level blocker to switching them or is it just someone sitting down
>>>> with each one?
>>>>
>>>> Kenn
>>>>
>>>>
>>>> On Wed, Mar 7, 2018 at 1:22 PM Lukasz Cwik <lc...@google.com> wrote:
>>>>
>>>>> Largest outstanding areas are:
>>>>> * Documentation relevant to the contributors guide/release
>>>>> process/testing
>>>>> * Performance tests
>>>>>
>>>>> There has been good progress towards:
>>>>> * Release artifact validations and generation
>>>>> * ValidatesRunner post commits
>>>>> * Pre commits
>>>>> * Container builds
>>>>>
>>>>>
>>>>> On Wed, Mar 7, 2018 at 1:19 PM, Reuven Lax <re...@google.com> wrote:
>>>>>
>>>>>> I think Alan was making progress on the Gradle build.
>>>>>>
>>>>>> What do people think of a "fixit" day for Gradle work? (or given that
>>>>>> people are distributed, maybe a fixit week, where everyone takes one day
>>>>>> from the week).
>>>>>>
>>>>>>
>>>>>> On Wed, Mar 7, 2018 at 1:17 PM Kenneth Knowles <kl...@google.com>
>>>>>> wrote:
>>>>>>
>>>>>>> I also cannot drop everything to work on Gradle build, but maybe it
>>>>>>> isn't that drastic anyhow. Now that we have ValidatesRunner and NeedsRunner
>>>>>>> tests and some progress on the release, is there any other known missing
>>>>>>> functionality in the Gradle builds? Archetypes? Docker container images?
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Mar 7, 2018 at 1:12 PM Lukasz Cwik <lc...@google.com> wrote:
>>>>>>>
>>>>>>>> I am working on various projects and may not be able to pause my
>>>>>>>> work for a couple of weeks while the build/test process is migrated.
>>>>>>>>
>>>>>>>> What is everyone thinking about Romain's suggestion because If I'm
>>>>>>>> the only person in such a situation, I would be willing to go along with
>>>>>>>> the plan.
>>>>>>>>
>>>>>>>> On Wed, Mar 7, 2018 at 12:59 PM, Romain Manni-Bucau <
>>>>>>>> rmannibucau@gmail.com> wrote:
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Le 7 mars 2018 20:21, "Lukasz Cwik" <lc...@google.com> a écrit :
>>>>>>>>>
>>>>>>>>> Note that Alan Myrvold has been making steady progress making the
>>>>>>>>> release process via Gradle a reality:
>>>>>>>>> 1) Creating a jenkins job which can run the quickstart validation
>>>>>>>>> against nightly snapshots and also can be used for release candidates (
>>>>>>>>> https://github.com/apache/beam/pull/4252)
>>>>>>>>> 2) Building a release candidate using Gradle (
>>>>>>>>> https://github.com/apache/beam/pull/4812)
>>>>>>>>>
>>>>>>>>> Also, Gradle is the tool that has been selected already and there
>>>>>>>>> was a community discussion about what was needed for the migration to occur
>>>>>>>>> with a clear set of criteria. Romain, it doesn't seem like we should ignore
>>>>>>>>> that criteria or are you suggesting we change that criteria (if yes, how)?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> No, no. My goal is just to quit this state.
>>>>>>>>>
>>>>>>>>> Let s draft a plan:
>>>>>>>>>
>>>>>>>>> 1. 2.4 is released - i assume it is done with mvn here
>>>>>>>>> 2. We drop all poms and jenkins mvn config
>>>>>>>>> 3. We fix all build issues if so (let say in a week)
>>>>>>>>> 4. Pr can nees updates but no more mvn merge
>>>>>>>>>
>>>>>>>>> April is gradle month :)
>>>>>>>>>
>>>>>>>>> Wdyt?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Mar 7, 2018 at 10:39 AM, Romain Manni-Bucau <
>>>>>>>>> rmannibucau@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Le 7 mars 2018 17:34, "Lukasz Cwik" <lc...@google.com> a écrit :
>>>>>>>>>>
>>>>>>>>>> Thanks for bringing this up Romain but I believe your data points
>>>>>>>>>> on pass rates are only partially correct.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Sure sure, it is mainly about my own PR which a very small % of
>>>>>>>>>> the whole project ;).
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> In the past week the Java Gradle precommit passed 46.34% of the
>>>>>>>>>> time compared to the Java Maven precommit which passed 46.15% of the time.
>>>>>>>>>> When I looked at these numbers in mid January they were around 37% so there
>>>>>>>>>> has been some improvement. Regardless of the build tool it seems that our
>>>>>>>>>> pass rates aren't stellar for the Java build and are causing the community
>>>>>>>>>> to not follow best practices (wait for precommits to be green before
>>>>>>>>>> merging). I know that on the website we used the mergebot to ensure that
>>>>>>>>>> things passed before they were merged, should we institute this on the
>>>>>>>>>> master branch or are their any other ideas?
>>>>>>>>>>
>>>>>>>>>> As a side note we had achieved the goals we set out to not need
>>>>>>>>>> to maintain the Maven precommit and have authored the first PR to drop the
>>>>>>>>>> Maven precommit: https://github.com/apache/beam/pull/4814
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Well, I'd be for a strong switch otherwise PR will keep using
>>>>>>>>>> maven, jenkins will not test the code and at the end we fail to deliver
>>>>>>>>>> something consistent. So whatever tool is selected I'm tempted to say drop
>>>>>>>>>> other build files and jenkins hooks references.
>>>>>>>>>>
>>>>>>>>>> What about doing it after 2.4 vote?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Wed, Mar 7, 2018 at 2:24 AM, Romain Manni-Bucau <
>>>>>>>>>> rmannibucau@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Up,
>>>>>>>>>>>
>>>>>>>>>>> We discussed to have a strong switch to gradle or rollback to
>>>>>>>>>>> maven around april to not be blocked by the build tool. I noticed gradle
>>>>>>>>>>> build rarely passes on PR and kind of blurry our vision - not sure why
>>>>>>>>>>> exactly. Also, PR don't always contain the gradle updates - generally
>>>>>>>>>>> dependencies+plugins are added in pom.xml AFAIK, so it seems the adoption
>>>>>>>>>>> is very slow to not say rejected.
>>>>>>>>>>>
>>>>>>>>>>> What do we do about that? When do we drop the double build
>>>>>>>>>>> maintenance - whatever is picked?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Romain Manni-Bucau
>>>>>>>>>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
>>>>>>>>>>> <https://rmannibucau.metawerx.net/> | Old Blog
>>>>>>>>>>> <http://rmannibucau.wordpress.com> | Github
>>>>>>>>>>> <https://github.com/rmannibucau> | LinkedIn
>>>>>>>>>>> <https://www.linkedin.com/in/rmannibucau> | Book
>>>>>>>>>>> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>>>>>>>>>>>
>>>>>>>>>>> 2018-01-12 6:30 GMT+01:00 Romain Manni-Bucau <
>>>>>>>>>>> rmannibucau@gmail.com>:
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Le 11 janv. 2018 23:13, "Kenneth Knowles" <kl...@google.com> a
>>>>>>>>>>>> écrit :
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Jan 11, 2018 at 8:43 AM, Romain Manni-Bucau <
>>>>>>>>>>>> rmannibucau@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> 2. gradle build doesn't use the same output directory than
>>>>>>>>>>>>> maven so it is not really smooth to have both and have to maintain both
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I also have an opinion on this. It is useful and reasonable to
>>>>>>>>>>>> be able to build even when the source is on a read-only filesystem. Maven's
>>>>>>>>>>>> defaults are undesirable and require workarounds. We shouldn't mimic the
>>>>>>>>>>>> behavior, but actually should set gradle up to build to a directory outside
>>>>>>>>>>>> the source tree always, if we can.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Hmm, which is something you can do with maven as well so not
>>>>>>>>>>>> sure I get it.
>>>>>>>>>>>>
>>>>>>>>>>>> Also note the thread is no more about the technical points but
>>>>>>>>>>>> more the sources maintenance and consistency.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Kenn
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>
>>>
>
> --
> -------
> Jason Kuster
> Apache Beam / Google Cloud Dataflow
>
Re: Gradle status
Posted by Jason Kuster <ja...@google.com>.
+1
On Wed, Mar 7, 2018 at 2:21 PM Robert Bradshaw <ro...@google.com> wrote:
> +1 to a fixit day. I'd be happy to help out myself.
>
>
> On Wed, Mar 7, 2018 at 1:49 PM Henning Rohde <he...@google.com> wrote:
>
>> +1 to a Gradle fixit day/week. I agree with Romain that we should make an
>> effort quit this dual state. The Go, Go PreCommit and various container
>> image Gradle builds, I think, are in a reasonable state (modulo some
>> documentation updates).
>>
>> On Wed, Mar 7, 2018 at 1:29 PM, Kenneth Knowles <kl...@google.com> wrote:
>>
>>> SGTM. I should also say that every day is Gradle fixit day for me, as I
>>> have been using only Gradle (with IntelliJ) for a while :-). If anyone is
>>> hesitant, definitely it is ready to be used for normal dev.
>>>
>>> Seems like changing the messaging in onboarding docs is the main thing
>>> to fixit.
>>>
>>> Based on https://builds.apache.org/view/A-D/view/Beam/ and our failure
>>> spam level the performance tests are mostly not healthy anyhow. So is there
>>> any high level blocker to switching them or is it just someone sitting down
>>> with each one?
>>>
>>> Kenn
>>>
>>>
>>> On Wed, Mar 7, 2018 at 1:22 PM Lukasz Cwik <lc...@google.com> wrote:
>>>
>>>> Largest outstanding areas are:
>>>> * Documentation relevant to the contributors guide/release
>>>> process/testing
>>>> * Performance tests
>>>>
>>>> There has been good progress towards:
>>>> * Release artifact validations and generation
>>>> * ValidatesRunner post commits
>>>> * Pre commits
>>>> * Container builds
>>>>
>>>>
>>>> On Wed, Mar 7, 2018 at 1:19 PM, Reuven Lax <re...@google.com> wrote:
>>>>
>>>>> I think Alan was making progress on the Gradle build.
>>>>>
>>>>> What do people think of a "fixit" day for Gradle work? (or given that
>>>>> people are distributed, maybe a fixit week, where everyone takes one day
>>>>> from the week).
>>>>>
>>>>>
>>>>> On Wed, Mar 7, 2018 at 1:17 PM Kenneth Knowles <kl...@google.com> wrote:
>>>>>
>>>>>> I also cannot drop everything to work on Gradle build, but maybe it
>>>>>> isn't that drastic anyhow. Now that we have ValidatesRunner and NeedsRunner
>>>>>> tests and some progress on the release, is there any other known missing
>>>>>> functionality in the Gradle builds? Archetypes? Docker container images?
>>>>>>
>>>>>>
>>>>>> On Wed, Mar 7, 2018 at 1:12 PM Lukasz Cwik <lc...@google.com> wrote:
>>>>>>
>>>>>>> I am working on various projects and may not be able to pause my
>>>>>>> work for a couple of weeks while the build/test process is migrated.
>>>>>>>
>>>>>>> What is everyone thinking about Romain's suggestion because If I'm
>>>>>>> the only person in such a situation, I would be willing to go along with
>>>>>>> the plan.
>>>>>>>
>>>>>>> On Wed, Mar 7, 2018 at 12:59 PM, Romain Manni-Bucau <
>>>>>>> rmannibucau@gmail.com> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Le 7 mars 2018 20:21, "Lukasz Cwik" <lc...@google.com> a écrit :
>>>>>>>>
>>>>>>>> Note that Alan Myrvold has been making steady progress making the
>>>>>>>> release process via Gradle a reality:
>>>>>>>> 1) Creating a jenkins job which can run the quickstart validation
>>>>>>>> against nightly snapshots and also can be used for release candidates (
>>>>>>>> https://github.com/apache/beam/pull/4252)
>>>>>>>> 2) Building a release candidate using Gradle (
>>>>>>>> https://github.com/apache/beam/pull/4812)
>>>>>>>>
>>>>>>>> Also, Gradle is the tool that has been selected already and there
>>>>>>>> was a community discussion about what was needed for the migration to occur
>>>>>>>> with a clear set of criteria. Romain, it doesn't seem like we should ignore
>>>>>>>> that criteria or are you suggesting we change that criteria (if yes, how)?
>>>>>>>>
>>>>>>>>
>>>>>>>> No, no. My goal is just to quit this state.
>>>>>>>>
>>>>>>>> Let s draft a plan:
>>>>>>>>
>>>>>>>> 1. 2.4 is released - i assume it is done with mvn here
>>>>>>>> 2. We drop all poms and jenkins mvn config
>>>>>>>> 3. We fix all build issues if so (let say in a week)
>>>>>>>> 4. Pr can nees updates but no more mvn merge
>>>>>>>>
>>>>>>>> April is gradle month :)
>>>>>>>>
>>>>>>>> Wdyt?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Mar 7, 2018 at 10:39 AM, Romain Manni-Bucau <
>>>>>>>> rmannibucau@gmail.com> wrote:
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Le 7 mars 2018 17:34, "Lukasz Cwik" <lc...@google.com> a écrit :
>>>>>>>>>
>>>>>>>>> Thanks for bringing this up Romain but I believe your data points
>>>>>>>>> on pass rates are only partially correct.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Sure sure, it is mainly about my own PR which a very small % of
>>>>>>>>> the whole project ;).
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> In the past week the Java Gradle precommit passed 46.34% of the
>>>>>>>>> time compared to the Java Maven precommit which passed 46.15% of the time.
>>>>>>>>> When I looked at these numbers in mid January they were around 37% so there
>>>>>>>>> has been some improvement. Regardless of the build tool it seems that our
>>>>>>>>> pass rates aren't stellar for the Java build and are causing the community
>>>>>>>>> to not follow best practices (wait for precommits to be green before
>>>>>>>>> merging). I know that on the website we used the mergebot to ensure that
>>>>>>>>> things passed before they were merged, should we institute this on the
>>>>>>>>> master branch or are their any other ideas?
>>>>>>>>>
>>>>>>>>> As a side note we had achieved the goals we set out to not need to
>>>>>>>>> maintain the Maven precommit and have authored the first PR to drop the
>>>>>>>>> Maven precommit: https://github.com/apache/beam/pull/4814
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Well, I'd be for a strong switch otherwise PR will keep using
>>>>>>>>> maven, jenkins will not test the code and at the end we fail to deliver
>>>>>>>>> something consistent. So whatever tool is selected I'm tempted to say drop
>>>>>>>>> other build files and jenkins hooks references.
>>>>>>>>>
>>>>>>>>> What about doing it after 2.4 vote?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Mar 7, 2018 at 2:24 AM, Romain Manni-Bucau <
>>>>>>>>> rmannibucau@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Up,
>>>>>>>>>>
>>>>>>>>>> We discussed to have a strong switch to gradle or rollback to
>>>>>>>>>> maven around april to not be blocked by the build tool. I noticed gradle
>>>>>>>>>> build rarely passes on PR and kind of blurry our vision - not sure why
>>>>>>>>>> exactly. Also, PR don't always contain the gradle updates - generally
>>>>>>>>>> dependencies+plugins are added in pom.xml AFAIK, so it seems the adoption
>>>>>>>>>> is very slow to not say rejected.
>>>>>>>>>>
>>>>>>>>>> What do we do about that? When do we drop the double build
>>>>>>>>>> maintenance - whatever is picked?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Romain Manni-Bucau
>>>>>>>>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
>>>>>>>>>> <https://rmannibucau.metawerx.net/> | Old Blog
>>>>>>>>>> <http://rmannibucau.wordpress.com> | Github
>>>>>>>>>> <https://github.com/rmannibucau> | LinkedIn
>>>>>>>>>> <https://www.linkedin.com/in/rmannibucau> | Book
>>>>>>>>>> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>>>>>>>>>>
>>>>>>>>>> 2018-01-12 6:30 GMT+01:00 Romain Manni-Bucau <
>>>>>>>>>> rmannibucau@gmail.com>:
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Le 11 janv. 2018 23:13, "Kenneth Knowles" <kl...@google.com> a
>>>>>>>>>>> écrit :
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Jan 11, 2018 at 8:43 AM, Romain Manni-Bucau <
>>>>>>>>>>> rmannibucau@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> 2. gradle build doesn't use the same output directory than
>>>>>>>>>>>> maven so it is not really smooth to have both and have to maintain both
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I also have an opinion on this. It is useful and reasonable to
>>>>>>>>>>> be able to build even when the source is on a read-only filesystem. Maven's
>>>>>>>>>>> defaults are undesirable and require workarounds. We shouldn't mimic the
>>>>>>>>>>> behavior, but actually should set gradle up to build to a directory outside
>>>>>>>>>>> the source tree always, if we can.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Hmm, which is something you can do with maven as well so not
>>>>>>>>>>> sure I get it.
>>>>>>>>>>>
>>>>>>>>>>> Also note the thread is no more about the technical points but
>>>>>>>>>>> more the sources maintenance and consistency.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Kenn
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>
>>
--
-------
Jason Kuster
Apache Beam / Google Cloud Dataflow
Re: Gradle status
Posted by Robert Bradshaw <ro...@google.com>.
+1 to a fixit day. I'd be happy to help out myself.
On Wed, Mar 7, 2018 at 1:49 PM Henning Rohde <he...@google.com> wrote:
> +1 to a Gradle fixit day/week. I agree with Romain that we should make an
> effort quit this dual state. The Go, Go PreCommit and various container
> image Gradle builds, I think, are in a reasonable state (modulo some
> documentation updates).
>
> On Wed, Mar 7, 2018 at 1:29 PM, Kenneth Knowles <kl...@google.com> wrote:
>
>> SGTM. I should also say that every day is Gradle fixit day for me, as I
>> have been using only Gradle (with IntelliJ) for a while :-). If anyone is
>> hesitant, definitely it is ready to be used for normal dev.
>>
>> Seems like changing the messaging in onboarding docs is the main thing to
>> fixit.
>>
>> Based on https://builds.apache.org/view/A-D/view/Beam/ and our failure
>> spam level the performance tests are mostly not healthy anyhow. So is there
>> any high level blocker to switching them or is it just someone sitting down
>> with each one?
>>
>> Kenn
>>
>>
>> On Wed, Mar 7, 2018 at 1:22 PM Lukasz Cwik <lc...@google.com> wrote:
>>
>>> Largest outstanding areas are:
>>> * Documentation relevant to the contributors guide/release
>>> process/testing
>>> * Performance tests
>>>
>>> There has been good progress towards:
>>> * Release artifact validations and generation
>>> * ValidatesRunner post commits
>>> * Pre commits
>>> * Container builds
>>>
>>>
>>> On Wed, Mar 7, 2018 at 1:19 PM, Reuven Lax <re...@google.com> wrote:
>>>
>>>> I think Alan was making progress on the Gradle build.
>>>>
>>>> What do people think of a "fixit" day for Gradle work? (or given that
>>>> people are distributed, maybe a fixit week, where everyone takes one day
>>>> from the week).
>>>>
>>>>
>>>> On Wed, Mar 7, 2018 at 1:17 PM Kenneth Knowles <kl...@google.com> wrote:
>>>>
>>>>> I also cannot drop everything to work on Gradle build, but maybe it
>>>>> isn't that drastic anyhow. Now that we have ValidatesRunner and NeedsRunner
>>>>> tests and some progress on the release, is there any other known missing
>>>>> functionality in the Gradle builds? Archetypes? Docker container images?
>>>>>
>>>>>
>>>>> On Wed, Mar 7, 2018 at 1:12 PM Lukasz Cwik <lc...@google.com> wrote:
>>>>>
>>>>>> I am working on various projects and may not be able to pause my work
>>>>>> for a couple of weeks while the build/test process is migrated.
>>>>>>
>>>>>> What is everyone thinking about Romain's suggestion because If I'm
>>>>>> the only person in such a situation, I would be willing to go along with
>>>>>> the plan.
>>>>>>
>>>>>> On Wed, Mar 7, 2018 at 12:59 PM, Romain Manni-Bucau <
>>>>>> rmannibucau@gmail.com> wrote:
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Le 7 mars 2018 20:21, "Lukasz Cwik" <lc...@google.com> a écrit :
>>>>>>>
>>>>>>> Note that Alan Myrvold has been making steady progress making the
>>>>>>> release process via Gradle a reality:
>>>>>>> 1) Creating a jenkins job which can run the quickstart validation
>>>>>>> against nightly snapshots and also can be used for release candidates (
>>>>>>> https://github.com/apache/beam/pull/4252)
>>>>>>> 2) Building a release candidate using Gradle (
>>>>>>> https://github.com/apache/beam/pull/4812)
>>>>>>>
>>>>>>> Also, Gradle is the tool that has been selected already and there
>>>>>>> was a community discussion about what was needed for the migration to occur
>>>>>>> with a clear set of criteria. Romain, it doesn't seem like we should ignore
>>>>>>> that criteria or are you suggesting we change that criteria (if yes, how)?
>>>>>>>
>>>>>>>
>>>>>>> No, no. My goal is just to quit this state.
>>>>>>>
>>>>>>> Let s draft a plan:
>>>>>>>
>>>>>>> 1. 2.4 is released - i assume it is done with mvn here
>>>>>>> 2. We drop all poms and jenkins mvn config
>>>>>>> 3. We fix all build issues if so (let say in a week)
>>>>>>> 4. Pr can nees updates but no more mvn merge
>>>>>>>
>>>>>>> April is gradle month :)
>>>>>>>
>>>>>>> Wdyt?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Mar 7, 2018 at 10:39 AM, Romain Manni-Bucau <
>>>>>>> rmannibucau@gmail.com> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Le 7 mars 2018 17:34, "Lukasz Cwik" <lc...@google.com> a écrit :
>>>>>>>>
>>>>>>>> Thanks for bringing this up Romain but I believe your data points
>>>>>>>> on pass rates are only partially correct.
>>>>>>>>
>>>>>>>>
>>>>>>>> Sure sure, it is mainly about my own PR which a very small % of the
>>>>>>>> whole project ;).
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> In the past week the Java Gradle precommit passed 46.34% of the
>>>>>>>> time compared to the Java Maven precommit which passed 46.15% of the time.
>>>>>>>> When I looked at these numbers in mid January they were around 37% so there
>>>>>>>> has been some improvement. Regardless of the build tool it seems that our
>>>>>>>> pass rates aren't stellar for the Java build and are causing the community
>>>>>>>> to not follow best practices (wait for precommits to be green before
>>>>>>>> merging). I know that on the website we used the mergebot to ensure that
>>>>>>>> things passed before they were merged, should we institute this on the
>>>>>>>> master branch or are their any other ideas?
>>>>>>>>
>>>>>>>> As a side note we had achieved the goals we set out to not need to
>>>>>>>> maintain the Maven precommit and have authored the first PR to drop the
>>>>>>>> Maven precommit: https://github.com/apache/beam/pull/4814
>>>>>>>>
>>>>>>>>
>>>>>>>> Well, I'd be for a strong switch otherwise PR will keep using
>>>>>>>> maven, jenkins will not test the code and at the end we fail to deliver
>>>>>>>> something consistent. So whatever tool is selected I'm tempted to say drop
>>>>>>>> other build files and jenkins hooks references.
>>>>>>>>
>>>>>>>> What about doing it after 2.4 vote?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Mar 7, 2018 at 2:24 AM, Romain Manni-Bucau <
>>>>>>>> rmannibucau@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Up,
>>>>>>>>>
>>>>>>>>> We discussed to have a strong switch to gradle or rollback to
>>>>>>>>> maven around april to not be blocked by the build tool. I noticed gradle
>>>>>>>>> build rarely passes on PR and kind of blurry our vision - not sure why
>>>>>>>>> exactly. Also, PR don't always contain the gradle updates - generally
>>>>>>>>> dependencies+plugins are added in pom.xml AFAIK, so it seems the adoption
>>>>>>>>> is very slow to not say rejected.
>>>>>>>>>
>>>>>>>>> What do we do about that? When do we drop the double build
>>>>>>>>> maintenance - whatever is picked?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Romain Manni-Bucau
>>>>>>>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
>>>>>>>>> <https://rmannibucau.metawerx.net/> | Old Blog
>>>>>>>>> <http://rmannibucau.wordpress.com> | Github
>>>>>>>>> <https://github.com/rmannibucau> | LinkedIn
>>>>>>>>> <https://www.linkedin.com/in/rmannibucau> | Book
>>>>>>>>> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>>>>>>>>>
>>>>>>>>> 2018-01-12 6:30 GMT+01:00 Romain Manni-Bucau <
>>>>>>>>> rmannibucau@gmail.com>:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Le 11 janv. 2018 23:13, "Kenneth Knowles" <kl...@google.com> a
>>>>>>>>>> écrit :
>>>>>>>>>>
>>>>>>>>>> On Thu, Jan 11, 2018 at 8:43 AM, Romain Manni-Bucau <
>>>>>>>>>> rmannibucau@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> 2. gradle build doesn't use the same output directory than maven
>>>>>>>>>>> so it is not really smooth to have both and have to maintain both
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I also have an opinion on this. It is useful and reasonable to be
>>>>>>>>>> able to build even when the source is on a read-only filesystem. Maven's
>>>>>>>>>> defaults are undesirable and require workarounds. We shouldn't mimic the
>>>>>>>>>> behavior, but actually should set gradle up to build to a directory outside
>>>>>>>>>> the source tree always, if we can.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Hmm, which is something you can do with maven as well so not sure
>>>>>>>>>> I get it.
>>>>>>>>>>
>>>>>>>>>> Also note the thread is no more about the technical points but
>>>>>>>>>> more the sources maintenance and consistency.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Kenn
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>
>
Re: Gradle status
Posted by Henning Rohde <he...@google.com>.
+1 to a Gradle fixit day/week. I agree with Romain that we should make an
effort quit this dual state. The Go, Go PreCommit and various container
image Gradle builds, I think, are in a reasonable state (modulo some
documentation updates).
On Wed, Mar 7, 2018 at 1:29 PM, Kenneth Knowles <kl...@google.com> wrote:
> SGTM. I should also say that every day is Gradle fixit day for me, as I
> have been using only Gradle (with IntelliJ) for a while :-). If anyone is
> hesitant, definitely it is ready to be used for normal dev.
>
> Seems like changing the messaging in onboarding docs is the main thing to
> fixit.
>
> Based on https://builds.apache.org/view/A-D/view/Beam/ and our failure
> spam level the performance tests are mostly not healthy anyhow. So is there
> any high level blocker to switching them or is it just someone sitting down
> with each one?
>
> Kenn
>
>
> On Wed, Mar 7, 2018 at 1:22 PM Lukasz Cwik <lc...@google.com> wrote:
>
>> Largest outstanding areas are:
>> * Documentation relevant to the contributors guide/release process/testing
>> * Performance tests
>>
>> There has been good progress towards:
>> * Release artifact validations and generation
>> * ValidatesRunner post commits
>> * Pre commits
>> * Container builds
>>
>>
>> On Wed, Mar 7, 2018 at 1:19 PM, Reuven Lax <re...@google.com> wrote:
>>
>>> I think Alan was making progress on the Gradle build.
>>>
>>> What do people think of a "fixit" day for Gradle work? (or given that
>>> people are distributed, maybe a fixit week, where everyone takes one day
>>> from the week).
>>>
>>>
>>> On Wed, Mar 7, 2018 at 1:17 PM Kenneth Knowles <kl...@google.com> wrote:
>>>
>>>> I also cannot drop everything to work on Gradle build, but maybe it
>>>> isn't that drastic anyhow. Now that we have ValidatesRunner and NeedsRunner
>>>> tests and some progress on the release, is there any other known missing
>>>> functionality in the Gradle builds? Archetypes? Docker container images?
>>>>
>>>>
>>>> On Wed, Mar 7, 2018 at 1:12 PM Lukasz Cwik <lc...@google.com> wrote:
>>>>
>>>>> I am working on various projects and may not be able to pause my work
>>>>> for a couple of weeks while the build/test process is migrated.
>>>>>
>>>>> What is everyone thinking about Romain's suggestion because If I'm the
>>>>> only person in such a situation, I would be willing to go along with the
>>>>> plan.
>>>>>
>>>>> On Wed, Mar 7, 2018 at 12:59 PM, Romain Manni-Bucau <
>>>>> rmannibucau@gmail.com> wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> Le 7 mars 2018 20:21, "Lukasz Cwik" <lc...@google.com> a écrit :
>>>>>>
>>>>>> Note that Alan Myrvold has been making steady progress making the
>>>>>> release process via Gradle a reality:
>>>>>> 1) Creating a jenkins job which can run the quickstart validation
>>>>>> against nightly snapshots and also can be used for release candidates (
>>>>>> https://github.com/apache/beam/pull/4252)
>>>>>> 2) Building a release candidate using Gradle (
>>>>>> https://github.com/apache/beam/pull/4812)
>>>>>>
>>>>>> Also, Gradle is the tool that has been selected already and there was
>>>>>> a community discussion about what was needed for the migration to occur
>>>>>> with a clear set of criteria. Romain, it doesn't seem like we should ignore
>>>>>> that criteria or are you suggesting we change that criteria (if yes, how)?
>>>>>>
>>>>>>
>>>>>> No, no. My goal is just to quit this state.
>>>>>>
>>>>>> Let s draft a plan:
>>>>>>
>>>>>> 1. 2.4 is released - i assume it is done with mvn here
>>>>>> 2. We drop all poms and jenkins mvn config
>>>>>> 3. We fix all build issues if so (let say in a week)
>>>>>> 4. Pr can nees updates but no more mvn merge
>>>>>>
>>>>>> April is gradle month :)
>>>>>>
>>>>>> Wdyt?
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Mar 7, 2018 at 10:39 AM, Romain Manni-Bucau <
>>>>>> rmannibucau@gmail.com> wrote:
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Le 7 mars 2018 17:34, "Lukasz Cwik" <lc...@google.com> a écrit :
>>>>>>>
>>>>>>> Thanks for bringing this up Romain but I believe your data points on
>>>>>>> pass rates are only partially correct.
>>>>>>>
>>>>>>>
>>>>>>> Sure sure, it is mainly about my own PR which a very small % of the
>>>>>>> whole project ;).
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> In the past week the Java Gradle precommit passed 46.34% of the time
>>>>>>> compared to the Java Maven precommit which passed 46.15% of the time. When
>>>>>>> I looked at these numbers in mid January they were around 37% so there has
>>>>>>> been some improvement. Regardless of the build tool it seems that our pass
>>>>>>> rates aren't stellar for the Java build and are causing the community to
>>>>>>> not follow best practices (wait for precommits to be green before merging).
>>>>>>> I know that on the website we used the mergebot to ensure that things
>>>>>>> passed before they were merged, should we institute this on the master
>>>>>>> branch or are their any other ideas?
>>>>>>>
>>>>>>> As a side note we had achieved the goals we set out to not need to
>>>>>>> maintain the Maven precommit and have authored the first PR to drop the
>>>>>>> Maven precommit: https://github.com/apache/beam/pull/4814
>>>>>>>
>>>>>>>
>>>>>>> Well, I'd be for a strong switch otherwise PR will keep using maven,
>>>>>>> jenkins will not test the code and at the end we fail to deliver something
>>>>>>> consistent. So whatever tool is selected I'm tempted to say drop other
>>>>>>> build files and jenkins hooks references.
>>>>>>>
>>>>>>> What about doing it after 2.4 vote?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Mar 7, 2018 at 2:24 AM, Romain Manni-Bucau <
>>>>>>> rmannibucau@gmail.com> wrote:
>>>>>>>
>>>>>>>> Up,
>>>>>>>>
>>>>>>>> We discussed to have a strong switch to gradle or rollback to maven
>>>>>>>> around april to not be blocked by the build tool. I noticed gradle build
>>>>>>>> rarely passes on PR and kind of blurry our vision - not sure why exactly.
>>>>>>>> Also, PR don't always contain the gradle updates - generally
>>>>>>>> dependencies+plugins are added in pom.xml AFAIK, so it seems the adoption
>>>>>>>> is very slow to not say rejected.
>>>>>>>>
>>>>>>>> What do we do about that? When do we drop the double build
>>>>>>>> maintenance - whatever is picked?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Romain Manni-Bucau
>>>>>>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
>>>>>>>> <https://rmannibucau.metawerx.net/> | Old Blog
>>>>>>>> <http://rmannibucau.wordpress.com> | Github
>>>>>>>> <https://github.com/rmannibucau> | LinkedIn
>>>>>>>> <https://www.linkedin.com/in/rmannibucau> | Book
>>>>>>>> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>>>>>>>>
>>>>>>>> 2018-01-12 6:30 GMT+01:00 Romain Manni-Bucau <rmannibucau@gmail.com
>>>>>>>> >:
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Le 11 janv. 2018 23:13, "Kenneth Knowles" <kl...@google.com> a
>>>>>>>>> écrit :
>>>>>>>>>
>>>>>>>>> On Thu, Jan 11, 2018 at 8:43 AM, Romain Manni-Bucau <
>>>>>>>>> rmannibucau@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> 2. gradle build doesn't use the same output directory than maven
>>>>>>>>>> so it is not really smooth to have both and have to maintain both
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I also have an opinion on this. It is useful and reasonable to be
>>>>>>>>> able to build even when the source is on a read-only filesystem. Maven's
>>>>>>>>> defaults are undesirable and require workarounds. We shouldn't mimic the
>>>>>>>>> behavior, but actually should set gradle up to build to a directory outside
>>>>>>>>> the source tree always, if we can.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hmm, which is something you can do with maven as well so not sure
>>>>>>>>> I get it.
>>>>>>>>>
>>>>>>>>> Also note the thread is no more about the technical points but
>>>>>>>>> more the sources maintenance and consistency.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Kenn
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>
Re: Gradle status
Posted by Lukasz Cwik <lc...@google.com>.
Romain, that is incorrect. Contributors are to maintain both systems until
Gradle replaces that functionality in Maven and then it only needs to be
maintained in Gradle.
On Thu, Mar 22, 2018 at 10:30 AM Romain Manni-Bucau <rm...@gmail.com>
wrote:
> Ok so to be clear for any contributor (which is the goal of this thread):
> maven is still the main build system and no need to maintain gradle in PR
> then until beam switches.
>
> Im more than fine with that.
>
> Le 22 mars 2018 18:22, "Alan Myrvold" <am...@google.com> a écrit :
>
>> I think the investment in gradle is worthwhile, and incrementally we will
>> continue to make progress. From what I've seem, gradle is a good fit for
>> this project and a path to a faster, more reliable build system.
>>
>> pull/4812 <https://github.com/apache/beam/pull/4812> creates the release
>> artifacts, although it is not hooked up yet with authentication.
>>
>> I expect to help make incremental progress over the next month converting
>> some of the integration tests, but welcome incremental improvements from
>> others.
>>
>>
>>
>> On Thu, Mar 22, 2018 at 9:57 AM Romain Manni-Bucau <rm...@gmail.com>
>> wrote:
>>
>>>
>>>
>>> 2018-03-22 17:45 GMT+01:00 Lukasz Cwik <lc...@google.com>:
>>>
>>>> what do we do? "Gradle migration will happen incrementally."
>>>>
>>>> "last months prooved beam cant maintain 2 systems, easier with that
>>>> state is then to drop gradle since it is a 0 investment compared to the
>>>> opposite"
>>>> Its unfortunate that you feel this way but many people do not share
>>>> your opinion.
>>>>
>>>
>>> And a lot do so when a project is 50-50 it is time to act.
>>>
>>> Incrementally kind of means never (makes 4 months and nothing really
>>> changed in PRs and habits, gradle maintener(s) are still alone)
>>>
>>>
>>>>
>>>>
>>>> On Thu, Mar 22, 2018 at 9:32 AM Romain Manni-Bucau <
>>>> rmannibucau@gmail.com> wrote:
>>>>
>>>>> @Valentyn: concretely any user can PR and be part of that process so
>>>>> anyone can do it wrong (me first)
>>>>> @Luskasz, Hennking: fine but what do we do? last months prooved beam
>>>>> cant maintain 2 systems, easier with that state is then to drop gradle
>>>>> since it is a 0 investment compared to the opposite
>>>>>
>>>>>
>>>>> Romain Manni-Bucau
>>>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
>>>>> <https://rmannibucau.metawerx.net/> | Old Blog
>>>>> <http://rmannibucau.wordpress.com> | Github
>>>>> <https://github.com/rmannibucau> | LinkedIn
>>>>> <https://www.linkedin.com/in/rmannibucau> | Book
>>>>> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>>>>>
>>>>> 2018-03-22 17:24 GMT+01:00 Lukasz Cwik <lc...@google.com>:
>>>>>
>>>>>> Romain, from the previous discussions several people agreed that
>>>>>> running a fixit that migrated Maven to Gradle over a 1 or 2 day period was
>>>>>> worthwhile but there was nobody in the community with the time commitment
>>>>>> to organize and run it so the status quo plan remained where the Gradle
>>>>>> migration will happen incrementally.
>>>>>>
>>>>>>
>>>>>> On Thu, Mar 22, 2018 at 8:53 AM Henning Rohde <he...@google.com>
>>>>>> wrote:
>>>>>>
>>>>>>> My understanding was the same as Ismaël's. I don't think breaking
>>>>>>> the build with a large known gaps (but not fully known cost) is practical.
>>>>>>> Also, most items in the jira are not even assigned yet.
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Mar 22, 2018 at 8:03 AM Romain Manni-Bucau <
>>>>>>> rmannibucau@gmail.com> wrote:
>>>>>>>
>>>>>>>> Not really Ismaël, this thread was about to do it at once and have
>>>>>>>> 1 day to fix it all.
>>>>>>>>
>>>>>>>> As mentionned at the very beginning nobody maintains the 2 system
>>>>>>>> so it must stop after months so either we drop maven or gradle *at once*
>>>>>>>> or we keep a state where each dev does what he wants and the build
>>>>>>>> system just doesn't work.
>>>>>>>>
>>>>>>>> 2018-03-22 15:42 GMT+01:00 Ismaël Mejía <ie...@gmail.com>:
>>>>>>>>
>>>>>>>>> I don't think that removing all maven descriptors was the expected
>>>>>>>>> path, no ? Or even a good idea at this moment.
>>>>>>>>>
>>>>>>>>> I understood that what we were going to do was to replace
>>>>>>>>> incrementally the CI until we cover the whole maven functionality
>>>>>>>>> and
>>>>>>>>> then remove it, from looking at the JIRA ticket
>>>>>>>>> https://issues.apache.org/jira/browse/BEAM-3249 we are still far
>>>>>>>>> from
>>>>>>>>> covering the complete maven functionality in particular for the
>>>>>>>>> release part that could be the biggest pain point.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Mar 22, 2018 at 9:30 AM, Romain Manni-Bucau
>>>>>>>>> <rm...@gmail.com> wrote:
>>>>>>>>> > hey guys,
>>>>>>>>> >
>>>>>>>>> > 2.4 is out, do we plan to drop all maven descriptors tomorrow or
>>>>>>>>> on monday?
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > Romain Manni-Bucau
>>>>>>>>> > @rmannibucau | Blog | Old Blog | Github | LinkedIn | Book
>>>>>>>>> >
>>>>>>>>> > 2018-03-09 21:42 GMT+01:00 Kenneth Knowles <kl...@google.com>:
>>>>>>>>> >>
>>>>>>>>> >> On Fri, Mar 9, 2018 at 12:16 PM Lukasz Cwik <lc...@google.com>
>>>>>>>>> wrote:
>>>>>>>>> >>>
>>>>>>>>> >>> Based upon your description it seems as though you would
>>>>>>>>> rather have a
>>>>>>>>> >>> way to run existing postcommits without it impacting
>>>>>>>>> dashboard/health
>>>>>>>>> >>> stats/notifications/.... (We have just run the PostCommits on
>>>>>>>>> PRs for
>>>>>>>>> >>> additional validation (like upgrading the Dataflow container
>>>>>>>>> image)).
>>>>>>>>> >>
>>>>>>>>> >>
>>>>>>>>> >> Yes, that is exactly what I have described.
>>>>>>>>> >>
>>>>>>>>> >>> I don't think that keeping the current Java PreCommit as a
>>>>>>>>> proxy for the
>>>>>>>>> >>> the Java PostCommit is the right way to go but I also don't
>>>>>>>>> have the time to
>>>>>>>>> >>> implement what your actually asking for.
>>>>>>>>> >>
>>>>>>>>> >>
>>>>>>>>> >> Mostly I thought this might be very easy based on the fact that
>>>>>>>>> they are
>>>>>>>>> >> nearly identical. If not, oh well.
>>>>>>>>> >>
>>>>>>>>> >> Kenn
>>>>>>>>> >>
>>>>>>>>> >>
>>>>>>>>> >>> It seems more likely that migrating the PostCommit to Gradle
>>>>>>>>> will be less
>>>>>>>>> >>> work then adding the functionality but your argument where the
>>>>>>>>> PreCommit is
>>>>>>>>> >>> a proxy for the Java PostCommit also applies to the
>>>>>>>>> ValidatesRunner
>>>>>>>>> >>> PostCommits and so forth requiring even more migration to
>>>>>>>>> happen before you
>>>>>>>>> >>> don't have to worry about maintaining Maven/breaking post
>>>>>>>>> commits.
>>>>>>>>> >>>
>>>>>>>>> >>> I'm fine with leaving both the Java/Gradle PreCommits running
>>>>>>>>> for now and
>>>>>>>>> >>> hopefully as more of the PostCommits are migrated off we will
>>>>>>>>> be able to
>>>>>>>>> >>> remove it.
>>>>>>>>> >>>
>>>>>>>>> >>> On Fri, Mar 9, 2018 at 11:39 AM, Kenneth Knowles <
>>>>>>>>> klk@google.com> wrote:
>>>>>>>>> >>>>
>>>>>>>>> >>>> Separate history (for easy dashboarding, health stats, etc)
>>>>>>>>> and
>>>>>>>>> >>>> notification (email to dev@ for postcommits, nothing for
>>>>>>>>> precommits) for pre
>>>>>>>>> >>>> & post commit targets.
>>>>>>>>> >>>>
>>>>>>>>> >>>> A post commit failure is always a problem to be triaged at
>>>>>>>>> high
>>>>>>>>> >>>> priority, while a precommit failure is just a natural
>>>>>>>>> occurrence.
>>>>>>>>> >>>>
>>>>>>>>> >>>> On Fri, Mar 9, 2018 at 11:33 AM Lukasz Cwik <lc...@google.com>
>>>>>>>>> wrote:
>>>>>>>>> >>>>>
>>>>>>>>> >>>>> Ken, I'm probably not seeing something but how does using
>>>>>>>>> the PreCommit
>>>>>>>>> >>>>> as a proxy improve upon just running the post commit via the
>>>>>>>>> phrase it
>>>>>>>>> >>>>> already supports ('Run Java PostCommit')?
>>>>>>>>> >>>>>
>>>>>>>>> >>>>> On Fri, Mar 9, 2018 at 11:22 AM, Kenneth Knowles <
>>>>>>>>> klk@google.com>
>>>>>>>>> >>>>> wrote:
>>>>>>>>> >>>>>>
>>>>>>>>> >>>>>> Indeed, we've already had the discussion a couple of times
>>>>>>>>> and I think
>>>>>>>>> >>>>>> the criteria are clearly met. Incremental progress is a
>>>>>>>>> good thing and we
>>>>>>>>> >>>>>> shouldn't block it.
>>>>>>>>> >>>>>>
>>>>>>>>> >>>>>> OTOH I see where Romain is coming from and I have a good
>>>>>>>>> example that
>>>>>>>>> >>>>>> supports a slightly different action. Consider
>>>>>>>>> >>>>>> https://github.com/apache/beam/pull/4740 which fixes some
>>>>>>>>> errors in how we
>>>>>>>>> >>>>>> use dependency mechanisms.
>>>>>>>>> >>>>>>
>>>>>>>>> >>>>>> This PR is green except that I need to fix some Maven pom
>>>>>>>>> slightly
>>>>>>>>> >>>>>> more. That is throwaway work. I would love to just not have
>>>>>>>>> to do it. But
>>>>>>>>> >>>>>> removing the precommit does not actually make the PR OK to
>>>>>>>>> merge. It would
>>>>>>>>> >>>>>> cause postcommits to fail.
>>>>>>>>> >>>>>>
>>>>>>>>> >>>>>> We can hope such situations are rare. I think I tend to be
>>>>>>>>> hit by this
>>>>>>>>> >>>>>> more often than most, as I work with the project build
>>>>>>>>> health quite a bit.
>>>>>>>>> >>>>>>
>>>>>>>>> >>>>>> Here is a proposal to support these things: instead of
>>>>>>>>> deleting the
>>>>>>>>> >>>>>> job in #4814, move it to not run automatically but only via
>>>>>>>>> a phrase. In
>>>>>>>>> >>>>>> fact, you could migrate it to be the manually-invoked
>>>>>>>>> version of the
>>>>>>>>> >>>>>> postcommit job as we've discussed a couple times. Then if
>>>>>>>>> someone is working
>>>>>>>>> >>>>>> on the build in something like #4740 they can invoke it
>>>>>>>>> manually.
>>>>>>>>> >>>>>>
>>>>>>>>> >>>>>> Kenn
>>>>>>>>> >>>>>>
>>>>>>>>> >>>>>> On Fri, Mar 9, 2018 at 10:25 AM Lukasz Cwik <
>>>>>>>>> lcwik@google.com> wrote:
>>>>>>>>> >>>>>>>
>>>>>>>>> >>>>>>> Based upon the criteria that was discussed on the mailing
>>>>>>>>> list[1], I
>>>>>>>>> >>>>>>> would agree with Kenn about merging PR/4814 (drop Java
>>>>>>>>> Maven precommit).
>>>>>>>>> >>>>>>>
>>>>>>>>> >>>>>>> 1:
>>>>>>>>> >>>>>>>
>>>>>>>>> https://lists.apache.org/thread.html/7eba5c77bc1a77b5046d915ab59f5f6fc41536c2c84863ad2efb5e99@%3Cdev.beam.apache.org%3E
>>>>>>>>> >>>>>>>
>>>>>>>>> >>>>>>> On Thu, Mar 8, 2018 at 9:10 PM, Romain Manni-Bucau
>>>>>>>>> >>>>>>> <rm...@gmail.com> wrote:
>>>>>>>>> >>>>>>>>
>>>>>>>>> >>>>>>>> Hi Kenneth,
>>>>>>>>> >>>>>>>>
>>>>>>>>> >>>>>>>> For now maven covers the full needs of beam. If we start
>>>>>>>>> to have
>>>>>>>>> >>>>>>>> this kind of PR we become dependent of the 2 builds which
>>>>>>>>> is what this
>>>>>>>>> >>>>>>>> thread is about avoiding so tempted to say it must be a
>>>>>>>>> PR drop completely
>>>>>>>>> >>>>>>>> maven or nothing as mentionned before.
>>>>>>>>> >>>>>>>>
>>>>>>>>> >>>>>>>> Le 9 mars 2018 04:48, "Kenneth Knowles" <kl...@google.com>
>>>>>>>>> a écrit :
>>>>>>>>> >>>>>>>>>
>>>>>>>>> >>>>>>>>> I would like to briefly re-focus this discussion and
>>>>>>>>> suggest that
>>>>>>>>> >>>>>>>>> we merge https://github.com/apache/beam/pull/4814.
>>>>>>>>> >>>>>>>>>
>>>>>>>>> >>>>>>>>> The only material objection I've heard is that it means
>>>>>>>>> the
>>>>>>>>> >>>>>>>>> precommit no longer tests exactly what is built for
>>>>>>>>> release. It is a valid
>>>>>>>>> >>>>>>>>> concern, but we have mvn postcommit so the coverage is
>>>>>>>>> not lost. The
>>>>>>>>> >>>>>>>>> benefits of quicker reviews and less Jenkins congestion
>>>>>>>>> seem worth it to me.
>>>>>>>>> >>>>>>>>>
>>>>>>>>> >>>>>>>>> Kenn
>>>>>>>>> >>>>>>>>>
>>>>>>>>> >>>>>>>>> On Thu, Mar 8, 2018 at 12:33 PM Romain Manni-Bucau
>>>>>>>>> >>>>>>>>> <rm...@gmail.com> wrote:
>>>>>>>>> >>>>>>>>>>
>>>>>>>>> >>>>>>>>>> @Luskasz: not sure Im the best to host it since I know
>>>>>>>>> more gradle
>>>>>>>>> >>>>>>>>>> internals that user interface/ecosystem but happy to
>>>>>>>>> help. Will also require
>>>>>>>>> >>>>>>>>>> a "sudo" merger for this day to merge fixes asap -
>>>>>>>>> guess we can bypass
>>>>>>>>> >>>>>>>>>> reviews or have a fast cycle plan for this day to avoid
>>>>>>>>> it to be a week?
>>>>>>>>> >>>>>>>>>>
>>>>>>>>> >>>>>>>>>> Le 8 mars 2018 21:08, "Kenneth Knowles" <kl...@google.com>
>>>>>>>>> a écrit :
>>>>>>>>> >>>>>>>>>>
>>>>>>>>> >>>>>>>>>> @Romain - Idea/IntelliJ is great with Gradle, way
>>>>>>>>> better than
>>>>>>>>> >>>>>>>>>> Maven, and what I mean is that I have added enough
>>>>>>>>> hints that it works OOTB
>>>>>>>>> >>>>>>>>>> already. The rest of my instructions are just how you
>>>>>>>>> should override
>>>>>>>>> >>>>>>>>>> IntelliJ's defaults to have a proper dev env - mostly
>>>>>>>>> just about storing
>>>>>>>>> >>>>>>>>>> files outside the git clone.
>>>>>>>>> >>>>>>>>>>
>>>>>>>>> >>>>>>>>>>
>>>>>>>>> >>>>>>>>>> Hmm it is always super slow here and not as integrated
>>>>>>>>> as maven
>>>>>>>>> >>>>>>>>>> which is smooth thanks to the combination of idea build
>>>>>>>>> and maven plugin
>>>>>>>>> >>>>>>>>>> config read. Import is also faster cause it just reads
>>>>>>>>> meta and doesnt run
>>>>>>>>> >>>>>>>>>> anything. Hope it is a "a few times" issues at the
>>>>>>>>> moment but not yet sure.
>>>>>>>>> >>>>>>>>>>
>>>>>>>>> >>>>>>>>>>
>>>>>>>>> >>>>>>>>>>
>>>>>>>>> >>>>>>>>>> @Łukasz - yea I just mean the ones that I find in my
>>>>>>>>> email and
>>>>>>>>> >>>>>>>>>> browsing https://builds.apache.org/view/A-D/view/Beam/
>>>>>>>>> which are the ones
>>>>>>>>> >>>>>>>>>> you classify as "totally not working". I would love to
>>>>>>>>> disable these.
>>>>>>>>> >>>>>>>>>>
>>>>>>>>> >>>>>>>>>> On the subject of running things on a pending PR - we
>>>>>>>>> should not
>>>>>>>>> >>>>>>>>>> run postcommit jobs on PRs. We should make separate
>>>>>>>>> (optional) precommit
>>>>>>>>> >>>>>>>>>> jobs that run the same build commands. That will give a
>>>>>>>>> more clear history
>>>>>>>>> >>>>>>>>>> and allow trivial configuration of which jobs deserve
>>>>>>>>> alert emails and which
>>>>>>>>> >>>>>>>>>> are not a problem. This is easy but I've been waiting
>>>>>>>>> to do it after Gradle
>>>>>>>>> >>>>>>>>>> migration.
>>>>>>>>> >>>>>>>>>>
>>>>>>>>> >>>>>>>>>> On Thu, Mar 8, 2018 at 11:37 AM Lukasz Cwik <
>>>>>>>>> lcwik@google.com>
>>>>>>>>> >>>>>>>>>> wrote:
>>>>>>>>> >>>>>>>>>>>
>>>>>>>>> >>>>>>>>>>> Romain, would you like to host/plan/run the Gradle
>>>>>>>>> fixit day?
>>>>>>>>> >>>>>>>>>>>
>>>>>>>>> >>>>>>>>>>> On Thu, Mar 8, 2018 at 11:24 AM, Chamikara Jayalath
>>>>>>>>> >>>>>>>>>>> <ch...@google.com> wrote:
>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>> >>>>>>>>>>>> +1 for the general idea of fixit day/week for Gradle.
>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>> >>>>>>>>>>>> Agree with what Łukasz said. Some of these
>>>>>>>>> performance tests are
>>>>>>>>> >>>>>>>>>>>> new and are flaky due to other issues that were
>>>>>>>>> discovered during the
>>>>>>>>> >>>>>>>>>>>> process of adding the test.
>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>> >>>>>>>>>>>> I think the high level blocker is updating
>>>>>>>>> performance testing
>>>>>>>>> >>>>>>>>>>>> framework to use Gradle. This will involve adding
>>>>>>>>> Gradle-based logic for
>>>>>>>>> >>>>>>>>>>>> invoking PerfKitBenchmaker, for example [1], and
>>>>>>>>> updating PerfKitBenchmarker
>>>>>>>>> >>>>>>>>>>>> to issue a Gradle command for running the benchmark
>>>>>>>>> [2]. First task will be
>>>>>>>>> >>>>>>>>>>>> to find the work needed here and updating the
>>>>>>>>> relevant JIRA [3].
>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>> >>>>>>>>>>>> Thanks,
>>>>>>>>> >>>>>>>>>>>> Cham
>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>> >>>>>>>>>>>> [1]
>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/pom.xml#L79
>>>>>>>>> >>>>>>>>>>>> [2]
>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>> https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/blob/master/perfkitbenchmarker/beam_benchmark_helper.py
>>>>>>>>> >>>>>>>>>>>> [3] https://issues.apache.org/jira/browse/BEAM-3251
>>>>>>>>> >>>>>>>>>>>>
>>>>>>>>> >>>>>>>>>>>> On Thu, Mar 8, 2018 at 3:10 AM Łukasz Gajowy
>>>>>>>>> >>>>>>>>>>>> <lu...@gmail.com> wrote:
>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>> >>>>>>>>>>>>> 2018-03-07 22:29 GMT+01:00 Kenneth Knowles <
>>>>>>>>> klk@google.com>:
>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>> >>>>>>>>>>>>>> Based on
>>>>>>>>> https://builds.apache.org/view/A-D/view/Beam/ and our
>>>>>>>>> >>>>>>>>>>>>>> failure spam level the performance tests are mostly
>>>>>>>>> not healthy anyhow. So
>>>>>>>>> >>>>>>>>>>>>>> is there any high level blocker to switching them
>>>>>>>>> or is it just someone
>>>>>>>>> >>>>>>>>>>>>>> sitting down with each one?
>>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>> >>>>>>>>>>>>> I thought I could share my point of view here as I
>>>>>>>>> am working
>>>>>>>>> >>>>>>>>>>>>> on the Performance Test part for a while now. I
>>>>>>>>> wouldn't say those are
>>>>>>>>> >>>>>>>>>>>>> "mostly not healthy". The situation is as follows:
>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>> >>>>>>>>>>>>> - totally not working: Python, Spark performance
>>>>>>>>> tests (don't
>>>>>>>>> >>>>>>>>>>>>> know why, I'm not maintaining those tests. Should we
>>>>>>>>> disable them?)
>>>>>>>>> >>>>>>>>>>>>> - flaky: the recently re-enabled JDBC Performance
>>>>>>>>> Test. It's
>>>>>>>>> >>>>>>>>>>>>> seems to be flaky mostly due to:
>>>>>>>>> >>>>>>>>>>>>> https://issues.apache.org/jira/browse/BEAM-3798
>>>>>>>>> >>>>>>>>>>>>> - working well, rarely failing: AvroIO, TextIO,
>>>>>>>>> Compressed
>>>>>>>>> >>>>>>>>>>>>> Text, TFRecordIO
>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>> >>>>>>>>>>>>> Also, some test failures are caused due to pending
>>>>>>>>> PRs (we
>>>>>>>>> >>>>>>>>>>>>> sometimes run concrete tests to check if PRs won't
>>>>>>>>> break them). This also
>>>>>>>>> >>>>>>>>>>>>> causes failures sometimes.
>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>> >>>>>>>>>>>>> I can help with switching the Performance tests to
>>>>>>>>> Gradle as
>>>>>>>>> >>>>>>>>>>>>> this part seems to be free to take.
>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>> >>>>>>>>>>>>> Łukasz
>>>>>>>>> >>>>>>>>>>>>>
>>>>>>>>> >>>>>>>>>>>
>>>>>>>>> >>>>>>>>>>
>>>>>>>>> >>>>>>>
>>>>>>>>> >>>>>
>>>>>>>>> >>>
>>>>>>>>> >
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>
>>>
Re: Gradle status
Posted by Romain Manni-Bucau <rm...@gmail.com>.
Ok so to be clear for any contributor (which is the goal of this thread):
maven is still the main build system and no need to maintain gradle in PR
then until beam switches.
Im more than fine with that.
Le 22 mars 2018 18:22, "Alan Myrvold" <am...@google.com> a écrit :
> I think the investment in gradle is worthwhile, and incrementally we will
> continue to make progress. From what I've seem, gradle is a good fit for
> this project and a path to a faster, more reliable build system.
>
> pull/4812 <https://github.com/apache/beam/pull/4812> creates the release
> artifacts, although it is not hooked up yet with authentication.
>
> I expect to help make incremental progress over the next month converting
> some of the integration tests, but welcome incremental improvements from
> others.
>
>
>
> On Thu, Mar 22, 2018 at 9:57 AM Romain Manni-Bucau <rm...@gmail.com>
> wrote:
>
>>
>>
>> 2018-03-22 17:45 GMT+01:00 Lukasz Cwik <lc...@google.com>:
>>
>>> what do we do? "Gradle migration will happen incrementally."
>>>
>>> "last months prooved beam cant maintain 2 systems, easier with that
>>> state is then to drop gradle since it is a 0 investment compared to the
>>> opposite"
>>> Its unfortunate that you feel this way but many people do not share your
>>> opinion.
>>>
>>
>> And a lot do so when a project is 50-50 it is time to act.
>>
>> Incrementally kind of means never (makes 4 months and nothing really
>> changed in PRs and habits, gradle maintener(s) are still alone)
>>
>>
>>>
>>>
>>> On Thu, Mar 22, 2018 at 9:32 AM Romain Manni-Bucau <
>>> rmannibucau@gmail.com> wrote:
>>>
>>>> @Valentyn: concretely any user can PR and be part of that process so
>>>> anyone can do it wrong (me first)
>>>> @Luskasz, Hennking: fine but what do we do? last months prooved beam
>>>> cant maintain 2 systems, easier with that state is then to drop gradle
>>>> since it is a 0 investment compared to the opposite
>>>>
>>>>
>>>> Romain Manni-Bucau
>>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
>>>> <https://rmannibucau.metawerx.net/> | Old Blog
>>>> <http://rmannibucau.wordpress.com> | Github
>>>> <https://github.com/rmannibucau> | LinkedIn
>>>> <https://www.linkedin.com/in/rmannibucau> | Book
>>>> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>>>>
>>>> 2018-03-22 17:24 GMT+01:00 Lukasz Cwik <lc...@google.com>:
>>>>
>>>>> Romain, from the previous discussions several people agreed that
>>>>> running a fixit that migrated Maven to Gradle over a 1 or 2 day period was
>>>>> worthwhile but there was nobody in the community with the time commitment
>>>>> to organize and run it so the status quo plan remained where the Gradle
>>>>> migration will happen incrementally.
>>>>>
>>>>>
>>>>> On Thu, Mar 22, 2018 at 8:53 AM Henning Rohde <he...@google.com>
>>>>> wrote:
>>>>>
>>>>>> My understanding was the same as Ismaël's. I don't think breaking the
>>>>>> build with a large known gaps (but not fully known cost) is practical.
>>>>>> Also, most items in the jira are not even assigned yet.
>>>>>>
>>>>>>
>>>>>> On Thu, Mar 22, 2018 at 8:03 AM Romain Manni-Bucau <
>>>>>> rmannibucau@gmail.com> wrote:
>>>>>>
>>>>>>> Not really Ismaël, this thread was about to do it at once and have 1
>>>>>>> day to fix it all.
>>>>>>>
>>>>>>> As mentionned at the very beginning nobody maintains the 2 system so
>>>>>>> it must stop after months so either we drop maven or gradle *at once*
>>>>>>> or we keep a state where each dev does what he wants and the build
>>>>>>> system just doesn't work.
>>>>>>>
>>>>>>> 2018-03-22 15:42 GMT+01:00 Ismaël Mejía <ie...@gmail.com>:
>>>>>>>
>>>>>>>> I don't think that removing all maven descriptors was the expected
>>>>>>>> path, no ? Or even a good idea at this moment.
>>>>>>>>
>>>>>>>> I understood that what we were going to do was to replace
>>>>>>>> incrementally the CI until we cover the whole maven functionality
>>>>>>>> and
>>>>>>>> then remove it, from looking at the JIRA ticket
>>>>>>>> https://issues.apache.org/jira/browse/BEAM-3249 we are still far
>>>>>>>> from
>>>>>>>> covering the complete maven functionality in particular for the
>>>>>>>> release part that could be the biggest pain point.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Mar 22, 2018 at 9:30 AM, Romain Manni-Bucau
>>>>>>>> <rm...@gmail.com> wrote:
>>>>>>>> > hey guys,
>>>>>>>> >
>>>>>>>> > 2.4 is out, do we plan to drop all maven descriptors tomorrow or
>>>>>>>> on monday?
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > Romain Manni-Bucau
>>>>>>>> > @rmannibucau | Blog | Old Blog | Github | LinkedIn | Book
>>>>>>>> >
>>>>>>>> > 2018-03-09 21:42 GMT+01:00 Kenneth Knowles <kl...@google.com>:
>>>>>>>> >>
>>>>>>>> >> On Fri, Mar 9, 2018 at 12:16 PM Lukasz Cwik <lc...@google.com>
>>>>>>>> wrote:
>>>>>>>> >>>
>>>>>>>> >>> Based upon your description it seems as though you would rather
>>>>>>>> have a
>>>>>>>> >>> way to run existing postcommits without it impacting
>>>>>>>> dashboard/health
>>>>>>>> >>> stats/notifications/.... (We have just run the PostCommits on
>>>>>>>> PRs for
>>>>>>>> >>> additional validation (like upgrading the Dataflow container
>>>>>>>> image)).
>>>>>>>> >>
>>>>>>>> >>
>>>>>>>> >> Yes, that is exactly what I have described.
>>>>>>>> >>
>>>>>>>> >>> I don't think that keeping the current Java PreCommit as a
>>>>>>>> proxy for the
>>>>>>>> >>> the Java PostCommit is the right way to go but I also don't
>>>>>>>> have the time to
>>>>>>>> >>> implement what your actually asking for.
>>>>>>>> >>
>>>>>>>> >>
>>>>>>>> >> Mostly I thought this might be very easy based on the fact that
>>>>>>>> they are
>>>>>>>> >> nearly identical. If not, oh well.
>>>>>>>> >>
>>>>>>>> >> Kenn
>>>>>>>> >>
>>>>>>>> >>
>>>>>>>> >>> It seems more likely that migrating the PostCommit to Gradle
>>>>>>>> will be less
>>>>>>>> >>> work then adding the functionality but your argument where the
>>>>>>>> PreCommit is
>>>>>>>> >>> a proxy for the Java PostCommit also applies to the
>>>>>>>> ValidatesRunner
>>>>>>>> >>> PostCommits and so forth requiring even more migration to
>>>>>>>> happen before you
>>>>>>>> >>> don't have to worry about maintaining Maven/breaking post
>>>>>>>> commits.
>>>>>>>> >>>
>>>>>>>> >>> I'm fine with leaving both the Java/Gradle PreCommits running
>>>>>>>> for now and
>>>>>>>> >>> hopefully as more of the PostCommits are migrated off we will
>>>>>>>> be able to
>>>>>>>> >>> remove it.
>>>>>>>> >>>
>>>>>>>> >>> On Fri, Mar 9, 2018 at 11:39 AM, Kenneth Knowles <
>>>>>>>> klk@google.com> wrote:
>>>>>>>> >>>>
>>>>>>>> >>>> Separate history (for easy dashboarding, health stats, etc) and
>>>>>>>> >>>> notification (email to dev@ for postcommits, nothing for
>>>>>>>> precommits) for pre
>>>>>>>> >>>> & post commit targets.
>>>>>>>> >>>>
>>>>>>>> >>>> A post commit failure is always a problem to be triaged at high
>>>>>>>> >>>> priority, while a precommit failure is just a natural
>>>>>>>> occurrence.
>>>>>>>> >>>>
>>>>>>>> >>>> On Fri, Mar 9, 2018 at 11:33 AM Lukasz Cwik <lc...@google.com>
>>>>>>>> wrote:
>>>>>>>> >>>>>
>>>>>>>> >>>>> Ken, I'm probably not seeing something but how does using the
>>>>>>>> PreCommit
>>>>>>>> >>>>> as a proxy improve upon just running the post commit via the
>>>>>>>> phrase it
>>>>>>>> >>>>> already supports ('Run Java PostCommit')?
>>>>>>>> >>>>>
>>>>>>>> >>>>> On Fri, Mar 9, 2018 at 11:22 AM, Kenneth Knowles <
>>>>>>>> klk@google.com>
>>>>>>>> >>>>> wrote:
>>>>>>>> >>>>>>
>>>>>>>> >>>>>> Indeed, we've already had the discussion a couple of times
>>>>>>>> and I think
>>>>>>>> >>>>>> the criteria are clearly met. Incremental progress is a good
>>>>>>>> thing and we
>>>>>>>> >>>>>> shouldn't block it.
>>>>>>>> >>>>>>
>>>>>>>> >>>>>> OTOH I see where Romain is coming from and I have a good
>>>>>>>> example that
>>>>>>>> >>>>>> supports a slightly different action. Consider
>>>>>>>> >>>>>> https://github.com/apache/beam/pull/4740 which fixes some
>>>>>>>> errors in how we
>>>>>>>> >>>>>> use dependency mechanisms.
>>>>>>>> >>>>>>
>>>>>>>> >>>>>> This PR is green except that I need to fix some Maven pom
>>>>>>>> slightly
>>>>>>>> >>>>>> more. That is throwaway work. I would love to just not have
>>>>>>>> to do it. But
>>>>>>>> >>>>>> removing the precommit does not actually make the PR OK to
>>>>>>>> merge. It would
>>>>>>>> >>>>>> cause postcommits to fail.
>>>>>>>> >>>>>>
>>>>>>>> >>>>>> We can hope such situations are rare. I think I tend to be
>>>>>>>> hit by this
>>>>>>>> >>>>>> more often than most, as I work with the project build
>>>>>>>> health quite a bit.
>>>>>>>> >>>>>>
>>>>>>>> >>>>>> Here is a proposal to support these things: instead of
>>>>>>>> deleting the
>>>>>>>> >>>>>> job in #4814, move it to not run automatically but only via
>>>>>>>> a phrase. In
>>>>>>>> >>>>>> fact, you could migrate it to be the manually-invoked
>>>>>>>> version of the
>>>>>>>> >>>>>> postcommit job as we've discussed a couple times. Then if
>>>>>>>> someone is working
>>>>>>>> >>>>>> on the build in something like #4740 they can invoke it
>>>>>>>> manually.
>>>>>>>> >>>>>>
>>>>>>>> >>>>>> Kenn
>>>>>>>> >>>>>>
>>>>>>>> >>>>>> On Fri, Mar 9, 2018 at 10:25 AM Lukasz Cwik <
>>>>>>>> lcwik@google.com> wrote:
>>>>>>>> >>>>>>>
>>>>>>>> >>>>>>> Based upon the criteria that was discussed on the mailing
>>>>>>>> list[1], I
>>>>>>>> >>>>>>> would agree with Kenn about merging PR/4814 (drop Java
>>>>>>>> Maven precommit).
>>>>>>>> >>>>>>>
>>>>>>>> >>>>>>> 1:
>>>>>>>> >>>>>>> https://lists.apache.org/thread.html/
>>>>>>>> 7eba5c77bc1a77b5046d915ab59f5f6fc41536c2c84863ad2efb5e99@%
>>>>>>>> 3Cdev.beam.apache.org%3E
>>>>>>>> >>>>>>>
>>>>>>>> >>>>>>> On Thu, Mar 8, 2018 at 9:10 PM, Romain Manni-Bucau
>>>>>>>> >>>>>>> <rm...@gmail.com> wrote:
>>>>>>>> >>>>>>>>
>>>>>>>> >>>>>>>> Hi Kenneth,
>>>>>>>> >>>>>>>>
>>>>>>>> >>>>>>>> For now maven covers the full needs of beam. If we start
>>>>>>>> to have
>>>>>>>> >>>>>>>> this kind of PR we become dependent of the 2 builds which
>>>>>>>> is what this
>>>>>>>> >>>>>>>> thread is about avoiding so tempted to say it must be a PR
>>>>>>>> drop completely
>>>>>>>> >>>>>>>> maven or nothing as mentionned before.
>>>>>>>> >>>>>>>>
>>>>>>>> >>>>>>>> Le 9 mars 2018 04:48, "Kenneth Knowles" <kl...@google.com>
>>>>>>>> a écrit :
>>>>>>>> >>>>>>>>>
>>>>>>>> >>>>>>>>> I would like to briefly re-focus this discussion and
>>>>>>>> suggest that
>>>>>>>> >>>>>>>>> we merge https://github.com/apache/beam/pull/4814.
>>>>>>>> >>>>>>>>>
>>>>>>>> >>>>>>>>> The only material objection I've heard is that it means
>>>>>>>> the
>>>>>>>> >>>>>>>>> precommit no longer tests exactly what is built for
>>>>>>>> release. It is a valid
>>>>>>>> >>>>>>>>> concern, but we have mvn postcommit so the coverage is
>>>>>>>> not lost. The
>>>>>>>> >>>>>>>>> benefits of quicker reviews and less Jenkins congestion
>>>>>>>> seem worth it to me.
>>>>>>>> >>>>>>>>>
>>>>>>>> >>>>>>>>> Kenn
>>>>>>>> >>>>>>>>>
>>>>>>>> >>>>>>>>> On Thu, Mar 8, 2018 at 12:33 PM Romain Manni-Bucau
>>>>>>>> >>>>>>>>> <rm...@gmail.com> wrote:
>>>>>>>> >>>>>>>>>>
>>>>>>>> >>>>>>>>>> @Luskasz: not sure Im the best to host it since I know
>>>>>>>> more gradle
>>>>>>>> >>>>>>>>>> internals that user interface/ecosystem but happy to
>>>>>>>> help. Will also require
>>>>>>>> >>>>>>>>>> a "sudo" merger for this day to merge fixes asap - guess
>>>>>>>> we can bypass
>>>>>>>> >>>>>>>>>> reviews or have a fast cycle plan for this day to avoid
>>>>>>>> it to be a week?
>>>>>>>> >>>>>>>>>>
>>>>>>>> >>>>>>>>>> Le 8 mars 2018 21:08, "Kenneth Knowles" <kl...@google.com>
>>>>>>>> a écrit :
>>>>>>>> >>>>>>>>>>
>>>>>>>> >>>>>>>>>> @Romain - Idea/IntelliJ is great with Gradle, way better
>>>>>>>> than
>>>>>>>> >>>>>>>>>> Maven, and what I mean is that I have added enough hints
>>>>>>>> that it works OOTB
>>>>>>>> >>>>>>>>>> already. The rest of my instructions are just how you
>>>>>>>> should override
>>>>>>>> >>>>>>>>>> IntelliJ's defaults to have a proper dev env - mostly
>>>>>>>> just about storing
>>>>>>>> >>>>>>>>>> files outside the git clone.
>>>>>>>> >>>>>>>>>>
>>>>>>>> >>>>>>>>>>
>>>>>>>> >>>>>>>>>> Hmm it is always super slow here and not as integrated
>>>>>>>> as maven
>>>>>>>> >>>>>>>>>> which is smooth thanks to the combination of idea build
>>>>>>>> and maven plugin
>>>>>>>> >>>>>>>>>> config read. Import is also faster cause it just reads
>>>>>>>> meta and doesnt run
>>>>>>>> >>>>>>>>>> anything. Hope it is a "a few times" issues at the
>>>>>>>> moment but not yet sure.
>>>>>>>> >>>>>>>>>>
>>>>>>>> >>>>>>>>>>
>>>>>>>> >>>>>>>>>>
>>>>>>>> >>>>>>>>>> @Łukasz - yea I just mean the ones that I find in my
>>>>>>>> email and
>>>>>>>> >>>>>>>>>> browsing https://builds.apache.org/view/A-D/view/Beam/
>>>>>>>> which are the ones
>>>>>>>> >>>>>>>>>> you classify as "totally not working". I would love to
>>>>>>>> disable these.
>>>>>>>> >>>>>>>>>>
>>>>>>>> >>>>>>>>>> On the subject of running things on a pending PR - we
>>>>>>>> should not
>>>>>>>> >>>>>>>>>> run postcommit jobs on PRs. We should make separate
>>>>>>>> (optional) precommit
>>>>>>>> >>>>>>>>>> jobs that run the same build commands. That will give a
>>>>>>>> more clear history
>>>>>>>> >>>>>>>>>> and allow trivial configuration of which jobs deserve
>>>>>>>> alert emails and which
>>>>>>>> >>>>>>>>>> are not a problem. This is easy but I've been waiting to
>>>>>>>> do it after Gradle
>>>>>>>> >>>>>>>>>> migration.
>>>>>>>> >>>>>>>>>>
>>>>>>>> >>>>>>>>>> On Thu, Mar 8, 2018 at 11:37 AM Lukasz Cwik <
>>>>>>>> lcwik@google.com>
>>>>>>>> >>>>>>>>>> wrote:
>>>>>>>> >>>>>>>>>>>
>>>>>>>> >>>>>>>>>>> Romain, would you like to host/plan/run the Gradle
>>>>>>>> fixit day?
>>>>>>>> >>>>>>>>>>>
>>>>>>>> >>>>>>>>>>> On Thu, Mar 8, 2018 at 11:24 AM, Chamikara Jayalath
>>>>>>>> >>>>>>>>>>> <ch...@google.com> wrote:
>>>>>>>> >>>>>>>>>>>>
>>>>>>>> >>>>>>>>>>>> +1 for the general idea of fixit day/week for Gradle.
>>>>>>>> >>>>>>>>>>>>
>>>>>>>> >>>>>>>>>>>> Agree with what Łukasz said. Some of these performance
>>>>>>>> tests are
>>>>>>>> >>>>>>>>>>>> new and are flaky due to other issues that were
>>>>>>>> discovered during the
>>>>>>>> >>>>>>>>>>>> process of adding the test.
>>>>>>>> >>>>>>>>>>>>
>>>>>>>> >>>>>>>>>>>> I think the high level blocker is updating performance
>>>>>>>> testing
>>>>>>>> >>>>>>>>>>>> framework to use Gradle. This will involve adding
>>>>>>>> Gradle-based logic for
>>>>>>>> >>>>>>>>>>>> invoking PerfKitBenchmaker, for example [1], and
>>>>>>>> updating PerfKitBenchmarker
>>>>>>>> >>>>>>>>>>>> to issue a Gradle command for running the benchmark
>>>>>>>> [2]. First task will be
>>>>>>>> >>>>>>>>>>>> to find the work needed here and updating the relevant
>>>>>>>> JIRA [3].
>>>>>>>> >>>>>>>>>>>>
>>>>>>>> >>>>>>>>>>>> Thanks,
>>>>>>>> >>>>>>>>>>>> Cham
>>>>>>>> >>>>>>>>>>>>
>>>>>>>> >>>>>>>>>>>> [1]
>>>>>>>> >>>>>>>>>>>> https://github.com/apache/
>>>>>>>> beam/blob/master/sdks/java/io/google-cloud-platform/pom.xml#L79
>>>>>>>> >>>>>>>>>>>> [2]
>>>>>>>> >>>>>>>>>>>> https://github.com/GoogleCloudPlatform/
>>>>>>>> PerfKitBenchmarker/blob/master/perfkitbenchmarker/
>>>>>>>> beam_benchmark_helper.py
>>>>>>>> >>>>>>>>>>>> [3] https://issues.apache.org/jira/browse/BEAM-3251
>>>>>>>> >>>>>>>>>>>>
>>>>>>>> >>>>>>>>>>>> On Thu, Mar 8, 2018 at 3:10 AM Łukasz Gajowy
>>>>>>>> >>>>>>>>>>>> <lu...@gmail.com> wrote:
>>>>>>>> >>>>>>>>>>>>>
>>>>>>>> >>>>>>>>>>>>> 2018-03-07 22:29 GMT+01:00 Kenneth Knowles <
>>>>>>>> klk@google.com>:
>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>> >>>>>>>>>>>>>> Based on https://builds.apache.org/
>>>>>>>> view/A-D/view/Beam/ and our
>>>>>>>> >>>>>>>>>>>>>> failure spam level the performance tests are mostly
>>>>>>>> not healthy anyhow. So
>>>>>>>> >>>>>>>>>>>>>> is there any high level blocker to switching them or
>>>>>>>> is it just someone
>>>>>>>> >>>>>>>>>>>>>> sitting down with each one?
>>>>>>>> >>>>>>>>>>>>>>
>>>>>>>> >>>>>>>>>>>>>
>>>>>>>> >>>>>>>>>>>>> I thought I could share my point of view here as I am
>>>>>>>> working
>>>>>>>> >>>>>>>>>>>>> on the Performance Test part for a while now. I
>>>>>>>> wouldn't say those are
>>>>>>>> >>>>>>>>>>>>> "mostly not healthy". The situation is as follows:
>>>>>>>> >>>>>>>>>>>>>
>>>>>>>> >>>>>>>>>>>>> - totally not working: Python, Spark performance
>>>>>>>> tests (don't
>>>>>>>> >>>>>>>>>>>>> know why, I'm not maintaining those tests. Should we
>>>>>>>> disable them?)
>>>>>>>> >>>>>>>>>>>>> - flaky: the recently re-enabled JDBC Performance
>>>>>>>> Test. It's
>>>>>>>> >>>>>>>>>>>>> seems to be flaky mostly due to:
>>>>>>>> >>>>>>>>>>>>> https://issues.apache.org/jira/browse/BEAM-3798
>>>>>>>> >>>>>>>>>>>>> - working well, rarely failing: AvroIO, TextIO,
>>>>>>>> Compressed
>>>>>>>> >>>>>>>>>>>>> Text, TFRecordIO
>>>>>>>> >>>>>>>>>>>>>
>>>>>>>> >>>>>>>>>>>>> Also, some test failures are caused due to pending
>>>>>>>> PRs (we
>>>>>>>> >>>>>>>>>>>>> sometimes run concrete tests to check if PRs won't
>>>>>>>> break them). This also
>>>>>>>> >>>>>>>>>>>>> causes failures sometimes.
>>>>>>>> >>>>>>>>>>>>>
>>>>>>>> >>>>>>>>>>>>> I can help with switching the Performance tests to
>>>>>>>> Gradle as
>>>>>>>> >>>>>>>>>>>>> this part seems to be free to take.
>>>>>>>> >>>>>>>>>>>>>
>>>>>>>> >>>>>>>>>>>>>
>>>>>>>> >>>>>>>>>>>>> Łukasz
>>>>>>>> >>>>>>>>>>>>>
>>>>>>>> >>>>>>>>>>>
>>>>>>>> >>>>>>>>>>
>>>>>>>> >>>>>>>
>>>>>>>> >>>>>
>>>>>>>> >>>
>>>>>>>> >
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>
>>
Re: Gradle status
Posted by Alan Myrvold <am...@google.com>.
I think the investment in gradle is worthwhile, and incrementally we will
continue to make progress. From what I've seem, gradle is a good fit for
this project and a path to a faster, more reliable build system.
pull/4812 <https://github.com/apache/beam/pull/4812> creates the release
artifacts, although it is not hooked up yet with authentication.
I expect to help make incremental progress over the next month converting
some of the integration tests, but welcome incremental improvements from
others.
On Thu, Mar 22, 2018 at 9:57 AM Romain Manni-Bucau <rm...@gmail.com>
wrote:
>
>
> 2018-03-22 17:45 GMT+01:00 Lukasz Cwik <lc...@google.com>:
>
>> what do we do? "Gradle migration will happen incrementally."
>>
>> "last months prooved beam cant maintain 2 systems, easier with that
>> state is then to drop gradle since it is a 0 investment compared to the
>> opposite"
>> Its unfortunate that you feel this way but many people do not share your
>> opinion.
>>
>
> And a lot do so when a project is 50-50 it is time to act.
>
> Incrementally kind of means never (makes 4 months and nothing really
> changed in PRs and habits, gradle maintener(s) are still alone)
>
>
>>
>>
>> On Thu, Mar 22, 2018 at 9:32 AM Romain Manni-Bucau <rm...@gmail.com>
>> wrote:
>>
>>> @Valentyn: concretely any user can PR and be part of that process so
>>> anyone can do it wrong (me first)
>>> @Luskasz, Hennking: fine but what do we do? last months prooved beam
>>> cant maintain 2 systems, easier with that state is then to drop gradle
>>> since it is a 0 investment compared to the opposite
>>>
>>>
>>> Romain Manni-Bucau
>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
>>> <https://rmannibucau.metawerx.net/> | Old Blog
>>> <http://rmannibucau.wordpress.com> | Github
>>> <https://github.com/rmannibucau> | LinkedIn
>>> <https://www.linkedin.com/in/rmannibucau> | Book
>>> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>>>
>>> 2018-03-22 17:24 GMT+01:00 Lukasz Cwik <lc...@google.com>:
>>>
>>>> Romain, from the previous discussions several people agreed that
>>>> running a fixit that migrated Maven to Gradle over a 1 or 2 day period was
>>>> worthwhile but there was nobody in the community with the time commitment
>>>> to organize and run it so the status quo plan remained where the Gradle
>>>> migration will happen incrementally.
>>>>
>>>>
>>>> On Thu, Mar 22, 2018 at 8:53 AM Henning Rohde <he...@google.com>
>>>> wrote:
>>>>
>>>>> My understanding was the same as Ismaël's. I don't think breaking the
>>>>> build with a large known gaps (but not fully known cost) is practical.
>>>>> Also, most items in the jira are not even assigned yet.
>>>>>
>>>>>
>>>>> On Thu, Mar 22, 2018 at 8:03 AM Romain Manni-Bucau <
>>>>> rmannibucau@gmail.com> wrote:
>>>>>
>>>>>> Not really Ismaël, this thread was about to do it at once and have 1
>>>>>> day to fix it all.
>>>>>>
>>>>>> As mentionned at the very beginning nobody maintains the 2 system so
>>>>>> it must stop after months so either we drop maven or gradle *at once*
>>>>>> or we keep a state where each dev does what he wants and the build
>>>>>> system just doesn't work.
>>>>>>
>>>>>> 2018-03-22 15:42 GMT+01:00 Ismaël Mejía <ie...@gmail.com>:
>>>>>>
>>>>>>> I don't think that removing all maven descriptors was the expected
>>>>>>> path, no ? Or even a good idea at this moment.
>>>>>>>
>>>>>>> I understood that what we were going to do was to replace
>>>>>>> incrementally the CI until we cover the whole maven functionality and
>>>>>>> then remove it, from looking at the JIRA ticket
>>>>>>> https://issues.apache.org/jira/browse/BEAM-3249 we are still far
>>>>>>> from
>>>>>>> covering the complete maven functionality in particular for the
>>>>>>> release part that could be the biggest pain point.
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Mar 22, 2018 at 9:30 AM, Romain Manni-Bucau
>>>>>>> <rm...@gmail.com> wrote:
>>>>>>> > hey guys,
>>>>>>> >
>>>>>>> > 2.4 is out, do we plan to drop all maven descriptors tomorrow or
>>>>>>> on monday?
>>>>>>> >
>>>>>>> >
>>>>>>> > Romain Manni-Bucau
>>>>>>> > @rmannibucau | Blog | Old Blog | Github | LinkedIn | Book
>>>>>>> >
>>>>>>> > 2018-03-09 21:42 GMT+01:00 Kenneth Knowles <kl...@google.com>:
>>>>>>> >>
>>>>>>> >> On Fri, Mar 9, 2018 at 12:16 PM Lukasz Cwik <lc...@google.com>
>>>>>>> wrote:
>>>>>>> >>>
>>>>>>> >>> Based upon your description it seems as though you would rather
>>>>>>> have a
>>>>>>> >>> way to run existing postcommits without it impacting
>>>>>>> dashboard/health
>>>>>>> >>> stats/notifications/.... (We have just run the PostCommits on
>>>>>>> PRs for
>>>>>>> >>> additional validation (like upgrading the Dataflow container
>>>>>>> image)).
>>>>>>> >>
>>>>>>> >>
>>>>>>> >> Yes, that is exactly what I have described.
>>>>>>> >>
>>>>>>> >>> I don't think that keeping the current Java PreCommit as a proxy
>>>>>>> for the
>>>>>>> >>> the Java PostCommit is the right way to go but I also don't have
>>>>>>> the time to
>>>>>>> >>> implement what your actually asking for.
>>>>>>> >>
>>>>>>> >>
>>>>>>> >> Mostly I thought this might be very easy based on the fact that
>>>>>>> they are
>>>>>>> >> nearly identical. If not, oh well.
>>>>>>> >>
>>>>>>> >> Kenn
>>>>>>> >>
>>>>>>> >>
>>>>>>> >>> It seems more likely that migrating the PostCommit to Gradle
>>>>>>> will be less
>>>>>>> >>> work then adding the functionality but your argument where the
>>>>>>> PreCommit is
>>>>>>> >>> a proxy for the Java PostCommit also applies to the
>>>>>>> ValidatesRunner
>>>>>>> >>> PostCommits and so forth requiring even more migration to happen
>>>>>>> before you
>>>>>>> >>> don't have to worry about maintaining Maven/breaking post
>>>>>>> commits.
>>>>>>> >>>
>>>>>>> >>> I'm fine with leaving both the Java/Gradle PreCommits running
>>>>>>> for now and
>>>>>>> >>> hopefully as more of the PostCommits are migrated off we will be
>>>>>>> able to
>>>>>>> >>> remove it.
>>>>>>> >>>
>>>>>>> >>> On Fri, Mar 9, 2018 at 11:39 AM, Kenneth Knowles <kl...@google.com>
>>>>>>> wrote:
>>>>>>> >>>>
>>>>>>> >>>> Separate history (for easy dashboarding, health stats, etc) and
>>>>>>> >>>> notification (email to dev@ for postcommits, nothing for
>>>>>>> precommits) for pre
>>>>>>> >>>> & post commit targets.
>>>>>>> >>>>
>>>>>>> >>>> A post commit failure is always a problem to be triaged at high
>>>>>>> >>>> priority, while a precommit failure is just a natural
>>>>>>> occurrence.
>>>>>>> >>>>
>>>>>>> >>>> On Fri, Mar 9, 2018 at 11:33 AM Lukasz Cwik <lc...@google.com>
>>>>>>> wrote:
>>>>>>> >>>>>
>>>>>>> >>>>> Ken, I'm probably not seeing something but how does using the
>>>>>>> PreCommit
>>>>>>> >>>>> as a proxy improve upon just running the post commit via the
>>>>>>> phrase it
>>>>>>> >>>>> already supports ('Run Java PostCommit')?
>>>>>>> >>>>>
>>>>>>> >>>>> On Fri, Mar 9, 2018 at 11:22 AM, Kenneth Knowles <
>>>>>>> klk@google.com>
>>>>>>> >>>>> wrote:
>>>>>>> >>>>>>
>>>>>>> >>>>>> Indeed, we've already had the discussion a couple of times
>>>>>>> and I think
>>>>>>> >>>>>> the criteria are clearly met. Incremental progress is a good
>>>>>>> thing and we
>>>>>>> >>>>>> shouldn't block it.
>>>>>>> >>>>>>
>>>>>>> >>>>>> OTOH I see where Romain is coming from and I have a good
>>>>>>> example that
>>>>>>> >>>>>> supports a slightly different action. Consider
>>>>>>> >>>>>> https://github.com/apache/beam/pull/4740 which fixes some
>>>>>>> errors in how we
>>>>>>> >>>>>> use dependency mechanisms.
>>>>>>> >>>>>>
>>>>>>> >>>>>> This PR is green except that I need to fix some Maven pom
>>>>>>> slightly
>>>>>>> >>>>>> more. That is throwaway work. I would love to just not have
>>>>>>> to do it. But
>>>>>>> >>>>>> removing the precommit does not actually make the PR OK to
>>>>>>> merge. It would
>>>>>>> >>>>>> cause postcommits to fail.
>>>>>>> >>>>>>
>>>>>>> >>>>>> We can hope such situations are rare. I think I tend to be
>>>>>>> hit by this
>>>>>>> >>>>>> more often than most, as I work with the project build health
>>>>>>> quite a bit.
>>>>>>> >>>>>>
>>>>>>> >>>>>> Here is a proposal to support these things: instead of
>>>>>>> deleting the
>>>>>>> >>>>>> job in #4814, move it to not run automatically but only via a
>>>>>>> phrase. In
>>>>>>> >>>>>> fact, you could migrate it to be the manually-invoked version
>>>>>>> of the
>>>>>>> >>>>>> postcommit job as we've discussed a couple times. Then if
>>>>>>> someone is working
>>>>>>> >>>>>> on the build in something like #4740 they can invoke it
>>>>>>> manually.
>>>>>>> >>>>>>
>>>>>>> >>>>>> Kenn
>>>>>>> >>>>>>
>>>>>>> >>>>>> On Fri, Mar 9, 2018 at 10:25 AM Lukasz Cwik <lc...@google.com>
>>>>>>> wrote:
>>>>>>> >>>>>>>
>>>>>>> >>>>>>> Based upon the criteria that was discussed on the mailing
>>>>>>> list[1], I
>>>>>>> >>>>>>> would agree with Kenn about merging PR/4814 (drop Java Maven
>>>>>>> precommit).
>>>>>>> >>>>>>>
>>>>>>> >>>>>>> 1:
>>>>>>> >>>>>>>
>>>>>>> https://lists.apache.org/thread.html/7eba5c77bc1a77b5046d915ab59f5f6fc41536c2c84863ad2efb5e99@%3Cdev.beam.apache.org%3E
>>>>>>> >>>>>>>
>>>>>>> >>>>>>> On Thu, Mar 8, 2018 at 9:10 PM, Romain Manni-Bucau
>>>>>>> >>>>>>> <rm...@gmail.com> wrote:
>>>>>>> >>>>>>>>
>>>>>>> >>>>>>>> Hi Kenneth,
>>>>>>> >>>>>>>>
>>>>>>> >>>>>>>> For now maven covers the full needs of beam. If we start to
>>>>>>> have
>>>>>>> >>>>>>>> this kind of PR we become dependent of the 2 builds which
>>>>>>> is what this
>>>>>>> >>>>>>>> thread is about avoiding so tempted to say it must be a PR
>>>>>>> drop completely
>>>>>>> >>>>>>>> maven or nothing as mentionned before.
>>>>>>> >>>>>>>>
>>>>>>> >>>>>>>> Le 9 mars 2018 04:48, "Kenneth Knowles" <kl...@google.com> a
>>>>>>> écrit :
>>>>>>> >>>>>>>>>
>>>>>>> >>>>>>>>> I would like to briefly re-focus this discussion and
>>>>>>> suggest that
>>>>>>> >>>>>>>>> we merge https://github.com/apache/beam/pull/4814.
>>>>>>> >>>>>>>>>
>>>>>>> >>>>>>>>> The only material objection I've heard is that it means the
>>>>>>> >>>>>>>>> precommit no longer tests exactly what is built for
>>>>>>> release. It is a valid
>>>>>>> >>>>>>>>> concern, but we have mvn postcommit so the coverage is not
>>>>>>> lost. The
>>>>>>> >>>>>>>>> benefits of quicker reviews and less Jenkins congestion
>>>>>>> seem worth it to me.
>>>>>>> >>>>>>>>>
>>>>>>> >>>>>>>>> Kenn
>>>>>>> >>>>>>>>>
>>>>>>> >>>>>>>>> On Thu, Mar 8, 2018 at 12:33 PM Romain Manni-Bucau
>>>>>>> >>>>>>>>> <rm...@gmail.com> wrote:
>>>>>>> >>>>>>>>>>
>>>>>>> >>>>>>>>>> @Luskasz: not sure Im the best to host it since I know
>>>>>>> more gradle
>>>>>>> >>>>>>>>>> internals that user interface/ecosystem but happy to
>>>>>>> help. Will also require
>>>>>>> >>>>>>>>>> a "sudo" merger for this day to merge fixes asap - guess
>>>>>>> we can bypass
>>>>>>> >>>>>>>>>> reviews or have a fast cycle plan for this day to avoid
>>>>>>> it to be a week?
>>>>>>> >>>>>>>>>>
>>>>>>> >>>>>>>>>> Le 8 mars 2018 21:08, "Kenneth Knowles" <kl...@google.com>
>>>>>>> a écrit :
>>>>>>> >>>>>>>>>>
>>>>>>> >>>>>>>>>> @Romain - Idea/IntelliJ is great with Gradle, way better
>>>>>>> than
>>>>>>> >>>>>>>>>> Maven, and what I mean is that I have added enough hints
>>>>>>> that it works OOTB
>>>>>>> >>>>>>>>>> already. The rest of my instructions are just how you
>>>>>>> should override
>>>>>>> >>>>>>>>>> IntelliJ's defaults to have a proper dev env - mostly
>>>>>>> just about storing
>>>>>>> >>>>>>>>>> files outside the git clone.
>>>>>>> >>>>>>>>>>
>>>>>>> >>>>>>>>>>
>>>>>>> >>>>>>>>>> Hmm it is always super slow here and not as integrated as
>>>>>>> maven
>>>>>>> >>>>>>>>>> which is smooth thanks to the combination of idea build
>>>>>>> and maven plugin
>>>>>>> >>>>>>>>>> config read. Import is also faster cause it just reads
>>>>>>> meta and doesnt run
>>>>>>> >>>>>>>>>> anything. Hope it is a "a few times" issues at the moment
>>>>>>> but not yet sure.
>>>>>>> >>>>>>>>>>
>>>>>>> >>>>>>>>>>
>>>>>>> >>>>>>>>>>
>>>>>>> >>>>>>>>>> @Łukasz - yea I just mean the ones that I find in my
>>>>>>> email and
>>>>>>> >>>>>>>>>> browsing https://builds.apache.org/view/A-D/view/Beam/
>>>>>>> which are the ones
>>>>>>> >>>>>>>>>> you classify as "totally not working". I would love to
>>>>>>> disable these.
>>>>>>> >>>>>>>>>>
>>>>>>> >>>>>>>>>> On the subject of running things on a pending PR - we
>>>>>>> should not
>>>>>>> >>>>>>>>>> run postcommit jobs on PRs. We should make separate
>>>>>>> (optional) precommit
>>>>>>> >>>>>>>>>> jobs that run the same build commands. That will give a
>>>>>>> more clear history
>>>>>>> >>>>>>>>>> and allow trivial configuration of which jobs deserve
>>>>>>> alert emails and which
>>>>>>> >>>>>>>>>> are not a problem. This is easy but I've been waiting to
>>>>>>> do it after Gradle
>>>>>>> >>>>>>>>>> migration.
>>>>>>> >>>>>>>>>>
>>>>>>> >>>>>>>>>> On Thu, Mar 8, 2018 at 11:37 AM Lukasz Cwik <
>>>>>>> lcwik@google.com>
>>>>>>> >>>>>>>>>> wrote:
>>>>>>> >>>>>>>>>>>
>>>>>>> >>>>>>>>>>> Romain, would you like to host/plan/run the Gradle fixit
>>>>>>> day?
>>>>>>> >>>>>>>>>>>
>>>>>>> >>>>>>>>>>> On Thu, Mar 8, 2018 at 11:24 AM, Chamikara Jayalath
>>>>>>> >>>>>>>>>>> <ch...@google.com> wrote:
>>>>>>> >>>>>>>>>>>>
>>>>>>> >>>>>>>>>>>> +1 for the general idea of fixit day/week for Gradle.
>>>>>>> >>>>>>>>>>>>
>>>>>>> >>>>>>>>>>>> Agree with what Łukasz said. Some of these performance
>>>>>>> tests are
>>>>>>> >>>>>>>>>>>> new and are flaky due to other issues that were
>>>>>>> discovered during the
>>>>>>> >>>>>>>>>>>> process of adding the test.
>>>>>>> >>>>>>>>>>>>
>>>>>>> >>>>>>>>>>>> I think the high level blocker is updating performance
>>>>>>> testing
>>>>>>> >>>>>>>>>>>> framework to use Gradle. This will involve adding
>>>>>>> Gradle-based logic for
>>>>>>> >>>>>>>>>>>> invoking PerfKitBenchmaker, for example [1], and
>>>>>>> updating PerfKitBenchmarker
>>>>>>> >>>>>>>>>>>> to issue a Gradle command for running the benchmark
>>>>>>> [2]. First task will be
>>>>>>> >>>>>>>>>>>> to find the work needed here and updating the relevant
>>>>>>> JIRA [3].
>>>>>>> >>>>>>>>>>>>
>>>>>>> >>>>>>>>>>>> Thanks,
>>>>>>> >>>>>>>>>>>> Cham
>>>>>>> >>>>>>>>>>>>
>>>>>>> >>>>>>>>>>>> [1]
>>>>>>> >>>>>>>>>>>>
>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/pom.xml#L79
>>>>>>> >>>>>>>>>>>> [2]
>>>>>>> >>>>>>>>>>>>
>>>>>>> https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/blob/master/perfkitbenchmarker/beam_benchmark_helper.py
>>>>>>> >>>>>>>>>>>> [3] https://issues.apache.org/jira/browse/BEAM-3251
>>>>>>> >>>>>>>>>>>>
>>>>>>> >>>>>>>>>>>> On Thu, Mar 8, 2018 at 3:10 AM Łukasz Gajowy
>>>>>>> >>>>>>>>>>>> <lu...@gmail.com> wrote:
>>>>>>> >>>>>>>>>>>>>
>>>>>>> >>>>>>>>>>>>> 2018-03-07 22:29 GMT+01:00 Kenneth Knowles <
>>>>>>> klk@google.com>:
>>>>>>> >>>>>>>>>>>>>>
>>>>>>> >>>>>>>>>>>>>>
>>>>>>> >>>>>>>>>>>>>> Based on
>>>>>>> https://builds.apache.org/view/A-D/view/Beam/ and our
>>>>>>> >>>>>>>>>>>>>> failure spam level the performance tests are mostly
>>>>>>> not healthy anyhow. So
>>>>>>> >>>>>>>>>>>>>> is there any high level blocker to switching them or
>>>>>>> is it just someone
>>>>>>> >>>>>>>>>>>>>> sitting down with each one?
>>>>>>> >>>>>>>>>>>>>>
>>>>>>> >>>>>>>>>>>>>
>>>>>>> >>>>>>>>>>>>> I thought I could share my point of view here as I am
>>>>>>> working
>>>>>>> >>>>>>>>>>>>> on the Performance Test part for a while now. I
>>>>>>> wouldn't say those are
>>>>>>> >>>>>>>>>>>>> "mostly not healthy". The situation is as follows:
>>>>>>> >>>>>>>>>>>>>
>>>>>>> >>>>>>>>>>>>> - totally not working: Python, Spark performance tests
>>>>>>> (don't
>>>>>>> >>>>>>>>>>>>> know why, I'm not maintaining those tests. Should we
>>>>>>> disable them?)
>>>>>>> >>>>>>>>>>>>> - flaky: the recently re-enabled JDBC Performance
>>>>>>> Test. It's
>>>>>>> >>>>>>>>>>>>> seems to be flaky mostly due to:
>>>>>>> >>>>>>>>>>>>> https://issues.apache.org/jira/browse/BEAM-3798
>>>>>>> >>>>>>>>>>>>> - working well, rarely failing: AvroIO, TextIO,
>>>>>>> Compressed
>>>>>>> >>>>>>>>>>>>> Text, TFRecordIO
>>>>>>> >>>>>>>>>>>>>
>>>>>>> >>>>>>>>>>>>> Also, some test failures are caused due to pending PRs
>>>>>>> (we
>>>>>>> >>>>>>>>>>>>> sometimes run concrete tests to check if PRs won't
>>>>>>> break them). This also
>>>>>>> >>>>>>>>>>>>> causes failures sometimes.
>>>>>>> >>>>>>>>>>>>>
>>>>>>> >>>>>>>>>>>>> I can help with switching the Performance tests to
>>>>>>> Gradle as
>>>>>>> >>>>>>>>>>>>> this part seems to be free to take.
>>>>>>> >>>>>>>>>>>>>
>>>>>>> >>>>>>>>>>>>>
>>>>>>> >>>>>>>>>>>>> Łukasz
>>>>>>> >>>>>>>>>>>>>
>>>>>>> >>>>>>>>>>>
>>>>>>> >>>>>>>>>>
>>>>>>> >>>>>>>
>>>>>>> >>>>>
>>>>>>> >>>
>>>>>>> >
>>>>>>>
>>>>>>
>>>>>>
>>>
>
Re: Gradle status
Posted by Romain Manni-Bucau <rm...@gmail.com>.
2018-03-22 17:45 GMT+01:00 Lukasz Cwik <lc...@google.com>:
> what do we do? "Gradle migration will happen incrementally."
>
> "last months prooved beam cant maintain 2 systems, easier with that state
> is then to drop gradle since it is a 0 investment compared to the opposite"
> Its unfortunate that you feel this way but many people do not share your
> opinion.
>
And a lot do so when a project is 50-50 it is time to act.
Incrementally kind of means never (makes 4 months and nothing really
changed in PRs and habits, gradle maintener(s) are still alone)
>
>
> On Thu, Mar 22, 2018 at 9:32 AM Romain Manni-Bucau <rm...@gmail.com>
> wrote:
>
>> @Valentyn: concretely any user can PR and be part of that process so
>> anyone can do it wrong (me first)
>> @Luskasz, Hennking: fine but what do we do? last months prooved beam cant
>> maintain 2 systems, easier with that state is then to drop gradle since it
>> is a 0 investment compared to the opposite
>>
>>
>> Romain Manni-Bucau
>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
>> <https://rmannibucau.metawerx.net/> | Old Blog
>> <http://rmannibucau.wordpress.com> | Github
>> <https://github.com/rmannibucau> | LinkedIn
>> <https://www.linkedin.com/in/rmannibucau> | Book
>> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>>
>> 2018-03-22 17:24 GMT+01:00 Lukasz Cwik <lc...@google.com>:
>>
>>> Romain, from the previous discussions several people agreed that running
>>> a fixit that migrated Maven to Gradle over a 1 or 2 day period was
>>> worthwhile but there was nobody in the community with the time commitment
>>> to organize and run it so the status quo plan remained where the Gradle
>>> migration will happen incrementally.
>>>
>>>
>>> On Thu, Mar 22, 2018 at 8:53 AM Henning Rohde <he...@google.com>
>>> wrote:
>>>
>>>> My understanding was the same as Ismaël's. I don't think breaking the
>>>> build with a large known gaps (but not fully known cost) is practical.
>>>> Also, most items in the jira are not even assigned yet.
>>>>
>>>>
>>>> On Thu, Mar 22, 2018 at 8:03 AM Romain Manni-Bucau <
>>>> rmannibucau@gmail.com> wrote:
>>>>
>>>>> Not really Ismaël, this thread was about to do it at once and have 1
>>>>> day to fix it all.
>>>>>
>>>>> As mentionned at the very beginning nobody maintains the 2 system so
>>>>> it must stop after months so either we drop maven or gradle *at once*
>>>>> or we keep a state where each dev does what he wants and the build
>>>>> system just doesn't work.
>>>>>
>>>>> 2018-03-22 15:42 GMT+01:00 Ismaël Mejía <ie...@gmail.com>:
>>>>>
>>>>>> I don't think that removing all maven descriptors was the expected
>>>>>> path, no ? Or even a good idea at this moment.
>>>>>>
>>>>>> I understood that what we were going to do was to replace
>>>>>> incrementally the CI until we cover the whole maven functionality and
>>>>>> then remove it, from looking at the JIRA ticket
>>>>>> https://issues.apache.org/jira/browse/BEAM-3249 we are still far from
>>>>>> covering the complete maven functionality in particular for the
>>>>>> release part that could be the biggest pain point.
>>>>>>
>>>>>>
>>>>>> On Thu, Mar 22, 2018 at 9:30 AM, Romain Manni-Bucau
>>>>>> <rm...@gmail.com> wrote:
>>>>>> > hey guys,
>>>>>> >
>>>>>> > 2.4 is out, do we plan to drop all maven descriptors tomorrow or on
>>>>>> monday?
>>>>>> >
>>>>>> >
>>>>>> > Romain Manni-Bucau
>>>>>> > @rmannibucau | Blog | Old Blog | Github | LinkedIn | Book
>>>>>> >
>>>>>> > 2018-03-09 21:42 GMT+01:00 Kenneth Knowles <kl...@google.com>:
>>>>>> >>
>>>>>> >> On Fri, Mar 9, 2018 at 12:16 PM Lukasz Cwik <lc...@google.com>
>>>>>> wrote:
>>>>>> >>>
>>>>>> >>> Based upon your description it seems as though you would rather
>>>>>> have a
>>>>>> >>> way to run existing postcommits without it impacting
>>>>>> dashboard/health
>>>>>> >>> stats/notifications/.... (We have just run the PostCommits on PRs
>>>>>> for
>>>>>> >>> additional validation (like upgrading the Dataflow container
>>>>>> image)).
>>>>>> >>
>>>>>> >>
>>>>>> >> Yes, that is exactly what I have described.
>>>>>> >>
>>>>>> >>> I don't think that keeping the current Java PreCommit as a proxy
>>>>>> for the
>>>>>> >>> the Java PostCommit is the right way to go but I also don't have
>>>>>> the time to
>>>>>> >>> implement what your actually asking for.
>>>>>> >>
>>>>>> >>
>>>>>> >> Mostly I thought this might be very easy based on the fact that
>>>>>> they are
>>>>>> >> nearly identical. If not, oh well.
>>>>>> >>
>>>>>> >> Kenn
>>>>>> >>
>>>>>> >>
>>>>>> >>> It seems more likely that migrating the PostCommit to Gradle will
>>>>>> be less
>>>>>> >>> work then adding the functionality but your argument where the
>>>>>> PreCommit is
>>>>>> >>> a proxy for the Java PostCommit also applies to the
>>>>>> ValidatesRunner
>>>>>> >>> PostCommits and so forth requiring even more migration to happen
>>>>>> before you
>>>>>> >>> don't have to worry about maintaining Maven/breaking post commits.
>>>>>> >>>
>>>>>> >>> I'm fine with leaving both the Java/Gradle PreCommits running for
>>>>>> now and
>>>>>> >>> hopefully as more of the PostCommits are migrated off we will be
>>>>>> able to
>>>>>> >>> remove it.
>>>>>> >>>
>>>>>> >>> On Fri, Mar 9, 2018 at 11:39 AM, Kenneth Knowles <kl...@google.com>
>>>>>> wrote:
>>>>>> >>>>
>>>>>> >>>> Separate history (for easy dashboarding, health stats, etc) and
>>>>>> >>>> notification (email to dev@ for postcommits, nothing for
>>>>>> precommits) for pre
>>>>>> >>>> & post commit targets.
>>>>>> >>>>
>>>>>> >>>> A post commit failure is always a problem to be triaged at high
>>>>>> >>>> priority, while a precommit failure is just a natural occurrence.
>>>>>> >>>>
>>>>>> >>>> On Fri, Mar 9, 2018 at 11:33 AM Lukasz Cwik <lc...@google.com>
>>>>>> wrote:
>>>>>> >>>>>
>>>>>> >>>>> Ken, I'm probably not seeing something but how does using the
>>>>>> PreCommit
>>>>>> >>>>> as a proxy improve upon just running the post commit via the
>>>>>> phrase it
>>>>>> >>>>> already supports ('Run Java PostCommit')?
>>>>>> >>>>>
>>>>>> >>>>> On Fri, Mar 9, 2018 at 11:22 AM, Kenneth Knowles <
>>>>>> klk@google.com>
>>>>>> >>>>> wrote:
>>>>>> >>>>>>
>>>>>> >>>>>> Indeed, we've already had the discussion a couple of times and
>>>>>> I think
>>>>>> >>>>>> the criteria are clearly met. Incremental progress is a good
>>>>>> thing and we
>>>>>> >>>>>> shouldn't block it.
>>>>>> >>>>>>
>>>>>> >>>>>> OTOH I see where Romain is coming from and I have a good
>>>>>> example that
>>>>>> >>>>>> supports a slightly different action. Consider
>>>>>> >>>>>> https://github.com/apache/beam/pull/4740 which fixes some
>>>>>> errors in how we
>>>>>> >>>>>> use dependency mechanisms.
>>>>>> >>>>>>
>>>>>> >>>>>> This PR is green except that I need to fix some Maven pom
>>>>>> slightly
>>>>>> >>>>>> more. That is throwaway work. I would love to just not have to
>>>>>> do it. But
>>>>>> >>>>>> removing the precommit does not actually make the PR OK to
>>>>>> merge. It would
>>>>>> >>>>>> cause postcommits to fail.
>>>>>> >>>>>>
>>>>>> >>>>>> We can hope such situations are rare. I think I tend to be hit
>>>>>> by this
>>>>>> >>>>>> more often than most, as I work with the project build health
>>>>>> quite a bit.
>>>>>> >>>>>>
>>>>>> >>>>>> Here is a proposal to support these things: instead of
>>>>>> deleting the
>>>>>> >>>>>> job in #4814, move it to not run automatically but only via a
>>>>>> phrase. In
>>>>>> >>>>>> fact, you could migrate it to be the manually-invoked version
>>>>>> of the
>>>>>> >>>>>> postcommit job as we've discussed a couple times. Then if
>>>>>> someone is working
>>>>>> >>>>>> on the build in something like #4740 they can invoke it
>>>>>> manually.
>>>>>> >>>>>>
>>>>>> >>>>>> Kenn
>>>>>> >>>>>>
>>>>>> >>>>>> On Fri, Mar 9, 2018 at 10:25 AM Lukasz Cwik <lc...@google.com>
>>>>>> wrote:
>>>>>> >>>>>>>
>>>>>> >>>>>>> Based upon the criteria that was discussed on the mailing
>>>>>> list[1], I
>>>>>> >>>>>>> would agree with Kenn about merging PR/4814 (drop Java Maven
>>>>>> precommit).
>>>>>> >>>>>>>
>>>>>> >>>>>>> 1:
>>>>>> >>>>>>> https://lists.apache.org/thread.html/7eba5c77bc1a77b5046d915
>>>>>> ab59f5f6fc41536c2c84863ad2efb5e99@%3Cdev.beam.apache.org%3E
>>>>>> >>>>>>>
>>>>>> >>>>>>> On Thu, Mar 8, 2018 at 9:10 PM, Romain Manni-Bucau
>>>>>> >>>>>>> <rm...@gmail.com> wrote:
>>>>>> >>>>>>>>
>>>>>> >>>>>>>> Hi Kenneth,
>>>>>> >>>>>>>>
>>>>>> >>>>>>>> For now maven covers the full needs of beam. If we start to
>>>>>> have
>>>>>> >>>>>>>> this kind of PR we become dependent of the 2 builds which is
>>>>>> what this
>>>>>> >>>>>>>> thread is about avoiding so tempted to say it must be a PR
>>>>>> drop completely
>>>>>> >>>>>>>> maven or nothing as mentionned before.
>>>>>> >>>>>>>>
>>>>>> >>>>>>>> Le 9 mars 2018 04:48, "Kenneth Knowles" <kl...@google.com> a
>>>>>> écrit :
>>>>>> >>>>>>>>>
>>>>>> >>>>>>>>> I would like to briefly re-focus this discussion and
>>>>>> suggest that
>>>>>> >>>>>>>>> we merge https://github.com/apache/beam/pull/4814.
>>>>>> >>>>>>>>>
>>>>>> >>>>>>>>> The only material objection I've heard is that it means the
>>>>>> >>>>>>>>> precommit no longer tests exactly what is built for
>>>>>> release. It is a valid
>>>>>> >>>>>>>>> concern, but we have mvn postcommit so the coverage is not
>>>>>> lost. The
>>>>>> >>>>>>>>> benefits of quicker reviews and less Jenkins congestion
>>>>>> seem worth it to me.
>>>>>> >>>>>>>>>
>>>>>> >>>>>>>>> Kenn
>>>>>> >>>>>>>>>
>>>>>> >>>>>>>>> On Thu, Mar 8, 2018 at 12:33 PM Romain Manni-Bucau
>>>>>> >>>>>>>>> <rm...@gmail.com> wrote:
>>>>>> >>>>>>>>>>
>>>>>> >>>>>>>>>> @Luskasz: not sure Im the best to host it since I know
>>>>>> more gradle
>>>>>> >>>>>>>>>> internals that user interface/ecosystem but happy to help.
>>>>>> Will also require
>>>>>> >>>>>>>>>> a "sudo" merger for this day to merge fixes asap - guess
>>>>>> we can bypass
>>>>>> >>>>>>>>>> reviews or have a fast cycle plan for this day to avoid it
>>>>>> to be a week?
>>>>>> >>>>>>>>>>
>>>>>> >>>>>>>>>> Le 8 mars 2018 21:08, "Kenneth Knowles" <kl...@google.com>
>>>>>> a écrit :
>>>>>> >>>>>>>>>>
>>>>>> >>>>>>>>>> @Romain - Idea/IntelliJ is great with Gradle, way better
>>>>>> than
>>>>>> >>>>>>>>>> Maven, and what I mean is that I have added enough hints
>>>>>> that it works OOTB
>>>>>> >>>>>>>>>> already. The rest of my instructions are just how you
>>>>>> should override
>>>>>> >>>>>>>>>> IntelliJ's defaults to have a proper dev env - mostly just
>>>>>> about storing
>>>>>> >>>>>>>>>> files outside the git clone.
>>>>>> >>>>>>>>>>
>>>>>> >>>>>>>>>>
>>>>>> >>>>>>>>>> Hmm it is always super slow here and not as integrated as
>>>>>> maven
>>>>>> >>>>>>>>>> which is smooth thanks to the combination of idea build
>>>>>> and maven plugin
>>>>>> >>>>>>>>>> config read. Import is also faster cause it just reads
>>>>>> meta and doesnt run
>>>>>> >>>>>>>>>> anything. Hope it is a "a few times" issues at the moment
>>>>>> but not yet sure.
>>>>>> >>>>>>>>>>
>>>>>> >>>>>>>>>>
>>>>>> >>>>>>>>>>
>>>>>> >>>>>>>>>> @Łukasz - yea I just mean the ones that I find in my email
>>>>>> and
>>>>>> >>>>>>>>>> browsing https://builds.apache.org/view/A-D/view/Beam/
>>>>>> which are the ones
>>>>>> >>>>>>>>>> you classify as "totally not working". I would love to
>>>>>> disable these.
>>>>>> >>>>>>>>>>
>>>>>> >>>>>>>>>> On the subject of running things on a pending PR - we
>>>>>> should not
>>>>>> >>>>>>>>>> run postcommit jobs on PRs. We should make separate
>>>>>> (optional) precommit
>>>>>> >>>>>>>>>> jobs that run the same build commands. That will give a
>>>>>> more clear history
>>>>>> >>>>>>>>>> and allow trivial configuration of which jobs deserve
>>>>>> alert emails and which
>>>>>> >>>>>>>>>> are not a problem. This is easy but I've been waiting to
>>>>>> do it after Gradle
>>>>>> >>>>>>>>>> migration.
>>>>>> >>>>>>>>>>
>>>>>> >>>>>>>>>> On Thu, Mar 8, 2018 at 11:37 AM Lukasz Cwik <
>>>>>> lcwik@google.com>
>>>>>> >>>>>>>>>> wrote:
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>> Romain, would you like to host/plan/run the Gradle fixit
>>>>>> day?
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>> On Thu, Mar 8, 2018 at 11:24 AM, Chamikara Jayalath
>>>>>> >>>>>>>>>>> <ch...@google.com> wrote:
>>>>>> >>>>>>>>>>>>
>>>>>> >>>>>>>>>>>> +1 for the general idea of fixit day/week for Gradle.
>>>>>> >>>>>>>>>>>>
>>>>>> >>>>>>>>>>>> Agree with what Łukasz said. Some of these performance
>>>>>> tests are
>>>>>> >>>>>>>>>>>> new and are flaky due to other issues that were
>>>>>> discovered during the
>>>>>> >>>>>>>>>>>> process of adding the test.
>>>>>> >>>>>>>>>>>>
>>>>>> >>>>>>>>>>>> I think the high level blocker is updating performance
>>>>>> testing
>>>>>> >>>>>>>>>>>> framework to use Gradle. This will involve adding
>>>>>> Gradle-based logic for
>>>>>> >>>>>>>>>>>> invoking PerfKitBenchmaker, for example [1], and
>>>>>> updating PerfKitBenchmarker
>>>>>> >>>>>>>>>>>> to issue a Gradle command for running the benchmark [2].
>>>>>> First task will be
>>>>>> >>>>>>>>>>>> to find the work needed here and updating the relevant
>>>>>> JIRA [3].
>>>>>> >>>>>>>>>>>>
>>>>>> >>>>>>>>>>>> Thanks,
>>>>>> >>>>>>>>>>>> Cham
>>>>>> >>>>>>>>>>>>
>>>>>> >>>>>>>>>>>> [1]
>>>>>> >>>>>>>>>>>> https://github.com/apache/beam
>>>>>> /blob/master/sdks/java/io/google-cloud-platform/pom.xml#L79
>>>>>> >>>>>>>>>>>> [2]
>>>>>> >>>>>>>>>>>> https://github.com/GoogleCloud
>>>>>> Platform/PerfKitBenchmarker/blob/master/perfkitbenchmarker/beam_
>>>>>> benchmark_helper.py
>>>>>> >>>>>>>>>>>> [3] https://issues.apache.org/jira/browse/BEAM-3251
>>>>>> >>>>>>>>>>>>
>>>>>> >>>>>>>>>>>> On Thu, Mar 8, 2018 at 3:10 AM Łukasz Gajowy
>>>>>> >>>>>>>>>>>> <lu...@gmail.com> wrote:
>>>>>> >>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>> 2018-03-07 22:29 GMT+01:00 Kenneth Knowles <
>>>>>> klk@google.com>:
>>>>>> >>>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>> Based on https://builds.apache.org/view/A-D/view/Beam/
>>>>>> and our
>>>>>> >>>>>>>>>>>>>> failure spam level the performance tests are mostly
>>>>>> not healthy anyhow. So
>>>>>> >>>>>>>>>>>>>> is there any high level blocker to switching them or
>>>>>> is it just someone
>>>>>> >>>>>>>>>>>>>> sitting down with each one?
>>>>>> >>>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>> I thought I could share my point of view here as I am
>>>>>> working
>>>>>> >>>>>>>>>>>>> on the Performance Test part for a while now. I
>>>>>> wouldn't say those are
>>>>>> >>>>>>>>>>>>> "mostly not healthy". The situation is as follows:
>>>>>> >>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>> - totally not working: Python, Spark performance tests
>>>>>> (don't
>>>>>> >>>>>>>>>>>>> know why, I'm not maintaining those tests. Should we
>>>>>> disable them?)
>>>>>> >>>>>>>>>>>>> - flaky: the recently re-enabled JDBC Performance Test.
>>>>>> It's
>>>>>> >>>>>>>>>>>>> seems to be flaky mostly due to:
>>>>>> >>>>>>>>>>>>> https://issues.apache.org/jira/browse/BEAM-3798
>>>>>> >>>>>>>>>>>>> - working well, rarely failing: AvroIO, TextIO,
>>>>>> Compressed
>>>>>> >>>>>>>>>>>>> Text, TFRecordIO
>>>>>> >>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>> Also, some test failures are caused due to pending PRs
>>>>>> (we
>>>>>> >>>>>>>>>>>>> sometimes run concrete tests to check if PRs won't
>>>>>> break them). This also
>>>>>> >>>>>>>>>>>>> causes failures sometimes.
>>>>>> >>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>> I can help with switching the Performance tests to
>>>>>> Gradle as
>>>>>> >>>>>>>>>>>>> this part seems to be free to take.
>>>>>> >>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>>> Łukasz
>>>>>> >>>>>>>>>>>>>
>>>>>> >>>>>>>>>>>
>>>>>> >>>>>>>>>>
>>>>>> >>>>>>>
>>>>>> >>>>>
>>>>>> >>>
>>>>>> >
>>>>>>
>>>>>
>>>>>
>>
Re: Gradle status
Posted by Lukasz Cwik <lc...@google.com>.
what do we do? "Gradle migration will happen incrementally."
"last months prooved beam cant maintain 2 systems, easier with that state
is then to drop gradle since it is a 0 investment compared to the opposite"
Its unfortunate that you feel this way but many people do not share your
opinion.
On Thu, Mar 22, 2018 at 9:32 AM Romain Manni-Bucau <rm...@gmail.com>
wrote:
> @Valentyn: concretely any user can PR and be part of that process so
> anyone can do it wrong (me first)
> @Luskasz, Hennking: fine but what do we do? last months prooved beam cant
> maintain 2 systems, easier with that state is then to drop gradle since it
> is a 0 investment compared to the opposite
>
>
> Romain Manni-Bucau
> @rmannibucau <https://twitter.com/rmannibucau> | Blog
> <https://rmannibucau.metawerx.net/> | Old Blog
> <http://rmannibucau.wordpress.com> | Github
> <https://github.com/rmannibucau> | LinkedIn
> <https://www.linkedin.com/in/rmannibucau> | Book
> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>
> 2018-03-22 17:24 GMT+01:00 Lukasz Cwik <lc...@google.com>:
>
>> Romain, from the previous discussions several people agreed that running
>> a fixit that migrated Maven to Gradle over a 1 or 2 day period was
>> worthwhile but there was nobody in the community with the time commitment
>> to organize and run it so the status quo plan remained where the Gradle
>> migration will happen incrementally.
>>
>>
>> On Thu, Mar 22, 2018 at 8:53 AM Henning Rohde <he...@google.com> wrote:
>>
>>> My understanding was the same as Ismaël's. I don't think breaking the
>>> build with a large known gaps (but not fully known cost) is practical.
>>> Also, most items in the jira are not even assigned yet.
>>>
>>>
>>> On Thu, Mar 22, 2018 at 8:03 AM Romain Manni-Bucau <
>>> rmannibucau@gmail.com> wrote:
>>>
>>>> Not really Ismaël, this thread was about to do it at once and have 1
>>>> day to fix it all.
>>>>
>>>> As mentionned at the very beginning nobody maintains the 2 system so it
>>>> must stop after months so either we drop maven or gradle *at once*
>>>> or we keep a state where each dev does what he wants and the build
>>>> system just doesn't work.
>>>>
>>>> 2018-03-22 15:42 GMT+01:00 Ismaël Mejía <ie...@gmail.com>:
>>>>
>>>>> I don't think that removing all maven descriptors was the expected
>>>>> path, no ? Or even a good idea at this moment.
>>>>>
>>>>> I understood that what we were going to do was to replace
>>>>> incrementally the CI until we cover the whole maven functionality and
>>>>> then remove it, from looking at the JIRA ticket
>>>>> https://issues.apache.org/jira/browse/BEAM-3249 we are still far from
>>>>> covering the complete maven functionality in particular for the
>>>>> release part that could be the biggest pain point.
>>>>>
>>>>>
>>>>> On Thu, Mar 22, 2018 at 9:30 AM, Romain Manni-Bucau
>>>>> <rm...@gmail.com> wrote:
>>>>> > hey guys,
>>>>> >
>>>>> > 2.4 is out, do we plan to drop all maven descriptors tomorrow or on
>>>>> monday?
>>>>> >
>>>>> >
>>>>> > Romain Manni-Bucau
>>>>> > @rmannibucau | Blog | Old Blog | Github | LinkedIn | Book
>>>>> >
>>>>> > 2018-03-09 21:42 GMT+01:00 Kenneth Knowles <kl...@google.com>:
>>>>> >>
>>>>> >> On Fri, Mar 9, 2018 at 12:16 PM Lukasz Cwik <lc...@google.com>
>>>>> wrote:
>>>>> >>>
>>>>> >>> Based upon your description it seems as though you would rather
>>>>> have a
>>>>> >>> way to run existing postcommits without it impacting
>>>>> dashboard/health
>>>>> >>> stats/notifications/.... (We have just run the PostCommits on PRs
>>>>> for
>>>>> >>> additional validation (like upgrading the Dataflow container
>>>>> image)).
>>>>> >>
>>>>> >>
>>>>> >> Yes, that is exactly what I have described.
>>>>> >>
>>>>> >>> I don't think that keeping the current Java PreCommit as a proxy
>>>>> for the
>>>>> >>> the Java PostCommit is the right way to go but I also don't have
>>>>> the time to
>>>>> >>> implement what your actually asking for.
>>>>> >>
>>>>> >>
>>>>> >> Mostly I thought this might be very easy based on the fact that
>>>>> they are
>>>>> >> nearly identical. If not, oh well.
>>>>> >>
>>>>> >> Kenn
>>>>> >>
>>>>> >>
>>>>> >>> It seems more likely that migrating the PostCommit to Gradle will
>>>>> be less
>>>>> >>> work then adding the functionality but your argument where the
>>>>> PreCommit is
>>>>> >>> a proxy for the Java PostCommit also applies to the ValidatesRunner
>>>>> >>> PostCommits and so forth requiring even more migration to happen
>>>>> before you
>>>>> >>> don't have to worry about maintaining Maven/breaking post commits.
>>>>> >>>
>>>>> >>> I'm fine with leaving both the Java/Gradle PreCommits running for
>>>>> now and
>>>>> >>> hopefully as more of the PostCommits are migrated off we will be
>>>>> able to
>>>>> >>> remove it.
>>>>> >>>
>>>>> >>> On Fri, Mar 9, 2018 at 11:39 AM, Kenneth Knowles <kl...@google.com>
>>>>> wrote:
>>>>> >>>>
>>>>> >>>> Separate history (for easy dashboarding, health stats, etc) and
>>>>> >>>> notification (email to dev@ for postcommits, nothing for
>>>>> precommits) for pre
>>>>> >>>> & post commit targets.
>>>>> >>>>
>>>>> >>>> A post commit failure is always a problem to be triaged at high
>>>>> >>>> priority, while a precommit failure is just a natural occurrence.
>>>>> >>>>
>>>>> >>>> On Fri, Mar 9, 2018 at 11:33 AM Lukasz Cwik <lc...@google.com>
>>>>> wrote:
>>>>> >>>>>
>>>>> >>>>> Ken, I'm probably not seeing something but how does using the
>>>>> PreCommit
>>>>> >>>>> as a proxy improve upon just running the post commit via the
>>>>> phrase it
>>>>> >>>>> already supports ('Run Java PostCommit')?
>>>>> >>>>>
>>>>> >>>>> On Fri, Mar 9, 2018 at 11:22 AM, Kenneth Knowles <klk@google.com
>>>>> >
>>>>> >>>>> wrote:
>>>>> >>>>>>
>>>>> >>>>>> Indeed, we've already had the discussion a couple of times and
>>>>> I think
>>>>> >>>>>> the criteria are clearly met. Incremental progress is a good
>>>>> thing and we
>>>>> >>>>>> shouldn't block it.
>>>>> >>>>>>
>>>>> >>>>>> OTOH I see where Romain is coming from and I have a good
>>>>> example that
>>>>> >>>>>> supports a slightly different action. Consider
>>>>> >>>>>> https://github.com/apache/beam/pull/4740 which fixes some
>>>>> errors in how we
>>>>> >>>>>> use dependency mechanisms.
>>>>> >>>>>>
>>>>> >>>>>> This PR is green except that I need to fix some Maven pom
>>>>> slightly
>>>>> >>>>>> more. That is throwaway work. I would love to just not have to
>>>>> do it. But
>>>>> >>>>>> removing the precommit does not actually make the PR OK to
>>>>> merge. It would
>>>>> >>>>>> cause postcommits to fail.
>>>>> >>>>>>
>>>>> >>>>>> We can hope such situations are rare. I think I tend to be hit
>>>>> by this
>>>>> >>>>>> more often than most, as I work with the project build health
>>>>> quite a bit.
>>>>> >>>>>>
>>>>> >>>>>> Here is a proposal to support these things: instead of deleting
>>>>> the
>>>>> >>>>>> job in #4814, move it to not run automatically but only via a
>>>>> phrase. In
>>>>> >>>>>> fact, you could migrate it to be the manually-invoked version
>>>>> of the
>>>>> >>>>>> postcommit job as we've discussed a couple times. Then if
>>>>> someone is working
>>>>> >>>>>> on the build in something like #4740 they can invoke it
>>>>> manually.
>>>>> >>>>>>
>>>>> >>>>>> Kenn
>>>>> >>>>>>
>>>>> >>>>>> On Fri, Mar 9, 2018 at 10:25 AM Lukasz Cwik <lc...@google.com>
>>>>> wrote:
>>>>> >>>>>>>
>>>>> >>>>>>> Based upon the criteria that was discussed on the mailing
>>>>> list[1], I
>>>>> >>>>>>> would agree with Kenn about merging PR/4814 (drop Java Maven
>>>>> precommit).
>>>>> >>>>>>>
>>>>> >>>>>>> 1:
>>>>> >>>>>>>
>>>>> https://lists.apache.org/thread.html/7eba5c77bc1a77b5046d915ab59f5f6fc41536c2c84863ad2efb5e99@%3Cdev.beam.apache.org%3E
>>>>> >>>>>>>
>>>>> >>>>>>> On Thu, Mar 8, 2018 at 9:10 PM, Romain Manni-Bucau
>>>>> >>>>>>> <rm...@gmail.com> wrote:
>>>>> >>>>>>>>
>>>>> >>>>>>>> Hi Kenneth,
>>>>> >>>>>>>>
>>>>> >>>>>>>> For now maven covers the full needs of beam. If we start to
>>>>> have
>>>>> >>>>>>>> this kind of PR we become dependent of the 2 builds which is
>>>>> what this
>>>>> >>>>>>>> thread is about avoiding so tempted to say it must be a PR
>>>>> drop completely
>>>>> >>>>>>>> maven or nothing as mentionned before.
>>>>> >>>>>>>>
>>>>> >>>>>>>> Le 9 mars 2018 04:48, "Kenneth Knowles" <kl...@google.com> a
>>>>> écrit :
>>>>> >>>>>>>>>
>>>>> >>>>>>>>> I would like to briefly re-focus this discussion and suggest
>>>>> that
>>>>> >>>>>>>>> we merge https://github.com/apache/beam/pull/4814.
>>>>> >>>>>>>>>
>>>>> >>>>>>>>> The only material objection I've heard is that it means the
>>>>> >>>>>>>>> precommit no longer tests exactly what is built for release.
>>>>> It is a valid
>>>>> >>>>>>>>> concern, but we have mvn postcommit so the coverage is not
>>>>> lost. The
>>>>> >>>>>>>>> benefits of quicker reviews and less Jenkins congestion seem
>>>>> worth it to me.
>>>>> >>>>>>>>>
>>>>> >>>>>>>>> Kenn
>>>>> >>>>>>>>>
>>>>> >>>>>>>>> On Thu, Mar 8, 2018 at 12:33 PM Romain Manni-Bucau
>>>>> >>>>>>>>> <rm...@gmail.com> wrote:
>>>>> >>>>>>>>>>
>>>>> >>>>>>>>>> @Luskasz: not sure Im the best to host it since I know more
>>>>> gradle
>>>>> >>>>>>>>>> internals that user interface/ecosystem but happy to help.
>>>>> Will also require
>>>>> >>>>>>>>>> a "sudo" merger for this day to merge fixes asap - guess we
>>>>> can bypass
>>>>> >>>>>>>>>> reviews or have a fast cycle plan for this day to avoid it
>>>>> to be a week?
>>>>> >>>>>>>>>>
>>>>> >>>>>>>>>> Le 8 mars 2018 21:08, "Kenneth Knowles" <kl...@google.com> a
>>>>> écrit :
>>>>> >>>>>>>>>>
>>>>> >>>>>>>>>> @Romain - Idea/IntelliJ is great with Gradle, way better
>>>>> than
>>>>> >>>>>>>>>> Maven, and what I mean is that I have added enough hints
>>>>> that it works OOTB
>>>>> >>>>>>>>>> already. The rest of my instructions are just how you
>>>>> should override
>>>>> >>>>>>>>>> IntelliJ's defaults to have a proper dev env - mostly just
>>>>> about storing
>>>>> >>>>>>>>>> files outside the git clone.
>>>>> >>>>>>>>>>
>>>>> >>>>>>>>>>
>>>>> >>>>>>>>>> Hmm it is always super slow here and not as integrated as
>>>>> maven
>>>>> >>>>>>>>>> which is smooth thanks to the combination of idea build and
>>>>> maven plugin
>>>>> >>>>>>>>>> config read. Import is also faster cause it just reads meta
>>>>> and doesnt run
>>>>> >>>>>>>>>> anything. Hope it is a "a few times" issues at the moment
>>>>> but not yet sure.
>>>>> >>>>>>>>>>
>>>>> >>>>>>>>>>
>>>>> >>>>>>>>>>
>>>>> >>>>>>>>>> @Łukasz - yea I just mean the ones that I find in my email
>>>>> and
>>>>> >>>>>>>>>> browsing https://builds.apache.org/view/A-D/view/Beam/
>>>>> which are the ones
>>>>> >>>>>>>>>> you classify as "totally not working". I would love to
>>>>> disable these.
>>>>> >>>>>>>>>>
>>>>> >>>>>>>>>> On the subject of running things on a pending PR - we
>>>>> should not
>>>>> >>>>>>>>>> run postcommit jobs on PRs. We should make separate
>>>>> (optional) precommit
>>>>> >>>>>>>>>> jobs that run the same build commands. That will give a
>>>>> more clear history
>>>>> >>>>>>>>>> and allow trivial configuration of which jobs deserve alert
>>>>> emails and which
>>>>> >>>>>>>>>> are not a problem. This is easy but I've been waiting to do
>>>>> it after Gradle
>>>>> >>>>>>>>>> migration.
>>>>> >>>>>>>>>>
>>>>> >>>>>>>>>> On Thu, Mar 8, 2018 at 11:37 AM Lukasz Cwik <
>>>>> lcwik@google.com>
>>>>> >>>>>>>>>> wrote:
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>> Romain, would you like to host/plan/run the Gradle fixit
>>>>> day?
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>> On Thu, Mar 8, 2018 at 11:24 AM, Chamikara Jayalath
>>>>> >>>>>>>>>>> <ch...@google.com> wrote:
>>>>> >>>>>>>>>>>>
>>>>> >>>>>>>>>>>> +1 for the general idea of fixit day/week for Gradle.
>>>>> >>>>>>>>>>>>
>>>>> >>>>>>>>>>>> Agree with what Łukasz said. Some of these performance
>>>>> tests are
>>>>> >>>>>>>>>>>> new and are flaky due to other issues that were
>>>>> discovered during the
>>>>> >>>>>>>>>>>> process of adding the test.
>>>>> >>>>>>>>>>>>
>>>>> >>>>>>>>>>>> I think the high level blocker is updating performance
>>>>> testing
>>>>> >>>>>>>>>>>> framework to use Gradle. This will involve adding
>>>>> Gradle-based logic for
>>>>> >>>>>>>>>>>> invoking PerfKitBenchmaker, for example [1], and updating
>>>>> PerfKitBenchmarker
>>>>> >>>>>>>>>>>> to issue a Gradle command for running the benchmark [2].
>>>>> First task will be
>>>>> >>>>>>>>>>>> to find the work needed here and updating the relevant
>>>>> JIRA [3].
>>>>> >>>>>>>>>>>>
>>>>> >>>>>>>>>>>> Thanks,
>>>>> >>>>>>>>>>>> Cham
>>>>> >>>>>>>>>>>>
>>>>> >>>>>>>>>>>> [1]
>>>>> >>>>>>>>>>>>
>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/pom.xml#L79
>>>>> >>>>>>>>>>>> [2]
>>>>> >>>>>>>>>>>>
>>>>> https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/blob/master/perfkitbenchmarker/beam_benchmark_helper.py
>>>>> >>>>>>>>>>>> [3] https://issues.apache.org/jira/browse/BEAM-3251
>>>>> >>>>>>>>>>>>
>>>>> >>>>>>>>>>>> On Thu, Mar 8, 2018 at 3:10 AM Łukasz Gajowy
>>>>> >>>>>>>>>>>> <lu...@gmail.com> wrote:
>>>>> >>>>>>>>>>>>>
>>>>> >>>>>>>>>>>>> 2018-03-07 22:29 GMT+01:00 Kenneth Knowles <
>>>>> klk@google.com>:
>>>>> >>>>>>>>>>>>>>
>>>>> >>>>>>>>>>>>>>
>>>>> >>>>>>>>>>>>>> Based on https://builds.apache.org/view/A-D/view/Beam/
>>>>> and our
>>>>> >>>>>>>>>>>>>> failure spam level the performance tests are mostly not
>>>>> healthy anyhow. So
>>>>> >>>>>>>>>>>>>> is there any high level blocker to switching them or is
>>>>> it just someone
>>>>> >>>>>>>>>>>>>> sitting down with each one?
>>>>> >>>>>>>>>>>>>>
>>>>> >>>>>>>>>>>>>
>>>>> >>>>>>>>>>>>> I thought I could share my point of view here as I am
>>>>> working
>>>>> >>>>>>>>>>>>> on the Performance Test part for a while now. I wouldn't
>>>>> say those are
>>>>> >>>>>>>>>>>>> "mostly not healthy". The situation is as follows:
>>>>> >>>>>>>>>>>>>
>>>>> >>>>>>>>>>>>> - totally not working: Python, Spark performance tests
>>>>> (don't
>>>>> >>>>>>>>>>>>> know why, I'm not maintaining those tests. Should we
>>>>> disable them?)
>>>>> >>>>>>>>>>>>> - flaky: the recently re-enabled JDBC Performance Test.
>>>>> It's
>>>>> >>>>>>>>>>>>> seems to be flaky mostly due to:
>>>>> >>>>>>>>>>>>> https://issues.apache.org/jira/browse/BEAM-3798
>>>>> >>>>>>>>>>>>> - working well, rarely failing: AvroIO, TextIO,
>>>>> Compressed
>>>>> >>>>>>>>>>>>> Text, TFRecordIO
>>>>> >>>>>>>>>>>>>
>>>>> >>>>>>>>>>>>> Also, some test failures are caused due to pending PRs
>>>>> (we
>>>>> >>>>>>>>>>>>> sometimes run concrete tests to check if PRs won't break
>>>>> them). This also
>>>>> >>>>>>>>>>>>> causes failures sometimes.
>>>>> >>>>>>>>>>>>>
>>>>> >>>>>>>>>>>>> I can help with switching the Performance tests to
>>>>> Gradle as
>>>>> >>>>>>>>>>>>> this part seems to be free to take.
>>>>> >>>>>>>>>>>>>
>>>>> >>>>>>>>>>>>>
>>>>> >>>>>>>>>>>>> Łukasz
>>>>> >>>>>>>>>>>>>
>>>>> >>>>>>>>>>>
>>>>> >>>>>>>>>>
>>>>> >>>>>>>
>>>>> >>>>>
>>>>> >>>
>>>>> >
>>>>>
>>>>
>>>>
>
Re: Gradle status
Posted by Romain Manni-Bucau <rm...@gmail.com>.
@Valentyn: concretely any user can PR and be part of that process so anyone
can do it wrong (me first)
@Luskasz, Hennking: fine but what do we do? last months prooved beam cant
maintain 2 systems, easier with that state is then to drop gradle since it
is a 0 investment compared to the opposite
Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> | Blog
<https://rmannibucau.metawerx.net/> | Old Blog
<http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
<https://www.packtpub.com/application-development/java-ee-8-high-performance>
2018-03-22 17:24 GMT+01:00 Lukasz Cwik <lc...@google.com>:
> Romain, from the previous discussions several people agreed that running a
> fixit that migrated Maven to Gradle over a 1 or 2 day period was worthwhile
> but there was nobody in the community with the time commitment to organize
> and run it so the status quo plan remained where the Gradle migration will
> happen incrementally.
>
>
> On Thu, Mar 22, 2018 at 8:53 AM Henning Rohde <he...@google.com> wrote:
>
>> My understanding was the same as Ismaël's. I don't think breaking the
>> build with a large known gaps (but not fully known cost) is practical.
>> Also, most items in the jira are not even assigned yet.
>>
>>
>> On Thu, Mar 22, 2018 at 8:03 AM Romain Manni-Bucau <rm...@gmail.com>
>> wrote:
>>
>>> Not really Ismaël, this thread was about to do it at once and have 1 day
>>> to fix it all.
>>>
>>> As mentionned at the very beginning nobody maintains the 2 system so it
>>> must stop after months so either we drop maven or gradle *at once*
>>> or we keep a state where each dev does what he wants and the build
>>> system just doesn't work.
>>>
>>> 2018-03-22 15:42 GMT+01:00 Ismaël Mejía <ie...@gmail.com>:
>>>
>>>> I don't think that removing all maven descriptors was the expected
>>>> path, no ? Or even a good idea at this moment.
>>>>
>>>> I understood that what we were going to do was to replace
>>>> incrementally the CI until we cover the whole maven functionality and
>>>> then remove it, from looking at the JIRA ticket
>>>> https://issues.apache.org/jira/browse/BEAM-3249 we are still far from
>>>> covering the complete maven functionality in particular for the
>>>> release part that could be the biggest pain point.
>>>>
>>>>
>>>> On Thu, Mar 22, 2018 at 9:30 AM, Romain Manni-Bucau
>>>> <rm...@gmail.com> wrote:
>>>> > hey guys,
>>>> >
>>>> > 2.4 is out, do we plan to drop all maven descriptors tomorrow or on
>>>> monday?
>>>> >
>>>> >
>>>> > Romain Manni-Bucau
>>>> > @rmannibucau | Blog | Old Blog | Github | LinkedIn | Book
>>>> >
>>>> > 2018-03-09 21:42 GMT+01:00 Kenneth Knowles <kl...@google.com>:
>>>> >>
>>>> >> On Fri, Mar 9, 2018 at 12:16 PM Lukasz Cwik <lc...@google.com>
>>>> wrote:
>>>> >>>
>>>> >>> Based upon your description it seems as though you would rather
>>>> have a
>>>> >>> way to run existing postcommits without it impacting
>>>> dashboard/health
>>>> >>> stats/notifications/.... (We have just run the PostCommits on PRs
>>>> for
>>>> >>> additional validation (like upgrading the Dataflow container
>>>> image)).
>>>> >>
>>>> >>
>>>> >> Yes, that is exactly what I have described.
>>>> >>
>>>> >>> I don't think that keeping the current Java PreCommit as a proxy
>>>> for the
>>>> >>> the Java PostCommit is the right way to go but I also don't have
>>>> the time to
>>>> >>> implement what your actually asking for.
>>>> >>
>>>> >>
>>>> >> Mostly I thought this might be very easy based on the fact that they
>>>> are
>>>> >> nearly identical. If not, oh well.
>>>> >>
>>>> >> Kenn
>>>> >>
>>>> >>
>>>> >>> It seems more likely that migrating the PostCommit to Gradle will
>>>> be less
>>>> >>> work then adding the functionality but your argument where the
>>>> PreCommit is
>>>> >>> a proxy for the Java PostCommit also applies to the ValidatesRunner
>>>> >>> PostCommits and so forth requiring even more migration to happen
>>>> before you
>>>> >>> don't have to worry about maintaining Maven/breaking post commits.
>>>> >>>
>>>> >>> I'm fine with leaving both the Java/Gradle PreCommits running for
>>>> now and
>>>> >>> hopefully as more of the PostCommits are migrated off we will be
>>>> able to
>>>> >>> remove it.
>>>> >>>
>>>> >>> On Fri, Mar 9, 2018 at 11:39 AM, Kenneth Knowles <kl...@google.com>
>>>> wrote:
>>>> >>>>
>>>> >>>> Separate history (for easy dashboarding, health stats, etc) and
>>>> >>>> notification (email to dev@ for postcommits, nothing for
>>>> precommits) for pre
>>>> >>>> & post commit targets.
>>>> >>>>
>>>> >>>> A post commit failure is always a problem to be triaged at high
>>>> >>>> priority, while a precommit failure is just a natural occurrence.
>>>> >>>>
>>>> >>>> On Fri, Mar 9, 2018 at 11:33 AM Lukasz Cwik <lc...@google.com>
>>>> wrote:
>>>> >>>>>
>>>> >>>>> Ken, I'm probably not seeing something but how does using the
>>>> PreCommit
>>>> >>>>> as a proxy improve upon just running the post commit via the
>>>> phrase it
>>>> >>>>> already supports ('Run Java PostCommit')?
>>>> >>>>>
>>>> >>>>> On Fri, Mar 9, 2018 at 11:22 AM, Kenneth Knowles <kl...@google.com>
>>>> >>>>> wrote:
>>>> >>>>>>
>>>> >>>>>> Indeed, we've already had the discussion a couple of times and I
>>>> think
>>>> >>>>>> the criteria are clearly met. Incremental progress is a good
>>>> thing and we
>>>> >>>>>> shouldn't block it.
>>>> >>>>>>
>>>> >>>>>> OTOH I see where Romain is coming from and I have a good example
>>>> that
>>>> >>>>>> supports a slightly different action. Consider
>>>> >>>>>> https://github.com/apache/beam/pull/4740 which fixes some
>>>> errors in how we
>>>> >>>>>> use dependency mechanisms.
>>>> >>>>>>
>>>> >>>>>> This PR is green except that I need to fix some Maven pom
>>>> slightly
>>>> >>>>>> more. That is throwaway work. I would love to just not have to
>>>> do it. But
>>>> >>>>>> removing the precommit does not actually make the PR OK to
>>>> merge. It would
>>>> >>>>>> cause postcommits to fail.
>>>> >>>>>>
>>>> >>>>>> We can hope such situations are rare. I think I tend to be hit
>>>> by this
>>>> >>>>>> more often than most, as I work with the project build health
>>>> quite a bit.
>>>> >>>>>>
>>>> >>>>>> Here is a proposal to support these things: instead of deleting
>>>> the
>>>> >>>>>> job in #4814, move it to not run automatically but only via a
>>>> phrase. In
>>>> >>>>>> fact, you could migrate it to be the manually-invoked version of
>>>> the
>>>> >>>>>> postcommit job as we've discussed a couple times. Then if
>>>> someone is working
>>>> >>>>>> on the build in something like #4740 they can invoke it manually.
>>>> >>>>>>
>>>> >>>>>> Kenn
>>>> >>>>>>
>>>> >>>>>> On Fri, Mar 9, 2018 at 10:25 AM Lukasz Cwik <lc...@google.com>
>>>> wrote:
>>>> >>>>>>>
>>>> >>>>>>> Based upon the criteria that was discussed on the mailing
>>>> list[1], I
>>>> >>>>>>> would agree with Kenn about merging PR/4814 (drop Java Maven
>>>> precommit).
>>>> >>>>>>>
>>>> >>>>>>> 1:
>>>> >>>>>>> https://lists.apache.org/thread.html/
>>>> 7eba5c77bc1a77b5046d915ab59f5f6fc41536c2c84863ad2efb5e99@%
>>>> 3Cdev.beam.apache.org%3E
>>>> >>>>>>>
>>>> >>>>>>> On Thu, Mar 8, 2018 at 9:10 PM, Romain Manni-Bucau
>>>> >>>>>>> <rm...@gmail.com> wrote:
>>>> >>>>>>>>
>>>> >>>>>>>> Hi Kenneth,
>>>> >>>>>>>>
>>>> >>>>>>>> For now maven covers the full needs of beam. If we start to
>>>> have
>>>> >>>>>>>> this kind of PR we become dependent of the 2 builds which is
>>>> what this
>>>> >>>>>>>> thread is about avoiding so tempted to say it must be a PR
>>>> drop completely
>>>> >>>>>>>> maven or nothing as mentionned before.
>>>> >>>>>>>>
>>>> >>>>>>>> Le 9 mars 2018 04:48, "Kenneth Knowles" <kl...@google.com> a
>>>> écrit :
>>>> >>>>>>>>>
>>>> >>>>>>>>> I would like to briefly re-focus this discussion and suggest
>>>> that
>>>> >>>>>>>>> we merge https://github.com/apache/beam/pull/4814.
>>>> >>>>>>>>>
>>>> >>>>>>>>> The only material objection I've heard is that it means the
>>>> >>>>>>>>> precommit no longer tests exactly what is built for release.
>>>> It is a valid
>>>> >>>>>>>>> concern, but we have mvn postcommit so the coverage is not
>>>> lost. The
>>>> >>>>>>>>> benefits of quicker reviews and less Jenkins congestion seem
>>>> worth it to me.
>>>> >>>>>>>>>
>>>> >>>>>>>>> Kenn
>>>> >>>>>>>>>
>>>> >>>>>>>>> On Thu, Mar 8, 2018 at 12:33 PM Romain Manni-Bucau
>>>> >>>>>>>>> <rm...@gmail.com> wrote:
>>>> >>>>>>>>>>
>>>> >>>>>>>>>> @Luskasz: not sure Im the best to host it since I know more
>>>> gradle
>>>> >>>>>>>>>> internals that user interface/ecosystem but happy to help.
>>>> Will also require
>>>> >>>>>>>>>> a "sudo" merger for this day to merge fixes asap - guess we
>>>> can bypass
>>>> >>>>>>>>>> reviews or have a fast cycle plan for this day to avoid it
>>>> to be a week?
>>>> >>>>>>>>>>
>>>> >>>>>>>>>> Le 8 mars 2018 21:08, "Kenneth Knowles" <kl...@google.com> a
>>>> écrit :
>>>> >>>>>>>>>>
>>>> >>>>>>>>>> @Romain - Idea/IntelliJ is great with Gradle, way better than
>>>> >>>>>>>>>> Maven, and what I mean is that I have added enough hints
>>>> that it works OOTB
>>>> >>>>>>>>>> already. The rest of my instructions are just how you should
>>>> override
>>>> >>>>>>>>>> IntelliJ's defaults to have a proper dev env - mostly just
>>>> about storing
>>>> >>>>>>>>>> files outside the git clone.
>>>> >>>>>>>>>>
>>>> >>>>>>>>>>
>>>> >>>>>>>>>> Hmm it is always super slow here and not as integrated as
>>>> maven
>>>> >>>>>>>>>> which is smooth thanks to the combination of idea build and
>>>> maven plugin
>>>> >>>>>>>>>> config read. Import is also faster cause it just reads meta
>>>> and doesnt run
>>>> >>>>>>>>>> anything. Hope it is a "a few times" issues at the moment
>>>> but not yet sure.
>>>> >>>>>>>>>>
>>>> >>>>>>>>>>
>>>> >>>>>>>>>>
>>>> >>>>>>>>>> @Łukasz - yea I just mean the ones that I find in my email
>>>> and
>>>> >>>>>>>>>> browsing https://builds.apache.org/view/A-D/view/Beam/
>>>> which are the ones
>>>> >>>>>>>>>> you classify as "totally not working". I would love to
>>>> disable these.
>>>> >>>>>>>>>>
>>>> >>>>>>>>>> On the subject of running things on a pending PR - we should
>>>> not
>>>> >>>>>>>>>> run postcommit jobs on PRs. We should make separate
>>>> (optional) precommit
>>>> >>>>>>>>>> jobs that run the same build commands. That will give a more
>>>> clear history
>>>> >>>>>>>>>> and allow trivial configuration of which jobs deserve alert
>>>> emails and which
>>>> >>>>>>>>>> are not a problem. This is easy but I've been waiting to do
>>>> it after Gradle
>>>> >>>>>>>>>> migration.
>>>> >>>>>>>>>>
>>>> >>>>>>>>>> On Thu, Mar 8, 2018 at 11:37 AM Lukasz Cwik <
>>>> lcwik@google.com>
>>>> >>>>>>>>>> wrote:
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>> Romain, would you like to host/plan/run the Gradle fixit
>>>> day?
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>> On Thu, Mar 8, 2018 at 11:24 AM, Chamikara Jayalath
>>>> >>>>>>>>>>> <ch...@google.com> wrote:
>>>> >>>>>>>>>>>>
>>>> >>>>>>>>>>>> +1 for the general idea of fixit day/week for Gradle.
>>>> >>>>>>>>>>>>
>>>> >>>>>>>>>>>> Agree with what Łukasz said. Some of these performance
>>>> tests are
>>>> >>>>>>>>>>>> new and are flaky due to other issues that were discovered
>>>> during the
>>>> >>>>>>>>>>>> process of adding the test.
>>>> >>>>>>>>>>>>
>>>> >>>>>>>>>>>> I think the high level blocker is updating performance
>>>> testing
>>>> >>>>>>>>>>>> framework to use Gradle. This will involve adding
>>>> Gradle-based logic for
>>>> >>>>>>>>>>>> invoking PerfKitBenchmaker, for example [1], and updating
>>>> PerfKitBenchmarker
>>>> >>>>>>>>>>>> to issue a Gradle command for running the benchmark [2].
>>>> First task will be
>>>> >>>>>>>>>>>> to find the work needed here and updating the relevant
>>>> JIRA [3].
>>>> >>>>>>>>>>>>
>>>> >>>>>>>>>>>> Thanks,
>>>> >>>>>>>>>>>> Cham
>>>> >>>>>>>>>>>>
>>>> >>>>>>>>>>>> [1]
>>>> >>>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/
>>>> google-cloud-platform/pom.xml#L79
>>>> >>>>>>>>>>>> [2]
>>>> >>>>>>>>>>>> https://github.com/GoogleCloudPlatform/
>>>> PerfKitBenchmarker/blob/master/perfkitbenchmarker/
>>>> beam_benchmark_helper.py
>>>> >>>>>>>>>>>> [3] https://issues.apache.org/jira/browse/BEAM-3251
>>>> >>>>>>>>>>>>
>>>> >>>>>>>>>>>> On Thu, Mar 8, 2018 at 3:10 AM Łukasz Gajowy
>>>> >>>>>>>>>>>> <lu...@gmail.com> wrote:
>>>> >>>>>>>>>>>>>
>>>> >>>>>>>>>>>>> 2018-03-07 22:29 GMT+01:00 Kenneth Knowles <
>>>> klk@google.com>:
>>>> >>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>> Based on https://builds.apache.org/view/A-D/view/Beam/
>>>> and our
>>>> >>>>>>>>>>>>>> failure spam level the performance tests are mostly not
>>>> healthy anyhow. So
>>>> >>>>>>>>>>>>>> is there any high level blocker to switching them or is
>>>> it just someone
>>>> >>>>>>>>>>>>>> sitting down with each one?
>>>> >>>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>
>>>> >>>>>>>>>>>>> I thought I could share my point of view here as I am
>>>> working
>>>> >>>>>>>>>>>>> on the Performance Test part for a while now. I wouldn't
>>>> say those are
>>>> >>>>>>>>>>>>> "mostly not healthy". The situation is as follows:
>>>> >>>>>>>>>>>>>
>>>> >>>>>>>>>>>>> - totally not working: Python, Spark performance tests
>>>> (don't
>>>> >>>>>>>>>>>>> know why, I'm not maintaining those tests. Should we
>>>> disable them?)
>>>> >>>>>>>>>>>>> - flaky: the recently re-enabled JDBC Performance Test.
>>>> It's
>>>> >>>>>>>>>>>>> seems to be flaky mostly due to:
>>>> >>>>>>>>>>>>> https://issues.apache.org/jira/browse/BEAM-3798
>>>> >>>>>>>>>>>>> - working well, rarely failing: AvroIO, TextIO, Compressed
>>>> >>>>>>>>>>>>> Text, TFRecordIO
>>>> >>>>>>>>>>>>>
>>>> >>>>>>>>>>>>> Also, some test failures are caused due to pending PRs (we
>>>> >>>>>>>>>>>>> sometimes run concrete tests to check if PRs won't break
>>>> them). This also
>>>> >>>>>>>>>>>>> causes failures sometimes.
>>>> >>>>>>>>>>>>>
>>>> >>>>>>>>>>>>> I can help with switching the Performance tests to Gradle
>>>> as
>>>> >>>>>>>>>>>>> this part seems to be free to take.
>>>> >>>>>>>>>>>>>
>>>> >>>>>>>>>>>>>
>>>> >>>>>>>>>>>>> Łukasz
>>>> >>>>>>>>>>>>>
>>>> >>>>>>>>>>>
>>>> >>>>>>>>>>
>>>> >>>>>>>
>>>> >>>>>
>>>> >>>
>>>> >
>>>>
>>>
>>>
Re: Gradle status
Posted by Lukasz Cwik <lc...@google.com>.
Romain, from the previous discussions several people agreed that running a
fixit that migrated Maven to Gradle over a 1 or 2 day period was worthwhile
but there was nobody in the community with the time commitment to organize
and run it so the status quo plan remained where the Gradle migration will
happen incrementally.
On Thu, Mar 22, 2018 at 8:53 AM Henning Rohde <he...@google.com> wrote:
> My understanding was the same as Ismaël's. I don't think breaking the
> build with a large known gaps (but not fully known cost) is practical.
> Also, most items in the jira are not even assigned yet.
>
>
> On Thu, Mar 22, 2018 at 8:03 AM Romain Manni-Bucau <rm...@gmail.com>
> wrote:
>
>> Not really Ismaël, this thread was about to do it at once and have 1 day
>> to fix it all.
>>
>> As mentionned at the very beginning nobody maintains the 2 system so it
>> must stop after months so either we drop maven or gradle *at once*
>> or we keep a state where each dev does what he wants and the build system
>> just doesn't work.
>>
>> 2018-03-22 15:42 GMT+01:00 Ismaël Mejía <ie...@gmail.com>:
>>
>>> I don't think that removing all maven descriptors was the expected
>>> path, no ? Or even a good idea at this moment.
>>>
>>> I understood that what we were going to do was to replace
>>> incrementally the CI until we cover the whole maven functionality and
>>> then remove it, from looking at the JIRA ticket
>>> https://issues.apache.org/jira/browse/BEAM-3249 we are still far from
>>> covering the complete maven functionality in particular for the
>>> release part that could be the biggest pain point.
>>>
>>>
>>> On Thu, Mar 22, 2018 at 9:30 AM, Romain Manni-Bucau
>>> <rm...@gmail.com> wrote:
>>> > hey guys,
>>> >
>>> > 2.4 is out, do we plan to drop all maven descriptors tomorrow or on
>>> monday?
>>> >
>>> >
>>> > Romain Manni-Bucau
>>> > @rmannibucau | Blog | Old Blog | Github | LinkedIn | Book
>>> >
>>> > 2018-03-09 21:42 GMT+01:00 Kenneth Knowles <kl...@google.com>:
>>> >>
>>> >> On Fri, Mar 9, 2018 at 12:16 PM Lukasz Cwik <lc...@google.com> wrote:
>>> >>>
>>> >>> Based upon your description it seems as though you would rather have
>>> a
>>> >>> way to run existing postcommits without it impacting dashboard/health
>>> >>> stats/notifications/.... (We have just run the PostCommits on PRs for
>>> >>> additional validation (like upgrading the Dataflow container image)).
>>> >>
>>> >>
>>> >> Yes, that is exactly what I have described.
>>> >>
>>> >>> I don't think that keeping the current Java PreCommit as a proxy for
>>> the
>>> >>> the Java PostCommit is the right way to go but I also don't have the
>>> time to
>>> >>> implement what your actually asking for.
>>> >>
>>> >>
>>> >> Mostly I thought this might be very easy based on the fact that they
>>> are
>>> >> nearly identical. If not, oh well.
>>> >>
>>> >> Kenn
>>> >>
>>> >>
>>> >>> It seems more likely that migrating the PostCommit to Gradle will be
>>> less
>>> >>> work then adding the functionality but your argument where the
>>> PreCommit is
>>> >>> a proxy for the Java PostCommit also applies to the ValidatesRunner
>>> >>> PostCommits and so forth requiring even more migration to happen
>>> before you
>>> >>> don't have to worry about maintaining Maven/breaking post commits.
>>> >>>
>>> >>> I'm fine with leaving both the Java/Gradle PreCommits running for
>>> now and
>>> >>> hopefully as more of the PostCommits are migrated off we will be
>>> able to
>>> >>> remove it.
>>> >>>
>>> >>> On Fri, Mar 9, 2018 at 11:39 AM, Kenneth Knowles <kl...@google.com>
>>> wrote:
>>> >>>>
>>> >>>> Separate history (for easy dashboarding, health stats, etc) and
>>> >>>> notification (email to dev@ for postcommits, nothing for
>>> precommits) for pre
>>> >>>> & post commit targets.
>>> >>>>
>>> >>>> A post commit failure is always a problem to be triaged at high
>>> >>>> priority, while a precommit failure is just a natural occurrence.
>>> >>>>
>>> >>>> On Fri, Mar 9, 2018 at 11:33 AM Lukasz Cwik <lc...@google.com>
>>> wrote:
>>> >>>>>
>>> >>>>> Ken, I'm probably not seeing something but how does using the
>>> PreCommit
>>> >>>>> as a proxy improve upon just running the post commit via the
>>> phrase it
>>> >>>>> already supports ('Run Java PostCommit')?
>>> >>>>>
>>> >>>>> On Fri, Mar 9, 2018 at 11:22 AM, Kenneth Knowles <kl...@google.com>
>>> >>>>> wrote:
>>> >>>>>>
>>> >>>>>> Indeed, we've already had the discussion a couple of times and I
>>> think
>>> >>>>>> the criteria are clearly met. Incremental progress is a good
>>> thing and we
>>> >>>>>> shouldn't block it.
>>> >>>>>>
>>> >>>>>> OTOH I see where Romain is coming from and I have a good example
>>> that
>>> >>>>>> supports a slightly different action. Consider
>>> >>>>>> https://github.com/apache/beam/pull/4740 which fixes some errors
>>> in how we
>>> >>>>>> use dependency mechanisms.
>>> >>>>>>
>>> >>>>>> This PR is green except that I need to fix some Maven pom slightly
>>> >>>>>> more. That is throwaway work. I would love to just not have to do
>>> it. But
>>> >>>>>> removing the precommit does not actually make the PR OK to merge.
>>> It would
>>> >>>>>> cause postcommits to fail.
>>> >>>>>>
>>> >>>>>> We can hope such situations are rare. I think I tend to be hit by
>>> this
>>> >>>>>> more often than most, as I work with the project build health
>>> quite a bit.
>>> >>>>>>
>>> >>>>>> Here is a proposal to support these things: instead of deleting
>>> the
>>> >>>>>> job in #4814, move it to not run automatically but only via a
>>> phrase. In
>>> >>>>>> fact, you could migrate it to be the manually-invoked version of
>>> the
>>> >>>>>> postcommit job as we've discussed a couple times. Then if someone
>>> is working
>>> >>>>>> on the build in something like #4740 they can invoke it manually.
>>> >>>>>>
>>> >>>>>> Kenn
>>> >>>>>>
>>> >>>>>> On Fri, Mar 9, 2018 at 10:25 AM Lukasz Cwik <lc...@google.com>
>>> wrote:
>>> >>>>>>>
>>> >>>>>>> Based upon the criteria that was discussed on the mailing
>>> list[1], I
>>> >>>>>>> would agree with Kenn about merging PR/4814 (drop Java Maven
>>> precommit).
>>> >>>>>>>
>>> >>>>>>> 1:
>>> >>>>>>>
>>> https://lists.apache.org/thread.html/7eba5c77bc1a77b5046d915ab59f5f6fc41536c2c84863ad2efb5e99@%3Cdev.beam.apache.org%3E
>>> >>>>>>>
>>> >>>>>>> On Thu, Mar 8, 2018 at 9:10 PM, Romain Manni-Bucau
>>> >>>>>>> <rm...@gmail.com> wrote:
>>> >>>>>>>>
>>> >>>>>>>> Hi Kenneth,
>>> >>>>>>>>
>>> >>>>>>>> For now maven covers the full needs of beam. If we start to have
>>> >>>>>>>> this kind of PR we become dependent of the 2 builds which is
>>> what this
>>> >>>>>>>> thread is about avoiding so tempted to say it must be a PR drop
>>> completely
>>> >>>>>>>> maven or nothing as mentionned before.
>>> >>>>>>>>
>>> >>>>>>>> Le 9 mars 2018 04:48, "Kenneth Knowles" <kl...@google.com> a
>>> écrit :
>>> >>>>>>>>>
>>> >>>>>>>>> I would like to briefly re-focus this discussion and suggest
>>> that
>>> >>>>>>>>> we merge https://github.com/apache/beam/pull/4814.
>>> >>>>>>>>>
>>> >>>>>>>>> The only material objection I've heard is that it means the
>>> >>>>>>>>> precommit no longer tests exactly what is built for release.
>>> It is a valid
>>> >>>>>>>>> concern, but we have mvn postcommit so the coverage is not
>>> lost. The
>>> >>>>>>>>> benefits of quicker reviews and less Jenkins congestion seem
>>> worth it to me.
>>> >>>>>>>>>
>>> >>>>>>>>> Kenn
>>> >>>>>>>>>
>>> >>>>>>>>> On Thu, Mar 8, 2018 at 12:33 PM Romain Manni-Bucau
>>> >>>>>>>>> <rm...@gmail.com> wrote:
>>> >>>>>>>>>>
>>> >>>>>>>>>> @Luskasz: not sure Im the best to host it since I know more
>>> gradle
>>> >>>>>>>>>> internals that user interface/ecosystem but happy to help.
>>> Will also require
>>> >>>>>>>>>> a "sudo" merger for this day to merge fixes asap - guess we
>>> can bypass
>>> >>>>>>>>>> reviews or have a fast cycle plan for this day to avoid it to
>>> be a week?
>>> >>>>>>>>>>
>>> >>>>>>>>>> Le 8 mars 2018 21:08, "Kenneth Knowles" <kl...@google.com> a
>>> écrit :
>>> >>>>>>>>>>
>>> >>>>>>>>>> @Romain - Idea/IntelliJ is great with Gradle, way better than
>>> >>>>>>>>>> Maven, and what I mean is that I have added enough hints that
>>> it works OOTB
>>> >>>>>>>>>> already. The rest of my instructions are just how you should
>>> override
>>> >>>>>>>>>> IntelliJ's defaults to have a proper dev env - mostly just
>>> about storing
>>> >>>>>>>>>> files outside the git clone.
>>> >>>>>>>>>>
>>> >>>>>>>>>>
>>> >>>>>>>>>> Hmm it is always super slow here and not as integrated as
>>> maven
>>> >>>>>>>>>> which is smooth thanks to the combination of idea build and
>>> maven plugin
>>> >>>>>>>>>> config read. Import is also faster cause it just reads meta
>>> and doesnt run
>>> >>>>>>>>>> anything. Hope it is a "a few times" issues at the moment but
>>> not yet sure.
>>> >>>>>>>>>>
>>> >>>>>>>>>>
>>> >>>>>>>>>>
>>> >>>>>>>>>> @Łukasz - yea I just mean the ones that I find in my email and
>>> >>>>>>>>>> browsing https://builds.apache.org/view/A-D/view/Beam/ which
>>> are the ones
>>> >>>>>>>>>> you classify as "totally not working". I would love to
>>> disable these.
>>> >>>>>>>>>>
>>> >>>>>>>>>> On the subject of running things on a pending PR - we should
>>> not
>>> >>>>>>>>>> run postcommit jobs on PRs. We should make separate
>>> (optional) precommit
>>> >>>>>>>>>> jobs that run the same build commands. That will give a more
>>> clear history
>>> >>>>>>>>>> and allow trivial configuration of which jobs deserve alert
>>> emails and which
>>> >>>>>>>>>> are not a problem. This is easy but I've been waiting to do
>>> it after Gradle
>>> >>>>>>>>>> migration.
>>> >>>>>>>>>>
>>> >>>>>>>>>> On Thu, Mar 8, 2018 at 11:37 AM Lukasz Cwik <lcwik@google.com
>>> >
>>> >>>>>>>>>> wrote:
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> Romain, would you like to host/plan/run the Gradle fixit day?
>>> >>>>>>>>>>>
>>> >>>>>>>>>>> On Thu, Mar 8, 2018 at 11:24 AM, Chamikara Jayalath
>>> >>>>>>>>>>> <ch...@google.com> wrote:
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>> +1 for the general idea of fixit day/week for Gradle.
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>> Agree with what Łukasz said. Some of these performance
>>> tests are
>>> >>>>>>>>>>>> new and are flaky due to other issues that were discovered
>>> during the
>>> >>>>>>>>>>>> process of adding the test.
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>> I think the high level blocker is updating performance
>>> testing
>>> >>>>>>>>>>>> framework to use Gradle. This will involve adding
>>> Gradle-based logic for
>>> >>>>>>>>>>>> invoking PerfKitBenchmaker, for example [1], and updating
>>> PerfKitBenchmarker
>>> >>>>>>>>>>>> to issue a Gradle command for running the benchmark [2].
>>> First task will be
>>> >>>>>>>>>>>> to find the work needed here and updating the relevant JIRA
>>> [3].
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>> Thanks,
>>> >>>>>>>>>>>> Cham
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>> [1]
>>> >>>>>>>>>>>>
>>> https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/pom.xml#L79
>>> >>>>>>>>>>>> [2]
>>> >>>>>>>>>>>>
>>> https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/blob/master/perfkitbenchmarker/beam_benchmark_helper.py
>>> >>>>>>>>>>>> [3] https://issues.apache.org/jira/browse/BEAM-3251
>>> >>>>>>>>>>>>
>>> >>>>>>>>>>>> On Thu, Mar 8, 2018 at 3:10 AM Łukasz Gajowy
>>> >>>>>>>>>>>> <lu...@gmail.com> wrote:
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>> 2018-03-07 22:29 GMT+01:00 Kenneth Knowles <klk@google.com
>>> >:
>>> >>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>> Based on https://builds.apache.org/view/A-D/view/Beam/
>>> and our
>>> >>>>>>>>>>>>>> failure spam level the performance tests are mostly not
>>> healthy anyhow. So
>>> >>>>>>>>>>>>>> is there any high level blocker to switching them or is
>>> it just someone
>>> >>>>>>>>>>>>>> sitting down with each one?
>>> >>>>>>>>>>>>>>
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>> I thought I could share my point of view here as I am
>>> working
>>> >>>>>>>>>>>>> on the Performance Test part for a while now. I wouldn't
>>> say those are
>>> >>>>>>>>>>>>> "mostly not healthy". The situation is as follows:
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>> - totally not working: Python, Spark performance tests
>>> (don't
>>> >>>>>>>>>>>>> know why, I'm not maintaining those tests. Should we
>>> disable them?)
>>> >>>>>>>>>>>>> - flaky: the recently re-enabled JDBC Performance Test.
>>> It's
>>> >>>>>>>>>>>>> seems to be flaky mostly due to:
>>> >>>>>>>>>>>>> https://issues.apache.org/jira/browse/BEAM-3798
>>> >>>>>>>>>>>>> - working well, rarely failing: AvroIO, TextIO, Compressed
>>> >>>>>>>>>>>>> Text, TFRecordIO
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>> Also, some test failures are caused due to pending PRs (we
>>> >>>>>>>>>>>>> sometimes run concrete tests to check if PRs won't break
>>> them). This also
>>> >>>>>>>>>>>>> causes failures sometimes.
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>> I can help with switching the Performance tests to Gradle
>>> as
>>> >>>>>>>>>>>>> this part seems to be free to take.
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>>> Łukasz
>>> >>>>>>>>>>>>>
>>> >>>>>>>>>>>
>>> >>>>>>>>>>
>>> >>>>>>>
>>> >>>>>
>>> >>>
>>> >
>>>
>>
>>
Re: Gradle status
Posted by Henning Rohde <he...@google.com>.
My understanding was the same as Ismaël's. I don't think breaking the build
with a large known gaps (but not fully known cost) is practical. Also, most
items in the jira are not even assigned yet.
On Thu, Mar 22, 2018 at 8:03 AM Romain Manni-Bucau <rm...@gmail.com>
wrote:
> Not really Ismaël, this thread was about to do it at once and have 1 day
> to fix it all.
>
> As mentionned at the very beginning nobody maintains the 2 system so it
> must stop after months so either we drop maven or gradle *at once*
> or we keep a state where each dev does what he wants and the build system
> just doesn't work.
>
> 2018-03-22 15:42 GMT+01:00 Ismaël Mejía <ie...@gmail.com>:
>
>> I don't think that removing all maven descriptors was the expected
>> path, no ? Or even a good idea at this moment.
>>
>> I understood that what we were going to do was to replace
>> incrementally the CI until we cover the whole maven functionality and
>> then remove it, from looking at the JIRA ticket
>> https://issues.apache.org/jira/browse/BEAM-3249 we are still far from
>> covering the complete maven functionality in particular for the
>> release part that could be the biggest pain point.
>>
>>
>> On Thu, Mar 22, 2018 at 9:30 AM, Romain Manni-Bucau
>> <rm...@gmail.com> wrote:
>> > hey guys,
>> >
>> > 2.4 is out, do we plan to drop all maven descriptors tomorrow or on
>> monday?
>> >
>> >
>> > Romain Manni-Bucau
>> > @rmannibucau | Blog | Old Blog | Github | LinkedIn | Book
>> >
>> > 2018-03-09 21:42 GMT+01:00 Kenneth Knowles <kl...@google.com>:
>> >>
>> >> On Fri, Mar 9, 2018 at 12:16 PM Lukasz Cwik <lc...@google.com> wrote:
>> >>>
>> >>> Based upon your description it seems as though you would rather have a
>> >>> way to run existing postcommits without it impacting dashboard/health
>> >>> stats/notifications/.... (We have just run the PostCommits on PRs for
>> >>> additional validation (like upgrading the Dataflow container image)).
>> >>
>> >>
>> >> Yes, that is exactly what I have described.
>> >>
>> >>> I don't think that keeping the current Java PreCommit as a proxy for
>> the
>> >>> the Java PostCommit is the right way to go but I also don't have the
>> time to
>> >>> implement what your actually asking for.
>> >>
>> >>
>> >> Mostly I thought this might be very easy based on the fact that they
>> are
>> >> nearly identical. If not, oh well.
>> >>
>> >> Kenn
>> >>
>> >>
>> >>> It seems more likely that migrating the PostCommit to Gradle will be
>> less
>> >>> work then adding the functionality but your argument where the
>> PreCommit is
>> >>> a proxy for the Java PostCommit also applies to the ValidatesRunner
>> >>> PostCommits and so forth requiring even more migration to happen
>> before you
>> >>> don't have to worry about maintaining Maven/breaking post commits.
>> >>>
>> >>> I'm fine with leaving both the Java/Gradle PreCommits running for now
>> and
>> >>> hopefully as more of the PostCommits are migrated off we will be able
>> to
>> >>> remove it.
>> >>>
>> >>> On Fri, Mar 9, 2018 at 11:39 AM, Kenneth Knowles <kl...@google.com>
>> wrote:
>> >>>>
>> >>>> Separate history (for easy dashboarding, health stats, etc) and
>> >>>> notification (email to dev@ for postcommits, nothing for
>> precommits) for pre
>> >>>> & post commit targets.
>> >>>>
>> >>>> A post commit failure is always a problem to be triaged at high
>> >>>> priority, while a precommit failure is just a natural occurrence.
>> >>>>
>> >>>> On Fri, Mar 9, 2018 at 11:33 AM Lukasz Cwik <lc...@google.com>
>> wrote:
>> >>>>>
>> >>>>> Ken, I'm probably not seeing something but how does using the
>> PreCommit
>> >>>>> as a proxy improve upon just running the post commit via the phrase
>> it
>> >>>>> already supports ('Run Java PostCommit')?
>> >>>>>
>> >>>>> On Fri, Mar 9, 2018 at 11:22 AM, Kenneth Knowles <kl...@google.com>
>> >>>>> wrote:
>> >>>>>>
>> >>>>>> Indeed, we've already had the discussion a couple of times and I
>> think
>> >>>>>> the criteria are clearly met. Incremental progress is a good thing
>> and we
>> >>>>>> shouldn't block it.
>> >>>>>>
>> >>>>>> OTOH I see where Romain is coming from and I have a good example
>> that
>> >>>>>> supports a slightly different action. Consider
>> >>>>>> https://github.com/apache/beam/pull/4740 which fixes some errors
>> in how we
>> >>>>>> use dependency mechanisms.
>> >>>>>>
>> >>>>>> This PR is green except that I need to fix some Maven pom slightly
>> >>>>>> more. That is throwaway work. I would love to just not have to do
>> it. But
>> >>>>>> removing the precommit does not actually make the PR OK to merge.
>> It would
>> >>>>>> cause postcommits to fail.
>> >>>>>>
>> >>>>>> We can hope such situations are rare. I think I tend to be hit by
>> this
>> >>>>>> more often than most, as I work with the project build health
>> quite a bit.
>> >>>>>>
>> >>>>>> Here is a proposal to support these things: instead of deleting the
>> >>>>>> job in #4814, move it to not run automatically but only via a
>> phrase. In
>> >>>>>> fact, you could migrate it to be the manually-invoked version of
>> the
>> >>>>>> postcommit job as we've discussed a couple times. Then if someone
>> is working
>> >>>>>> on the build in something like #4740 they can invoke it manually.
>> >>>>>>
>> >>>>>> Kenn
>> >>>>>>
>> >>>>>> On Fri, Mar 9, 2018 at 10:25 AM Lukasz Cwik <lc...@google.com>
>> wrote:
>> >>>>>>>
>> >>>>>>> Based upon the criteria that was discussed on the mailing
>> list[1], I
>> >>>>>>> would agree with Kenn about merging PR/4814 (drop Java Maven
>> precommit).
>> >>>>>>>
>> >>>>>>> 1:
>> >>>>>>>
>> https://lists.apache.org/thread.html/7eba5c77bc1a77b5046d915ab59f5f6fc41536c2c84863ad2efb5e99@%3Cdev.beam.apache.org%3E
>> >>>>>>>
>> >>>>>>> On Thu, Mar 8, 2018 at 9:10 PM, Romain Manni-Bucau
>> >>>>>>> <rm...@gmail.com> wrote:
>> >>>>>>>>
>> >>>>>>>> Hi Kenneth,
>> >>>>>>>>
>> >>>>>>>> For now maven covers the full needs of beam. If we start to have
>> >>>>>>>> this kind of PR we become dependent of the 2 builds which is
>> what this
>> >>>>>>>> thread is about avoiding so tempted to say it must be a PR drop
>> completely
>> >>>>>>>> maven or nothing as mentionned before.
>> >>>>>>>>
>> >>>>>>>> Le 9 mars 2018 04:48, "Kenneth Knowles" <kl...@google.com> a
>> écrit :
>> >>>>>>>>>
>> >>>>>>>>> I would like to briefly re-focus this discussion and suggest
>> that
>> >>>>>>>>> we merge https://github.com/apache/beam/pull/4814.
>> >>>>>>>>>
>> >>>>>>>>> The only material objection I've heard is that it means the
>> >>>>>>>>> precommit no longer tests exactly what is built for release. It
>> is a valid
>> >>>>>>>>> concern, but we have mvn postcommit so the coverage is not
>> lost. The
>> >>>>>>>>> benefits of quicker reviews and less Jenkins congestion seem
>> worth it to me.
>> >>>>>>>>>
>> >>>>>>>>> Kenn
>> >>>>>>>>>
>> >>>>>>>>> On Thu, Mar 8, 2018 at 12:33 PM Romain Manni-Bucau
>> >>>>>>>>> <rm...@gmail.com> wrote:
>> >>>>>>>>>>
>> >>>>>>>>>> @Luskasz: not sure Im the best to host it since I know more
>> gradle
>> >>>>>>>>>> internals that user interface/ecosystem but happy to help.
>> Will also require
>> >>>>>>>>>> a "sudo" merger for this day to merge fixes asap - guess we
>> can bypass
>> >>>>>>>>>> reviews or have a fast cycle plan for this day to avoid it to
>> be a week?
>> >>>>>>>>>>
>> >>>>>>>>>> Le 8 mars 2018 21:08, "Kenneth Knowles" <kl...@google.com> a
>> écrit :
>> >>>>>>>>>>
>> >>>>>>>>>> @Romain - Idea/IntelliJ is great with Gradle, way better than
>> >>>>>>>>>> Maven, and what I mean is that I have added enough hints that
>> it works OOTB
>> >>>>>>>>>> already. The rest of my instructions are just how you should
>> override
>> >>>>>>>>>> IntelliJ's defaults to have a proper dev env - mostly just
>> about storing
>> >>>>>>>>>> files outside the git clone.
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> Hmm it is always super slow here and not as integrated as maven
>> >>>>>>>>>> which is smooth thanks to the combination of idea build and
>> maven plugin
>> >>>>>>>>>> config read. Import is also faster cause it just reads meta
>> and doesnt run
>> >>>>>>>>>> anything. Hope it is a "a few times" issues at the moment but
>> not yet sure.
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> @Łukasz - yea I just mean the ones that I find in my email and
>> >>>>>>>>>> browsing https://builds.apache.org/view/A-D/view/Beam/ which
>> are the ones
>> >>>>>>>>>> you classify as "totally not working". I would love to disable
>> these.
>> >>>>>>>>>>
>> >>>>>>>>>> On the subject of running things on a pending PR - we should
>> not
>> >>>>>>>>>> run postcommit jobs on PRs. We should make separate (optional)
>> precommit
>> >>>>>>>>>> jobs that run the same build commands. That will give a more
>> clear history
>> >>>>>>>>>> and allow trivial configuration of which jobs deserve alert
>> emails and which
>> >>>>>>>>>> are not a problem. This is easy but I've been waiting to do it
>> after Gradle
>> >>>>>>>>>> migration.
>> >>>>>>>>>>
>> >>>>>>>>>> On Thu, Mar 8, 2018 at 11:37 AM Lukasz Cwik <lc...@google.com>
>> >>>>>>>>>> wrote:
>> >>>>>>>>>>>
>> >>>>>>>>>>> Romain, would you like to host/plan/run the Gradle fixit day?
>> >>>>>>>>>>>
>> >>>>>>>>>>> On Thu, Mar 8, 2018 at 11:24 AM, Chamikara Jayalath
>> >>>>>>>>>>> <ch...@google.com> wrote:
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> +1 for the general idea of fixit day/week for Gradle.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> Agree with what Łukasz said. Some of these performance tests
>> are
>> >>>>>>>>>>>> new and are flaky due to other issues that were discovered
>> during the
>> >>>>>>>>>>>> process of adding the test.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> I think the high level blocker is updating performance
>> testing
>> >>>>>>>>>>>> framework to use Gradle. This will involve adding
>> Gradle-based logic for
>> >>>>>>>>>>>> invoking PerfKitBenchmaker, for example [1], and updating
>> PerfKitBenchmarker
>> >>>>>>>>>>>> to issue a Gradle command for running the benchmark [2].
>> First task will be
>> >>>>>>>>>>>> to find the work needed here and updating the relevant JIRA
>> [3].
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> Thanks,
>> >>>>>>>>>>>> Cham
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> [1]
>> >>>>>>>>>>>>
>> https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/pom.xml#L79
>> >>>>>>>>>>>> [2]
>> >>>>>>>>>>>>
>> https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/blob/master/perfkitbenchmarker/beam_benchmark_helper.py
>> >>>>>>>>>>>> [3] https://issues.apache.org/jira/browse/BEAM-3251
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> On Thu, Mar 8, 2018 at 3:10 AM Łukasz Gajowy
>> >>>>>>>>>>>> <lu...@gmail.com> wrote:
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> 2018-03-07 22:29 GMT+01:00 Kenneth Knowles <klk@google.com
>> >:
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>> Based on https://builds.apache.org/view/A-D/view/Beam/
>> and our
>> >>>>>>>>>>>>>> failure spam level the performance tests are mostly not
>> healthy anyhow. So
>> >>>>>>>>>>>>>> is there any high level blocker to switching them or is it
>> just someone
>> >>>>>>>>>>>>>> sitting down with each one?
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> I thought I could share my point of view here as I am
>> working
>> >>>>>>>>>>>>> on the Performance Test part for a while now. I wouldn't
>> say those are
>> >>>>>>>>>>>>> "mostly not healthy". The situation is as follows:
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> - totally not working: Python, Spark performance tests
>> (don't
>> >>>>>>>>>>>>> know why, I'm not maintaining those tests. Should we
>> disable them?)
>> >>>>>>>>>>>>> - flaky: the recently re-enabled JDBC Performance Test. It's
>> >>>>>>>>>>>>> seems to be flaky mostly due to:
>> >>>>>>>>>>>>> https://issues.apache.org/jira/browse/BEAM-3798
>> >>>>>>>>>>>>> - working well, rarely failing: AvroIO, TextIO, Compressed
>> >>>>>>>>>>>>> Text, TFRecordIO
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Also, some test failures are caused due to pending PRs (we
>> >>>>>>>>>>>>> sometimes run concrete tests to check if PRs won't break
>> them). This also
>> >>>>>>>>>>>>> causes failures sometimes.
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> I can help with switching the Performance tests to Gradle as
>> >>>>>>>>>>>>> this part seems to be free to take.
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Łukasz
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>
>> >>>>>
>> >>>
>> >
>>
>
>
Re: Gradle status
Posted by Romain Manni-Bucau <rm...@gmail.com>.
Not really Ismaël, this thread was about to do it at once and have 1 day to
fix it all.
As mentionned at the very beginning nobody maintains the 2 system so it
must stop after months so either we drop maven or gradle *at once*
or we keep a state where each dev does what he wants and the build system
just doesn't work.
2018-03-22 15:42 GMT+01:00 Ismaël Mejía <ie...@gmail.com>:
> I don't think that removing all maven descriptors was the expected
> path, no ? Or even a good idea at this moment.
>
> I understood that what we were going to do was to replace
> incrementally the CI until we cover the whole maven functionality and
> then remove it, from looking at the JIRA ticket
> https://issues.apache.org/jira/browse/BEAM-3249 we are still far from
> covering the complete maven functionality in particular for the
> release part that could be the biggest pain point.
>
>
> On Thu, Mar 22, 2018 at 9:30 AM, Romain Manni-Bucau
> <rm...@gmail.com> wrote:
> > hey guys,
> >
> > 2.4 is out, do we plan to drop all maven descriptors tomorrow or on
> monday?
> >
> >
> > Romain Manni-Bucau
> > @rmannibucau | Blog | Old Blog | Github | LinkedIn | Book
> >
> > 2018-03-09 21:42 GMT+01:00 Kenneth Knowles <kl...@google.com>:
> >>
> >> On Fri, Mar 9, 2018 at 12:16 PM Lukasz Cwik <lc...@google.com> wrote:
> >>>
> >>> Based upon your description it seems as though you would rather have a
> >>> way to run existing postcommits without it impacting dashboard/health
> >>> stats/notifications/.... (We have just run the PostCommits on PRs for
> >>> additional validation (like upgrading the Dataflow container image)).
> >>
> >>
> >> Yes, that is exactly what I have described.
> >>
> >>> I don't think that keeping the current Java PreCommit as a proxy for
> the
> >>> the Java PostCommit is the right way to go but I also don't have the
> time to
> >>> implement what your actually asking for.
> >>
> >>
> >> Mostly I thought this might be very easy based on the fact that they are
> >> nearly identical. If not, oh well.
> >>
> >> Kenn
> >>
> >>
> >>> It seems more likely that migrating the PostCommit to Gradle will be
> less
> >>> work then adding the functionality but your argument where the
> PreCommit is
> >>> a proxy for the Java PostCommit also applies to the ValidatesRunner
> >>> PostCommits and so forth requiring even more migration to happen
> before you
> >>> don't have to worry about maintaining Maven/breaking post commits.
> >>>
> >>> I'm fine with leaving both the Java/Gradle PreCommits running for now
> and
> >>> hopefully as more of the PostCommits are migrated off we will be able
> to
> >>> remove it.
> >>>
> >>> On Fri, Mar 9, 2018 at 11:39 AM, Kenneth Knowles <kl...@google.com>
> wrote:
> >>>>
> >>>> Separate history (for easy dashboarding, health stats, etc) and
> >>>> notification (email to dev@ for postcommits, nothing for precommits)
> for pre
> >>>> & post commit targets.
> >>>>
> >>>> A post commit failure is always a problem to be triaged at high
> >>>> priority, while a precommit failure is just a natural occurrence.
> >>>>
> >>>> On Fri, Mar 9, 2018 at 11:33 AM Lukasz Cwik <lc...@google.com> wrote:
> >>>>>
> >>>>> Ken, I'm probably not seeing something but how does using the
> PreCommit
> >>>>> as a proxy improve upon just running the post commit via the phrase
> it
> >>>>> already supports ('Run Java PostCommit')?
> >>>>>
> >>>>> On Fri, Mar 9, 2018 at 11:22 AM, Kenneth Knowles <kl...@google.com>
> >>>>> wrote:
> >>>>>>
> >>>>>> Indeed, we've already had the discussion a couple of times and I
> think
> >>>>>> the criteria are clearly met. Incremental progress is a good thing
> and we
> >>>>>> shouldn't block it.
> >>>>>>
> >>>>>> OTOH I see where Romain is coming from and I have a good example
> that
> >>>>>> supports a slightly different action. Consider
> >>>>>> https://github.com/apache/beam/pull/4740 which fixes some errors
> in how we
> >>>>>> use dependency mechanisms.
> >>>>>>
> >>>>>> This PR is green except that I need to fix some Maven pom slightly
> >>>>>> more. That is throwaway work. I would love to just not have to do
> it. But
> >>>>>> removing the precommit does not actually make the PR OK to merge.
> It would
> >>>>>> cause postcommits to fail.
> >>>>>>
> >>>>>> We can hope such situations are rare. I think I tend to be hit by
> this
> >>>>>> more often than most, as I work with the project build health quite
> a bit.
> >>>>>>
> >>>>>> Here is a proposal to support these things: instead of deleting the
> >>>>>> job in #4814, move it to not run automatically but only via a
> phrase. In
> >>>>>> fact, you could migrate it to be the manually-invoked version of the
> >>>>>> postcommit job as we've discussed a couple times. Then if someone
> is working
> >>>>>> on the build in something like #4740 they can invoke it manually.
> >>>>>>
> >>>>>> Kenn
> >>>>>>
> >>>>>> On Fri, Mar 9, 2018 at 10:25 AM Lukasz Cwik <lc...@google.com>
> wrote:
> >>>>>>>
> >>>>>>> Based upon the criteria that was discussed on the mailing list[1],
> I
> >>>>>>> would agree with Kenn about merging PR/4814 (drop Java Maven
> precommit).
> >>>>>>>
> >>>>>>> 1:
> >>>>>>> https://lists.apache.org/thread.html/
> 7eba5c77bc1a77b5046d915ab59f5f6fc41536c2c84863ad2efb5e99@%
> 3Cdev.beam.apache.org%3E
> >>>>>>>
> >>>>>>> On Thu, Mar 8, 2018 at 9:10 PM, Romain Manni-Bucau
> >>>>>>> <rm...@gmail.com> wrote:
> >>>>>>>>
> >>>>>>>> Hi Kenneth,
> >>>>>>>>
> >>>>>>>> For now maven covers the full needs of beam. If we start to have
> >>>>>>>> this kind of PR we become dependent of the 2 builds which is what
> this
> >>>>>>>> thread is about avoiding so tempted to say it must be a PR drop
> completely
> >>>>>>>> maven or nothing as mentionned before.
> >>>>>>>>
> >>>>>>>> Le 9 mars 2018 04:48, "Kenneth Knowles" <kl...@google.com> a écrit
> :
> >>>>>>>>>
> >>>>>>>>> I would like to briefly re-focus this discussion and suggest that
> >>>>>>>>> we merge https://github.com/apache/beam/pull/4814.
> >>>>>>>>>
> >>>>>>>>> The only material objection I've heard is that it means the
> >>>>>>>>> precommit no longer tests exactly what is built for release. It
> is a valid
> >>>>>>>>> concern, but we have mvn postcommit so the coverage is not lost.
> The
> >>>>>>>>> benefits of quicker reviews and less Jenkins congestion seem
> worth it to me.
> >>>>>>>>>
> >>>>>>>>> Kenn
> >>>>>>>>>
> >>>>>>>>> On Thu, Mar 8, 2018 at 12:33 PM Romain Manni-Bucau
> >>>>>>>>> <rm...@gmail.com> wrote:
> >>>>>>>>>>
> >>>>>>>>>> @Luskasz: not sure Im the best to host it since I know more
> gradle
> >>>>>>>>>> internals that user interface/ecosystem but happy to help. Will
> also require
> >>>>>>>>>> a "sudo" merger for this day to merge fixes asap - guess we can
> bypass
> >>>>>>>>>> reviews or have a fast cycle plan for this day to avoid it to
> be a week?
> >>>>>>>>>>
> >>>>>>>>>> Le 8 mars 2018 21:08, "Kenneth Knowles" <kl...@google.com> a
> écrit :
> >>>>>>>>>>
> >>>>>>>>>> @Romain - Idea/IntelliJ is great with Gradle, way better than
> >>>>>>>>>> Maven, and what I mean is that I have added enough hints that
> it works OOTB
> >>>>>>>>>> already. The rest of my instructions are just how you should
> override
> >>>>>>>>>> IntelliJ's defaults to have a proper dev env - mostly just
> about storing
> >>>>>>>>>> files outside the git clone.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Hmm it is always super slow here and not as integrated as maven
> >>>>>>>>>> which is smooth thanks to the combination of idea build and
> maven plugin
> >>>>>>>>>> config read. Import is also faster cause it just reads meta and
> doesnt run
> >>>>>>>>>> anything. Hope it is a "a few times" issues at the moment but
> not yet sure.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> @Łukasz - yea I just mean the ones that I find in my email and
> >>>>>>>>>> browsing https://builds.apache.org/view/A-D/view/Beam/ which
> are the ones
> >>>>>>>>>> you classify as "totally not working". I would love to disable
> these.
> >>>>>>>>>>
> >>>>>>>>>> On the subject of running things on a pending PR - we should not
> >>>>>>>>>> run postcommit jobs on PRs. We should make separate (optional)
> precommit
> >>>>>>>>>> jobs that run the same build commands. That will give a more
> clear history
> >>>>>>>>>> and allow trivial configuration of which jobs deserve alert
> emails and which
> >>>>>>>>>> are not a problem. This is easy but I've been waiting to do it
> after Gradle
> >>>>>>>>>> migration.
> >>>>>>>>>>
> >>>>>>>>>> On Thu, Mar 8, 2018 at 11:37 AM Lukasz Cwik <lc...@google.com>
> >>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> Romain, would you like to host/plan/run the Gradle fixit day?
> >>>>>>>>>>>
> >>>>>>>>>>> On Thu, Mar 8, 2018 at 11:24 AM, Chamikara Jayalath
> >>>>>>>>>>> <ch...@google.com> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>> +1 for the general idea of fixit day/week for Gradle.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Agree with what Łukasz said. Some of these performance tests
> are
> >>>>>>>>>>>> new and are flaky due to other issues that were discovered
> during the
> >>>>>>>>>>>> process of adding the test.
> >>>>>>>>>>>>
> >>>>>>>>>>>> I think the high level blocker is updating performance testing
> >>>>>>>>>>>> framework to use Gradle. This will involve adding
> Gradle-based logic for
> >>>>>>>>>>>> invoking PerfKitBenchmaker, for example [1], and updating
> PerfKitBenchmarker
> >>>>>>>>>>>> to issue a Gradle command for running the benchmark [2].
> First task will be
> >>>>>>>>>>>> to find the work needed here and updating the relevant JIRA
> [3].
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks,
> >>>>>>>>>>>> Cham
> >>>>>>>>>>>>
> >>>>>>>>>>>> [1]
> >>>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/
> google-cloud-platform/pom.xml#L79
> >>>>>>>>>>>> [2]
> >>>>>>>>>>>> https://github.com/GoogleCloudPlatform/
> PerfKitBenchmarker/blob/master/perfkitbenchmarker/beam_benchmark_helper.py
> >>>>>>>>>>>> [3] https://issues.apache.org/jira/browse/BEAM-3251
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Thu, Mar 8, 2018 at 3:10 AM Łukasz Gajowy
> >>>>>>>>>>>> <lu...@gmail.com> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> 2018-03-07 22:29 GMT+01:00 Kenneth Knowles <kl...@google.com>:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Based on https://builds.apache.org/view/A-D/view/Beam/ and
> our
> >>>>>>>>>>>>>> failure spam level the performance tests are mostly not
> healthy anyhow. So
> >>>>>>>>>>>>>> is there any high level blocker to switching them or is it
> just someone
> >>>>>>>>>>>>>> sitting down with each one?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I thought I could share my point of view here as I am working
> >>>>>>>>>>>>> on the Performance Test part for a while now. I wouldn't say
> those are
> >>>>>>>>>>>>> "mostly not healthy". The situation is as follows:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> - totally not working: Python, Spark performance tests (don't
> >>>>>>>>>>>>> know why, I'm not maintaining those tests. Should we disable
> them?)
> >>>>>>>>>>>>> - flaky: the recently re-enabled JDBC Performance Test. It's
> >>>>>>>>>>>>> seems to be flaky mostly due to:
> >>>>>>>>>>>>> https://issues.apache.org/jira/browse/BEAM-3798
> >>>>>>>>>>>>> - working well, rarely failing: AvroIO, TextIO, Compressed
> >>>>>>>>>>>>> Text, TFRecordIO
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Also, some test failures are caused due to pending PRs (we
> >>>>>>>>>>>>> sometimes run concrete tests to check if PRs won't break
> them). This also
> >>>>>>>>>>>>> causes failures sometimes.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I can help with switching the Performance tests to Gradle as
> >>>>>>>>>>>>> this part seems to be free to take.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Łukasz
> >>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>
> >>>>>
> >>>
> >
>
Re: Gradle status
Posted by Ismaël Mejía <ie...@gmail.com>.
I don't think that removing all maven descriptors was the expected
path, no ? Or even a good idea at this moment.
I understood that what we were going to do was to replace
incrementally the CI until we cover the whole maven functionality and
then remove it, from looking at the JIRA ticket
https://issues.apache.org/jira/browse/BEAM-3249 we are still far from
covering the complete maven functionality in particular for the
release part that could be the biggest pain point.
On Thu, Mar 22, 2018 at 9:30 AM, Romain Manni-Bucau
<rm...@gmail.com> wrote:
> hey guys,
>
> 2.4 is out, do we plan to drop all maven descriptors tomorrow or on monday?
>
>
> Romain Manni-Bucau
> @rmannibucau | Blog | Old Blog | Github | LinkedIn | Book
>
> 2018-03-09 21:42 GMT+01:00 Kenneth Knowles <kl...@google.com>:
>>
>> On Fri, Mar 9, 2018 at 12:16 PM Lukasz Cwik <lc...@google.com> wrote:
>>>
>>> Based upon your description it seems as though you would rather have a
>>> way to run existing postcommits without it impacting dashboard/health
>>> stats/notifications/.... (We have just run the PostCommits on PRs for
>>> additional validation (like upgrading the Dataflow container image)).
>>
>>
>> Yes, that is exactly what I have described.
>>
>>> I don't think that keeping the current Java PreCommit as a proxy for the
>>> the Java PostCommit is the right way to go but I also don't have the time to
>>> implement what your actually asking for.
>>
>>
>> Mostly I thought this might be very easy based on the fact that they are
>> nearly identical. If not, oh well.
>>
>> Kenn
>>
>>
>>> It seems more likely that migrating the PostCommit to Gradle will be less
>>> work then adding the functionality but your argument where the PreCommit is
>>> a proxy for the Java PostCommit also applies to the ValidatesRunner
>>> PostCommits and so forth requiring even more migration to happen before you
>>> don't have to worry about maintaining Maven/breaking post commits.
>>>
>>> I'm fine with leaving both the Java/Gradle PreCommits running for now and
>>> hopefully as more of the PostCommits are migrated off we will be able to
>>> remove it.
>>>
>>> On Fri, Mar 9, 2018 at 11:39 AM, Kenneth Knowles <kl...@google.com> wrote:
>>>>
>>>> Separate history (for easy dashboarding, health stats, etc) and
>>>> notification (email to dev@ for postcommits, nothing for precommits) for pre
>>>> & post commit targets.
>>>>
>>>> A post commit failure is always a problem to be triaged at high
>>>> priority, while a precommit failure is just a natural occurrence.
>>>>
>>>> On Fri, Mar 9, 2018 at 11:33 AM Lukasz Cwik <lc...@google.com> wrote:
>>>>>
>>>>> Ken, I'm probably not seeing something but how does using the PreCommit
>>>>> as a proxy improve upon just running the post commit via the phrase it
>>>>> already supports ('Run Java PostCommit')?
>>>>>
>>>>> On Fri, Mar 9, 2018 at 11:22 AM, Kenneth Knowles <kl...@google.com>
>>>>> wrote:
>>>>>>
>>>>>> Indeed, we've already had the discussion a couple of times and I think
>>>>>> the criteria are clearly met. Incremental progress is a good thing and we
>>>>>> shouldn't block it.
>>>>>>
>>>>>> OTOH I see where Romain is coming from and I have a good example that
>>>>>> supports a slightly different action. Consider
>>>>>> https://github.com/apache/beam/pull/4740 which fixes some errors in how we
>>>>>> use dependency mechanisms.
>>>>>>
>>>>>> This PR is green except that I need to fix some Maven pom slightly
>>>>>> more. That is throwaway work. I would love to just not have to do it. But
>>>>>> removing the precommit does not actually make the PR OK to merge. It would
>>>>>> cause postcommits to fail.
>>>>>>
>>>>>> We can hope such situations are rare. I think I tend to be hit by this
>>>>>> more often than most, as I work with the project build health quite a bit.
>>>>>>
>>>>>> Here is a proposal to support these things: instead of deleting the
>>>>>> job in #4814, move it to not run automatically but only via a phrase. In
>>>>>> fact, you could migrate it to be the manually-invoked version of the
>>>>>> postcommit job as we've discussed a couple times. Then if someone is working
>>>>>> on the build in something like #4740 they can invoke it manually.
>>>>>>
>>>>>> Kenn
>>>>>>
>>>>>> On Fri, Mar 9, 2018 at 10:25 AM Lukasz Cwik <lc...@google.com> wrote:
>>>>>>>
>>>>>>> Based upon the criteria that was discussed on the mailing list[1], I
>>>>>>> would agree with Kenn about merging PR/4814 (drop Java Maven precommit).
>>>>>>>
>>>>>>> 1:
>>>>>>> https://lists.apache.org/thread.html/7eba5c77bc1a77b5046d915ab59f5f6fc41536c2c84863ad2efb5e99@%3Cdev.beam.apache.org%3E
>>>>>>>
>>>>>>> On Thu, Mar 8, 2018 at 9:10 PM, Romain Manni-Bucau
>>>>>>> <rm...@gmail.com> wrote:
>>>>>>>>
>>>>>>>> Hi Kenneth,
>>>>>>>>
>>>>>>>> For now maven covers the full needs of beam. If we start to have
>>>>>>>> this kind of PR we become dependent of the 2 builds which is what this
>>>>>>>> thread is about avoiding so tempted to say it must be a PR drop completely
>>>>>>>> maven or nothing as mentionned before.
>>>>>>>>
>>>>>>>> Le 9 mars 2018 04:48, "Kenneth Knowles" <kl...@google.com> a écrit :
>>>>>>>>>
>>>>>>>>> I would like to briefly re-focus this discussion and suggest that
>>>>>>>>> we merge https://github.com/apache/beam/pull/4814.
>>>>>>>>>
>>>>>>>>> The only material objection I've heard is that it means the
>>>>>>>>> precommit no longer tests exactly what is built for release. It is a valid
>>>>>>>>> concern, but we have mvn postcommit so the coverage is not lost. The
>>>>>>>>> benefits of quicker reviews and less Jenkins congestion seem worth it to me.
>>>>>>>>>
>>>>>>>>> Kenn
>>>>>>>>>
>>>>>>>>> On Thu, Mar 8, 2018 at 12:33 PM Romain Manni-Bucau
>>>>>>>>> <rm...@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>> @Luskasz: not sure Im the best to host it since I know more gradle
>>>>>>>>>> internals that user interface/ecosystem but happy to help. Will also require
>>>>>>>>>> a "sudo" merger for this day to merge fixes asap - guess we can bypass
>>>>>>>>>> reviews or have a fast cycle plan for this day to avoid it to be a week?
>>>>>>>>>>
>>>>>>>>>> Le 8 mars 2018 21:08, "Kenneth Knowles" <kl...@google.com> a écrit :
>>>>>>>>>>
>>>>>>>>>> @Romain - Idea/IntelliJ is great with Gradle, way better than
>>>>>>>>>> Maven, and what I mean is that I have added enough hints that it works OOTB
>>>>>>>>>> already. The rest of my instructions are just how you should override
>>>>>>>>>> IntelliJ's defaults to have a proper dev env - mostly just about storing
>>>>>>>>>> files outside the git clone.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Hmm it is always super slow here and not as integrated as maven
>>>>>>>>>> which is smooth thanks to the combination of idea build and maven plugin
>>>>>>>>>> config read. Import is also faster cause it just reads meta and doesnt run
>>>>>>>>>> anything. Hope it is a "a few times" issues at the moment but not yet sure.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> @Łukasz - yea I just mean the ones that I find in my email and
>>>>>>>>>> browsing https://builds.apache.org/view/A-D/view/Beam/ which are the ones
>>>>>>>>>> you classify as "totally not working". I would love to disable these.
>>>>>>>>>>
>>>>>>>>>> On the subject of running things on a pending PR - we should not
>>>>>>>>>> run postcommit jobs on PRs. We should make separate (optional) precommit
>>>>>>>>>> jobs that run the same build commands. That will give a more clear history
>>>>>>>>>> and allow trivial configuration of which jobs deserve alert emails and which
>>>>>>>>>> are not a problem. This is easy but I've been waiting to do it after Gradle
>>>>>>>>>> migration.
>>>>>>>>>>
>>>>>>>>>> On Thu, Mar 8, 2018 at 11:37 AM Lukasz Cwik <lc...@google.com>
>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Romain, would you like to host/plan/run the Gradle fixit day?
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Mar 8, 2018 at 11:24 AM, Chamikara Jayalath
>>>>>>>>>>> <ch...@google.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> +1 for the general idea of fixit day/week for Gradle.
>>>>>>>>>>>>
>>>>>>>>>>>> Agree with what Łukasz said. Some of these performance tests are
>>>>>>>>>>>> new and are flaky due to other issues that were discovered during the
>>>>>>>>>>>> process of adding the test.
>>>>>>>>>>>>
>>>>>>>>>>>> I think the high level blocker is updating performance testing
>>>>>>>>>>>> framework to use Gradle. This will involve adding Gradle-based logic for
>>>>>>>>>>>> invoking PerfKitBenchmaker, for example [1], and updating PerfKitBenchmarker
>>>>>>>>>>>> to issue a Gradle command for running the benchmark [2]. First task will be
>>>>>>>>>>>> to find the work needed here and updating the relevant JIRA [3].
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Cham
>>>>>>>>>>>>
>>>>>>>>>>>> [1]
>>>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/pom.xml#L79
>>>>>>>>>>>> [2]
>>>>>>>>>>>> https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/blob/master/perfkitbenchmarker/beam_benchmark_helper.py
>>>>>>>>>>>> [3] https://issues.apache.org/jira/browse/BEAM-3251
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Mar 8, 2018 at 3:10 AM Łukasz Gajowy
>>>>>>>>>>>> <lu...@gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2018-03-07 22:29 GMT+01:00 Kenneth Knowles <kl...@google.com>:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Based on https://builds.apache.org/view/A-D/view/Beam/ and our
>>>>>>>>>>>>>> failure spam level the performance tests are mostly not healthy anyhow. So
>>>>>>>>>>>>>> is there any high level blocker to switching them or is it just someone
>>>>>>>>>>>>>> sitting down with each one?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> I thought I could share my point of view here as I am working
>>>>>>>>>>>>> on the Performance Test part for a while now. I wouldn't say those are
>>>>>>>>>>>>> "mostly not healthy". The situation is as follows:
>>>>>>>>>>>>>
>>>>>>>>>>>>> - totally not working: Python, Spark performance tests (don't
>>>>>>>>>>>>> know why, I'm not maintaining those tests. Should we disable them?)
>>>>>>>>>>>>> - flaky: the recently re-enabled JDBC Performance Test. It's
>>>>>>>>>>>>> seems to be flaky mostly due to:
>>>>>>>>>>>>> https://issues.apache.org/jira/browse/BEAM-3798
>>>>>>>>>>>>> - working well, rarely failing: AvroIO, TextIO, Compressed
>>>>>>>>>>>>> Text, TFRecordIO
>>>>>>>>>>>>>
>>>>>>>>>>>>> Also, some test failures are caused due to pending PRs (we
>>>>>>>>>>>>> sometimes run concrete tests to check if PRs won't break them). This also
>>>>>>>>>>>>> causes failures sometimes.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I can help with switching the Performance tests to Gradle as
>>>>>>>>>>>>> this part seems to be free to take.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Łukasz
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>
>>>>>
>>>
>
Re: Gradle status
Posted by Romain Manni-Bucau <rm...@gmail.com>.
hey guys,
2.4 is out, do we plan to drop all maven descriptors tomorrow or on monday?
Romain Manni-Bucau
@rmannibucau <https://twitter.com/rmannibucau> | Blog
<https://rmannibucau.metawerx.net/> | Old Blog
<http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> |
LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book
<https://www.packtpub.com/application-development/java-ee-8-high-performance>
2018-03-09 21:42 GMT+01:00 Kenneth Knowles <kl...@google.com>:
> On Fri, Mar 9, 2018 at 12:16 PM Lukasz Cwik <lc...@google.com> wrote:
>
>> Based upon your description it seems as though you would rather have a
>> way to run existing postcommits without it impacting dashboard/health
>> stats/notifications/.... (We have just run the PostCommits on PRs for
>> additional validation (like upgrading the Dataflow container image)).
>>
>
> Yes, that is exactly what I have described.
>
> I don't think that keeping the current Java PreCommit as a proxy for the
>> the Java PostCommit is the right way to go but I also don't have the time
>> to implement what your actually asking for.
>>
>
> Mostly I thought this might be very easy based on the fact that they are
> nearly identical. If not, oh well.
>
> Kenn
>
>
> It seems more likely that migrating the PostCommit to Gradle will be less
>> work then adding the functionality but your argument where the PreCommit is
>> a proxy for the Java PostCommit also applies to the ValidatesRunner
>> PostCommits and so forth requiring even more migration to happen before you
>> don't have to worry about maintaining Maven/breaking post commits.
>>
>> I'm fine with leaving both the Java/Gradle PreCommits running for now and
>> hopefully as more of the PostCommits are migrated off we will be able to
>> remove it.
>>
>> On Fri, Mar 9, 2018 at 11:39 AM, Kenneth Knowles <kl...@google.com> wrote:
>>
>>> Separate history (for easy dashboarding, health stats, etc) and
>>> notification (email to dev@ for postcommits, nothing for precommits)
>>> for pre & post commit targets.
>>>
>>> A post commit failure is always a problem to be triaged at high
>>> priority, while a precommit failure is just a natural occurrence.
>>>
>>> On Fri, Mar 9, 2018 at 11:33 AM Lukasz Cwik <lc...@google.com> wrote:
>>>
>>>> Ken, I'm probably not seeing something but how does using the PreCommit
>>>> as a proxy improve upon just running the post commit via the phrase it
>>>> already supports ('Run Java PostCommit')?
>>>>
>>>> On Fri, Mar 9, 2018 at 11:22 AM, Kenneth Knowles <kl...@google.com>
>>>> wrote:
>>>>
>>>>> Indeed, we've already had the discussion a couple of times and I think
>>>>> the criteria are clearly met. Incremental progress is a good thing and we
>>>>> shouldn't block it.
>>>>>
>>>>> OTOH I see where Romain is coming from and I have a good example that
>>>>> supports a slightly different action. Consider
>>>>> https://github.com/apache/beam/pull/4740 which fixes some errors in
>>>>> how we use dependency mechanisms.
>>>>>
>>>>> This PR is green except that I need to fix some Maven pom slightly
>>>>> more. That is throwaway work. I would love to just not have to do it. But
>>>>> removing the precommit does not actually make the PR OK to merge. It would
>>>>> cause postcommits to fail.
>>>>>
>>>>> We can hope such situations are rare. I think I tend to be hit by this
>>>>> more often than most, as I work with the project build health quite a bit.
>>>>>
>>>>> Here is a proposal to support these things: instead of deleting the
>>>>> job in #4814, move it to not run automatically but only via a phrase. In
>>>>> fact, you could migrate it to be the manually-invoked version of the
>>>>> postcommit job as we've discussed a couple times. Then if someone is
>>>>> working on the build in something like #4740 they can invoke it manually.
>>>>>
>>>>> Kenn
>>>>>
>>>>> On Fri, Mar 9, 2018 at 10:25 AM Lukasz Cwik <lc...@google.com> wrote:
>>>>>
>>>>>> Based upon the criteria that was discussed on the mailing list[1], I
>>>>>> would agree with Kenn about merging PR/4814 (drop Java Maven precommit).
>>>>>>
>>>>>> 1: https://lists.apache.org/thread.html/
>>>>>> 7eba5c77bc1a77b5046d915ab59f5f6fc41536c2c84863ad2efb5e99@%
>>>>>> 3Cdev.beam.apache.org%3E
>>>>>>
>>>>>> On Thu, Mar 8, 2018 at 9:10 PM, Romain Manni-Bucau <
>>>>>> rmannibucau@gmail.com> wrote:
>>>>>>
>>>>>>> Hi Kenneth,
>>>>>>>
>>>>>>> For now maven covers the full needs of beam. If we start to have
>>>>>>> this kind of PR we become dependent of the 2 builds which is what this
>>>>>>> thread is about avoiding so tempted to say it must be a PR drop completely
>>>>>>> maven or nothing as mentionned before.
>>>>>>>
>>>>>>> Le 9 mars 2018 04:48, "Kenneth Knowles" <kl...@google.com> a écrit :
>>>>>>>
>>>>>>>> I would like to briefly re-focus this discussion and suggest that
>>>>>>>> we merge https://github.com/apache/beam/pull/4814.
>>>>>>>>
>>>>>>>> The only material objection I've heard is that it means the
>>>>>>>> precommit no longer tests exactly what is built for release. It is a valid
>>>>>>>> concern, but we have mvn postcommit so the coverage is not lost. The
>>>>>>>> benefits of quicker reviews and less Jenkins congestion seem worth it to me.
>>>>>>>>
>>>>>>>> Kenn
>>>>>>>>
>>>>>>>> On Thu, Mar 8, 2018 at 12:33 PM Romain Manni-Bucau <
>>>>>>>> rmannibucau@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> @Luskasz: not sure Im the best to host it since I know more gradle
>>>>>>>>> internals that user interface/ecosystem but happy to help. Will also
>>>>>>>>> require a "sudo" merger for this day to merge fixes asap - guess we can
>>>>>>>>> bypass reviews or have a fast cycle plan for this day to avoid it to be a
>>>>>>>>> week?
>>>>>>>>>
>>>>>>>>> Le 8 mars 2018 21:08, "Kenneth Knowles" <kl...@google.com> a écrit :
>>>>>>>>>
>>>>>>>>> @Romain - Idea/IntelliJ is great with Gradle, way better than
>>>>>>>>> Maven, and what I mean is that I have added enough hints that it works OOTB
>>>>>>>>> already. The rest of my instructions are just how you should override
>>>>>>>>> IntelliJ's defaults to have a proper dev env - mostly just about storing
>>>>>>>>> files outside the git clone.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hmm it is always super slow here and not as integrated as maven
>>>>>>>>> which is smooth thanks to the combination of idea build and maven plugin
>>>>>>>>> config read. Import is also faster cause it just reads meta and doesnt run
>>>>>>>>> anything. Hope it is a "a few times" issues at the moment but not yet sure.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> @Łukasz - yea I just mean the ones that I find in my email and
>>>>>>>>> browsing https://builds.apache.org/view/A-D/view/Beam/ which are
>>>>>>>>> the ones you classify as "totally not working". I would love to disable
>>>>>>>>> these.
>>>>>>>>>
>>>>>>>>> On the subject of running things on a pending PR - we should not
>>>>>>>>> run postcommit jobs on PRs. We should make separate (optional) precommit
>>>>>>>>> jobs that run the same build commands. That will give a more clear history
>>>>>>>>> and allow trivial configuration of which jobs deserve alert emails and
>>>>>>>>> which are not a problem. This is easy but I've been waiting to do it after
>>>>>>>>> Gradle migration.
>>>>>>>>>
>>>>>>>>> On Thu, Mar 8, 2018 at 11:37 AM Lukasz Cwik <lc...@google.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Romain, would you like to host/plan/run the Gradle fixit day?
>>>>>>>>>>
>>>>>>>>>> On Thu, Mar 8, 2018 at 11:24 AM, Chamikara Jayalath <
>>>>>>>>>> chamikara@google.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> +1 for the general idea of fixit day/week for Gradle.
>>>>>>>>>>>
>>>>>>>>>>> Agree with what Łukasz said. Some of these performance tests are
>>>>>>>>>>> new and are flaky due to other issues that were discovered during the
>>>>>>>>>>> process of adding the test.
>>>>>>>>>>>
>>>>>>>>>>> I think the high level blocker is updating performance testing
>>>>>>>>>>> framework to use Gradle. This will involve adding Gradle-based logic for
>>>>>>>>>>> invoking PerfKitBenchmaker, for example [1], and updating
>>>>>>>>>>> PerfKitBenchmarker to issue a Gradle command for running the benchmark [2].
>>>>>>>>>>> First task will be to find the work needed here and updating the relevant
>>>>>>>>>>> JIRA [3].
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Cham
>>>>>>>>>>>
>>>>>>>>>>> [1] https://github.com/apache/beam/blob/master/sdks/
>>>>>>>>>>> java/io/google-cloud-platform/pom.xml#L79
>>>>>>>>>>> [2] https://github.com/GoogleCloudPlatform/
>>>>>>>>>>> PerfKitBenchmarker/blob/master/perfkitbenchmarker/
>>>>>>>>>>> beam_benchmark_helper.py
>>>>>>>>>>> [3] https://issues.apache.org/jira/browse/BEAM-3251
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Mar 8, 2018 at 3:10 AM Łukasz Gajowy <
>>>>>>>>>>> lukasz.gajowy@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> 2018-03-07 22:29 GMT+01:00 Kenneth Knowles <kl...@google.com>:
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Based on https://builds.apache.org/view/A-D/view/Beam/ and
>>>>>>>>>>>>> our failure spam level the performance tests are mostly not healthy anyhow.
>>>>>>>>>>>>> So is there any high level blocker to switching them or is it just someone
>>>>>>>>>>>>> sitting down with each one?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>> I thought I could share my point of view here as I am working
>>>>>>>>>>>> on the Performance Test part for a while now. I wouldn't say those are
>>>>>>>>>>>> "mostly not healthy". The situation is as follows:
>>>>>>>>>>>>
>>>>>>>>>>>> - totally not working: Python, Spark performance tests (don't
>>>>>>>>>>>> know why, I'm not maintaining those tests. Should we disable them?)
>>>>>>>>>>>> - flaky: the recently re-enabled JDBC Performance Test. It's
>>>>>>>>>>>> seems to be flaky mostly due to: https://issues.apache.org/
>>>>>>>>>>>> jira/browse/BEAM-3798
>>>>>>>>>>>> - working well, rarely failing: AvroIO, TextIO, Compressed
>>>>>>>>>>>> Text, TFRecordIO
>>>>>>>>>>>>
>>>>>>>>>>>> Also, some test failures are caused due to pending PRs (we
>>>>>>>>>>>> sometimes run concrete tests to check if PRs won't break them). This also
>>>>>>>>>>>> causes failures sometimes.
>>>>>>>>>>>>
>>>>>>>>>>>> I can help with switching the Performance tests to Gradle as
>>>>>>>>>>>> this part seems to be free to take.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Łukasz
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>
>>>>
>>
Re: Gradle status
Posted by Kenneth Knowles <kl...@google.com>.
On Fri, Mar 9, 2018 at 12:16 PM Lukasz Cwik <lc...@google.com> wrote:
> Based upon your description it seems as though you would rather have a way
> to run existing postcommits without it impacting dashboard/health
> stats/notifications/.... (We have just run the PostCommits on PRs for
> additional validation (like upgrading the Dataflow container image)).
>
Yes, that is exactly what I have described.
I don't think that keeping the current Java PreCommit as a proxy for the
> the Java PostCommit is the right way to go but I also don't have the time
> to implement what your actually asking for.
>
Mostly I thought this might be very easy based on the fact that they are
nearly identical. If not, oh well.
Kenn
It seems more likely that migrating the PostCommit to Gradle will be less
> work then adding the functionality but your argument where the PreCommit is
> a proxy for the Java PostCommit also applies to the ValidatesRunner
> PostCommits and so forth requiring even more migration to happen before you
> don't have to worry about maintaining Maven/breaking post commits.
>
> I'm fine with leaving both the Java/Gradle PreCommits running for now and
> hopefully as more of the PostCommits are migrated off we will be able to
> remove it.
>
> On Fri, Mar 9, 2018 at 11:39 AM, Kenneth Knowles <kl...@google.com> wrote:
>
>> Separate history (for easy dashboarding, health stats, etc) and
>> notification (email to dev@ for postcommits, nothing for precommits) for
>> pre & post commit targets.
>>
>> A post commit failure is always a problem to be triaged at high priority,
>> while a precommit failure is just a natural occurrence.
>>
>> On Fri, Mar 9, 2018 at 11:33 AM Lukasz Cwik <lc...@google.com> wrote:
>>
>>> Ken, I'm probably not seeing something but how does using the PreCommit
>>> as a proxy improve upon just running the post commit via the phrase it
>>> already supports ('Run Java PostCommit')?
>>>
>>> On Fri, Mar 9, 2018 at 11:22 AM, Kenneth Knowles <kl...@google.com> wrote:
>>>
>>>> Indeed, we've already had the discussion a couple of times and I think
>>>> the criteria are clearly met. Incremental progress is a good thing and we
>>>> shouldn't block it.
>>>>
>>>> OTOH I see where Romain is coming from and I have a good example that
>>>> supports a slightly different action. Consider
>>>> https://github.com/apache/beam/pull/4740 which fixes some errors in
>>>> how we use dependency mechanisms.
>>>>
>>>> This PR is green except that I need to fix some Maven pom slightly
>>>> more. That is throwaway work. I would love to just not have to do it. But
>>>> removing the precommit does not actually make the PR OK to merge. It would
>>>> cause postcommits to fail.
>>>>
>>>> We can hope such situations are rare. I think I tend to be hit by this
>>>> more often than most, as I work with the project build health quite a bit.
>>>>
>>>> Here is a proposal to support these things: instead of deleting the job
>>>> in #4814, move it to not run automatically but only via a phrase. In fact,
>>>> you could migrate it to be the manually-invoked version of the postcommit
>>>> job as we've discussed a couple times. Then if someone is working on the
>>>> build in something like #4740 they can invoke it manually.
>>>>
>>>> Kenn
>>>>
>>>> On Fri, Mar 9, 2018 at 10:25 AM Lukasz Cwik <lc...@google.com> wrote:
>>>>
>>>>> Based upon the criteria that was discussed on the mailing list[1], I
>>>>> would agree with Kenn about merging PR/4814 (drop Java Maven precommit).
>>>>>
>>>>> 1:
>>>>> https://lists.apache.org/thread.html/7eba5c77bc1a77b5046d915ab59f5f6fc41536c2c84863ad2efb5e99@%3Cdev.beam.apache.org%3E
>>>>>
>>>>> On Thu, Mar 8, 2018 at 9:10 PM, Romain Manni-Bucau <
>>>>> rmannibucau@gmail.com> wrote:
>>>>>
>>>>>> Hi Kenneth,
>>>>>>
>>>>>> For now maven covers the full needs of beam. If we start to have this
>>>>>> kind of PR we become dependent of the 2 builds which is what this thread is
>>>>>> about avoiding so tempted to say it must be a PR drop completely maven or
>>>>>> nothing as mentionned before.
>>>>>>
>>>>>> Le 9 mars 2018 04:48, "Kenneth Knowles" <kl...@google.com> a écrit :
>>>>>>
>>>>>>> I would like to briefly re-focus this discussion and suggest that we
>>>>>>> merge https://github.com/apache/beam/pull/4814.
>>>>>>>
>>>>>>> The only material objection I've heard is that it means the
>>>>>>> precommit no longer tests exactly what is built for release. It is a valid
>>>>>>> concern, but we have mvn postcommit so the coverage is not lost. The
>>>>>>> benefits of quicker reviews and less Jenkins congestion seem worth it to me.
>>>>>>>
>>>>>>> Kenn
>>>>>>>
>>>>>>> On Thu, Mar 8, 2018 at 12:33 PM Romain Manni-Bucau <
>>>>>>> rmannibucau@gmail.com> wrote:
>>>>>>>
>>>>>>>> @Luskasz: not sure Im the best to host it since I know more gradle
>>>>>>>> internals that user interface/ecosystem but happy to help. Will also
>>>>>>>> require a "sudo" merger for this day to merge fixes asap - guess we can
>>>>>>>> bypass reviews or have a fast cycle plan for this day to avoid it to be a
>>>>>>>> week?
>>>>>>>>
>>>>>>>> Le 8 mars 2018 21:08, "Kenneth Knowles" <kl...@google.com> a écrit :
>>>>>>>>
>>>>>>>> @Romain - Idea/IntelliJ is great with Gradle, way better than
>>>>>>>> Maven, and what I mean is that I have added enough hints that it works OOTB
>>>>>>>> already. The rest of my instructions are just how you should override
>>>>>>>> IntelliJ's defaults to have a proper dev env - mostly just about storing
>>>>>>>> files outside the git clone.
>>>>>>>>
>>>>>>>>
>>>>>>>> Hmm it is always super slow here and not as integrated as maven
>>>>>>>> which is smooth thanks to the combination of idea build and maven plugin
>>>>>>>> config read. Import is also faster cause it just reads meta and doesnt run
>>>>>>>> anything. Hope it is a "a few times" issues at the moment but not yet sure.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> @Łukasz - yea I just mean the ones that I find in my email and
>>>>>>>> browsing https://builds.apache.org/view/A-D/view/Beam/ which are
>>>>>>>> the ones you classify as "totally not working". I would love to disable
>>>>>>>> these.
>>>>>>>>
>>>>>>>> On the subject of running things on a pending PR - we should not
>>>>>>>> run postcommit jobs on PRs. We should make separate (optional) precommit
>>>>>>>> jobs that run the same build commands. That will give a more clear history
>>>>>>>> and allow trivial configuration of which jobs deserve alert emails and
>>>>>>>> which are not a problem. This is easy but I've been waiting to do it after
>>>>>>>> Gradle migration.
>>>>>>>>
>>>>>>>> On Thu, Mar 8, 2018 at 11:37 AM Lukasz Cwik <lc...@google.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Romain, would you like to host/plan/run the Gradle fixit day?
>>>>>>>>>
>>>>>>>>> On Thu, Mar 8, 2018 at 11:24 AM, Chamikara Jayalath <
>>>>>>>>> chamikara@google.com> wrote:
>>>>>>>>>
>>>>>>>>>> +1 for the general idea of fixit day/week for Gradle.
>>>>>>>>>>
>>>>>>>>>> Agree with what Łukasz said. Some of these performance tests are
>>>>>>>>>> new and are flaky due to other issues that were discovered during the
>>>>>>>>>> process of adding the test.
>>>>>>>>>>
>>>>>>>>>> I think the high level blocker is updating performance testing
>>>>>>>>>> framework to use Gradle. This will involve adding Gradle-based logic for
>>>>>>>>>> invoking PerfKitBenchmaker, for example [1], and updating
>>>>>>>>>> PerfKitBenchmarker to issue a Gradle command for running the benchmark [2].
>>>>>>>>>> First task will be to find the work needed here and updating the relevant
>>>>>>>>>> JIRA [3].
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Cham
>>>>>>>>>>
>>>>>>>>>> [1]
>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/pom.xml#L79
>>>>>>>>>> [2]
>>>>>>>>>> https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/blob/master/perfkitbenchmarker/beam_benchmark_helper.py
>>>>>>>>>> [3] https://issues.apache.org/jira/browse/BEAM-3251
>>>>>>>>>>
>>>>>>>>>> On Thu, Mar 8, 2018 at 3:10 AM Łukasz Gajowy <
>>>>>>>>>> lukasz.gajowy@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> 2018-03-07 22:29 GMT+01:00 Kenneth Knowles <kl...@google.com>:
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Based on https://builds.apache.org/view/A-D/view/Beam/ and our
>>>>>>>>>>>> failure spam level the performance tests are mostly not healthy anyhow. So
>>>>>>>>>>>> is there any high level blocker to switching them or is it just someone
>>>>>>>>>>>> sitting down with each one?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> I thought I could share my point of view here as I am working on
>>>>>>>>>>> the Performance Test part for a while now. I wouldn't say those are "mostly
>>>>>>>>>>> not healthy". The situation is as follows:
>>>>>>>>>>>
>>>>>>>>>>> - totally not working: Python, Spark performance tests (don't
>>>>>>>>>>> know why, I'm not maintaining those tests. Should we disable them?)
>>>>>>>>>>> - flaky: the recently re-enabled JDBC Performance Test. It's
>>>>>>>>>>> seems to be flaky mostly due to:
>>>>>>>>>>> https://issues.apache.org/jira/browse/BEAM-3798
>>>>>>>>>>> - working well, rarely failing: AvroIO, TextIO, Compressed Text,
>>>>>>>>>>> TFRecordIO
>>>>>>>>>>>
>>>>>>>>>>> Also, some test failures are caused due to pending PRs (we
>>>>>>>>>>> sometimes run concrete tests to check if PRs won't break them). This also
>>>>>>>>>>> causes failures sometimes.
>>>>>>>>>>>
>>>>>>>>>>> I can help with switching the Performance tests to Gradle as
>>>>>>>>>>> this part seems to be free to take.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Łukasz
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>
>>>
>
Re: Gradle status
Posted by Lukasz Cwik <lc...@google.com>.
Based upon your description it seems as though you would rather have a way
to run existing postcommits without it impacting dashboard/health
stats/notifications/.... (We have just run the PostCommits on PRs for
additional validation (like upgrading the Dataflow container image)).
I don't think that keeping the current Java PreCommit as a proxy for the
the Java PostCommit is the right way to go but I also don't have the time
to implement what your actually asking for.
It seems more likely that migrating the PostCommit to Gradle will be less
work then adding the functionality but your argument where the PreCommit is
a proxy for the Java PostCommit also applies to the ValidatesRunner
PostCommits and so forth requiring even more migration to happen before you
don't have to worry about maintaining Maven/breaking post commits.
I'm fine with leaving both the Java/Gradle PreCommits running for now and
hopefully as more of the PostCommits are migrated off we will be able to
remove it.
On Fri, Mar 9, 2018 at 11:39 AM, Kenneth Knowles <kl...@google.com> wrote:
> Separate history (for easy dashboarding, health stats, etc) and
> notification (email to dev@ for postcommits, nothing for precommits) for
> pre & post commit targets.
>
> A post commit failure is always a problem to be triaged at high priority,
> while a precommit failure is just a natural occurrence.
>
> On Fri, Mar 9, 2018 at 11:33 AM Lukasz Cwik <lc...@google.com> wrote:
>
>> Ken, I'm probably not seeing something but how does using the PreCommit
>> as a proxy improve upon just running the post commit via the phrase it
>> already supports ('Run Java PostCommit')?
>>
>> On Fri, Mar 9, 2018 at 11:22 AM, Kenneth Knowles <kl...@google.com> wrote:
>>
>>> Indeed, we've already had the discussion a couple of times and I think
>>> the criteria are clearly met. Incremental progress is a good thing and we
>>> shouldn't block it.
>>>
>>> OTOH I see where Romain is coming from and I have a good example that
>>> supports a slightly different action. Consider
>>> https://github.com/apache/beam/pull/4740 which fixes some errors in how
>>> we use dependency mechanisms.
>>>
>>> This PR is green except that I need to fix some Maven pom slightly more.
>>> That is throwaway work. I would love to just not have to do it. But
>>> removing the precommit does not actually make the PR OK to merge. It would
>>> cause postcommits to fail.
>>>
>>> We can hope such situations are rare. I think I tend to be hit by this
>>> more often than most, as I work with the project build health quite a bit.
>>>
>>> Here is a proposal to support these things: instead of deleting the job
>>> in #4814, move it to not run automatically but only via a phrase. In fact,
>>> you could migrate it to be the manually-invoked version of the postcommit
>>> job as we've discussed a couple times. Then if someone is working on the
>>> build in something like #4740 they can invoke it manually.
>>>
>>> Kenn
>>>
>>> On Fri, Mar 9, 2018 at 10:25 AM Lukasz Cwik <lc...@google.com> wrote:
>>>
>>>> Based upon the criteria that was discussed on the mailing list[1], I
>>>> would agree with Kenn about merging PR/4814 (drop Java Maven precommit).
>>>>
>>>> 1: https://lists.apache.org/thread.html/7eba5c77bc1a77b5046d915ab59f5f
>>>> 6fc41536c2c84863ad2efb5e99@%3Cdev.beam.apache.org%3E
>>>>
>>>> On Thu, Mar 8, 2018 at 9:10 PM, Romain Manni-Bucau <
>>>> rmannibucau@gmail.com> wrote:
>>>>
>>>>> Hi Kenneth,
>>>>>
>>>>> For now maven covers the full needs of beam. If we start to have this
>>>>> kind of PR we become dependent of the 2 builds which is what this thread is
>>>>> about avoiding so tempted to say it must be a PR drop completely maven or
>>>>> nothing as mentionned before.
>>>>>
>>>>> Le 9 mars 2018 04:48, "Kenneth Knowles" <kl...@google.com> a écrit :
>>>>>
>>>>>> I would like to briefly re-focus this discussion and suggest that we
>>>>>> merge https://github.com/apache/beam/pull/4814.
>>>>>>
>>>>>> The only material objection I've heard is that it means the precommit
>>>>>> no longer tests exactly what is built for release. It is a valid concern,
>>>>>> but we have mvn postcommit so the coverage is not lost. The benefits of
>>>>>> quicker reviews and less Jenkins congestion seem worth it to me.
>>>>>>
>>>>>> Kenn
>>>>>>
>>>>>> On Thu, Mar 8, 2018 at 12:33 PM Romain Manni-Bucau <
>>>>>> rmannibucau@gmail.com> wrote:
>>>>>>
>>>>>>> @Luskasz: not sure Im the best to host it since I know more gradle
>>>>>>> internals that user interface/ecosystem but happy to help. Will also
>>>>>>> require a "sudo" merger for this day to merge fixes asap - guess we can
>>>>>>> bypass reviews or have a fast cycle plan for this day to avoid it to be a
>>>>>>> week?
>>>>>>>
>>>>>>> Le 8 mars 2018 21:08, "Kenneth Knowles" <kl...@google.com> a écrit :
>>>>>>>
>>>>>>> @Romain - Idea/IntelliJ is great with Gradle, way better than Maven,
>>>>>>> and what I mean is that I have added enough hints that it works OOTB
>>>>>>> already. The rest of my instructions are just how you should override
>>>>>>> IntelliJ's defaults to have a proper dev env - mostly just about storing
>>>>>>> files outside the git clone.
>>>>>>>
>>>>>>>
>>>>>>> Hmm it is always super slow here and not as integrated as maven
>>>>>>> which is smooth thanks to the combination of idea build and maven plugin
>>>>>>> config read. Import is also faster cause it just reads meta and doesnt run
>>>>>>> anything. Hope it is a "a few times" issues at the moment but not yet sure.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> @Łukasz - yea I just mean the ones that I find in my email and
>>>>>>> browsing https://builds.apache.org/view/A-D/view/Beam/ which are
>>>>>>> the ones you classify as "totally not working". I would love to disable
>>>>>>> these.
>>>>>>>
>>>>>>> On the subject of running things on a pending PR - we should not run
>>>>>>> postcommit jobs on PRs. We should make separate (optional) precommit jobs
>>>>>>> that run the same build commands. That will give a more clear history and
>>>>>>> allow trivial configuration of which jobs deserve alert emails and which
>>>>>>> are not a problem. This is easy but I've been waiting to do it after Gradle
>>>>>>> migration.
>>>>>>>
>>>>>>> On Thu, Mar 8, 2018 at 11:37 AM Lukasz Cwik <lc...@google.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Romain, would you like to host/plan/run the Gradle fixit day?
>>>>>>>>
>>>>>>>> On Thu, Mar 8, 2018 at 11:24 AM, Chamikara Jayalath <
>>>>>>>> chamikara@google.com> wrote:
>>>>>>>>
>>>>>>>>> +1 for the general idea of fixit day/week for Gradle.
>>>>>>>>>
>>>>>>>>> Agree with what Łukasz said. Some of these performance tests are
>>>>>>>>> new and are flaky due to other issues that were discovered during the
>>>>>>>>> process of adding the test.
>>>>>>>>>
>>>>>>>>> I think the high level blocker is updating performance testing
>>>>>>>>> framework to use Gradle. This will involve adding Gradle-based logic for
>>>>>>>>> invoking PerfKitBenchmaker, for example [1], and updating
>>>>>>>>> PerfKitBenchmarker to issue a Gradle command for running the benchmark [2].
>>>>>>>>> First task will be to find the work needed here and updating the relevant
>>>>>>>>> JIRA [3].
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Cham
>>>>>>>>>
>>>>>>>>> [1] https://github.com/apache/beam/blob/master/sdks/
>>>>>>>>> java/io/google-cloud-platform/pom.xml#L79
>>>>>>>>> [2] https://github.com/GoogleCloudPlatform/
>>>>>>>>> PerfKitBenchmarker/blob/master/perfkitbenchmarker/
>>>>>>>>> beam_benchmark_helper.py
>>>>>>>>> [3] https://issues.apache.org/jira/browse/BEAM-3251
>>>>>>>>>
>>>>>>>>> On Thu, Mar 8, 2018 at 3:10 AM Łukasz Gajowy <
>>>>>>>>> lukasz.gajowy@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> 2018-03-07 22:29 GMT+01:00 Kenneth Knowles <kl...@google.com>:
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Based on https://builds.apache.org/view/A-D/view/Beam/ and our
>>>>>>>>>>> failure spam level the performance tests are mostly not healthy anyhow. So
>>>>>>>>>>> is there any high level blocker to switching them or is it just someone
>>>>>>>>>>> sitting down with each one?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> I thought I could share my point of view here as I am working on
>>>>>>>>>> the Performance Test part for a while now. I wouldn't say those are "mostly
>>>>>>>>>> not healthy". The situation is as follows:
>>>>>>>>>>
>>>>>>>>>> - totally not working: Python, Spark performance tests (don't
>>>>>>>>>> know why, I'm not maintaining those tests. Should we disable them?)
>>>>>>>>>> - flaky: the recently re-enabled JDBC Performance Test. It's
>>>>>>>>>> seems to be flaky mostly due to: https://issues.apache.org/
>>>>>>>>>> jira/browse/BEAM-3798
>>>>>>>>>> - working well, rarely failing: AvroIO, TextIO, Compressed Text,
>>>>>>>>>> TFRecordIO
>>>>>>>>>>
>>>>>>>>>> Also, some test failures are caused due to pending PRs (we
>>>>>>>>>> sometimes run concrete tests to check if PRs won't break them). This also
>>>>>>>>>> causes failures sometimes.
>>>>>>>>>>
>>>>>>>>>> I can help with switching the Performance tests to Gradle as this
>>>>>>>>>> part seems to be free to take.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Łukasz
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>
>>
Re: Gradle status
Posted by Romain Manni-Bucau <rm...@gmail.com>.
Is having a 2 days period to "merge all mvn related pr we can" next week an
option? If so it can be enough and well fix gradle in the fix day (goal
being to reduce pr number a lot).
Le 9 mars 2018 20:40, "Kenneth Knowles" <kl...@google.com> a écrit :
> Separate history (for easy dashboarding, health stats, etc) and
> notification (email to dev@ for postcommits, nothing for precommits) for
> pre & post commit targets.
>
> A post commit failure is always a problem to be triaged at high priority,
> while a precommit failure is just a natural occurrence.
>
> On Fri, Mar 9, 2018 at 11:33 AM Lukasz Cwik <lc...@google.com> wrote:
>
>> Ken, I'm probably not seeing something but how does using the PreCommit
>> as a proxy improve upon just running the post commit via the phrase it
>> already supports ('Run Java PostCommit')?
>>
>> On Fri, Mar 9, 2018 at 11:22 AM, Kenneth Knowles <kl...@google.com> wrote:
>>
>>> Indeed, we've already had the discussion a couple of times and I think
>>> the criteria are clearly met. Incremental progress is a good thing and we
>>> shouldn't block it.
>>>
>>> OTOH I see where Romain is coming from and I have a good example that
>>> supports a slightly different action. Consider
>>> https://github.com/apache/beam/pull/4740 which fixes some errors in how
>>> we use dependency mechanisms.
>>>
>>> This PR is green except that I need to fix some Maven pom slightly more.
>>> That is throwaway work. I would love to just not have to do it. But
>>> removing the precommit does not actually make the PR OK to merge. It would
>>> cause postcommits to fail.
>>>
>>> We can hope such situations are rare. I think I tend to be hit by this
>>> more often than most, as I work with the project build health quite a bit.
>>>
>>> Here is a proposal to support these things: instead of deleting the job
>>> in #4814, move it to not run automatically but only via a phrase. In fact,
>>> you could migrate it to be the manually-invoked version of the postcommit
>>> job as we've discussed a couple times. Then if someone is working on the
>>> build in something like #4740 they can invoke it manually.
>>>
>>> Kenn
>>>
>>> On Fri, Mar 9, 2018 at 10:25 AM Lukasz Cwik <lc...@google.com> wrote:
>>>
>>>> Based upon the criteria that was discussed on the mailing list[1], I
>>>> would agree with Kenn about merging PR/4814 (drop Java Maven precommit).
>>>>
>>>> 1: https://lists.apache.org/thread.html/7eba5c77bc1a77b5046d915ab59f5f
>>>> 6fc41536c2c84863ad2efb5e99@%3Cdev.beam.apache.org%3E
>>>>
>>>> On Thu, Mar 8, 2018 at 9:10 PM, Romain Manni-Bucau <
>>>> rmannibucau@gmail.com> wrote:
>>>>
>>>>> Hi Kenneth,
>>>>>
>>>>> For now maven covers the full needs of beam. If we start to have this
>>>>> kind of PR we become dependent of the 2 builds which is what this thread is
>>>>> about avoiding so tempted to say it must be a PR drop completely maven or
>>>>> nothing as mentionned before.
>>>>>
>>>>> Le 9 mars 2018 04:48, "Kenneth Knowles" <kl...@google.com> a écrit :
>>>>>
>>>>>> I would like to briefly re-focus this discussion and suggest that we
>>>>>> merge https://github.com/apache/beam/pull/4814.
>>>>>>
>>>>>> The only material objection I've heard is that it means the precommit
>>>>>> no longer tests exactly what is built for release. It is a valid concern,
>>>>>> but we have mvn postcommit so the coverage is not lost. The benefits of
>>>>>> quicker reviews and less Jenkins congestion seem worth it to me.
>>>>>>
>>>>>> Kenn
>>>>>>
>>>>>> On Thu, Mar 8, 2018 at 12:33 PM Romain Manni-Bucau <
>>>>>> rmannibucau@gmail.com> wrote:
>>>>>>
>>>>>>> @Luskasz: not sure Im the best to host it since I know more gradle
>>>>>>> internals that user interface/ecosystem but happy to help. Will also
>>>>>>> require a "sudo" merger for this day to merge fixes asap - guess we can
>>>>>>> bypass reviews or have a fast cycle plan for this day to avoid it to be a
>>>>>>> week?
>>>>>>>
>>>>>>> Le 8 mars 2018 21:08, "Kenneth Knowles" <kl...@google.com> a écrit :
>>>>>>>
>>>>>>> @Romain - Idea/IntelliJ is great with Gradle, way better than Maven,
>>>>>>> and what I mean is that I have added enough hints that it works OOTB
>>>>>>> already. The rest of my instructions are just how you should override
>>>>>>> IntelliJ's defaults to have a proper dev env - mostly just about storing
>>>>>>> files outside the git clone.
>>>>>>>
>>>>>>>
>>>>>>> Hmm it is always super slow here and not as integrated as maven
>>>>>>> which is smooth thanks to the combination of idea build and maven plugin
>>>>>>> config read. Import is also faster cause it just reads meta and doesnt run
>>>>>>> anything. Hope it is a "a few times" issues at the moment but not yet sure.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> @Łukasz - yea I just mean the ones that I find in my email and
>>>>>>> browsing https://builds.apache.org/view/A-D/view/Beam/ which are
>>>>>>> the ones you classify as "totally not working". I would love to disable
>>>>>>> these.
>>>>>>>
>>>>>>> On the subject of running things on a pending PR - we should not run
>>>>>>> postcommit jobs on PRs. We should make separate (optional) precommit jobs
>>>>>>> that run the same build commands. That will give a more clear history and
>>>>>>> allow trivial configuration of which jobs deserve alert emails and which
>>>>>>> are not a problem. This is easy but I've been waiting to do it after Gradle
>>>>>>> migration.
>>>>>>>
>>>>>>> On Thu, Mar 8, 2018 at 11:37 AM Lukasz Cwik <lc...@google.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Romain, would you like to host/plan/run the Gradle fixit day?
>>>>>>>>
>>>>>>>> On Thu, Mar 8, 2018 at 11:24 AM, Chamikara Jayalath <
>>>>>>>> chamikara@google.com> wrote:
>>>>>>>>
>>>>>>>>> +1 for the general idea of fixit day/week for Gradle.
>>>>>>>>>
>>>>>>>>> Agree with what Łukasz said. Some of these performance tests are
>>>>>>>>> new and are flaky due to other issues that were discovered during the
>>>>>>>>> process of adding the test.
>>>>>>>>>
>>>>>>>>> I think the high level blocker is updating performance testing
>>>>>>>>> framework to use Gradle. This will involve adding Gradle-based logic for
>>>>>>>>> invoking PerfKitBenchmaker, for example [1], and updating
>>>>>>>>> PerfKitBenchmarker to issue a Gradle command for running the benchmark [2].
>>>>>>>>> First task will be to find the work needed here and updating the relevant
>>>>>>>>> JIRA [3].
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Cham
>>>>>>>>>
>>>>>>>>> [1] https://github.com/apache/beam/blob/master/sdks/
>>>>>>>>> java/io/google-cloud-platform/pom.xml#L79
>>>>>>>>> [2] https://github.com/GoogleCloudPlatform/
>>>>>>>>> PerfKitBenchmarker/blob/master/perfkitbenchmarker/
>>>>>>>>> beam_benchmark_helper.py
>>>>>>>>> [3] https://issues.apache.org/jira/browse/BEAM-3251
>>>>>>>>>
>>>>>>>>> On Thu, Mar 8, 2018 at 3:10 AM Łukasz Gajowy <
>>>>>>>>> lukasz.gajowy@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> 2018-03-07 22:29 GMT+01:00 Kenneth Knowles <kl...@google.com>:
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Based on https://builds.apache.org/view/A-D/view/Beam/ and our
>>>>>>>>>>> failure spam level the performance tests are mostly not healthy anyhow. So
>>>>>>>>>>> is there any high level blocker to switching them or is it just someone
>>>>>>>>>>> sitting down with each one?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> I thought I could share my point of view here as I am working on
>>>>>>>>>> the Performance Test part for a while now. I wouldn't say those are "mostly
>>>>>>>>>> not healthy". The situation is as follows:
>>>>>>>>>>
>>>>>>>>>> - totally not working: Python, Spark performance tests (don't
>>>>>>>>>> know why, I'm not maintaining those tests. Should we disable them?)
>>>>>>>>>> - flaky: the recently re-enabled JDBC Performance Test. It's
>>>>>>>>>> seems to be flaky mostly due to: https://issues.apache.org/
>>>>>>>>>> jira/browse/BEAM-3798
>>>>>>>>>> - working well, rarely failing: AvroIO, TextIO, Compressed Text,
>>>>>>>>>> TFRecordIO
>>>>>>>>>>
>>>>>>>>>> Also, some test failures are caused due to pending PRs (we
>>>>>>>>>> sometimes run concrete tests to check if PRs won't break them). This also
>>>>>>>>>> causes failures sometimes.
>>>>>>>>>>
>>>>>>>>>> I can help with switching the Performance tests to Gradle as this
>>>>>>>>>> part seems to be free to take.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Łukasz
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>
>>
Re: Gradle status
Posted by Kenneth Knowles <kl...@google.com>.
Separate history (for easy dashboarding, health stats, etc) and
notification (email to dev@ for postcommits, nothing for precommits) for
pre & post commit targets.
A post commit failure is always a problem to be triaged at high priority,
while a precommit failure is just a natural occurrence.
On Fri, Mar 9, 2018 at 11:33 AM Lukasz Cwik <lc...@google.com> wrote:
> Ken, I'm probably not seeing something but how does using the PreCommit as
> a proxy improve upon just running the post commit via the phrase it already
> supports ('Run Java PostCommit')?
>
> On Fri, Mar 9, 2018 at 11:22 AM, Kenneth Knowles <kl...@google.com> wrote:
>
>> Indeed, we've already had the discussion a couple of times and I think
>> the criteria are clearly met. Incremental progress is a good thing and we
>> shouldn't block it.
>>
>> OTOH I see where Romain is coming from and I have a good example that
>> supports a slightly different action. Consider
>> https://github.com/apache/beam/pull/4740 which fixes some errors in how
>> we use dependency mechanisms.
>>
>> This PR is green except that I need to fix some Maven pom slightly more.
>> That is throwaway work. I would love to just not have to do it. But
>> removing the precommit does not actually make the PR OK to merge. It would
>> cause postcommits to fail.
>>
>> We can hope such situations are rare. I think I tend to be hit by this
>> more often than most, as I work with the project build health quite a bit.
>>
>> Here is a proposal to support these things: instead of deleting the job
>> in #4814, move it to not run automatically but only via a phrase. In fact,
>> you could migrate it to be the manually-invoked version of the postcommit
>> job as we've discussed a couple times. Then if someone is working on the
>> build in something like #4740 they can invoke it manually.
>>
>> Kenn
>>
>> On Fri, Mar 9, 2018 at 10:25 AM Lukasz Cwik <lc...@google.com> wrote:
>>
>>> Based upon the criteria that was discussed on the mailing list[1], I
>>> would agree with Kenn about merging PR/4814 (drop Java Maven precommit).
>>>
>>> 1:
>>> https://lists.apache.org/thread.html/7eba5c77bc1a77b5046d915ab59f5f6fc41536c2c84863ad2efb5e99@%3Cdev.beam.apache.org%3E
>>>
>>> On Thu, Mar 8, 2018 at 9:10 PM, Romain Manni-Bucau <
>>> rmannibucau@gmail.com> wrote:
>>>
>>>> Hi Kenneth,
>>>>
>>>> For now maven covers the full needs of beam. If we start to have this
>>>> kind of PR we become dependent of the 2 builds which is what this thread is
>>>> about avoiding so tempted to say it must be a PR drop completely maven or
>>>> nothing as mentionned before.
>>>>
>>>> Le 9 mars 2018 04:48, "Kenneth Knowles" <kl...@google.com> a écrit :
>>>>
>>>>> I would like to briefly re-focus this discussion and suggest that we
>>>>> merge https://github.com/apache/beam/pull/4814.
>>>>>
>>>>> The only material objection I've heard is that it means the precommit
>>>>> no longer tests exactly what is built for release. It is a valid concern,
>>>>> but we have mvn postcommit so the coverage is not lost. The benefits of
>>>>> quicker reviews and less Jenkins congestion seem worth it to me.
>>>>>
>>>>> Kenn
>>>>>
>>>>> On Thu, Mar 8, 2018 at 12:33 PM Romain Manni-Bucau <
>>>>> rmannibucau@gmail.com> wrote:
>>>>>
>>>>>> @Luskasz: not sure Im the best to host it since I know more gradle
>>>>>> internals that user interface/ecosystem but happy to help. Will also
>>>>>> require a "sudo" merger for this day to merge fixes asap - guess we can
>>>>>> bypass reviews or have a fast cycle plan for this day to avoid it to be a
>>>>>> week?
>>>>>>
>>>>>> Le 8 mars 2018 21:08, "Kenneth Knowles" <kl...@google.com> a écrit :
>>>>>>
>>>>>> @Romain - Idea/IntelliJ is great with Gradle, way better than Maven,
>>>>>> and what I mean is that I have added enough hints that it works OOTB
>>>>>> already. The rest of my instructions are just how you should override
>>>>>> IntelliJ's defaults to have a proper dev env - mostly just about storing
>>>>>> files outside the git clone.
>>>>>>
>>>>>>
>>>>>> Hmm it is always super slow here and not as integrated as maven which
>>>>>> is smooth thanks to the combination of idea build and maven plugin config
>>>>>> read. Import is also faster cause it just reads meta and doesnt run
>>>>>> anything. Hope it is a "a few times" issues at the moment but not yet sure.
>>>>>>
>>>>>>
>>>>>>
>>>>>> @Łukasz - yea I just mean the ones that I find in my email and
>>>>>> browsing https://builds.apache.org/view/A-D/view/Beam/ which are the
>>>>>> ones you classify as "totally not working". I would love to disable these.
>>>>>>
>>>>>> On the subject of running things on a pending PR - we should not run
>>>>>> postcommit jobs on PRs. We should make separate (optional) precommit jobs
>>>>>> that run the same build commands. That will give a more clear history and
>>>>>> allow trivial configuration of which jobs deserve alert emails and which
>>>>>> are not a problem. This is easy but I've been waiting to do it after Gradle
>>>>>> migration.
>>>>>>
>>>>>> On Thu, Mar 8, 2018 at 11:37 AM Lukasz Cwik <lc...@google.com> wrote:
>>>>>>
>>>>>>> Romain, would you like to host/plan/run the Gradle fixit day?
>>>>>>>
>>>>>>> On Thu, Mar 8, 2018 at 11:24 AM, Chamikara Jayalath <
>>>>>>> chamikara@google.com> wrote:
>>>>>>>
>>>>>>>> +1 for the general idea of fixit day/week for Gradle.
>>>>>>>>
>>>>>>>> Agree with what Łukasz said. Some of these performance tests are
>>>>>>>> new and are flaky due to other issues that were discovered during the
>>>>>>>> process of adding the test.
>>>>>>>>
>>>>>>>> I think the high level blocker is updating performance testing
>>>>>>>> framework to use Gradle. This will involve adding Gradle-based logic for
>>>>>>>> invoking PerfKitBenchmaker, for example [1], and updating
>>>>>>>> PerfKitBenchmarker to issue a Gradle command for running the benchmark [2].
>>>>>>>> First task will be to find the work needed here and updating the relevant
>>>>>>>> JIRA [3].
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Cham
>>>>>>>>
>>>>>>>> [1]
>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/pom.xml#L79
>>>>>>>> [2]
>>>>>>>> https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/blob/master/perfkitbenchmarker/beam_benchmark_helper.py
>>>>>>>> [3] https://issues.apache.org/jira/browse/BEAM-3251
>>>>>>>>
>>>>>>>> On Thu, Mar 8, 2018 at 3:10 AM Łukasz Gajowy <
>>>>>>>> lukasz.gajowy@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> 2018-03-07 22:29 GMT+01:00 Kenneth Knowles <kl...@google.com>:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Based on https://builds.apache.org/view/A-D/view/Beam/ and our
>>>>>>>>>> failure spam level the performance tests are mostly not healthy anyhow. So
>>>>>>>>>> is there any high level blocker to switching them or is it just someone
>>>>>>>>>> sitting down with each one?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> I thought I could share my point of view here as I am working on
>>>>>>>>> the Performance Test part for a while now. I wouldn't say those are "mostly
>>>>>>>>> not healthy". The situation is as follows:
>>>>>>>>>
>>>>>>>>> - totally not working: Python, Spark performance tests (don't know
>>>>>>>>> why, I'm not maintaining those tests. Should we disable them?)
>>>>>>>>> - flaky: the recently re-enabled JDBC Performance Test. It's seems
>>>>>>>>> to be flaky mostly due to:
>>>>>>>>> https://issues.apache.org/jira/browse/BEAM-3798
>>>>>>>>> - working well, rarely failing: AvroIO, TextIO, Compressed Text,
>>>>>>>>> TFRecordIO
>>>>>>>>>
>>>>>>>>> Also, some test failures are caused due to pending PRs (we
>>>>>>>>> sometimes run concrete tests to check if PRs won't break them). This also
>>>>>>>>> causes failures sometimes.
>>>>>>>>>
>>>>>>>>> I can help with switching the Performance tests to Gradle as this
>>>>>>>>> part seems to be free to take.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Łukasz
>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>
>>>
>
Re: Gradle status
Posted by Lukasz Cwik <lc...@google.com>.
Ken, I'm probably not seeing something but how does using the PreCommit as
a proxy improve upon just running the post commit via the phrase it already
supports ('Run Java PostCommit')?
On Fri, Mar 9, 2018 at 11:22 AM, Kenneth Knowles <kl...@google.com> wrote:
> Indeed, we've already had the discussion a couple of times and I think the
> criteria are clearly met. Incremental progress is a good thing and we
> shouldn't block it.
>
> OTOH I see where Romain is coming from and I have a good example that
> supports a slightly different action. Consider https://github.com/apache/
> beam/pull/4740 which fixes some errors in how we use dependency
> mechanisms.
>
> This PR is green except that I need to fix some Maven pom slightly more.
> That is throwaway work. I would love to just not have to do it. But
> removing the precommit does not actually make the PR OK to merge. It would
> cause postcommits to fail.
>
> We can hope such situations are rare. I think I tend to be hit by this
> more often than most, as I work with the project build health quite a bit.
>
> Here is a proposal to support these things: instead of deleting the job in
> #4814, move it to not run automatically but only via a phrase. In fact, you
> could migrate it to be the manually-invoked version of the postcommit job
> as we've discussed a couple times. Then if someone is working on the build
> in something like #4740 they can invoke it manually.
>
> Kenn
>
> On Fri, Mar 9, 2018 at 10:25 AM Lukasz Cwik <lc...@google.com> wrote:
>
>> Based upon the criteria that was discussed on the mailing list[1], I
>> would agree with Kenn about merging PR/4814 (drop Java Maven precommit).
>>
>> 1: https://lists.apache.org/thread.html/7eba5c77bc1a77b5046d915ab59f5f
>> 6fc41536c2c84863ad2efb5e99@%3Cdev.beam.apache.org%3E
>>
>> On Thu, Mar 8, 2018 at 9:10 PM, Romain Manni-Bucau <rmannibucau@gmail.com
>> > wrote:
>>
>>> Hi Kenneth,
>>>
>>> For now maven covers the full needs of beam. If we start to have this
>>> kind of PR we become dependent of the 2 builds which is what this thread is
>>> about avoiding so tempted to say it must be a PR drop completely maven or
>>> nothing as mentionned before.
>>>
>>> Le 9 mars 2018 04:48, "Kenneth Knowles" <kl...@google.com> a écrit :
>>>
>>>> I would like to briefly re-focus this discussion and suggest that we
>>>> merge https://github.com/apache/beam/pull/4814.
>>>>
>>>> The only material objection I've heard is that it means the precommit
>>>> no longer tests exactly what is built for release. It is a valid concern,
>>>> but we have mvn postcommit so the coverage is not lost. The benefits of
>>>> quicker reviews and less Jenkins congestion seem worth it to me.
>>>>
>>>> Kenn
>>>>
>>>> On Thu, Mar 8, 2018 at 12:33 PM Romain Manni-Bucau <
>>>> rmannibucau@gmail.com> wrote:
>>>>
>>>>> @Luskasz: not sure Im the best to host it since I know more gradle
>>>>> internals that user interface/ecosystem but happy to help. Will also
>>>>> require a "sudo" merger for this day to merge fixes asap - guess we can
>>>>> bypass reviews or have a fast cycle plan for this day to avoid it to be a
>>>>> week?
>>>>>
>>>>> Le 8 mars 2018 21:08, "Kenneth Knowles" <kl...@google.com> a écrit :
>>>>>
>>>>> @Romain - Idea/IntelliJ is great with Gradle, way better than Maven,
>>>>> and what I mean is that I have added enough hints that it works OOTB
>>>>> already. The rest of my instructions are just how you should override
>>>>> IntelliJ's defaults to have a proper dev env - mostly just about storing
>>>>> files outside the git clone.
>>>>>
>>>>>
>>>>> Hmm it is always super slow here and not as integrated as maven which
>>>>> is smooth thanks to the combination of idea build and maven plugin config
>>>>> read. Import is also faster cause it just reads meta and doesnt run
>>>>> anything. Hope it is a "a few times" issues at the moment but not yet sure.
>>>>>
>>>>>
>>>>>
>>>>> @Łukasz - yea I just mean the ones that I find in my email and
>>>>> browsing https://builds.apache.org/view/A-D/view/Beam/ which are the
>>>>> ones you classify as "totally not working". I would love to disable these.
>>>>>
>>>>> On the subject of running things on a pending PR - we should not run
>>>>> postcommit jobs on PRs. We should make separate (optional) precommit jobs
>>>>> that run the same build commands. That will give a more clear history and
>>>>> allow trivial configuration of which jobs deserve alert emails and which
>>>>> are not a problem. This is easy but I've been waiting to do it after Gradle
>>>>> migration.
>>>>>
>>>>> On Thu, Mar 8, 2018 at 11:37 AM Lukasz Cwik <lc...@google.com> wrote:
>>>>>
>>>>>> Romain, would you like to host/plan/run the Gradle fixit day?
>>>>>>
>>>>>> On Thu, Mar 8, 2018 at 11:24 AM, Chamikara Jayalath <
>>>>>> chamikara@google.com> wrote:
>>>>>>
>>>>>>> +1 for the general idea of fixit day/week for Gradle.
>>>>>>>
>>>>>>> Agree with what Łukasz said. Some of these performance tests are new
>>>>>>> and are flaky due to other issues that were discovered during the process
>>>>>>> of adding the test.
>>>>>>>
>>>>>>> I think the high level blocker is updating performance testing
>>>>>>> framework to use Gradle. This will involve adding Gradle-based logic for
>>>>>>> invoking PerfKitBenchmaker, for example [1], and updating
>>>>>>> PerfKitBenchmarker to issue a Gradle command for running the benchmark [2].
>>>>>>> First task will be to find the work needed here and updating the relevant
>>>>>>> JIRA [3].
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Cham
>>>>>>>
>>>>>>> [1] https://github.com/apache/beam/blob/master/sdks/
>>>>>>> java/io/google-cloud-platform/pom.xml#L79
>>>>>>> [2] https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/blob/
>>>>>>> master/perfkitbenchmarker/beam_benchmark_helper.py
>>>>>>> [3] https://issues.apache.org/jira/browse/BEAM-3251
>>>>>>>
>>>>>>> On Thu, Mar 8, 2018 at 3:10 AM Łukasz Gajowy <
>>>>>>> lukasz.gajowy@gmail.com> wrote:
>>>>>>>
>>>>>>>> 2018-03-07 22:29 GMT+01:00 Kenneth Knowles <kl...@google.com>:
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Based on https://builds.apache.org/view/A-D/view/Beam/ and our
>>>>>>>>> failure spam level the performance tests are mostly not healthy anyhow. So
>>>>>>>>> is there any high level blocker to switching them or is it just someone
>>>>>>>>> sitting down with each one?
>>>>>>>>>
>>>>>>>>>
>>>>>>>> I thought I could share my point of view here as I am working on
>>>>>>>> the Performance Test part for a while now. I wouldn't say those are "mostly
>>>>>>>> not healthy". The situation is as follows:
>>>>>>>>
>>>>>>>> - totally not working: Python, Spark performance tests (don't know
>>>>>>>> why, I'm not maintaining those tests. Should we disable them?)
>>>>>>>> - flaky: the recently re-enabled JDBC Performance Test. It's seems
>>>>>>>> to be flaky mostly due to: https://issues.apache.org/
>>>>>>>> jira/browse/BEAM-3798
>>>>>>>> - working well, rarely failing: AvroIO, TextIO, Compressed Text,
>>>>>>>> TFRecordIO
>>>>>>>>
>>>>>>>> Also, some test failures are caused due to pending PRs (we
>>>>>>>> sometimes run concrete tests to check if PRs won't break them). This also
>>>>>>>> causes failures sometimes.
>>>>>>>>
>>>>>>>> I can help with switching the Performance tests to Gradle as this
>>>>>>>> part seems to be free to take.
>>>>>>>>
>>>>>>>>
>>>>>>>> Łukasz
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>
>>
Re: Gradle status
Posted by Kenneth Knowles <kl...@google.com>.
Indeed, we've already had the discussion a couple of times and I think the
criteria are clearly met. Incremental progress is a good thing and we
shouldn't block it.
OTOH I see where Romain is coming from and I have a good example that
supports a slightly different action. Consider
https://github.com/apache/beam/pull/4740 which fixes some errors in how we
use dependency mechanisms.
This PR is green except that I need to fix some Maven pom slightly more.
That is throwaway work. I would love to just not have to do it. But
removing the precommit does not actually make the PR OK to merge. It would
cause postcommits to fail.
We can hope such situations are rare. I think I tend to be hit by this more
often than most, as I work with the project build health quite a bit.
Here is a proposal to support these things: instead of deleting the job in
#4814, move it to not run automatically but only via a phrase. In fact, you
could migrate it to be the manually-invoked version of the postcommit job
as we've discussed a couple times. Then if someone is working on the build
in something like #4740 they can invoke it manually.
Kenn
On Fri, Mar 9, 2018 at 10:25 AM Lukasz Cwik <lc...@google.com> wrote:
> Based upon the criteria that was discussed on the mailing list[1], I would
> agree with Kenn about merging PR/4814 (drop Java Maven precommit).
>
> 1:
> https://lists.apache.org/thread.html/7eba5c77bc1a77b5046d915ab59f5f6fc41536c2c84863ad2efb5e99@%3Cdev.beam.apache.org%3E
>
> On Thu, Mar 8, 2018 at 9:10 PM, Romain Manni-Bucau <rm...@gmail.com>
> wrote:
>
>> Hi Kenneth,
>>
>> For now maven covers the full needs of beam. If we start to have this
>> kind of PR we become dependent of the 2 builds which is what this thread is
>> about avoiding so tempted to say it must be a PR drop completely maven or
>> nothing as mentionned before.
>>
>> Le 9 mars 2018 04:48, "Kenneth Knowles" <kl...@google.com> a écrit :
>>
>>> I would like to briefly re-focus this discussion and suggest that we
>>> merge https://github.com/apache/beam/pull/4814.
>>>
>>> The only material objection I've heard is that it means the precommit no
>>> longer tests exactly what is built for release. It is a valid concern, but
>>> we have mvn postcommit so the coverage is not lost. The benefits of quicker
>>> reviews and less Jenkins congestion seem worth it to me.
>>>
>>> Kenn
>>>
>>> On Thu, Mar 8, 2018 at 12:33 PM Romain Manni-Bucau <
>>> rmannibucau@gmail.com> wrote:
>>>
>>>> @Luskasz: not sure Im the best to host it since I know more gradle
>>>> internals that user interface/ecosystem but happy to help. Will also
>>>> require a "sudo" merger for this day to merge fixes asap - guess we can
>>>> bypass reviews or have a fast cycle plan for this day to avoid it to be a
>>>> week?
>>>>
>>>> Le 8 mars 2018 21:08, "Kenneth Knowles" <kl...@google.com> a écrit :
>>>>
>>>> @Romain - Idea/IntelliJ is great with Gradle, way better than Maven,
>>>> and what I mean is that I have added enough hints that it works OOTB
>>>> already. The rest of my instructions are just how you should override
>>>> IntelliJ's defaults to have a proper dev env - mostly just about storing
>>>> files outside the git clone.
>>>>
>>>>
>>>> Hmm it is always super slow here and not as integrated as maven which
>>>> is smooth thanks to the combination of idea build and maven plugin config
>>>> read. Import is also faster cause it just reads meta and doesnt run
>>>> anything. Hope it is a "a few times" issues at the moment but not yet sure.
>>>>
>>>>
>>>>
>>>> @Łukasz - yea I just mean the ones that I find in my email and
>>>> browsing https://builds.apache.org/view/A-D/view/Beam/ which are the
>>>> ones you classify as "totally not working". I would love to disable these.
>>>>
>>>> On the subject of running things on a pending PR - we should not run
>>>> postcommit jobs on PRs. We should make separate (optional) precommit jobs
>>>> that run the same build commands. That will give a more clear history and
>>>> allow trivial configuration of which jobs deserve alert emails and which
>>>> are not a problem. This is easy but I've been waiting to do it after Gradle
>>>> migration.
>>>>
>>>> On Thu, Mar 8, 2018 at 11:37 AM Lukasz Cwik <lc...@google.com> wrote:
>>>>
>>>>> Romain, would you like to host/plan/run the Gradle fixit day?
>>>>>
>>>>> On Thu, Mar 8, 2018 at 11:24 AM, Chamikara Jayalath <
>>>>> chamikara@google.com> wrote:
>>>>>
>>>>>> +1 for the general idea of fixit day/week for Gradle.
>>>>>>
>>>>>> Agree with what Łukasz said. Some of these performance tests are new
>>>>>> and are flaky due to other issues that were discovered during the process
>>>>>> of adding the test.
>>>>>>
>>>>>> I think the high level blocker is updating performance testing
>>>>>> framework to use Gradle. This will involve adding Gradle-based logic for
>>>>>> invoking PerfKitBenchmaker, for example [1], and updating
>>>>>> PerfKitBenchmarker to issue a Gradle command for running the benchmark [2].
>>>>>> First task will be to find the work needed here and updating the relevant
>>>>>> JIRA [3].
>>>>>>
>>>>>> Thanks,
>>>>>> Cham
>>>>>>
>>>>>> [1]
>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/pom.xml#L79
>>>>>> [2]
>>>>>> https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/blob/master/perfkitbenchmarker/beam_benchmark_helper.py
>>>>>> [3] https://issues.apache.org/jira/browse/BEAM-3251
>>>>>>
>>>>>> On Thu, Mar 8, 2018 at 3:10 AM Łukasz Gajowy <lu...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> 2018-03-07 22:29 GMT+01:00 Kenneth Knowles <kl...@google.com>:
>>>>>>>
>>>>>>>>
>>>>>>>> Based on https://builds.apache.org/view/A-D/view/Beam/ and our
>>>>>>>> failure spam level the performance tests are mostly not healthy anyhow. So
>>>>>>>> is there any high level blocker to switching them or is it just someone
>>>>>>>> sitting down with each one?
>>>>>>>>
>>>>>>>>
>>>>>>> I thought I could share my point of view here as I am working on the
>>>>>>> Performance Test part for a while now. I wouldn't say those are "mostly not
>>>>>>> healthy". The situation is as follows:
>>>>>>>
>>>>>>> - totally not working: Python, Spark performance tests (don't know
>>>>>>> why, I'm not maintaining those tests. Should we disable them?)
>>>>>>> - flaky: the recently re-enabled JDBC Performance Test. It's seems
>>>>>>> to be flaky mostly due to:
>>>>>>> https://issues.apache.org/jira/browse/BEAM-3798
>>>>>>> - working well, rarely failing: AvroIO, TextIO, Compressed Text,
>>>>>>> TFRecordIO
>>>>>>>
>>>>>>> Also, some test failures are caused due to pending PRs (we sometimes
>>>>>>> run concrete tests to check if PRs won't break them). This also causes
>>>>>>> failures sometimes.
>>>>>>>
>>>>>>> I can help with switching the Performance tests to Gradle as this
>>>>>>> part seems to be free to take.
>>>>>>>
>>>>>>>
>>>>>>> Łukasz
>>>>>>>
>>>>>>>
>>>>>
>>>>
>
Re: Gradle status
Posted by Lukasz Cwik <lc...@google.com>.
Based upon the criteria that was discussed on the mailing list[1], I would
agree with Kenn about merging PR/4814 (drop Java Maven precommit).
1:
https://lists.apache.org/thread.html/7eba5c77bc1a77b5046d915ab59f5f6fc41536c2c84863ad2efb5e99@%3Cdev.beam.apache.org%3E
On Thu, Mar 8, 2018 at 9:10 PM, Romain Manni-Bucau <rm...@gmail.com>
wrote:
> Hi Kenneth,
>
> For now maven covers the full needs of beam. If we start to have this kind
> of PR we become dependent of the 2 builds which is what this thread is
> about avoiding so tempted to say it must be a PR drop completely maven or
> nothing as mentionned before.
>
> Le 9 mars 2018 04:48, "Kenneth Knowles" <kl...@google.com> a écrit :
>
>> I would like to briefly re-focus this discussion and suggest that we
>> merge https://github.com/apache/beam/pull/4814.
>>
>> The only material objection I've heard is that it means the precommit no
>> longer tests exactly what is built for release. It is a valid concern, but
>> we have mvn postcommit so the coverage is not lost. The benefits of quicker
>> reviews and less Jenkins congestion seem worth it to me.
>>
>> Kenn
>>
>> On Thu, Mar 8, 2018 at 12:33 PM Romain Manni-Bucau <rm...@gmail.com>
>> wrote:
>>
>>> @Luskasz: not sure Im the best to host it since I know more gradle
>>> internals that user interface/ecosystem but happy to help. Will also
>>> require a "sudo" merger for this day to merge fixes asap - guess we can
>>> bypass reviews or have a fast cycle plan for this day to avoid it to be a
>>> week?
>>>
>>> Le 8 mars 2018 21:08, "Kenneth Knowles" <kl...@google.com> a écrit :
>>>
>>> @Romain - Idea/IntelliJ is great with Gradle, way better than Maven, and
>>> what I mean is that I have added enough hints that it works OOTB already.
>>> The rest of my instructions are just how you should override IntelliJ's
>>> defaults to have a proper dev env - mostly just about storing files outside
>>> the git clone.
>>>
>>>
>>> Hmm it is always super slow here and not as integrated as maven which is
>>> smooth thanks to the combination of idea build and maven plugin config
>>> read. Import is also faster cause it just reads meta and doesnt run
>>> anything. Hope it is a "a few times" issues at the moment but not yet sure.
>>>
>>>
>>>
>>> @Łukasz - yea I just mean the ones that I find in my email and browsing
>>> https://builds.apache.org/view/A-D/view/Beam/ which are the ones you
>>> classify as "totally not working". I would love to disable these.
>>>
>>> On the subject of running things on a pending PR - we should not run
>>> postcommit jobs on PRs. We should make separate (optional) precommit jobs
>>> that run the same build commands. That will give a more clear history and
>>> allow trivial configuration of which jobs deserve alert emails and which
>>> are not a problem. This is easy but I've been waiting to do it after Gradle
>>> migration.
>>>
>>> On Thu, Mar 8, 2018 at 11:37 AM Lukasz Cwik <lc...@google.com> wrote:
>>>
>>>> Romain, would you like to host/plan/run the Gradle fixit day?
>>>>
>>>> On Thu, Mar 8, 2018 at 11:24 AM, Chamikara Jayalath <
>>>> chamikara@google.com> wrote:
>>>>
>>>>> +1 for the general idea of fixit day/week for Gradle.
>>>>>
>>>>> Agree with what Łukasz said. Some of these performance tests are new
>>>>> and are flaky due to other issues that were discovered during the process
>>>>> of adding the test.
>>>>>
>>>>> I think the high level blocker is updating performance testing
>>>>> framework to use Gradle. This will involve adding Gradle-based logic for
>>>>> invoking PerfKitBenchmaker, for example [1], and updating
>>>>> PerfKitBenchmarker to issue a Gradle command for running the benchmark [2].
>>>>> First task will be to find the work needed here and updating the relevant
>>>>> JIRA [3].
>>>>>
>>>>> Thanks,
>>>>> Cham
>>>>>
>>>>> [1] https://github.com/apache/beam/blob/master/sdks/java/
>>>>> io/google-cloud-platform/pom.xml#L79
>>>>> [2] https://github.com/GoogleCloudPlatform/PerfKitBenchmarke
>>>>> r/blob/master/perfkitbenchmarker/beam_benchmark_helper.py
>>>>> [3] https://issues.apache.org/jira/browse/BEAM-3251
>>>>>
>>>>> On Thu, Mar 8, 2018 at 3:10 AM Łukasz Gajowy <lu...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> 2018-03-07 22:29 GMT+01:00 Kenneth Knowles <kl...@google.com>:
>>>>>>
>>>>>>>
>>>>>>> Based on https://builds.apache.org/view/A-D/view/Beam/ and our
>>>>>>> failure spam level the performance tests are mostly not healthy anyhow. So
>>>>>>> is there any high level blocker to switching them or is it just someone
>>>>>>> sitting down with each one?
>>>>>>>
>>>>>>>
>>>>>> I thought I could share my point of view here as I am working on the
>>>>>> Performance Test part for a while now. I wouldn't say those are "mostly not
>>>>>> healthy". The situation is as follows:
>>>>>>
>>>>>> - totally not working: Python, Spark performance tests (don't know
>>>>>> why, I'm not maintaining those tests. Should we disable them?)
>>>>>> - flaky: the recently re-enabled JDBC Performance Test. It's seems to
>>>>>> be flaky mostly due to: https://issues.apache.org/
>>>>>> jira/browse/BEAM-3798
>>>>>> - working well, rarely failing: AvroIO, TextIO, Compressed Text,
>>>>>> TFRecordIO
>>>>>>
>>>>>> Also, some test failures are caused due to pending PRs (we sometimes
>>>>>> run concrete tests to check if PRs won't break them). This also causes
>>>>>> failures sometimes.
>>>>>>
>>>>>> I can help with switching the Performance tests to Gradle as this
>>>>>> part seems to be free to take.
>>>>>>
>>>>>>
>>>>>> Łukasz
>>>>>>
>>>>>>
>>>>
>>>
Re: Gradle status
Posted by Romain Manni-Bucau <rm...@gmail.com>.
Hi Kenneth,
For now maven covers the full needs of beam. If we start to have this kind
of PR we become dependent of the 2 builds which is what this thread is
about avoiding so tempted to say it must be a PR drop completely maven or
nothing as mentionned before.
Le 9 mars 2018 04:48, "Kenneth Knowles" <kl...@google.com> a écrit :
> I would like to briefly re-focus this discussion and suggest that we merge
> https://github.com/apache/beam/pull/4814.
>
> The only material objection I've heard is that it means the precommit no
> longer tests exactly what is built for release. It is a valid concern, but
> we have mvn postcommit so the coverage is not lost. The benefits of quicker
> reviews and less Jenkins congestion seem worth it to me.
>
> Kenn
>
> On Thu, Mar 8, 2018 at 12:33 PM Romain Manni-Bucau <rm...@gmail.com>
> wrote:
>
>> @Luskasz: not sure Im the best to host it since I know more gradle
>> internals that user interface/ecosystem but happy to help. Will also
>> require a "sudo" merger for this day to merge fixes asap - guess we can
>> bypass reviews or have a fast cycle plan for this day to avoid it to be a
>> week?
>>
>> Le 8 mars 2018 21:08, "Kenneth Knowles" <kl...@google.com> a écrit :
>>
>> @Romain - Idea/IntelliJ is great with Gradle, way better than Maven, and
>> what I mean is that I have added enough hints that it works OOTB already.
>> The rest of my instructions are just how you should override IntelliJ's
>> defaults to have a proper dev env - mostly just about storing files outside
>> the git clone.
>>
>>
>> Hmm it is always super slow here and not as integrated as maven which is
>> smooth thanks to the combination of idea build and maven plugin config
>> read. Import is also faster cause it just reads meta and doesnt run
>> anything. Hope it is a "a few times" issues at the moment but not yet sure.
>>
>>
>>
>> @Łukasz - yea I just mean the ones that I find in my email and browsing
>> https://builds.apache.org/view/A-D/view/Beam/ which are the ones you
>> classify as "totally not working". I would love to disable these.
>>
>> On the subject of running things on a pending PR - we should not run
>> postcommit jobs on PRs. We should make separate (optional) precommit jobs
>> that run the same build commands. That will give a more clear history and
>> allow trivial configuration of which jobs deserve alert emails and which
>> are not a problem. This is easy but I've been waiting to do it after Gradle
>> migration.
>>
>> On Thu, Mar 8, 2018 at 11:37 AM Lukasz Cwik <lc...@google.com> wrote:
>>
>>> Romain, would you like to host/plan/run the Gradle fixit day?
>>>
>>> On Thu, Mar 8, 2018 at 11:24 AM, Chamikara Jayalath <
>>> chamikara@google.com> wrote:
>>>
>>>> +1 for the general idea of fixit day/week for Gradle.
>>>>
>>>> Agree with what Łukasz said. Some of these performance tests are new
>>>> and are flaky due to other issues that were discovered during the process
>>>> of adding the test.
>>>>
>>>> I think the high level blocker is updating performance testing
>>>> framework to use Gradle. This will involve adding Gradle-based logic for
>>>> invoking PerfKitBenchmaker, for example [1], and updating
>>>> PerfKitBenchmarker to issue a Gradle command for running the benchmark [2].
>>>> First task will be to find the work needed here and updating the relevant
>>>> JIRA [3].
>>>>
>>>> Thanks,
>>>> Cham
>>>>
>>>> [1] https://github.com/apache/beam/blob/master/sdks/
>>>> java/io/google-cloud-platform/pom.xml#L79
>>>> [2] https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/blob/
>>>> master/perfkitbenchmarker/beam_benchmark_helper.py
>>>> [3] https://issues.apache.org/jira/browse/BEAM-3251
>>>>
>>>> On Thu, Mar 8, 2018 at 3:10 AM Łukasz Gajowy <lu...@gmail.com>
>>>> wrote:
>>>>
>>>>> 2018-03-07 22:29 GMT+01:00 Kenneth Knowles <kl...@google.com>:
>>>>>
>>>>>>
>>>>>> Based on https://builds.apache.org/view/A-D/view/Beam/ and our
>>>>>> failure spam level the performance tests are mostly not healthy anyhow. So
>>>>>> is there any high level blocker to switching them or is it just someone
>>>>>> sitting down with each one?
>>>>>>
>>>>>>
>>>>> I thought I could share my point of view here as I am working on the
>>>>> Performance Test part for a while now. I wouldn't say those are "mostly not
>>>>> healthy". The situation is as follows:
>>>>>
>>>>> - totally not working: Python, Spark performance tests (don't know
>>>>> why, I'm not maintaining those tests. Should we disable them?)
>>>>> - flaky: the recently re-enabled JDBC Performance Test. It's seems to
>>>>> be flaky mostly due to: https://issues.apache.org/
>>>>> jira/browse/BEAM-3798
>>>>> - working well, rarely failing: AvroIO, TextIO, Compressed Text,
>>>>> TFRecordIO
>>>>>
>>>>> Also, some test failures are caused due to pending PRs (we sometimes
>>>>> run concrete tests to check if PRs won't break them). This also causes
>>>>> failures sometimes.
>>>>>
>>>>> I can help with switching the Performance tests to Gradle as this part
>>>>> seems to be free to take.
>>>>>
>>>>>
>>>>> Łukasz
>>>>>
>>>>>
>>>
>>
Re: Gradle status
Posted by Kenneth Knowles <kl...@google.com>.
I would like to briefly re-focus this discussion and suggest that we merge
https://github.com/apache/beam/pull/4814.
The only material objection I've heard is that it means the precommit no
longer tests exactly what is built for release. It is a valid concern, but
we have mvn postcommit so the coverage is not lost. The benefits of quicker
reviews and less Jenkins congestion seem worth it to me.
Kenn
On Thu, Mar 8, 2018 at 12:33 PM Romain Manni-Bucau <rm...@gmail.com>
wrote:
> @Luskasz: not sure Im the best to host it since I know more gradle
> internals that user interface/ecosystem but happy to help. Will also
> require a "sudo" merger for this day to merge fixes asap - guess we can
> bypass reviews or have a fast cycle plan for this day to avoid it to be a
> week?
>
> Le 8 mars 2018 21:08, "Kenneth Knowles" <kl...@google.com> a écrit :
>
> @Romain - Idea/IntelliJ is great with Gradle, way better than Maven, and
> what I mean is that I have added enough hints that it works OOTB already.
> The rest of my instructions are just how you should override IntelliJ's
> defaults to have a proper dev env - mostly just about storing files outside
> the git clone.
>
>
> Hmm it is always super slow here and not as integrated as maven which is
> smooth thanks to the combination of idea build and maven plugin config
> read. Import is also faster cause it just reads meta and doesnt run
> anything. Hope it is a "a few times" issues at the moment but not yet sure.
>
>
>
> @Łukasz - yea I just mean the ones that I find in my email and browsing
> https://builds.apache.org/view/A-D/view/Beam/ which are the ones you
> classify as "totally not working". I would love to disable these.
>
> On the subject of running things on a pending PR - we should not run
> postcommit jobs on PRs. We should make separate (optional) precommit jobs
> that run the same build commands. That will give a more clear history and
> allow trivial configuration of which jobs deserve alert emails and which
> are not a problem. This is easy but I've been waiting to do it after Gradle
> migration.
>
> On Thu, Mar 8, 2018 at 11:37 AM Lukasz Cwik <lc...@google.com> wrote:
>
>> Romain, would you like to host/plan/run the Gradle fixit day?
>>
>> On Thu, Mar 8, 2018 at 11:24 AM, Chamikara Jayalath <chamikara@google.com
>> > wrote:
>>
>>> +1 for the general idea of fixit day/week for Gradle.
>>>
>>> Agree with what Łukasz said. Some of these performance tests are new and
>>> are flaky due to other issues that were discovered during the process of
>>> adding the test.
>>>
>>> I think the high level blocker is updating performance testing framework
>>> to use Gradle. This will involve adding Gradle-based logic for invoking
>>> PerfKitBenchmaker, for example [1], and updating PerfKitBenchmarker to
>>> issue a Gradle command for running the benchmark [2]. First task will be to
>>> find the work needed here and updating the relevant JIRA [3].
>>>
>>> Thanks,
>>> Cham
>>>
>>> [1]
>>> https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/pom.xml#L79
>>> [2]
>>> https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/blob/master/perfkitbenchmarker/beam_benchmark_helper.py
>>> [3] https://issues.apache.org/jira/browse/BEAM-3251
>>>
>>> On Thu, Mar 8, 2018 at 3:10 AM Łukasz Gajowy <lu...@gmail.com>
>>> wrote:
>>>
>>>> 2018-03-07 22:29 GMT+01:00 Kenneth Knowles <kl...@google.com>:
>>>>
>>>>>
>>>>> Based on https://builds.apache.org/view/A-D/view/Beam/ and our
>>>>> failure spam level the performance tests are mostly not healthy anyhow. So
>>>>> is there any high level blocker to switching them or is it just someone
>>>>> sitting down with each one?
>>>>>
>>>>>
>>>> I thought I could share my point of view here as I am working on the
>>>> Performance Test part for a while now. I wouldn't say those are "mostly not
>>>> healthy". The situation is as follows:
>>>>
>>>> - totally not working: Python, Spark performance tests (don't know why,
>>>> I'm not maintaining those tests. Should we disable them?)
>>>> - flaky: the recently re-enabled JDBC Performance Test. It's seems to
>>>> be flaky mostly due to: https://issues.apache.org/jira/browse/BEAM-3798
>>>> - working well, rarely failing: AvroIO, TextIO, Compressed Text,
>>>> TFRecordIO
>>>>
>>>> Also, some test failures are caused due to pending PRs (we sometimes
>>>> run concrete tests to check if PRs won't break them). This also causes
>>>> failures sometimes.
>>>>
>>>> I can help with switching the Performance tests to Gradle as this part
>>>> seems to be free to take.
>>>>
>>>>
>>>> Łukasz
>>>>
>>>>
>>
>
Re: Gradle status
Posted by Romain Manni-Bucau <rm...@gmail.com>.
@Luskasz: not sure Im the best to host it since I know more gradle
internals that user interface/ecosystem but happy to help. Will also
require a "sudo" merger for this day to merge fixes asap - guess we can
bypass reviews or have a fast cycle plan for this day to avoid it to be a
week?
Le 8 mars 2018 21:08, "Kenneth Knowles" <kl...@google.com> a écrit :
@Romain - Idea/IntelliJ is great with Gradle, way better than Maven, and
what I mean is that I have added enough hints that it works OOTB already.
The rest of my instructions are just how you should override IntelliJ's
defaults to have a proper dev env - mostly just about storing files outside
the git clone.
Hmm it is always super slow here and not as integrated as maven which is
smooth thanks to the combination of idea build and maven plugin config
read. Import is also faster cause it just reads meta and doesnt run
anything. Hope it is a "a few times" issues at the moment but not yet sure.
@Łukasz - yea I just mean the ones that I find in my email and browsing
https://builds.apache.org/view/A-D/view/Beam/ which are the ones you
classify as "totally not working". I would love to disable these.
On the subject of running things on a pending PR - we should not run
postcommit jobs on PRs. We should make separate (optional) precommit jobs
that run the same build commands. That will give a more clear history and
allow trivial configuration of which jobs deserve alert emails and which
are not a problem. This is easy but I've been waiting to do it after Gradle
migration.
On Thu, Mar 8, 2018 at 11:37 AM Lukasz Cwik <lc...@google.com> wrote:
> Romain, would you like to host/plan/run the Gradle fixit day?
>
> On Thu, Mar 8, 2018 at 11:24 AM, Chamikara Jayalath <ch...@google.com>
> wrote:
>
>> +1 for the general idea of fixit day/week for Gradle.
>>
>> Agree with what Łukasz said. Some of these performance tests are new and
>> are flaky due to other issues that were discovered during the process of
>> adding the test.
>>
>> I think the high level blocker is updating performance testing framework
>> to use Gradle. This will involve adding Gradle-based logic for invoking
>> PerfKitBenchmaker, for example [1], and updating PerfKitBenchmarker to
>> issue a Gradle command for running the benchmark [2]. First task will be to
>> find the work needed here and updating the relevant JIRA [3].
>>
>> Thanks,
>> Cham
>>
>> [1] https://github.com/apache/beam/blob/master/sdks/
>> java/io/google-cloud-platform/pom.xml#L79
>> [2] https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/blob/
>> master/perfkitbenchmarker/beam_benchmark_helper.py
>> [3] https://issues.apache.org/jira/browse/BEAM-3251
>>
>> On Thu, Mar 8, 2018 at 3:10 AM Łukasz Gajowy <lu...@gmail.com>
>> wrote:
>>
>>> 2018-03-07 22:29 GMT+01:00 Kenneth Knowles <kl...@google.com>:
>>>
>>>>
>>>> Based on https://builds.apache.org/view/A-D/view/Beam/ and our failure
>>>> spam level the performance tests are mostly not healthy anyhow. So is there
>>>> any high level blocker to switching them or is it just someone sitting down
>>>> with each one?
>>>>
>>>>
>>> I thought I could share my point of view here as I am working on the
>>> Performance Test part for a while now. I wouldn't say those are "mostly not
>>> healthy". The situation is as follows:
>>>
>>> - totally not working: Python, Spark performance tests (don't know why,
>>> I'm not maintaining those tests. Should we disable them?)
>>> - flaky: the recently re-enabled JDBC Performance Test. It's seems to be
>>> flaky mostly due to: https://issues.apache.org/jira/browse/BEAM-3798
>>> - working well, rarely failing: AvroIO, TextIO, Compressed Text,
>>> TFRecordIO
>>>
>>> Also, some test failures are caused due to pending PRs (we sometimes run
>>> concrete tests to check if PRs won't break them). This also causes failures
>>> sometimes.
>>>
>>> I can help with switching the Performance tests to Gradle as this part
>>> seems to be free to take.
>>>
>>>
>>> Łukasz
>>>
>>>
>
Re: Gradle status
Posted by Kenneth Knowles <kl...@google.com>.
@Romain - Idea/IntelliJ is great with Gradle, way better than Maven, and
what I mean is that I have added enough hints that it works OOTB already.
The rest of my instructions are just how you should override IntelliJ's
defaults to have a proper dev env - mostly just about storing files outside
the git clone.
@Łukasz - yea I just mean the ones that I find in my email and browsing
https://builds.apache.org/view/A-D/view/Beam/ which are the ones you
classify as "totally not working". I would love to disable these.
On the subject of running things on a pending PR - we should not run
postcommit jobs on PRs. We should make separate (optional) precommit jobs
that run the same build commands. That will give a more clear history and
allow trivial configuration of which jobs deserve alert emails and which
are not a problem. This is easy but I've been waiting to do it after Gradle
migration.
On Thu, Mar 8, 2018 at 11:37 AM Lukasz Cwik <lc...@google.com> wrote:
> Romain, would you like to host/plan/run the Gradle fixit day?
>
> On Thu, Mar 8, 2018 at 11:24 AM, Chamikara Jayalath <ch...@google.com>
> wrote:
>
>> +1 for the general idea of fixit day/week for Gradle.
>>
>> Agree with what Łukasz said. Some of these performance tests are new and
>> are flaky due to other issues that were discovered during the process of
>> adding the test.
>>
>> I think the high level blocker is updating performance testing framework
>> to use Gradle. This will involve adding Gradle-based logic for invoking
>> PerfKitBenchmaker, for example [1], and updating PerfKitBenchmarker to
>> issue a Gradle command for running the benchmark [2]. First task will be to
>> find the work needed here and updating the relevant JIRA [3].
>>
>> Thanks,
>> Cham
>>
>> [1]
>> https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/pom.xml#L79
>> [2]
>> https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/blob/master/perfkitbenchmarker/beam_benchmark_helper.py
>> [3] https://issues.apache.org/jira/browse/BEAM-3251
>>
>> On Thu, Mar 8, 2018 at 3:10 AM Łukasz Gajowy <lu...@gmail.com>
>> wrote:
>>
>>> 2018-03-07 22:29 GMT+01:00 Kenneth Knowles <kl...@google.com>:
>>>
>>>>
>>>> Based on https://builds.apache.org/view/A-D/view/Beam/ and our failure
>>>> spam level the performance tests are mostly not healthy anyhow. So is there
>>>> any high level blocker to switching them or is it just someone sitting down
>>>> with each one?
>>>>
>>>>
>>> I thought I could share my point of view here as I am working on the
>>> Performance Test part for a while now. I wouldn't say those are "mostly not
>>> healthy". The situation is as follows:
>>>
>>> - totally not working: Python, Spark performance tests (don't know why,
>>> I'm not maintaining those tests. Should we disable them?)
>>> - flaky: the recently re-enabled JDBC Performance Test. It's seems to be
>>> flaky mostly due to: https://issues.apache.org/jira/browse/BEAM-3798
>>> - working well, rarely failing: AvroIO, TextIO, Compressed Text,
>>> TFRecordIO
>>>
>>> Also, some test failures are caused due to pending PRs (we sometimes run
>>> concrete tests to check if PRs won't break them). This also causes failures
>>> sometimes.
>>>
>>> I can help with switching the Performance tests to Gradle as this part
>>> seems to be free to take.
>>>
>>>
>>> Łukasz
>>>
>>>
>
Re: Gradle status
Posted by Lukasz Cwik <lc...@google.com>.
Romain, would you like to host/plan/run the Gradle fixit day?
On Thu, Mar 8, 2018 at 11:24 AM, Chamikara Jayalath <ch...@google.com>
wrote:
> +1 for the general idea of fixit day/week for Gradle.
>
> Agree with what Łukasz said. Some of these performance tests are new and
> are flaky due to other issues that were discovered during the process of
> adding the test.
>
> I think the high level blocker is updating performance testing framework
> to use Gradle. This will involve adding Gradle-based logic for invoking
> PerfKitBenchmaker, for example [1], and updating PerfKitBenchmarker to
> issue a Gradle command for running the benchmark [2]. First task will be to
> find the work needed here and updating the relevant JIRA [3].
>
> Thanks,
> Cham
>
> [1] https://github.com/apache/beam/blob/master/sdks/
> java/io/google-cloud-platform/pom.xml#L79
> [2] https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/blob/
> master/perfkitbenchmarker/beam_benchmark_helper.py
> [3] https://issues.apache.org/jira/browse/BEAM-3251
>
> On Thu, Mar 8, 2018 at 3:10 AM Łukasz Gajowy <lu...@gmail.com>
> wrote:
>
>> 2018-03-07 22:29 GMT+01:00 Kenneth Knowles <kl...@google.com>:
>>
>>>
>>> Based on https://builds.apache.org/view/A-D/view/Beam/ and our failure
>>> spam level the performance tests are mostly not healthy anyhow. So is there
>>> any high level blocker to switching them or is it just someone sitting down
>>> with each one?
>>>
>>>
>> I thought I could share my point of view here as I am working on the
>> Performance Test part for a while now. I wouldn't say those are "mostly not
>> healthy". The situation is as follows:
>>
>> - totally not working: Python, Spark performance tests (don't know why,
>> I'm not maintaining those tests. Should we disable them?)
>> - flaky: the recently re-enabled JDBC Performance Test. It's seems to be
>> flaky mostly due to: https://issues.apache.org/jira/browse/BEAM-3798
>> - working well, rarely failing: AvroIO, TextIO, Compressed Text,
>> TFRecordIO
>>
>> Also, some test failures are caused due to pending PRs (we sometimes run
>> concrete tests to check if PRs won't break them). This also causes failures
>> sometimes.
>>
>> I can help with switching the Performance tests to Gradle as this part
>> seems to be free to take.
>>
>>
>> Łukasz
>>
>>
Re: Gradle status
Posted by Chamikara Jayalath <ch...@google.com>.
+1 for the general idea of fixit day/week for Gradle.
Agree with what Łukasz said. Some of these performance tests are new and
are flaky due to other issues that were discovered during the process of
adding the test.
I think the high level blocker is updating performance testing framework to
use Gradle. This will involve adding Gradle-based logic for invoking
PerfKitBenchmaker, for example [1], and updating PerfKitBenchmarker to
issue a Gradle command for running the benchmark [2]. First task will be to
find the work needed here and updating the relevant JIRA [3].
Thanks,
Cham
[1]
https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/pom.xml#L79
[2]
https://github.com/GoogleCloudPlatform/PerfKitBenchmarker/blob/master/perfkitbenchmarker/beam_benchmark_helper.py
[3] https://issues.apache.org/jira/browse/BEAM-3251
On Thu, Mar 8, 2018 at 3:10 AM Łukasz Gajowy <lu...@gmail.com>
wrote:
> 2018-03-07 22:29 GMT+01:00 Kenneth Knowles <kl...@google.com>:
>
>>
>> Based on https://builds.apache.org/view/A-D/view/Beam/ and our failure
>> spam level the performance tests are mostly not healthy anyhow. So is there
>> any high level blocker to switching them or is it just someone sitting down
>> with each one?
>>
>>
> I thought I could share my point of view here as I am working on the
> Performance Test part for a while now. I wouldn't say those are "mostly not
> healthy". The situation is as follows:
>
> - totally not working: Python, Spark performance tests (don't know why,
> I'm not maintaining those tests. Should we disable them?)
> - flaky: the recently re-enabled JDBC Performance Test. It's seems to be
> flaky mostly due to: https://issues.apache.org/jira/browse/BEAM-3798
> - working well, rarely failing: AvroIO, TextIO, Compressed Text, TFRecordIO
>
> Also, some test failures are caused due to pending PRs (we sometimes run
> concrete tests to check if PRs won't break them). This also causes failures
> sometimes.
>
> I can help with switching the Performance tests to Gradle as this part
> seems to be free to take.
>
>
> Łukasz
>
>
Re: Gradle status
Posted by Łukasz Gajowy <lu...@gmail.com>.
2018-03-07 22:29 GMT+01:00 Kenneth Knowles <kl...@google.com>:
>
> Based on https://builds.apache.org/view/A-D/view/Beam/ and our failure
> spam level the performance tests are mostly not healthy anyhow. So is there
> any high level blocker to switching them or is it just someone sitting down
> with each one?
>
>
I thought I could share my point of view here as I am working on the
Performance Test part for a while now. I wouldn't say those are "mostly not
healthy". The situation is as follows:
- totally not working: Python, Spark performance tests (don't know why, I'm
not maintaining those tests. Should we disable them?)
- flaky: the recently re-enabled JDBC Performance Test. It's seems to be
flaky mostly due to: https://issues.apache.org/jira/browse/BEAM-3798
- working well, rarely failing: AvroIO, TextIO, Compressed Text, TFRecordIO
Also, some test failures are caused due to pending PRs (we sometimes run
concrete tests to check if PRs won't break them). This also causes failures
sometimes.
I can help with switching the Performance tests to Gradle as this part
seems to be free to take.
Łukasz
Re: Gradle status
Posted by Kenneth Knowles <kl...@google.com>.
SGTM. I should also say that every day is Gradle fixit day for me, as I
have been using only Gradle (with IntelliJ) for a while :-). If anyone is
hesitant, definitely it is ready to be used for normal dev.
Seems like changing the messaging in onboarding docs is the main thing to
fixit.
Based on https://builds.apache.org/view/A-D/view/Beam/ and our failure spam
level the performance tests are mostly not healthy anyhow. So is there any
high level blocker to switching them or is it just someone sitting down
with each one?
Kenn
On Wed, Mar 7, 2018 at 1:22 PM Lukasz Cwik <lc...@google.com> wrote:
> Largest outstanding areas are:
> * Documentation relevant to the contributors guide/release process/testing
> * Performance tests
>
> There has been good progress towards:
> * Release artifact validations and generation
> * ValidatesRunner post commits
> * Pre commits
> * Container builds
>
>
> On Wed, Mar 7, 2018 at 1:19 PM, Reuven Lax <re...@google.com> wrote:
>
>> I think Alan was making progress on the Gradle build.
>>
>> What do people think of a "fixit" day for Gradle work? (or given that
>> people are distributed, maybe a fixit week, where everyone takes one day
>> from the week).
>>
>>
>> On Wed, Mar 7, 2018 at 1:17 PM Kenneth Knowles <kl...@google.com> wrote:
>>
>>> I also cannot drop everything to work on Gradle build, but maybe it
>>> isn't that drastic anyhow. Now that we have ValidatesRunner and NeedsRunner
>>> tests and some progress on the release, is there any other known missing
>>> functionality in the Gradle builds? Archetypes? Docker container images?
>>>
>>>
>>> On Wed, Mar 7, 2018 at 1:12 PM Lukasz Cwik <lc...@google.com> wrote:
>>>
>>>> I am working on various projects and may not be able to pause my work
>>>> for a couple of weeks while the build/test process is migrated.
>>>>
>>>> What is everyone thinking about Romain's suggestion because If I'm the
>>>> only person in such a situation, I would be willing to go along with the
>>>> plan.
>>>>
>>>> On Wed, Mar 7, 2018 at 12:59 PM, Romain Manni-Bucau <
>>>> rmannibucau@gmail.com> wrote:
>>>>
>>>>>
>>>>>
>>>>> Le 7 mars 2018 20:21, "Lukasz Cwik" <lc...@google.com> a écrit :
>>>>>
>>>>> Note that Alan Myrvold has been making steady progress making the
>>>>> release process via Gradle a reality:
>>>>> 1) Creating a jenkins job which can run the quickstart validation
>>>>> against nightly snapshots and also can be used for release candidates (
>>>>> https://github.com/apache/beam/pull/4252)
>>>>> 2) Building a release candidate using Gradle (
>>>>> https://github.com/apache/beam/pull/4812)
>>>>>
>>>>> Also, Gradle is the tool that has been selected already and there was
>>>>> a community discussion about what was needed for the migration to occur
>>>>> with a clear set of criteria. Romain, it doesn't seem like we should ignore
>>>>> that criteria or are you suggesting we change that criteria (if yes, how)?
>>>>>
>>>>>
>>>>> No, no. My goal is just to quit this state.
>>>>>
>>>>> Let s draft a plan:
>>>>>
>>>>> 1. 2.4 is released - i assume it is done with mvn here
>>>>> 2. We drop all poms and jenkins mvn config
>>>>> 3. We fix all build issues if so (let say in a week)
>>>>> 4. Pr can nees updates but no more mvn merge
>>>>>
>>>>> April is gradle month :)
>>>>>
>>>>> Wdyt?
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Mar 7, 2018 at 10:39 AM, Romain Manni-Bucau <
>>>>> rmannibucau@gmail.com> wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> Le 7 mars 2018 17:34, "Lukasz Cwik" <lc...@google.com> a écrit :
>>>>>>
>>>>>> Thanks for bringing this up Romain but I believe your data points on
>>>>>> pass rates are only partially correct.
>>>>>>
>>>>>>
>>>>>> Sure sure, it is mainly about my own PR which a very small % of the
>>>>>> whole project ;).
>>>>>>
>>>>>>
>>>>>>
>>>>>> In the past week the Java Gradle precommit passed 46.34% of the time
>>>>>> compared to the Java Maven precommit which passed 46.15% of the time. When
>>>>>> I looked at these numbers in mid January they were around 37% so there has
>>>>>> been some improvement. Regardless of the build tool it seems that our pass
>>>>>> rates aren't stellar for the Java build and are causing the community to
>>>>>> not follow best practices (wait for precommits to be green before merging).
>>>>>> I know that on the website we used the mergebot to ensure that things
>>>>>> passed before they were merged, should we institute this on the master
>>>>>> branch or are their any other ideas?
>>>>>>
>>>>>> As a side note we had achieved the goals we set out to not need to
>>>>>> maintain the Maven precommit and have authored the first PR to drop the
>>>>>> Maven precommit: https://github.com/apache/beam/pull/4814
>>>>>>
>>>>>>
>>>>>> Well, I'd be for a strong switch otherwise PR will keep using maven,
>>>>>> jenkins will not test the code and at the end we fail to deliver something
>>>>>> consistent. So whatever tool is selected I'm tempted to say drop other
>>>>>> build files and jenkins hooks references.
>>>>>>
>>>>>> What about doing it after 2.4 vote?
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Mar 7, 2018 at 2:24 AM, Romain Manni-Bucau <
>>>>>> rmannibucau@gmail.com> wrote:
>>>>>>
>>>>>>> Up,
>>>>>>>
>>>>>>> We discussed to have a strong switch to gradle or rollback to maven
>>>>>>> around april to not be blocked by the build tool. I noticed gradle build
>>>>>>> rarely passes on PR and kind of blurry our vision - not sure why exactly.
>>>>>>> Also, PR don't always contain the gradle updates - generally
>>>>>>> dependencies+plugins are added in pom.xml AFAIK, so it seems the adoption
>>>>>>> is very slow to not say rejected.
>>>>>>>
>>>>>>> What do we do about that? When do we drop the double build
>>>>>>> maintenance - whatever is picked?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Romain Manni-Bucau
>>>>>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
>>>>>>> <https://rmannibucau.metawerx.net/> | Old Blog
>>>>>>> <http://rmannibucau.wordpress.com> | Github
>>>>>>> <https://github.com/rmannibucau> | LinkedIn
>>>>>>> <https://www.linkedin.com/in/rmannibucau> | Book
>>>>>>> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>>>>>>>
>>>>>>> 2018-01-12 6:30 GMT+01:00 Romain Manni-Bucau <rm...@gmail.com>
>>>>>>> :
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Le 11 janv. 2018 23:13, "Kenneth Knowles" <kl...@google.com> a
>>>>>>>> écrit :
>>>>>>>>
>>>>>>>> On Thu, Jan 11, 2018 at 8:43 AM, Romain Manni-Bucau <
>>>>>>>> rmannibucau@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> 2. gradle build doesn't use the same output directory than maven
>>>>>>>>> so it is not really smooth to have both and have to maintain both
>>>>>>>>>
>>>>>>>>
>>>>>>>> I also have an opinion on this. It is useful and reasonable to be
>>>>>>>> able to build even when the source is on a read-only filesystem. Maven's
>>>>>>>> defaults are undesirable and require workarounds. We shouldn't mimic the
>>>>>>>> behavior, but actually should set gradle up to build to a directory outside
>>>>>>>> the source tree always, if we can.
>>>>>>>>
>>>>>>>>
>>>>>>>> Hmm, which is something you can do with maven as well so not sure I
>>>>>>>> get it.
>>>>>>>>
>>>>>>>> Also note the thread is no more about the technical points but more
>>>>>>>> the sources maintenance and consistency.
>>>>>>>>
>>>>>>>>
>>>>>>>> Kenn
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>
Re: Gradle status
Posted by Lukasz Cwik <lc...@google.com>.
Largest outstanding areas are:
* Documentation relevant to the contributors guide/release process/testing
* Performance tests
There has been good progress towards:
* Release artifact validations and generation
* ValidatesRunner post commits
* Pre commits
* Container builds
On Wed, Mar 7, 2018 at 1:19 PM, Reuven Lax <re...@google.com> wrote:
> I think Alan was making progress on the Gradle build.
>
> What do people think of a "fixit" day for Gradle work? (or given that
> people are distributed, maybe a fixit week, where everyone takes one day
> from the week).
>
>
> On Wed, Mar 7, 2018 at 1:17 PM Kenneth Knowles <kl...@google.com> wrote:
>
>> I also cannot drop everything to work on Gradle build, but maybe it isn't
>> that drastic anyhow. Now that we have ValidatesRunner and NeedsRunner tests
>> and some progress on the release, is there any other known missing
>> functionality in the Gradle builds? Archetypes? Docker container images?
>>
>>
>> On Wed, Mar 7, 2018 at 1:12 PM Lukasz Cwik <lc...@google.com> wrote:
>>
>>> I am working on various projects and may not be able to pause my work
>>> for a couple of weeks while the build/test process is migrated.
>>>
>>> What is everyone thinking about Romain's suggestion because If I'm the
>>> only person in such a situation, I would be willing to go along with the
>>> plan.
>>>
>>> On Wed, Mar 7, 2018 at 12:59 PM, Romain Manni-Bucau <
>>> rmannibucau@gmail.com> wrote:
>>>
>>>>
>>>>
>>>> Le 7 mars 2018 20:21, "Lukasz Cwik" <lc...@google.com> a écrit :
>>>>
>>>> Note that Alan Myrvold has been making steady progress making the
>>>> release process via Gradle a reality:
>>>> 1) Creating a jenkins job which can run the quickstart validation
>>>> against nightly snapshots and also can be used for release candidates (
>>>> https://github.com/apache/beam/pull/4252)
>>>> 2) Building a release candidate using Gradle (
>>>> https://github.com/apache/beam/pull/4812)
>>>>
>>>> Also, Gradle is the tool that has been selected already and there was a
>>>> community discussion about what was needed for the migration to occur with
>>>> a clear set of criteria. Romain, it doesn't seem like we should ignore that
>>>> criteria or are you suggesting we change that criteria (if yes, how)?
>>>>
>>>>
>>>> No, no. My goal is just to quit this state.
>>>>
>>>> Let s draft a plan:
>>>>
>>>> 1. 2.4 is released - i assume it is done with mvn here
>>>> 2. We drop all poms and jenkins mvn config
>>>> 3. We fix all build issues if so (let say in a week)
>>>> 4. Pr can nees updates but no more mvn merge
>>>>
>>>> April is gradle month :)
>>>>
>>>> Wdyt?
>>>>
>>>>
>>>>
>>>>
>>>> On Wed, Mar 7, 2018 at 10:39 AM, Romain Manni-Bucau <
>>>> rmannibucau@gmail.com> wrote:
>>>>
>>>>>
>>>>>
>>>>> Le 7 mars 2018 17:34, "Lukasz Cwik" <lc...@google.com> a écrit :
>>>>>
>>>>> Thanks for bringing this up Romain but I believe your data points on
>>>>> pass rates are only partially correct.
>>>>>
>>>>>
>>>>> Sure sure, it is mainly about my own PR which a very small % of the
>>>>> whole project ;).
>>>>>
>>>>>
>>>>>
>>>>> In the past week the Java Gradle precommit passed 46.34% of the time
>>>>> compared to the Java Maven precommit which passed 46.15% of the time. When
>>>>> I looked at these numbers in mid January they were around 37% so there has
>>>>> been some improvement. Regardless of the build tool it seems that our pass
>>>>> rates aren't stellar for the Java build and are causing the community to
>>>>> not follow best practices (wait for precommits to be green before merging).
>>>>> I know that on the website we used the mergebot to ensure that things
>>>>> passed before they were merged, should we institute this on the master
>>>>> branch or are their any other ideas?
>>>>>
>>>>> As a side note we had achieved the goals we set out to not need to
>>>>> maintain the Maven precommit and have authored the first PR to drop the
>>>>> Maven precommit: https://github.com/apache/beam/pull/4814
>>>>>
>>>>>
>>>>> Well, I'd be for a strong switch otherwise PR will keep using maven,
>>>>> jenkins will not test the code and at the end we fail to deliver something
>>>>> consistent. So whatever tool is selected I'm tempted to say drop other
>>>>> build files and jenkins hooks references.
>>>>>
>>>>> What about doing it after 2.4 vote?
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Mar 7, 2018 at 2:24 AM, Romain Manni-Bucau <
>>>>> rmannibucau@gmail.com> wrote:
>>>>>
>>>>>> Up,
>>>>>>
>>>>>> We discussed to have a strong switch to gradle or rollback to maven
>>>>>> around april to not be blocked by the build tool. I noticed gradle build
>>>>>> rarely passes on PR and kind of blurry our vision - not sure why exactly.
>>>>>> Also, PR don't always contain the gradle updates - generally
>>>>>> dependencies+plugins are added in pom.xml AFAIK, so it seems the adoption
>>>>>> is very slow to not say rejected.
>>>>>>
>>>>>> What do we do about that? When do we drop the double build
>>>>>> maintenance - whatever is picked?
>>>>>>
>>>>>>
>>>>>>
>>>>>> Romain Manni-Bucau
>>>>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
>>>>>> <https://rmannibucau.metawerx.net/> | Old Blog
>>>>>> <http://rmannibucau.wordpress.com> | Github
>>>>>> <https://github.com/rmannibucau> | LinkedIn
>>>>>> <https://www.linkedin.com/in/rmannibucau> | Book
>>>>>> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>>>>>>
>>>>>> 2018-01-12 6:30 GMT+01:00 Romain Manni-Bucau <rm...@gmail.com>:
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Le 11 janv. 2018 23:13, "Kenneth Knowles" <kl...@google.com> a écrit :
>>>>>>>
>>>>>>> On Thu, Jan 11, 2018 at 8:43 AM, Romain Manni-Bucau <
>>>>>>> rmannibucau@gmail.com> wrote:
>>>>>>>
>>>>>>>> 2. gradle build doesn't use the same output directory than maven so
>>>>>>>> it is not really smooth to have both and have to maintain both
>>>>>>>>
>>>>>>>
>>>>>>> I also have an opinion on this. It is useful and reasonable to be
>>>>>>> able to build even when the source is on a read-only filesystem. Maven's
>>>>>>> defaults are undesirable and require workarounds. We shouldn't mimic the
>>>>>>> behavior, but actually should set gradle up to build to a directory outside
>>>>>>> the source tree always, if we can.
>>>>>>>
>>>>>>>
>>>>>>> Hmm, which is something you can do with maven as well so not sure I
>>>>>>> get it.
>>>>>>>
>>>>>>> Also note the thread is no more about the technical points but more
>>>>>>> the sources maintenance and consistency.
>>>>>>>
>>>>>>>
>>>>>>> Kenn
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
Re: Gradle status
Posted by Reuven Lax <re...@google.com>.
I think Alan was making progress on the Gradle build.
What do people think of a "fixit" day for Gradle work? (or given that
people are distributed, maybe a fixit week, where everyone takes one day
from the week).
On Wed, Mar 7, 2018 at 1:17 PM Kenneth Knowles <kl...@google.com> wrote:
> I also cannot drop everything to work on Gradle build, but maybe it isn't
> that drastic anyhow. Now that we have ValidatesRunner and NeedsRunner tests
> and some progress on the release, is there any other known missing
> functionality in the Gradle builds? Archetypes? Docker container images?
>
>
> On Wed, Mar 7, 2018 at 1:12 PM Lukasz Cwik <lc...@google.com> wrote:
>
>> I am working on various projects and may not be able to pause my work for
>> a couple of weeks while the build/test process is migrated.
>>
>> What is everyone thinking about Romain's suggestion because If I'm the
>> only person in such a situation, I would be willing to go along with the
>> plan.
>>
>> On Wed, Mar 7, 2018 at 12:59 PM, Romain Manni-Bucau <
>> rmannibucau@gmail.com> wrote:
>>
>>>
>>>
>>> Le 7 mars 2018 20:21, "Lukasz Cwik" <lc...@google.com> a écrit :
>>>
>>> Note that Alan Myrvold has been making steady progress making the
>>> release process via Gradle a reality:
>>> 1) Creating a jenkins job which can run the quickstart validation
>>> against nightly snapshots and also can be used for release candidates (
>>> https://github.com/apache/beam/pull/4252)
>>> 2) Building a release candidate using Gradle (
>>> https://github.com/apache/beam/pull/4812)
>>>
>>> Also, Gradle is the tool that has been selected already and there was a
>>> community discussion about what was needed for the migration to occur with
>>> a clear set of criteria. Romain, it doesn't seem like we should ignore that
>>> criteria or are you suggesting we change that criteria (if yes, how)?
>>>
>>>
>>> No, no. My goal is just to quit this state.
>>>
>>> Let s draft a plan:
>>>
>>> 1. 2.4 is released - i assume it is done with mvn here
>>> 2. We drop all poms and jenkins mvn config
>>> 3. We fix all build issues if so (let say in a week)
>>> 4. Pr can nees updates but no more mvn merge
>>>
>>> April is gradle month :)
>>>
>>> Wdyt?
>>>
>>>
>>>
>>>
>>> On Wed, Mar 7, 2018 at 10:39 AM, Romain Manni-Bucau <
>>> rmannibucau@gmail.com> wrote:
>>>
>>>>
>>>>
>>>> Le 7 mars 2018 17:34, "Lukasz Cwik" <lc...@google.com> a écrit :
>>>>
>>>> Thanks for bringing this up Romain but I believe your data points on
>>>> pass rates are only partially correct.
>>>>
>>>>
>>>> Sure sure, it is mainly about my own PR which a very small % of the
>>>> whole project ;).
>>>>
>>>>
>>>>
>>>> In the past week the Java Gradle precommit passed 46.34% of the time
>>>> compared to the Java Maven precommit which passed 46.15% of the time. When
>>>> I looked at these numbers in mid January they were around 37% so there has
>>>> been some improvement. Regardless of the build tool it seems that our pass
>>>> rates aren't stellar for the Java build and are causing the community to
>>>> not follow best practices (wait for precommits to be green before merging).
>>>> I know that on the website we used the mergebot to ensure that things
>>>> passed before they were merged, should we institute this on the master
>>>> branch or are their any other ideas?
>>>>
>>>> As a side note we had achieved the goals we set out to not need to
>>>> maintain the Maven precommit and have authored the first PR to drop the
>>>> Maven precommit: https://github.com/apache/beam/pull/4814
>>>>
>>>>
>>>> Well, I'd be for a strong switch otherwise PR will keep using maven,
>>>> jenkins will not test the code and at the end we fail to deliver something
>>>> consistent. So whatever tool is selected I'm tempted to say drop other
>>>> build files and jenkins hooks references.
>>>>
>>>> What about doing it after 2.4 vote?
>>>>
>>>>
>>>>
>>>> On Wed, Mar 7, 2018 at 2:24 AM, Romain Manni-Bucau <
>>>> rmannibucau@gmail.com> wrote:
>>>>
>>>>> Up,
>>>>>
>>>>> We discussed to have a strong switch to gradle or rollback to maven
>>>>> around april to not be blocked by the build tool. I noticed gradle build
>>>>> rarely passes on PR and kind of blurry our vision - not sure why exactly.
>>>>> Also, PR don't always contain the gradle updates - generally
>>>>> dependencies+plugins are added in pom.xml AFAIK, so it seems the adoption
>>>>> is very slow to not say rejected.
>>>>>
>>>>> What do we do about that? When do we drop the double build maintenance
>>>>> - whatever is picked?
>>>>>
>>>>>
>>>>>
>>>>> Romain Manni-Bucau
>>>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
>>>>> <https://rmannibucau.metawerx.net/> | Old Blog
>>>>> <http://rmannibucau.wordpress.com> | Github
>>>>> <https://github.com/rmannibucau> | LinkedIn
>>>>> <https://www.linkedin.com/in/rmannibucau> | Book
>>>>> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>>>>>
>>>>> 2018-01-12 6:30 GMT+01:00 Romain Manni-Bucau <rm...@gmail.com>:
>>>>>
>>>>>>
>>>>>>
>>>>>> Le 11 janv. 2018 23:13, "Kenneth Knowles" <kl...@google.com> a écrit :
>>>>>>
>>>>>> On Thu, Jan 11, 2018 at 8:43 AM, Romain Manni-Bucau <
>>>>>> rmannibucau@gmail.com> wrote:
>>>>>>
>>>>>>> 2. gradle build doesn't use the same output directory than maven so
>>>>>>> it is not really smooth to have both and have to maintain both
>>>>>>>
>>>>>>
>>>>>> I also have an opinion on this. It is useful and reasonable to be
>>>>>> able to build even when the source is on a read-only filesystem. Maven's
>>>>>> defaults are undesirable and require workarounds. We shouldn't mimic the
>>>>>> behavior, but actually should set gradle up to build to a directory outside
>>>>>> the source tree always, if we can.
>>>>>>
>>>>>>
>>>>>> Hmm, which is something you can do with maven as well so not sure I
>>>>>> get it.
>>>>>>
>>>>>> Also note the thread is no more about the technical points but more
>>>>>> the sources maintenance and consistency.
>>>>>>
>>>>>>
>>>>>> Kenn
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>
Re: Gradle status
Posted by Kenneth Knowles <kl...@google.com>.
I also cannot drop everything to work on Gradle build, but maybe it isn't
that drastic anyhow. Now that we have ValidatesRunner and NeedsRunner tests
and some progress on the release, is there any other known missing
functionality in the Gradle builds? Archetypes? Docker container images?
On Wed, Mar 7, 2018 at 1:12 PM Lukasz Cwik <lc...@google.com> wrote:
> I am working on various projects and may not be able to pause my work for
> a couple of weeks while the build/test process is migrated.
>
> What is everyone thinking about Romain's suggestion because If I'm the
> only person in such a situation, I would be willing to go along with the
> plan.
>
> On Wed, Mar 7, 2018 at 12:59 PM, Romain Manni-Bucau <rmannibucau@gmail.com
> > wrote:
>
>>
>>
>> Le 7 mars 2018 20:21, "Lukasz Cwik" <lc...@google.com> a écrit :
>>
>> Note that Alan Myrvold has been making steady progress making the release
>> process via Gradle a reality:
>> 1) Creating a jenkins job which can run the quickstart validation against
>> nightly snapshots and also can be used for release candidates (
>> https://github.com/apache/beam/pull/4252)
>> 2) Building a release candidate using Gradle (
>> https://github.com/apache/beam/pull/4812)
>>
>> Also, Gradle is the tool that has been selected already and there was a
>> community discussion about what was needed for the migration to occur with
>> a clear set of criteria. Romain, it doesn't seem like we should ignore that
>> criteria or are you suggesting we change that criteria (if yes, how)?
>>
>>
>> No, no. My goal is just to quit this state.
>>
>> Let s draft a plan:
>>
>> 1. 2.4 is released - i assume it is done with mvn here
>> 2. We drop all poms and jenkins mvn config
>> 3. We fix all build issues if so (let say in a week)
>> 4. Pr can nees updates but no more mvn merge
>>
>> April is gradle month :)
>>
>> Wdyt?
>>
>>
>>
>>
>> On Wed, Mar 7, 2018 at 10:39 AM, Romain Manni-Bucau <
>> rmannibucau@gmail.com> wrote:
>>
>>>
>>>
>>> Le 7 mars 2018 17:34, "Lukasz Cwik" <lc...@google.com> a écrit :
>>>
>>> Thanks for bringing this up Romain but I believe your data points on
>>> pass rates are only partially correct.
>>>
>>>
>>> Sure sure, it is mainly about my own PR which a very small % of the
>>> whole project ;).
>>>
>>>
>>>
>>> In the past week the Java Gradle precommit passed 46.34% of the time
>>> compared to the Java Maven precommit which passed 46.15% of the time. When
>>> I looked at these numbers in mid January they were around 37% so there has
>>> been some improvement. Regardless of the build tool it seems that our pass
>>> rates aren't stellar for the Java build and are causing the community to
>>> not follow best practices (wait for precommits to be green before merging).
>>> I know that on the website we used the mergebot to ensure that things
>>> passed before they were merged, should we institute this on the master
>>> branch or are their any other ideas?
>>>
>>> As a side note we had achieved the goals we set out to not need to
>>> maintain the Maven precommit and have authored the first PR to drop the
>>> Maven precommit: https://github.com/apache/beam/pull/4814
>>>
>>>
>>> Well, I'd be for a strong switch otherwise PR will keep using maven,
>>> jenkins will not test the code and at the end we fail to deliver something
>>> consistent. So whatever tool is selected I'm tempted to say drop other
>>> build files and jenkins hooks references.
>>>
>>> What about doing it after 2.4 vote?
>>>
>>>
>>>
>>> On Wed, Mar 7, 2018 at 2:24 AM, Romain Manni-Bucau <
>>> rmannibucau@gmail.com> wrote:
>>>
>>>> Up,
>>>>
>>>> We discussed to have a strong switch to gradle or rollback to maven
>>>> around april to not be blocked by the build tool. I noticed gradle build
>>>> rarely passes on PR and kind of blurry our vision - not sure why exactly.
>>>> Also, PR don't always contain the gradle updates - generally
>>>> dependencies+plugins are added in pom.xml AFAIK, so it seems the adoption
>>>> is very slow to not say rejected.
>>>>
>>>> What do we do about that? When do we drop the double build maintenance
>>>> - whatever is picked?
>>>>
>>>>
>>>>
>>>> Romain Manni-Bucau
>>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
>>>> <https://rmannibucau.metawerx.net/> | Old Blog
>>>> <http://rmannibucau.wordpress.com> | Github
>>>> <https://github.com/rmannibucau> | LinkedIn
>>>> <https://www.linkedin.com/in/rmannibucau> | Book
>>>> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>>>>
>>>> 2018-01-12 6:30 GMT+01:00 Romain Manni-Bucau <rm...@gmail.com>:
>>>>
>>>>>
>>>>>
>>>>> Le 11 janv. 2018 23:13, "Kenneth Knowles" <kl...@google.com> a écrit :
>>>>>
>>>>> On Thu, Jan 11, 2018 at 8:43 AM, Romain Manni-Bucau <
>>>>> rmannibucau@gmail.com> wrote:
>>>>>
>>>>>> 2. gradle build doesn't use the same output directory than maven so
>>>>>> it is not really smooth to have both and have to maintain both
>>>>>>
>>>>>
>>>>> I also have an opinion on this. It is useful and reasonable to be able
>>>>> to build even when the source is on a read-only filesystem. Maven's
>>>>> defaults are undesirable and require workarounds. We shouldn't mimic the
>>>>> behavior, but actually should set gradle up to build to a directory outside
>>>>> the source tree always, if we can.
>>>>>
>>>>>
>>>>> Hmm, which is something you can do with maven as well so not sure I
>>>>> get it.
>>>>>
>>>>> Also note the thread is no more about the technical points but more
>>>>> the sources maintenance and consistency.
>>>>>
>>>>>
>>>>> Kenn
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>
>>
>
Re: Gradle status
Posted by Lukasz Cwik <lc...@google.com>.
I am working on various projects and may not be able to pause my work for a
couple of weeks while the build/test process is migrated.
What is everyone thinking about Romain's suggestion because If I'm the only
person in such a situation, I would be willing to go along with the plan.
On Wed, Mar 7, 2018 at 12:59 PM, Romain Manni-Bucau <rm...@gmail.com>
wrote:
>
>
> Le 7 mars 2018 20:21, "Lukasz Cwik" <lc...@google.com> a écrit :
>
> Note that Alan Myrvold has been making steady progress making the release
> process via Gradle a reality:
> 1) Creating a jenkins job which can run the quickstart validation against
> nightly snapshots and also can be used for release candidates (
> https://github.com/apache/beam/pull/4252)
> 2) Building a release candidate using Gradle (
> https://github.com/apache/beam/pull/4812)
>
> Also, Gradle is the tool that has been selected already and there was a
> community discussion about what was needed for the migration to occur with
> a clear set of criteria. Romain, it doesn't seem like we should ignore that
> criteria or are you suggesting we change that criteria (if yes, how)?
>
>
> No, no. My goal is just to quit this state.
>
> Let s draft a plan:
>
> 1. 2.4 is released - i assume it is done with mvn here
> 2. We drop all poms and jenkins mvn config
> 3. We fix all build issues if so (let say in a week)
> 4. Pr can nees updates but no more mvn merge
>
> April is gradle month :)
>
> Wdyt?
>
>
>
>
> On Wed, Mar 7, 2018 at 10:39 AM, Romain Manni-Bucau <rmannibucau@gmail.com
> > wrote:
>
>>
>>
>> Le 7 mars 2018 17:34, "Lukasz Cwik" <lc...@google.com> a écrit :
>>
>> Thanks for bringing this up Romain but I believe your data points on pass
>> rates are only partially correct.
>>
>>
>> Sure sure, it is mainly about my own PR which a very small % of the whole
>> project ;).
>>
>>
>>
>> In the past week the Java Gradle precommit passed 46.34% of the time
>> compared to the Java Maven precommit which passed 46.15% of the time. When
>> I looked at these numbers in mid January they were around 37% so there has
>> been some improvement. Regardless of the build tool it seems that our pass
>> rates aren't stellar for the Java build and are causing the community to
>> not follow best practices (wait for precommits to be green before merging).
>> I know that on the website we used the mergebot to ensure that things
>> passed before they were merged, should we institute this on the master
>> branch or are their any other ideas?
>>
>> As a side note we had achieved the goals we set out to not need to
>> maintain the Maven precommit and have authored the first PR to drop the
>> Maven precommit: https://github.com/apache/beam/pull/4814
>>
>>
>> Well, I'd be for a strong switch otherwise PR will keep using maven,
>> jenkins will not test the code and at the end we fail to deliver something
>> consistent. So whatever tool is selected I'm tempted to say drop other
>> build files and jenkins hooks references.
>>
>> What about doing it after 2.4 vote?
>>
>>
>>
>> On Wed, Mar 7, 2018 at 2:24 AM, Romain Manni-Bucau <rmannibucau@gmail.com
>> > wrote:
>>
>>> Up,
>>>
>>> We discussed to have a strong switch to gradle or rollback to maven
>>> around april to not be blocked by the build tool. I noticed gradle build
>>> rarely passes on PR and kind of blurry our vision - not sure why exactly.
>>> Also, PR don't always contain the gradle updates - generally
>>> dependencies+plugins are added in pom.xml AFAIK, so it seems the adoption
>>> is very slow to not say rejected.
>>>
>>> What do we do about that? When do we drop the double build maintenance -
>>> whatever is picked?
>>>
>>>
>>>
>>> Romain Manni-Bucau
>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
>>> <https://rmannibucau.metawerx.net/> | Old Blog
>>> <http://rmannibucau.wordpress.com> | Github
>>> <https://github.com/rmannibucau> | LinkedIn
>>> <https://www.linkedin.com/in/rmannibucau> | Book
>>> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>>>
>>> 2018-01-12 6:30 GMT+01:00 Romain Manni-Bucau <rm...@gmail.com>:
>>>
>>>>
>>>>
>>>> Le 11 janv. 2018 23:13, "Kenneth Knowles" <kl...@google.com> a écrit :
>>>>
>>>> On Thu, Jan 11, 2018 at 8:43 AM, Romain Manni-Bucau <
>>>> rmannibucau@gmail.com> wrote:
>>>>
>>>>> 2. gradle build doesn't use the same output directory than maven so it
>>>>> is not really smooth to have both and have to maintain both
>>>>>
>>>>
>>>> I also have an opinion on this. It is useful and reasonable to be able
>>>> to build even when the source is on a read-only filesystem. Maven's
>>>> defaults are undesirable and require workarounds. We shouldn't mimic the
>>>> behavior, but actually should set gradle up to build to a directory outside
>>>> the source tree always, if we can.
>>>>
>>>>
>>>> Hmm, which is something you can do with maven as well so not sure I get
>>>> it.
>>>>
>>>> Also note the thread is no more about the technical points but more the
>>>> sources maintenance and consistency.
>>>>
>>>>
>>>> Kenn
>>>>
>>>>
>>>>
>>>
>>
>>
>
>
Re: Gradle status
Posted by Romain Manni-Bucau <rm...@gmail.com>.
Le 7 mars 2018 20:21, "Lukasz Cwik" <lc...@google.com> a écrit :
Note that Alan Myrvold has been making steady progress making the release
process via Gradle a reality:
1) Creating a jenkins job which can run the quickstart validation against
nightly snapshots and also can be used for release candidates (
https://github.com/apache/beam/pull/4252)
2) Building a release candidate using Gradle (https://github.com/apache/
beam/pull/4812)
Also, Gradle is the tool that has been selected already and there was a
community discussion about what was needed for the migration to occur with
a clear set of criteria. Romain, it doesn't seem like we should ignore that
criteria or are you suggesting we change that criteria (if yes, how)?
No, no. My goal is just to quit this state.
Let s draft a plan:
1. 2.4 is released - i assume it is done with mvn here
2. We drop all poms and jenkins mvn config
3. We fix all build issues if so (let say in a week)
4. Pr can nees updates but no more mvn merge
April is gradle month :)
Wdyt?
On Wed, Mar 7, 2018 at 10:39 AM, Romain Manni-Bucau <rm...@gmail.com>
wrote:
>
>
> Le 7 mars 2018 17:34, "Lukasz Cwik" <lc...@google.com> a écrit :
>
> Thanks for bringing this up Romain but I believe your data points on pass
> rates are only partially correct.
>
>
> Sure sure, it is mainly about my own PR which a very small % of the whole
> project ;).
>
>
>
> In the past week the Java Gradle precommit passed 46.34% of the time
> compared to the Java Maven precommit which passed 46.15% of the time. When
> I looked at these numbers in mid January they were around 37% so there has
> been some improvement. Regardless of the build tool it seems that our pass
> rates aren't stellar for the Java build and are causing the community to
> not follow best practices (wait for precommits to be green before merging).
> I know that on the website we used the mergebot to ensure that things
> passed before they were merged, should we institute this on the master
> branch or are their any other ideas?
>
> As a side note we had achieved the goals we set out to not need to
> maintain the Maven precommit and have authored the first PR to drop the
> Maven precommit: https://github.com/apache/beam/pull/4814
>
>
> Well, I'd be for a strong switch otherwise PR will keep using maven,
> jenkins will not test the code and at the end we fail to deliver something
> consistent. So whatever tool is selected I'm tempted to say drop other
> build files and jenkins hooks references.
>
> What about doing it after 2.4 vote?
>
>
>
> On Wed, Mar 7, 2018 at 2:24 AM, Romain Manni-Bucau <rm...@gmail.com>
> wrote:
>
>> Up,
>>
>> We discussed to have a strong switch to gradle or rollback to maven
>> around april to not be blocked by the build tool. I noticed gradle build
>> rarely passes on PR and kind of blurry our vision - not sure why exactly.
>> Also, PR don't always contain the gradle updates - generally
>> dependencies+plugins are added in pom.xml AFAIK, so it seems the adoption
>> is very slow to not say rejected.
>>
>> What do we do about that? When do we drop the double build maintenance -
>> whatever is picked?
>>
>>
>>
>> Romain Manni-Bucau
>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
>> <https://rmannibucau.metawerx.net/> | Old Blog
>> <http://rmannibucau.wordpress.com> | Github
>> <https://github.com/rmannibucau> | LinkedIn
>> <https://www.linkedin.com/in/rmannibucau> | Book
>> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>>
>> 2018-01-12 6:30 GMT+01:00 Romain Manni-Bucau <rm...@gmail.com>:
>>
>>>
>>>
>>> Le 11 janv. 2018 23:13, "Kenneth Knowles" <kl...@google.com> a écrit :
>>>
>>> On Thu, Jan 11, 2018 at 8:43 AM, Romain Manni-Bucau <
>>> rmannibucau@gmail.com> wrote:
>>>
>>>> 2. gradle build doesn't use the same output directory than maven so it
>>>> is not really smooth to have both and have to maintain both
>>>>
>>>
>>> I also have an opinion on this. It is useful and reasonable to be able
>>> to build even when the source is on a read-only filesystem. Maven's
>>> defaults are undesirable and require workarounds. We shouldn't mimic the
>>> behavior, but actually should set gradle up to build to a directory outside
>>> the source tree always, if we can.
>>>
>>>
>>> Hmm, which is something you can do with maven as well so not sure I get
>>> it.
>>>
>>> Also note the thread is no more about the technical points but more the
>>> sources maintenance and consistency.
>>>
>>>
>>> Kenn
>>>
>>>
>>>
>>
>
>
Re: Gradle status
Posted by Lukasz Cwik <lc...@google.com>.
Note that Alan Myrvold has been making steady progress making the release
process via Gradle a reality:
1) Creating a jenkins job which can run the quickstart validation against
nightly snapshots and also can be used for release candidates (
https://github.com/apache/beam/pull/4252)
2) Building a release candidate using Gradle (
https://github.com/apache/beam/pull/4812)
Also, Gradle is the tool that has been selected already and there was a
community discussion about what was needed for the migration to occur with
a clear set of criteria. Romain, it doesn't seem like we should ignore that
criteria or are you suggesting we change that criteria (if yes, how)?
On Wed, Mar 7, 2018 at 10:39 AM, Romain Manni-Bucau <rm...@gmail.com>
wrote:
>
>
> Le 7 mars 2018 17:34, "Lukasz Cwik" <lc...@google.com> a écrit :
>
> Thanks for bringing this up Romain but I believe your data points on pass
> rates are only partially correct.
>
>
> Sure sure, it is mainly about my own PR which a very small % of the whole
> project ;).
>
>
>
> In the past week the Java Gradle precommit passed 46.34% of the time
> compared to the Java Maven precommit which passed 46.15% of the time. When
> I looked at these numbers in mid January they were around 37% so there has
> been some improvement. Regardless of the build tool it seems that our pass
> rates aren't stellar for the Java build and are causing the community to
> not follow best practices (wait for precommits to be green before merging).
> I know that on the website we used the mergebot to ensure that things
> passed before they were merged, should we institute this on the master
> branch or are their any other ideas?
>
> As a side note we had achieved the goals we set out to not need to
> maintain the Maven precommit and have authored the first PR to drop the
> Maven precommit: https://github.com/apache/beam/pull/4814
>
>
> Well, I'd be for a strong switch otherwise PR will keep using maven,
> jenkins will not test the code and at the end we fail to deliver something
> consistent. So whatever tool is selected I'm tempted to say drop other
> build files and jenkins hooks references.
>
> What about doing it after 2.4 vote?
>
>
>
> On Wed, Mar 7, 2018 at 2:24 AM, Romain Manni-Bucau <rm...@gmail.com>
> wrote:
>
>> Up,
>>
>> We discussed to have a strong switch to gradle or rollback to maven
>> around april to not be blocked by the build tool. I noticed gradle build
>> rarely passes on PR and kind of blurry our vision - not sure why exactly.
>> Also, PR don't always contain the gradle updates - generally
>> dependencies+plugins are added in pom.xml AFAIK, so it seems the adoption
>> is very slow to not say rejected.
>>
>> What do we do about that? When do we drop the double build maintenance -
>> whatever is picked?
>>
>>
>>
>> Romain Manni-Bucau
>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
>> <https://rmannibucau.metawerx.net/> | Old Blog
>> <http://rmannibucau.wordpress.com> | Github
>> <https://github.com/rmannibucau> | LinkedIn
>> <https://www.linkedin.com/in/rmannibucau> | Book
>> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>>
>> 2018-01-12 6:30 GMT+01:00 Romain Manni-Bucau <rm...@gmail.com>:
>>
>>>
>>>
>>> Le 11 janv. 2018 23:13, "Kenneth Knowles" <kl...@google.com> a écrit :
>>>
>>> On Thu, Jan 11, 2018 at 8:43 AM, Romain Manni-Bucau <
>>> rmannibucau@gmail.com> wrote:
>>>
>>>> 2. gradle build doesn't use the same output directory than maven so it
>>>> is not really smooth to have both and have to maintain both
>>>>
>>>
>>> I also have an opinion on this. It is useful and reasonable to be able
>>> to build even when the source is on a read-only filesystem. Maven's
>>> defaults are undesirable and require workarounds. We shouldn't mimic the
>>> behavior, but actually should set gradle up to build to a directory outside
>>> the source tree always, if we can.
>>>
>>>
>>> Hmm, which is something you can do with maven as well so not sure I get
>>> it.
>>>
>>> Also note the thread is no more about the technical points but more the
>>> sources maintenance and consistency.
>>>
>>>
>>> Kenn
>>>
>>>
>>>
>>
>
>
Re: Gradle status
Posted by Romain Manni-Bucau <rm...@gmail.com>.
Le 7 mars 2018 17:34, "Lukasz Cwik" <lc...@google.com> a écrit :
Thanks for bringing this up Romain but I believe your data points on pass
rates are only partially correct.
Sure sure, it is mainly about my own PR which a very small % of the whole
project ;).
In the past week the Java Gradle precommit passed 46.34% of the time
compared to the Java Maven precommit which passed 46.15% of the time. When
I looked at these numbers in mid January they were around 37% so there has
been some improvement. Regardless of the build tool it seems that our pass
rates aren't stellar for the Java build and are causing the community to
not follow best practices (wait for precommits to be green before merging).
I know that on the website we used the mergebot to ensure that things
passed before they were merged, should we institute this on the master
branch or are their any other ideas?
As a side note we had achieved the goals we set out to not need to maintain
the Maven precommit and have authored the first PR to drop the Maven
precommit: https://github.com/apache/beam/pull/4814
Well, I'd be for a strong switch otherwise PR will keep using maven,
jenkins will not test the code and at the end we fail to deliver something
consistent. So whatever tool is selected I'm tempted to say drop other
build files and jenkins hooks references.
What about doing it after 2.4 vote?
On Wed, Mar 7, 2018 at 2:24 AM, Romain Manni-Bucau <rm...@gmail.com>
wrote:
> Up,
>
> We discussed to have a strong switch to gradle or rollback to maven around
> april to not be blocked by the build tool. I noticed gradle build rarely
> passes on PR and kind of blurry our vision - not sure why exactly. Also, PR
> don't always contain the gradle updates - generally dependencies+plugins
> are added in pom.xml AFAIK, so it seems the adoption is very slow to not
> say rejected.
>
> What do we do about that? When do we drop the double build maintenance -
> whatever is picked?
>
>
>
> Romain Manni-Bucau
> @rmannibucau <https://twitter.com/rmannibucau> | Blog
> <https://rmannibucau.metawerx.net/> | Old Blog
> <http://rmannibucau.wordpress.com> | Github
> <https://github.com/rmannibucau> | LinkedIn
> <https://www.linkedin.com/in/rmannibucau> | Book
> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>
> 2018-01-12 6:30 GMT+01:00 Romain Manni-Bucau <rm...@gmail.com>:
>
>>
>>
>> Le 11 janv. 2018 23:13, "Kenneth Knowles" <kl...@google.com> a écrit :
>>
>> On Thu, Jan 11, 2018 at 8:43 AM, Romain Manni-Bucau <
>> rmannibucau@gmail.com> wrote:
>>
>>> 2. gradle build doesn't use the same output directory than maven so it
>>> is not really smooth to have both and have to maintain both
>>>
>>
>> I also have an opinion on this. It is useful and reasonable to be able to
>> build even when the source is on a read-only filesystem. Maven's defaults
>> are undesirable and require workarounds. We shouldn't mimic the behavior,
>> but actually should set gradle up to build to a directory outside the
>> source tree always, if we can.
>>
>>
>> Hmm, which is something you can do with maven as well so not sure I get
>> it.
>>
>> Also note the thread is no more about the technical points but more the
>> sources maintenance and consistency.
>>
>>
>> Kenn
>>
>>
>>
>
Re: Gradle status
Posted by Lukasz Cwik <lc...@google.com>.
Thanks for bringing this up Romain but I believe your data points on pass
rates are only partially correct.
In the past week the Java Gradle precommit passed 46.34% of the time
compared to the Java Maven precommit which passed 46.15% of the time. When
I looked at these numbers in mid January they were around 37% so there has
been some improvement. Regardless of the build tool it seems that our pass
rates aren't stellar for the Java build and are causing the community to
not follow best practices (wait for precommits to be green before merging).
I know that on the website we used the mergebot to ensure that things
passed before they were merged, should we institute this on the master
branch or are their any other ideas?
As a side note we had achieved the goals we set out to not need to maintain
the Maven precommit and have authored the first PR to drop the Maven
precommit: https://github.com/apache/beam/pull/4814
On Wed, Mar 7, 2018 at 2:24 AM, Romain Manni-Bucau <rm...@gmail.com>
wrote:
> Up,
>
> We discussed to have a strong switch to gradle or rollback to maven around
> april to not be blocked by the build tool. I noticed gradle build rarely
> passes on PR and kind of blurry our vision - not sure why exactly. Also, PR
> don't always contain the gradle updates - generally dependencies+plugins
> are added in pom.xml AFAIK, so it seems the adoption is very slow to not
> say rejected.
>
> What do we do about that? When do we drop the double build maintenance -
> whatever is picked?
>
>
>
> Romain Manni-Bucau
> @rmannibucau <https://twitter.com/rmannibucau> | Blog
> <https://rmannibucau.metawerx.net/> | Old Blog
> <http://rmannibucau.wordpress.com> | Github
> <https://github.com/rmannibucau> | LinkedIn
> <https://www.linkedin.com/in/rmannibucau> | Book
> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>
> 2018-01-12 6:30 GMT+01:00 Romain Manni-Bucau <rm...@gmail.com>:
>
>>
>>
>> Le 11 janv. 2018 23:13, "Kenneth Knowles" <kl...@google.com> a écrit :
>>
>> On Thu, Jan 11, 2018 at 8:43 AM, Romain Manni-Bucau <
>> rmannibucau@gmail.com> wrote:
>>
>>> 2. gradle build doesn't use the same output directory than maven so it
>>> is not really smooth to have both and have to maintain both
>>>
>>
>> I also have an opinion on this. It is useful and reasonable to be able to
>> build even when the source is on a read-only filesystem. Maven's defaults
>> are undesirable and require workarounds. We shouldn't mimic the behavior,
>> but actually should set gradle up to build to a directory outside the
>> source tree always, if we can.
>>
>>
>> Hmm, which is something you can do with maven as well so not sure I get
>> it.
>>
>> Also note the thread is no more about the technical points but more the
>> sources maintenance and consistency.
>>
>>
>> Kenn
>>
>>
>>
>