You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Lukasz Cwik <lc...@google.com> on 2018/04/02 17:03:16 UTC
Re: Gradle migration fixit: April 3
Aljoscha, to add things to be shaded they need to be part of the "compile"
configuration and you also need to tell the shadow plugin to add these and
how to relocate them like this:
https://github.com/apache/beam/blob/cc2022157865d44e8dba070a766de78853c508bc/sdks/java/core/build.gradle#L77
Romain, the subtasks of BEAM-3249 seem to be centered around:
1) Migrating Jenkins
2) Making the artifacts/release (shading, jars, javadoc all fall under this)
I would defer things that are optional like "ensure concurrency is not
hardcoded" till after the migration.
On Sat, Mar 31, 2018 at 2:28 AM Romain Manni-Bucau <rm...@gmail.com>
wrote:
> Thanks for the doc Lukasz
>
> So we do 3 groups?
>
> 1. Jenkins
> 2. Release
> 3. Outputs/behavior (shades, jars, javadoc, ensure concurrency is not
> hardcoded, ...)
>
> ?
>
> Happy to be in 3 or 2 (by preference) if you agree.
>
> Le 31 mars 2018 10:54, "Aljoscha Krettek" <al...@apache.org> a écrit :
>
>> Thanks Luke, that was helpful!
>>
>> I've been playing around with gradle a bit and now have a question: By
>> default the shadow plugin will include dependencies that are "compile" or
>> "runtime" (compile dependencies are by default also runtime dependencies)
>> into the shaded jar. Is that correct? The reason our build doesn't do that
>> is that we have custom include/exclude rules that prevent that from
>> happening. I just noticed that when I change the Flink dependencies from
>> shadow to compile in the Flink Runner build file the shaded jar will not
>> contain the Flink classes.
>>
>> Btw, I'm not saying that I want to include the Flink classes, I was just
>> messing around with things to get a feeling for how things work.
>>
>> On 31. Mar 2018, at 00:15, Lukasz Cwik <lc...@google.com> wrote:
>>
>> I have started a doc[1] containing a Gradle primer to help people be more
>> productive during the fixit day. Feel free to add/update comments and
>> content.
>>
>> 1:
>> https://docs.google.com/document/d/1EiTwEMD8FNhU4Ok6jthASpmK3-1hiAYzVTrdl8qBLrs/edit?usp=sharing
>>
>> On Fri, Mar 30, 2018 at 8:33 AM Reuven Lax <re...@google.com> wrote:
>>
>>> Here is the Kanban board tracking all the current tasks. It looks like 7
>>> of them have already been closed over the past two days, so we're down to
>>> 19!
>>>
>>> https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=242
>>>
>>> On Thu, Mar 29, 2018 at 11:21 PM Romain Manni-Bucau <
>>> rmannibucau@gmail.com> wrote:
>>>
>>>> Yep - sorry if it was unclear. I know linux distro often do it (never
>>>> understood why though).
>>>>
>>>>
>>>> Romain Manni-Bucau
>>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
>>>> <https://rmannibucau.metawerx.net/> | Old Blog
>>>> <http://rmannibucau.wordpress.com/> | Github
>>>> <https://github.com/rmannibucau> | LinkedIn
>>>> <https://www.linkedin.com/in/rmannibucau> | Book
>>>> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>>>>
>>>> 2018-03-30 7:54 GMT+02:00 Reuven Lax <re...@google.com>:
>>>>
>>>>>
>>>>>
>>>>> On Thu, Mar 29, 2018 at 10:28 PM Romain Manni-Bucau <
>>>>> rmannibucau@gmail.com> wrote:
>>>>>
>>>>>> It was more about the drop of poms (same case as you for dataflow).
>>>>>>
>>>>>
>>>>> Ah - you're worried that some external users are building directly
>>>>> from the poms rather than using the published artifact.
>>>>>
>>>>> I think this is a valid concern, and I agree we should announce on
>>>>> users@ before (probably some time before) deleting the poms .
>>>>>
>>>>>
>>>>>> On that there is a missing but highly important task: gradle to mvn
>>>>>> descriptors. All the one I saw were corrupted poms so we must take care of
>>>>>> that as part of the release work (I can work on it on the 3rd if you want).
>>>>>>
>>>>>> Le 29 mars 2018 23:36, "Reuven Lax" <re...@google.com> a écrit :
>>>>>>
>>>>>>> I don't mind notifying users@, but this does seem more interesting
>>>>>>> for dev@. We will continue to publish Maven artifacts from our
>>>>>>> Gradle build, so users are still free to use either Maven or Gradle.
>>>>>>>
>>>>>>> That being said, if this is interesting to users@ we can notify
>>>>>>> them as well.
>>>>>>>
>>>>>>> Reuven
>>>>>>>
>>>>>>> On Thu, Mar 29, 2018 at 1:45 PM Romain Manni-Bucau <
>>>>>>> rmannibucau@gmail.com> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Le 29 mars 2018 21:20, "Reuven Lax" <re...@google.com> a écrit :
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Mar 29, 2018 at 12:17 PM Romain Manni-Bucau <
>>>>>>>> rmannibucau@gmail.com> wrote:
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Le 29 mars 2018 20:35, "Reuven Lax" <re...@google.com> a écrit :
>>>>>>>>>
>>>>>>>>> 1. As Luke already mentioned, we should first have a subgoal of
>>>>>>>>> the Gradle jenkins jobs being equivalent to the Maven jobs. Hopefully
>>>>>>>>> toward the end of the day, we'll make this change.
>>>>>>>>>
>>>>>>>>> 2. Let's see how much progress we make on the third. There is a
>>>>>>>>> side problem we have here at Google - we have an internal product called
>>>>>>>>> Dataflow built on Beam, and the Dataflow build still depends on those pom
>>>>>>>>> files. I would request leaving the pom files around just a little bit
>>>>>>>>> longer even if Beam no longer needs them, just so that we don't breakd
>>>>>>>>> Dataflow (and I think we would do this for any community members with a
>>>>>>>>> similar issue). We will prioritize moving Dataflow ASAP, it's just that the
>>>>>>>>> people who will do so will also be in the April 3 fixit so it can't happen
>>>>>>>>> until after. I think the delay should only be one or two weeks to delete
>>>>>>>>> the poms (assuming that Beam is ready at the end of the day).
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Can you try to put a date of that and we will communicate on it
>>>>>>>>> publicly if anyone else does (i dont think but i didnt expect you to do it
>>>>>>>>> too ;)).
>>>>>>>>>
>>>>>>>>
>>>>>>>> What do you mean by communicate publicly? The dev list is already
>>>>>>>> cced here :)
>>>>>>>>
>>>>>>>>
>>>>>>>> Misses a (not too far) date and users@ pby to be safe ;)
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Mar 29, 2018 at 5:20 AM Romain Manni-Bucau <
>>>>>>>>> rmannibucau@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Reuven, a few questions:
>>>>>>>>>>
>>>>>>>>>> 1. any inputs on how we can work on the jenkins part? Do we test
>>>>>>>>>> it live wiht "fake" PRs?
>>>>>>>>>> 2. What's the rational to not start by deleting the poms? Sounds
>>>>>>>>>> like it will be a day working on gradle and on the 4th we'll be back on
>>>>>>>>>> maven
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Romain Manni-Bucau
>>>>>>>>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
>>>>>>>>>> <https://rmannibucau.metawerx.net/> | Old Blog
>>>>>>>>>> <http://rmannibucau.wordpress.com/> | Github
>>>>>>>>>> <https://github.com/rmannibucau> | LinkedIn
>>>>>>>>>> <https://www.linkedin.com/in/rmannibucau> | Book
>>>>>>>>>> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>>>>>>>>>>
>>>>>>>>>> 2018-03-29 4:46 GMT+02:00 Reuven Lax <re...@google.com>:
>>>>>>>>>>
>>>>>>>>>>> Hi all,
>>>>>>>>>>>
>>>>>>>>>>> Last week we discussed having a "fixit" day for Gradle, and I
>>>>>>>>>>> volunteered to organize it. A number of people volunteered to help, from
>>>>>>>>>>> multiple organization. I'd like to say that it's great to see such a
>>>>>>>>>>> diverse set of people volunteering to help here - this is a great way to
>>>>>>>>>>> build community! Everyone who explicitly volunteered is directly cced on
>>>>>>>>>>> this email, though we'd love for more of the community to help.
>>>>>>>>>>>
>>>>>>>>>>> The agreed upon date is April 3. The top-level JIRA tracking
>>>>>>>>>>> this work is
>>>>>>>>>>> ttps://issues.apache.org/jira/browse/BEAM-3249
>>>>>>>>>>> <https://issues.apache.org/jira/browse/BEAM-3249>, and we
>>>>>>>>>>> currently have 26 subtasks linked to it. I've created a Kanban board to
>>>>>>>>>>> track these issues, which I'll share out soon. We will use Slack the day of
>>>>>>>>>>> the fixit for collaboration and for questions.
>>>>>>>>>>>
>>>>>>>>>>> Two major goals for this fixit should be to 1. Remove Maven runs
>>>>>>>>>>> from our Jenkins executors and 2. to migrate our release process fully over
>>>>>>>>>>> to Gradle. A lot of work has already been done on 1., and we've made some
>>>>>>>>>>> progress on 2.. Slightly longer-term the goal is to delete all of the pom
>>>>>>>>>>> files; I'm not sure we'll get as far as completely deleting Maven in one
>>>>>>>>>>> day, but we should get within striking distance!
>>>>>>>>>>>
>>>>>>>>>>> Thanks in advance to everyone who's helping out!
>>>>>>>>>>>
>>>>>>>>>>> Reuven
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>
>>
Re: Gradle migration fixit: April 3
Posted by Lukasz Cwik <lc...@google.com>.
I opened up PR/4992[1] to disable all maven based Jenkins jobs and leave
them around so that people working on the migration can update them to be
Gradle based. This also ensures that there is plenty of Jenkins capacity
for testing migration to Gradle.
It is important for Committers to be aware that this is being disabled and
to be careful about merging non Gradle migration fixit tasks.
1: https://github.com/apache/beam/pull/4992
On Mon, Apr 2, 2018 at 10:03 AM Lukasz Cwik <lc...@google.com> wrote:
> Aljoscha, to add things to be shaded they need to be part of the "compile"
> configuration and you also need to tell the shadow plugin to add these and
> how to relocate them like this:
>
> https://github.com/apache/beam/blob/cc2022157865d44e8dba070a766de78853c508bc/sdks/java/core/build.gradle#L77
>
> Romain, the subtasks of BEAM-3249 seem to be centered around:
> 1) Migrating Jenkins
> 2) Making the artifacts/release (shading, jars, javadoc all fall under
> this)
>
> I would defer things that are optional like "ensure concurrency is not
> hardcoded" till after the migration.
>
> On Sat, Mar 31, 2018 at 2:28 AM Romain Manni-Bucau <rm...@gmail.com>
> wrote:
>
>> Thanks for the doc Lukasz
>>
>> So we do 3 groups?
>>
>> 1. Jenkins
>> 2. Release
>> 3. Outputs/behavior (shades, jars, javadoc, ensure concurrency is not
>> hardcoded, ...)
>>
>> ?
>>
>> Happy to be in 3 or 2 (by preference) if you agree.
>>
>> Le 31 mars 2018 10:54, "Aljoscha Krettek" <al...@apache.org> a écrit :
>>
>>> Thanks Luke, that was helpful!
>>>
>>> I've been playing around with gradle a bit and now have a question: By
>>> default the shadow plugin will include dependencies that are "compile" or
>>> "runtime" (compile dependencies are by default also runtime dependencies)
>>> into the shaded jar. Is that correct? The reason our build doesn't do that
>>> is that we have custom include/exclude rules that prevent that from
>>> happening. I just noticed that when I change the Flink dependencies from
>>> shadow to compile in the Flink Runner build file the shaded jar will not
>>> contain the Flink classes.
>>>
>>> Btw, I'm not saying that I want to include the Flink classes, I was just
>>> messing around with things to get a feeling for how things work.
>>>
>>> On 31. Mar 2018, at 00:15, Lukasz Cwik <lc...@google.com> wrote:
>>>
>>> I have started a doc[1] containing a Gradle primer to help people be
>>> more productive during the fixit day. Feel free to add/update comments and
>>> content.
>>>
>>> 1:
>>> https://docs.google.com/document/d/1EiTwEMD8FNhU4Ok6jthASpmK3-1hiAYzVTrdl8qBLrs/edit?usp=sharing
>>>
>>> On Fri, Mar 30, 2018 at 8:33 AM Reuven Lax <re...@google.com> wrote:
>>>
>>>> Here is the Kanban board tracking all the current tasks. It looks like
>>>> 7 of them have already been closed over the past two days, so we're down to
>>>> 19!
>>>>
>>>> https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=242
>>>>
>>>> On Thu, Mar 29, 2018 at 11:21 PM Romain Manni-Bucau <
>>>> rmannibucau@gmail.com> wrote:
>>>>
>>>>> Yep - sorry if it was unclear. I know linux distro often do it (never
>>>>> understood why though).
>>>>>
>>>>>
>>>>> Romain Manni-Bucau
>>>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
>>>>> <https://rmannibucau.metawerx.net/> | Old Blog
>>>>> <http://rmannibucau.wordpress.com/> | Github
>>>>> <https://github.com/rmannibucau> | LinkedIn
>>>>> <https://www.linkedin.com/in/rmannibucau> | Book
>>>>> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>>>>>
>>>>> 2018-03-30 7:54 GMT+02:00 Reuven Lax <re...@google.com>:
>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Mar 29, 2018 at 10:28 PM Romain Manni-Bucau <
>>>>>> rmannibucau@gmail.com> wrote:
>>>>>>
>>>>>>> It was more about the drop of poms (same case as you for dataflow).
>>>>>>>
>>>>>>
>>>>>> Ah - you're worried that some external users are building directly
>>>>>> from the poms rather than using the published artifact.
>>>>>>
>>>>>> I think this is a valid concern, and I agree we should announce on
>>>>>> users@ before (probably some time before) deleting the poms .
>>>>>>
>>>>>>
>>>>>>> On that there is a missing but highly important task: gradle to mvn
>>>>>>> descriptors. All the one I saw were corrupted poms so we must take care of
>>>>>>> that as part of the release work (I can work on it on the 3rd if you want).
>>>>>>>
>>>>>>> Le 29 mars 2018 23:36, "Reuven Lax" <re...@google.com> a écrit :
>>>>>>>
>>>>>>>> I don't mind notifying users@, but this does seem more interesting
>>>>>>>> for dev@. We will continue to publish Maven artifacts from our
>>>>>>>> Gradle build, so users are still free to use either Maven or Gradle.
>>>>>>>>
>>>>>>>> That being said, if this is interesting to users@ we can notify
>>>>>>>> them as well.
>>>>>>>>
>>>>>>>> Reuven
>>>>>>>>
>>>>>>>> On Thu, Mar 29, 2018 at 1:45 PM Romain Manni-Bucau <
>>>>>>>> rmannibucau@gmail.com> wrote:
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Le 29 mars 2018 21:20, "Reuven Lax" <re...@google.com> a écrit :
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Mar 29, 2018 at 12:17 PM Romain Manni-Bucau <
>>>>>>>>> rmannibucau@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Le 29 mars 2018 20:35, "Reuven Lax" <re...@google.com> a écrit :
>>>>>>>>>>
>>>>>>>>>> 1. As Luke already mentioned, we should first have a subgoal of
>>>>>>>>>> the Gradle jenkins jobs being equivalent to the Maven jobs. Hopefully
>>>>>>>>>> toward the end of the day, we'll make this change.
>>>>>>>>>>
>>>>>>>>>> 2. Let's see how much progress we make on the third. There is a
>>>>>>>>>> side problem we have here at Google - we have an internal product called
>>>>>>>>>> Dataflow built on Beam, and the Dataflow build still depends on those pom
>>>>>>>>>> files. I would request leaving the pom files around just a little bit
>>>>>>>>>> longer even if Beam no longer needs them, just so that we don't breakd
>>>>>>>>>> Dataflow (and I think we would do this for any community members with a
>>>>>>>>>> similar issue). We will prioritize moving Dataflow ASAP, it's just that the
>>>>>>>>>> people who will do so will also be in the April 3 fixit so it can't happen
>>>>>>>>>> until after. I think the delay should only be one or two weeks to delete
>>>>>>>>>> the poms (assuming that Beam is ready at the end of the day).
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Can you try to put a date of that and we will communicate on it
>>>>>>>>>> publicly if anyone else does (i dont think but i didnt expect you to do it
>>>>>>>>>> too ;)).
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> What do you mean by communicate publicly? The dev list is already
>>>>>>>>> cced here :)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Misses a (not too far) date and users@ pby to be safe ;)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Thu, Mar 29, 2018 at 5:20 AM Romain Manni-Bucau <
>>>>>>>>>> rmannibucau@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Reuven, a few questions:
>>>>>>>>>>>
>>>>>>>>>>> 1. any inputs on how we can work on the jenkins part? Do we test
>>>>>>>>>>> it live wiht "fake" PRs?
>>>>>>>>>>> 2. What's the rational to not start by deleting the poms? Sounds
>>>>>>>>>>> like it will be a day working on gradle and on the 4th we'll be back on
>>>>>>>>>>> maven
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Romain Manni-Bucau
>>>>>>>>>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog
>>>>>>>>>>> <https://rmannibucau.metawerx.net/> | Old Blog
>>>>>>>>>>> <http://rmannibucau.wordpress.com/> | Github
>>>>>>>>>>> <https://github.com/rmannibucau> | LinkedIn
>>>>>>>>>>> <https://www.linkedin.com/in/rmannibucau> | Book
>>>>>>>>>>> <https://www.packtpub.com/application-development/java-ee-8-high-performance>
>>>>>>>>>>>
>>>>>>>>>>> 2018-03-29 4:46 GMT+02:00 Reuven Lax <re...@google.com>:
>>>>>>>>>>>
>>>>>>>>>>>> Hi all,
>>>>>>>>>>>>
>>>>>>>>>>>> Last week we discussed having a "fixit" day for Gradle, and I
>>>>>>>>>>>> volunteered to organize it. A number of people volunteered to help, from
>>>>>>>>>>>> multiple organization. I'd like to say that it's great to see such a
>>>>>>>>>>>> diverse set of people volunteering to help here - this is a great way to
>>>>>>>>>>>> build community! Everyone who explicitly volunteered is directly cced on
>>>>>>>>>>>> this email, though we'd love for more of the community to help.
>>>>>>>>>>>>
>>>>>>>>>>>> The agreed upon date is April 3. The top-level JIRA tracking
>>>>>>>>>>>> this work is
>>>>>>>>>>>> ttps://issues.apache.org/jira/browse/BEAM-3249
>>>>>>>>>>>> <https://issues.apache.org/jira/browse/BEAM-3249>, and we
>>>>>>>>>>>> currently have 26 subtasks linked to it. I've created a Kanban board to
>>>>>>>>>>>> track these issues, which I'll share out soon. We will use Slack the day of
>>>>>>>>>>>> the fixit for collaboration and for questions.
>>>>>>>>>>>>
>>>>>>>>>>>> Two major goals for this fixit should be to 1. Remove Maven
>>>>>>>>>>>> runs from our Jenkins executors and 2. to migrate our release process fully
>>>>>>>>>>>> over to Gradle. A lot of work has already been done on 1., and we've made
>>>>>>>>>>>> some progress on 2.. Slightly longer-term the goal is to delete all of the
>>>>>>>>>>>> pom files; I'm not sure we'll get as far as completely deleting Maven in
>>>>>>>>>>>> one day, but we should get within striking distance!
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks in advance to everyone who's helping out!
>>>>>>>>>>>>
>>>>>>>>>>>> Reuven
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>
>>>