You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Tomo Suzuki <su...@google.com> on 2020/03/04 22:16:23 UTC
Proposal: Beam to use GCP Libraries BOM
Hi Beam developers,
Shall we use GCP Libraries BOM [1] to specify the Google-related library
versions in Beam?
I've been working on Beam's dependency upgrades in the past few months.
It's time to consider a long-term solution to keep the libraries up-to-date
with small maintenance effort. To achieve that, I propose Beam to use GCP
Libraries BOM to set the Google-related library versions, rather than the
current way of making changes in each of ~30 Google libraries with
individual PRs [2].
After the proposal is implemented, Beam project upgrades the BOM version to
upgrade these Google-related libraries. This still needs to ensure the
libraries in GCP Library BOM are compatible with Beam's other dependencies.
(Linkage Checker will help with this job.) I believe onboarding GCP
Libraries BOM will solve lots of incompatibilities which we have seen in
gax, gRPC, google-cloud-core, and so on with minimal effort in Beam's
developers.
Created an issue to track this: BEAM-9444 [3]. I appreciate if you can
share questions or feedback (thumbs-up / concerns).
[1]:
https://github.com/GoogleCloudPlatform/cloud-opensource-java/wiki/The-Google-Cloud-Platform-Libraries-BOM
[2]: https://github.com/apache/beam/pulls?page=1&q=is%3Apr+author%3Asuztomo
[3]: https://issues.apache.org/jira/browse/BEAM-9444
--
Regards,
Tomo
Re: Proposal: Beam to use GCP Libraries BOM
Posted by Tomo Suzuki <su...@google.com>.
> How would the Apache Beam BOM and GCP BOM work together?
I envision there will be (new) "Beam GCP BOM" that imports (existing) Beam
BOM and GCP Libraries BOM with necessary overwrites (such as Guava version).
This clarifies which versions of Google libraries should be compatible with
Beam's version.
On Thu, Mar 5, 2020 at 1:05 PM Luke Cwik <lc...@google.com> wrote:
> How would the Apache Beam BOM and GCP BOM work together?
>
> On Thu, Mar 5, 2020 at 7:25 AM Filipe Regadas <fi...@gmail.com>
> wrote:
>
>> Big +1, this is a step in the right direction and checking with other
>> Beam's direct and transitive deps is crucial since the referred bom only
>> convers a small part of it. Apache Commons, Jackson, `com.google.{api,
>> apis, cloud}`, slf4j comes to mind.
>>
>> Filipe Regadas
>>
>>
>> On Thu, Mar 5, 2020 at 3:33 AM Ismaël Mejía <ie...@gmail.com> wrote:
>>
>>> +1 Sounds like a good improvement for users and maintainers !
>>>
>>> On Thu, Mar 5, 2020 at 6:59 AM Alex Van Boxel <al...@vanboxel.be> wrote:
>>> >
>>> > +1, I can remember the countless hours that we fought with Google
>>> dependencies.
>>> >
>>> > On Thu, Mar 5, 2020, 04:07 Chamikara Jayalath <ch...@google.com>
>>> wrote:
>>> >>
>>> >> +1 for this.
>>> >>
>>> >> This will make life easy for many of our users and will help us keep
>>> GCP related dependencies compatible (which has not been easy in the past).
>>> >>
>>> >> On Wed, Mar 4, 2020 at 2:16 PM Tomo Suzuki <su...@google.com>
>>> wrote:
>>> >>>
>>> >>> Hi Beam developers,
>>> >>>
>>> >>> Shall we use GCP Libraries BOM [1] to specify the Google-related
>>> library versions in Beam?
>>> >>>
>>> >>> I've been working on Beam's dependency upgrades in the past few
>>> months. It's time to consider a long-term solution to keep the libraries
>>> up-to-date with small maintenance effort. To achieve that, I propose Beam
>>> to use GCP Libraries BOM to set the Google-related library versions, rather
>>> than the current way of making changes in each of ~30 Google libraries with
>>> individual PRs [2].
>>> >>>
>>> >>> After the proposal is implemented, Beam project upgrades the BOM
>>> version to upgrade these Google-related libraries. This still needs to
>>> ensure the libraries in GCP Library BOM are compatible with Beam's other
>>> dependencies. (Linkage Checker will help with this job.) I believe
>>> onboarding GCP Libraries BOM will solve lots of incompatibilities which we
>>> have seen in gax, gRPC, google-cloud-core, and so on with minimal effort in
>>> Beam's developers.
>>> >>>
>>> >>> Created an issue to track this: BEAM-9444 [3]. I appreciate if you
>>> can share questions or feedback (thumbs-up / concerns).
>>> >>>
>>> >>> [1]:
>>> https://github.com/GoogleCloudPlatform/cloud-opensource-java/wiki/The-Google-Cloud-Platform-Libraries-BOM
>>> >>> [2]:
>>> https://github.com/apache/beam/pulls?page=1&q=is%3Apr+author%3Asuztomo
>>> >>> [3]: https://issues.apache.org/jira/browse/BEAM-9444
>>> >>>
>>> >>> --
>>> >>> Regards,
>>> >>> Tomo
>>>
>>
--
Regards,
Tomo
Re: Proposal: Beam to use GCP Libraries BOM
Posted by Tomo Suzuki <su...@google.com>.
This has been revised as https://github.com/apache/beam/pull/13075 and it's
merged yesterday. Now many GCP-related dependencies are managed by the BOM.
Thank you Kiley!
On Mon, Mar 30, 2020 at 10:59 AM Tomo Suzuki <su...@google.com> wrote:
> Hi Chamikara,
>
> Ahmet is asking for your input in
> https://github.com/apache/beam/pull/11156#issuecomment-602716275 . Would
> you check this?
>
> On Mon, Mar 23, 2020 at 3:05 PM Tomo Suzuki <su...@google.com> wrote:
>
>> (Applied Ahmet's feedback in https://github.com/apache/beam/pull/11156)
>>
>> Hi Luke,
>>
>> > the pom files that are produced
>>
>> Build.gradle that uses the BOM will produce pom files that have
>> corresponding dependencies without versions, and have a
>> "dependencyManagement" section through which they import the GCP Libraries
>> BOM. This fills the versions of the versionless dependencies when build
>> systems evaluates the pom files.
>>
>> > how the GCP BOM impacts the release process
>>
>> I'm afraid I don't have knowledge on how dependencies affect the Beam
>> release process. Would you be willing to break down this concern into some
>> questions I can answer?
>>
>>
>>
>>
>> On Fri, Mar 20, 2020 at 6:13 PM Luke Cwik <lc...@google.com> wrote:
>>
>>> My concern would be to validate how the GCP BOM impacts the release
>>> process and the pom files that are produced otherwise the next person
>>> running a release may run into trouble.
>>>
>>> On Fri, Mar 20, 2020 at 3:00 PM Pablo Estrada <pa...@google.com>
>>> wrote:
>>>
>>>> +1
>>>>
>>>> On Fri, Mar 20, 2020 at 2:41 PM Ismaël Mejía <ie...@gmail.com> wrote:
>>>>
>>>>> Well why we just don't merge it? I am unfamiliar with GCP deps to be
>>>>> confident to LGTM it. but given that 22 checks pass and given that
>>>>> Tomo addressed most comments and he has already a consistent track of
>>>>> good work on dependency improvements I think it is worth to merge it
>>>>> as it is. We still have some time to fix stuff if we find any
>>>>> regression. WDYT?
>>>>>
>>>>> On Thu, Mar 19, 2020 at 9:36 PM Luke Cwik <lc...@google.com> wrote:
>>>>> >
>>>>> > As much as I would like to spend time on these reviews, I believe
>>>>> I'll be delayed from reviewing them thoroughly till I finish other work
>>>>> that I'm targeting for the 2.21 release related to portability. It would be
>>>>> greatly appreciated if there are others that could review this in the
>>>>> meantime.
>>>>> >
>>>>> > On Thu, Mar 19, 2020 at 7:09 AM Tomo Suzuki <su...@google.com>
>>>>> wrote:
>>>>> >>
>>>>> >> PR is ready (22 successful check)
>>>>> >> https://github.com/apache/beam/pull/11156
>>>>> >> (Luke assigned himself as a reviewer)
>>>>> >>
>>>>> >> Regards,
>>>>> >> Tomo
>>>>> >>
>>>>> >> On Wed, Mar 11, 2020 at 3:50 PM Tomo Suzuki <su...@google.com>
>>>>> wrote:
>>>>> >>>
>>>>> >>> Thank you for favorable responses. I'll start implementation.
>>>>> >>>
>>>>> >>> On Thu, Mar 5, 2020 at 2:22 PM Tomo Suzuki <su...@google.com>
>>>>> wrote:
>>>>> >>>>
>>>>> >>>> > Do Spark or Flink have BOMs?
>>>>> >>>>
>>>>> >>>> Not that I know of. I couldn't find "bom" in their artifacts [1,
>>>>> 2].
>>>>> >>>>
>>>>> >>>> [1]: https://search.maven.org/search?q=g:org.apache.flink
>>>>> >>>> [2]: https://search.maven.org/search?q=g:org.apache.spark
>>>>> >>>>
>>>>> >>>>
>>>>> >>>> On Thu, Mar 5, 2020 at 1:46 PM Kenneth Knowles <ke...@apache.org>
>>>>> wrote:
>>>>> >>>>>
>>>>> >>>>> +1 and you have phrased the benefits and limitations well. We
>>>>> have plenty of not-Google-related dependencies that use Guava and protobuf
>>>>> (I know of Calcite, Cassandra, Kinesis, and Spark) so there's still work in
>>>>> managing deps, but the BOM should make it a lot easier to upgrade all these
>>>>> tightly coupled libraries that Google ships from their head.
>>>>> >>>>>
>>>>> >>>>> Do Spark or Flink have BOMs? I wonder if there's an opportunity
>>>>> to catch incompatible deps at a larger scale by comparing and merging a
>>>>> half dozen BOMs (although in the limit it approximately expands to one per
>>>>> runner and one per IO and projects mature and become independent)
>>>>> >>>>>
>>>>> >>>>> Kenn
>>>>> >>>>>
>>>>> >>>>> On Thu, Mar 5, 2020 at 10:05 AM Luke Cwik <lc...@google.com>
>>>>> wrote:
>>>>> >>>>>>
>>>>> >>>>>> How would the Apache Beam BOM and GCP BOM work together?
>>>>> >>>>>>
>>>>> >>>>>> On Thu, Mar 5, 2020 at 7:25 AM Filipe Regadas <
>>>>> filiperegadas@gmail.com> wrote:
>>>>> >>>>>>>
>>>>> >>>>>>> Big +1, this is a step in the right direction and checking
>>>>> with other Beam's direct and transitive deps is crucial since the referred
>>>>> bom only convers a small part of it. Apache Commons, Jackson,
>>>>> `com.google.{api, apis, cloud}`, slf4j comes to mind.
>>>>> >>>>>>>
>>>>> >>>>>>> Filipe Regadas
>>>>> >>>>>>>
>>>>> >>>>>>>
>>>>> >>>>>>> On Thu, Mar 5, 2020 at 3:33 AM Ismaël Mejía <ie...@gmail.com>
>>>>> wrote:
>>>>> >>>>>>>>
>>>>> >>>>>>>> +1 Sounds like a good improvement for users and maintainers !
>>>>> >>>>>>>>
>>>>> >>>>>>>> On Thu, Mar 5, 2020 at 6:59 AM Alex Van Boxel <
>>>>> alex@vanboxel.be> wrote:
>>>>> >>>>>>>> >
>>>>> >>>>>>>> > +1, I can remember the countless hours that we fought with
>>>>> Google dependencies.
>>>>> >>>>>>>> >
>>>>> >>>>>>>> > On Thu, Mar 5, 2020, 04:07 Chamikara Jayalath <
>>>>> chamikara@google.com> wrote:
>>>>> >>>>>>>> >>
>>>>> >>>>>>>> >> +1 for this.
>>>>> >>>>>>>> >>
>>>>> >>>>>>>> >> This will make life easy for many of our users and will
>>>>> help us keep GCP related dependencies compatible (which has not been easy
>>>>> in the past).
>>>>> >>>>>>>> >>
>>>>> >>>>>>>> >> On Wed, Mar 4, 2020 at 2:16 PM Tomo Suzuki <
>>>>> suztomo@google.com> wrote:
>>>>> >>>>>>>> >>>
>>>>> >>>>>>>> >>> Hi Beam developers,
>>>>> >>>>>>>> >>>
>>>>> >>>>>>>> >>> Shall we use GCP Libraries BOM [1] to specify the
>>>>> Google-related library versions in Beam?
>>>>> >>>>>>>> >>>
>>>>> >>>>>>>> >>> I've been working on Beam's dependency upgrades in the
>>>>> past few months. It's time to consider a long-term solution to keep the
>>>>> libraries up-to-date with small maintenance effort. To achieve that, I
>>>>> propose Beam to use GCP Libraries BOM to set the Google-related library
>>>>> versions, rather than the current way of making changes in each of ~30
>>>>> Google libraries with individual PRs [2].
>>>>> >>>>>>>> >>>
>>>>> >>>>>>>> >>> After the proposal is implemented, Beam project upgrades
>>>>> the BOM version to upgrade these Google-related libraries. This still needs
>>>>> to ensure the libraries in GCP Library BOM are compatible with Beam's other
>>>>> dependencies. (Linkage Checker will help with this job.) I believe
>>>>> onboarding GCP Libraries BOM will solve lots of incompatibilities which we
>>>>> have seen in gax, gRPC, google-cloud-core, and so on with minimal effort in
>>>>> Beam's developers.
>>>>> >>>>>>>> >>>
>>>>> >>>>>>>> >>> Created an issue to track this: BEAM-9444 [3]. I
>>>>> appreciate if you can share questions or feedback (thumbs-up / concerns).
>>>>> >>>>>>>> >>>
>>>>> >>>>>>>> >>> [1]:
>>>>> https://github.com/GoogleCloudPlatform/cloud-opensource-java/wiki/The-Google-Cloud-Platform-Libraries-BOM
>>>>> >>>>>>>> >>> [2]:
>>>>> https://github.com/apache/beam/pulls?page=1&q=is%3Apr+author%3Asuztomo
>>>>> >>>>>>>> >>> [3]: https://issues.apache.org/jira/browse/BEAM-9444
>>>>> >>>>>>>> >>>
>>>>> >>>>>>>> >>> --
>>>>> >>>>>>>> >>> Regards,
>>>>> >>>>>>>> >>> Tomo
>>>>> >>>>
>>>>> >>>>
>>>>> >>>>
>>>>> >>>> --
>>>>> >>>> Regards,
>>>>> >>>> Tomo
>>>>> >>>
>>>>> >>>
>>>>> >>>
>>>>> >>> --
>>>>> >>> Regards,
>>>>> >>> Tomo
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >> --
>>>>> >> Regards,
>>>>> >> Tomo
>>>>>
>>>>
>>
>> --
>> Regards,
>> Tomo
>>
>
>
> --
> Regards,
> Tomo
>
--
Regards,
Tomo
Re: Proposal: Beam to use GCP Libraries BOM
Posted by Tomo Suzuki <su...@google.com>.
Hi Chamikara,
Ahmet is asking for your input in
https://github.com/apache/beam/pull/11156#issuecomment-602716275 . Would
you check this?
On Mon, Mar 23, 2020 at 3:05 PM Tomo Suzuki <su...@google.com> wrote:
> (Applied Ahmet's feedback in https://github.com/apache/beam/pull/11156)
>
> Hi Luke,
>
> > the pom files that are produced
>
> Build.gradle that uses the BOM will produce pom files that have
> corresponding dependencies without versions, and have a
> "dependencyManagement" section through which they import the GCP Libraries
> BOM. This fills the versions of the versionless dependencies when build
> systems evaluates the pom files.
>
> > how the GCP BOM impacts the release process
>
> I'm afraid I don't have knowledge on how dependencies affect the Beam
> release process. Would you be willing to break down this concern into some
> questions I can answer?
>
>
>
>
> On Fri, Mar 20, 2020 at 6:13 PM Luke Cwik <lc...@google.com> wrote:
>
>> My concern would be to validate how the GCP BOM impacts the release
>> process and the pom files that are produced otherwise the next person
>> running a release may run into trouble.
>>
>> On Fri, Mar 20, 2020 at 3:00 PM Pablo Estrada <pa...@google.com> wrote:
>>
>>> +1
>>>
>>> On Fri, Mar 20, 2020 at 2:41 PM Ismaël Mejía <ie...@gmail.com> wrote:
>>>
>>>> Well why we just don't merge it? I am unfamiliar with GCP deps to be
>>>> confident to LGTM it. but given that 22 checks pass and given that
>>>> Tomo addressed most comments and he has already a consistent track of
>>>> good work on dependency improvements I think it is worth to merge it
>>>> as it is. We still have some time to fix stuff if we find any
>>>> regression. WDYT?
>>>>
>>>> On Thu, Mar 19, 2020 at 9:36 PM Luke Cwik <lc...@google.com> wrote:
>>>> >
>>>> > As much as I would like to spend time on these reviews, I believe
>>>> I'll be delayed from reviewing them thoroughly till I finish other work
>>>> that I'm targeting for the 2.21 release related to portability. It would be
>>>> greatly appreciated if there are others that could review this in the
>>>> meantime.
>>>> >
>>>> > On Thu, Mar 19, 2020 at 7:09 AM Tomo Suzuki <su...@google.com>
>>>> wrote:
>>>> >>
>>>> >> PR is ready (22 successful check)
>>>> >> https://github.com/apache/beam/pull/11156
>>>> >> (Luke assigned himself as a reviewer)
>>>> >>
>>>> >> Regards,
>>>> >> Tomo
>>>> >>
>>>> >> On Wed, Mar 11, 2020 at 3:50 PM Tomo Suzuki <su...@google.com>
>>>> wrote:
>>>> >>>
>>>> >>> Thank you for favorable responses. I'll start implementation.
>>>> >>>
>>>> >>> On Thu, Mar 5, 2020 at 2:22 PM Tomo Suzuki <su...@google.com>
>>>> wrote:
>>>> >>>>
>>>> >>>> > Do Spark or Flink have BOMs?
>>>> >>>>
>>>> >>>> Not that I know of. I couldn't find "bom" in their artifacts [1,
>>>> 2].
>>>> >>>>
>>>> >>>> [1]: https://search.maven.org/search?q=g:org.apache.flink
>>>> >>>> [2]: https://search.maven.org/search?q=g:org.apache.spark
>>>> >>>>
>>>> >>>>
>>>> >>>> On Thu, Mar 5, 2020 at 1:46 PM Kenneth Knowles <ke...@apache.org>
>>>> wrote:
>>>> >>>>>
>>>> >>>>> +1 and you have phrased the benefits and limitations well. We
>>>> have plenty of not-Google-related dependencies that use Guava and protobuf
>>>> (I know of Calcite, Cassandra, Kinesis, and Spark) so there's still work in
>>>> managing deps, but the BOM should make it a lot easier to upgrade all these
>>>> tightly coupled libraries that Google ships from their head.
>>>> >>>>>
>>>> >>>>> Do Spark or Flink have BOMs? I wonder if there's an opportunity
>>>> to catch incompatible deps at a larger scale by comparing and merging a
>>>> half dozen BOMs (although in the limit it approximately expands to one per
>>>> runner and one per IO and projects mature and become independent)
>>>> >>>>>
>>>> >>>>> Kenn
>>>> >>>>>
>>>> >>>>> On Thu, Mar 5, 2020 at 10:05 AM Luke Cwik <lc...@google.com>
>>>> wrote:
>>>> >>>>>>
>>>> >>>>>> How would the Apache Beam BOM and GCP BOM work together?
>>>> >>>>>>
>>>> >>>>>> On Thu, Mar 5, 2020 at 7:25 AM Filipe Regadas <
>>>> filiperegadas@gmail.com> wrote:
>>>> >>>>>>>
>>>> >>>>>>> Big +1, this is a step in the right direction and checking with
>>>> other Beam's direct and transitive deps is crucial since the referred bom
>>>> only convers a small part of it. Apache Commons, Jackson, `com.google.{api,
>>>> apis, cloud}`, slf4j comes to mind.
>>>> >>>>>>>
>>>> >>>>>>> Filipe Regadas
>>>> >>>>>>>
>>>> >>>>>>>
>>>> >>>>>>> On Thu, Mar 5, 2020 at 3:33 AM Ismaël Mejía <ie...@gmail.com>
>>>> wrote:
>>>> >>>>>>>>
>>>> >>>>>>>> +1 Sounds like a good improvement for users and maintainers !
>>>> >>>>>>>>
>>>> >>>>>>>> On Thu, Mar 5, 2020 at 6:59 AM Alex Van Boxel <
>>>> alex@vanboxel.be> wrote:
>>>> >>>>>>>> >
>>>> >>>>>>>> > +1, I can remember the countless hours that we fought with
>>>> Google dependencies.
>>>> >>>>>>>> >
>>>> >>>>>>>> > On Thu, Mar 5, 2020, 04:07 Chamikara Jayalath <
>>>> chamikara@google.com> wrote:
>>>> >>>>>>>> >>
>>>> >>>>>>>> >> +1 for this.
>>>> >>>>>>>> >>
>>>> >>>>>>>> >> This will make life easy for many of our users and will
>>>> help us keep GCP related dependencies compatible (which has not been easy
>>>> in the past).
>>>> >>>>>>>> >>
>>>> >>>>>>>> >> On Wed, Mar 4, 2020 at 2:16 PM Tomo Suzuki <
>>>> suztomo@google.com> wrote:
>>>> >>>>>>>> >>>
>>>> >>>>>>>> >>> Hi Beam developers,
>>>> >>>>>>>> >>>
>>>> >>>>>>>> >>> Shall we use GCP Libraries BOM [1] to specify the
>>>> Google-related library versions in Beam?
>>>> >>>>>>>> >>>
>>>> >>>>>>>> >>> I've been working on Beam's dependency upgrades in the
>>>> past few months. It's time to consider a long-term solution to keep the
>>>> libraries up-to-date with small maintenance effort. To achieve that, I
>>>> propose Beam to use GCP Libraries BOM to set the Google-related library
>>>> versions, rather than the current way of making changes in each of ~30
>>>> Google libraries with individual PRs [2].
>>>> >>>>>>>> >>>
>>>> >>>>>>>> >>> After the proposal is implemented, Beam project upgrades
>>>> the BOM version to upgrade these Google-related libraries. This still needs
>>>> to ensure the libraries in GCP Library BOM are compatible with Beam's other
>>>> dependencies. (Linkage Checker will help with this job.) I believe
>>>> onboarding GCP Libraries BOM will solve lots of incompatibilities which we
>>>> have seen in gax, gRPC, google-cloud-core, and so on with minimal effort in
>>>> Beam's developers.
>>>> >>>>>>>> >>>
>>>> >>>>>>>> >>> Created an issue to track this: BEAM-9444 [3]. I
>>>> appreciate if you can share questions or feedback (thumbs-up / concerns).
>>>> >>>>>>>> >>>
>>>> >>>>>>>> >>> [1]:
>>>> https://github.com/GoogleCloudPlatform/cloud-opensource-java/wiki/The-Google-Cloud-Platform-Libraries-BOM
>>>> >>>>>>>> >>> [2]:
>>>> https://github.com/apache/beam/pulls?page=1&q=is%3Apr+author%3Asuztomo
>>>> >>>>>>>> >>> [3]: https://issues.apache.org/jira/browse/BEAM-9444
>>>> >>>>>>>> >>>
>>>> >>>>>>>> >>> --
>>>> >>>>>>>> >>> Regards,
>>>> >>>>>>>> >>> Tomo
>>>> >>>>
>>>> >>>>
>>>> >>>>
>>>> >>>> --
>>>> >>>> Regards,
>>>> >>>> Tomo
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> --
>>>> >>> Regards,
>>>> >>> Tomo
>>>> >>
>>>> >>
>>>> >>
>>>> >> --
>>>> >> Regards,
>>>> >> Tomo
>>>>
>>>
>
> --
> Regards,
> Tomo
>
--
Regards,
Tomo
Re: Proposal: Beam to use GCP Libraries BOM
Posted by Tomo Suzuki <su...@google.com>.
(Applied Ahmet's feedback in https://github.com/apache/beam/pull/11156)
Hi Luke,
> the pom files that are produced
Build.gradle that uses the BOM will produce pom files that have
corresponding dependencies without versions, and have a
"dependencyManagement" section through which they import the GCP Libraries
BOM. This fills the versions of the versionless dependencies when build
systems evaluates the pom files.
> how the GCP BOM impacts the release process
I'm afraid I don't have knowledge on how dependencies affect the Beam
release process. Would you be willing to break down this concern into some
questions I can answer?
On Fri, Mar 20, 2020 at 6:13 PM Luke Cwik <lc...@google.com> wrote:
> My concern would be to validate how the GCP BOM impacts the release
> process and the pom files that are produced otherwise the next person
> running a release may run into trouble.
>
> On Fri, Mar 20, 2020 at 3:00 PM Pablo Estrada <pa...@google.com> wrote:
>
>> +1
>>
>> On Fri, Mar 20, 2020 at 2:41 PM Ismaël Mejía <ie...@gmail.com> wrote:
>>
>>> Well why we just don't merge it? I am unfamiliar with GCP deps to be
>>> confident to LGTM it. but given that 22 checks pass and given that
>>> Tomo addressed most comments and he has already a consistent track of
>>> good work on dependency improvements I think it is worth to merge it
>>> as it is. We still have some time to fix stuff if we find any
>>> regression. WDYT?
>>>
>>> On Thu, Mar 19, 2020 at 9:36 PM Luke Cwik <lc...@google.com> wrote:
>>> >
>>> > As much as I would like to spend time on these reviews, I believe I'll
>>> be delayed from reviewing them thoroughly till I finish other work that I'm
>>> targeting for the 2.21 release related to portability. It would be greatly
>>> appreciated if there are others that could review this in the meantime.
>>> >
>>> > On Thu, Mar 19, 2020 at 7:09 AM Tomo Suzuki <su...@google.com>
>>> wrote:
>>> >>
>>> >> PR is ready (22 successful check)
>>> >> https://github.com/apache/beam/pull/11156
>>> >> (Luke assigned himself as a reviewer)
>>> >>
>>> >> Regards,
>>> >> Tomo
>>> >>
>>> >> On Wed, Mar 11, 2020 at 3:50 PM Tomo Suzuki <su...@google.com>
>>> wrote:
>>> >>>
>>> >>> Thank you for favorable responses. I'll start implementation.
>>> >>>
>>> >>> On Thu, Mar 5, 2020 at 2:22 PM Tomo Suzuki <su...@google.com>
>>> wrote:
>>> >>>>
>>> >>>> > Do Spark or Flink have BOMs?
>>> >>>>
>>> >>>> Not that I know of. I couldn't find "bom" in their artifacts [1, 2].
>>> >>>>
>>> >>>> [1]: https://search.maven.org/search?q=g:org.apache.flink
>>> >>>> [2]: https://search.maven.org/search?q=g:org.apache.spark
>>> >>>>
>>> >>>>
>>> >>>> On Thu, Mar 5, 2020 at 1:46 PM Kenneth Knowles <ke...@apache.org>
>>> wrote:
>>> >>>>>
>>> >>>>> +1 and you have phrased the benefits and limitations well. We have
>>> plenty of not-Google-related dependencies that use Guava and protobuf (I
>>> know of Calcite, Cassandra, Kinesis, and Spark) so there's still work in
>>> managing deps, but the BOM should make it a lot easier to upgrade all these
>>> tightly coupled libraries that Google ships from their head.
>>> >>>>>
>>> >>>>> Do Spark or Flink have BOMs? I wonder if there's an opportunity to
>>> catch incompatible deps at a larger scale by comparing and merging a half
>>> dozen BOMs (although in the limit it approximately expands to one per
>>> runner and one per IO and projects mature and become independent)
>>> >>>>>
>>> >>>>> Kenn
>>> >>>>>
>>> >>>>> On Thu, Mar 5, 2020 at 10:05 AM Luke Cwik <lc...@google.com>
>>> wrote:
>>> >>>>>>
>>> >>>>>> How would the Apache Beam BOM and GCP BOM work together?
>>> >>>>>>
>>> >>>>>> On Thu, Mar 5, 2020 at 7:25 AM Filipe Regadas <
>>> filiperegadas@gmail.com> wrote:
>>> >>>>>>>
>>> >>>>>>> Big +1, this is a step in the right direction and checking with
>>> other Beam's direct and transitive deps is crucial since the referred bom
>>> only convers a small part of it. Apache Commons, Jackson, `com.google.{api,
>>> apis, cloud}`, slf4j comes to mind.
>>> >>>>>>>
>>> >>>>>>> Filipe Regadas
>>> >>>>>>>
>>> >>>>>>>
>>> >>>>>>> On Thu, Mar 5, 2020 at 3:33 AM Ismaël Mejía <ie...@gmail.com>
>>> wrote:
>>> >>>>>>>>
>>> >>>>>>>> +1 Sounds like a good improvement for users and maintainers !
>>> >>>>>>>>
>>> >>>>>>>> On Thu, Mar 5, 2020 at 6:59 AM Alex Van Boxel <al...@vanboxel.be>
>>> wrote:
>>> >>>>>>>> >
>>> >>>>>>>> > +1, I can remember the countless hours that we fought with
>>> Google dependencies.
>>> >>>>>>>> >
>>> >>>>>>>> > On Thu, Mar 5, 2020, 04:07 Chamikara Jayalath <
>>> chamikara@google.com> wrote:
>>> >>>>>>>> >>
>>> >>>>>>>> >> +1 for this.
>>> >>>>>>>> >>
>>> >>>>>>>> >> This will make life easy for many of our users and will help
>>> us keep GCP related dependencies compatible (which has not been easy in the
>>> past).
>>> >>>>>>>> >>
>>> >>>>>>>> >> On Wed, Mar 4, 2020 at 2:16 PM Tomo Suzuki <
>>> suztomo@google.com> wrote:
>>> >>>>>>>> >>>
>>> >>>>>>>> >>> Hi Beam developers,
>>> >>>>>>>> >>>
>>> >>>>>>>> >>> Shall we use GCP Libraries BOM [1] to specify the
>>> Google-related library versions in Beam?
>>> >>>>>>>> >>>
>>> >>>>>>>> >>> I've been working on Beam's dependency upgrades in the past
>>> few months. It's time to consider a long-term solution to keep the
>>> libraries up-to-date with small maintenance effort. To achieve that, I
>>> propose Beam to use GCP Libraries BOM to set the Google-related library
>>> versions, rather than the current way of making changes in each of ~30
>>> Google libraries with individual PRs [2].
>>> >>>>>>>> >>>
>>> >>>>>>>> >>> After the proposal is implemented, Beam project upgrades
>>> the BOM version to upgrade these Google-related libraries. This still needs
>>> to ensure the libraries in GCP Library BOM are compatible with Beam's other
>>> dependencies. (Linkage Checker will help with this job.) I believe
>>> onboarding GCP Libraries BOM will solve lots of incompatibilities which we
>>> have seen in gax, gRPC, google-cloud-core, and so on with minimal effort in
>>> Beam's developers.
>>> >>>>>>>> >>>
>>> >>>>>>>> >>> Created an issue to track this: BEAM-9444 [3]. I appreciate
>>> if you can share questions or feedback (thumbs-up / concerns).
>>> >>>>>>>> >>>
>>> >>>>>>>> >>> [1]:
>>> https://github.com/GoogleCloudPlatform/cloud-opensource-java/wiki/The-Google-Cloud-Platform-Libraries-BOM
>>> >>>>>>>> >>> [2]:
>>> https://github.com/apache/beam/pulls?page=1&q=is%3Apr+author%3Asuztomo
>>> >>>>>>>> >>> [3]: https://issues.apache.org/jira/browse/BEAM-9444
>>> >>>>>>>> >>>
>>> >>>>>>>> >>> --
>>> >>>>>>>> >>> Regards,
>>> >>>>>>>> >>> Tomo
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> --
>>> >>>> Regards,
>>> >>>> Tomo
>>> >>>
>>> >>>
>>> >>>
>>> >>> --
>>> >>> Regards,
>>> >>> Tomo
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> Regards,
>>> >> Tomo
>>>
>>
--
Regards,
Tomo
Re: Proposal: Beam to use GCP Libraries BOM
Posted by Luke Cwik <lc...@google.com>.
My concern would be to validate how the GCP BOM impacts the release process
and the pom files that are produced otherwise the next person running a
release may run into trouble.
On Fri, Mar 20, 2020 at 3:00 PM Pablo Estrada <pa...@google.com> wrote:
> +1
>
> On Fri, Mar 20, 2020 at 2:41 PM Ismaël Mejía <ie...@gmail.com> wrote:
>
>> Well why we just don't merge it? I am unfamiliar with GCP deps to be
>> confident to LGTM it. but given that 22 checks pass and given that
>> Tomo addressed most comments and he has already a consistent track of
>> good work on dependency improvements I think it is worth to merge it
>> as it is. We still have some time to fix stuff if we find any
>> regression. WDYT?
>>
>> On Thu, Mar 19, 2020 at 9:36 PM Luke Cwik <lc...@google.com> wrote:
>> >
>> > As much as I would like to spend time on these reviews, I believe I'll
>> be delayed from reviewing them thoroughly till I finish other work that I'm
>> targeting for the 2.21 release related to portability. It would be greatly
>> appreciated if there are others that could review this in the meantime.
>> >
>> > On Thu, Mar 19, 2020 at 7:09 AM Tomo Suzuki <su...@google.com> wrote:
>> >>
>> >> PR is ready (22 successful check)
>> >> https://github.com/apache/beam/pull/11156
>> >> (Luke assigned himself as a reviewer)
>> >>
>> >> Regards,
>> >> Tomo
>> >>
>> >> On Wed, Mar 11, 2020 at 3:50 PM Tomo Suzuki <su...@google.com>
>> wrote:
>> >>>
>> >>> Thank you for favorable responses. I'll start implementation.
>> >>>
>> >>> On Thu, Mar 5, 2020 at 2:22 PM Tomo Suzuki <su...@google.com>
>> wrote:
>> >>>>
>> >>>> > Do Spark or Flink have BOMs?
>> >>>>
>> >>>> Not that I know of. I couldn't find "bom" in their artifacts [1, 2].
>> >>>>
>> >>>> [1]: https://search.maven.org/search?q=g:org.apache.flink
>> >>>> [2]: https://search.maven.org/search?q=g:org.apache.spark
>> >>>>
>> >>>>
>> >>>> On Thu, Mar 5, 2020 at 1:46 PM Kenneth Knowles <ke...@apache.org>
>> wrote:
>> >>>>>
>> >>>>> +1 and you have phrased the benefits and limitations well. We have
>> plenty of not-Google-related dependencies that use Guava and protobuf (I
>> know of Calcite, Cassandra, Kinesis, and Spark) so there's still work in
>> managing deps, but the BOM should make it a lot easier to upgrade all these
>> tightly coupled libraries that Google ships from their head.
>> >>>>>
>> >>>>> Do Spark or Flink have BOMs? I wonder if there's an opportunity to
>> catch incompatible deps at a larger scale by comparing and merging a half
>> dozen BOMs (although in the limit it approximately expands to one per
>> runner and one per IO and projects mature and become independent)
>> >>>>>
>> >>>>> Kenn
>> >>>>>
>> >>>>> On Thu, Mar 5, 2020 at 10:05 AM Luke Cwik <lc...@google.com> wrote:
>> >>>>>>
>> >>>>>> How would the Apache Beam BOM and GCP BOM work together?
>> >>>>>>
>> >>>>>> On Thu, Mar 5, 2020 at 7:25 AM Filipe Regadas <
>> filiperegadas@gmail.com> wrote:
>> >>>>>>>
>> >>>>>>> Big +1, this is a step in the right direction and checking with
>> other Beam's direct and transitive deps is crucial since the referred bom
>> only convers a small part of it. Apache Commons, Jackson, `com.google.{api,
>> apis, cloud}`, slf4j comes to mind.
>> >>>>>>>
>> >>>>>>> Filipe Regadas
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> On Thu, Mar 5, 2020 at 3:33 AM Ismaël Mejía <ie...@gmail.com>
>> wrote:
>> >>>>>>>>
>> >>>>>>>> +1 Sounds like a good improvement for users and maintainers !
>> >>>>>>>>
>> >>>>>>>> On Thu, Mar 5, 2020 at 6:59 AM Alex Van Boxel <al...@vanboxel.be>
>> wrote:
>> >>>>>>>> >
>> >>>>>>>> > +1, I can remember the countless hours that we fought with
>> Google dependencies.
>> >>>>>>>> >
>> >>>>>>>> > On Thu, Mar 5, 2020, 04:07 Chamikara Jayalath <
>> chamikara@google.com> wrote:
>> >>>>>>>> >>
>> >>>>>>>> >> +1 for this.
>> >>>>>>>> >>
>> >>>>>>>> >> This will make life easy for many of our users and will help
>> us keep GCP related dependencies compatible (which has not been easy in the
>> past).
>> >>>>>>>> >>
>> >>>>>>>> >> On Wed, Mar 4, 2020 at 2:16 PM Tomo Suzuki <
>> suztomo@google.com> wrote:
>> >>>>>>>> >>>
>> >>>>>>>> >>> Hi Beam developers,
>> >>>>>>>> >>>
>> >>>>>>>> >>> Shall we use GCP Libraries BOM [1] to specify the
>> Google-related library versions in Beam?
>> >>>>>>>> >>>
>> >>>>>>>> >>> I've been working on Beam's dependency upgrades in the past
>> few months. It's time to consider a long-term solution to keep the
>> libraries up-to-date with small maintenance effort. To achieve that, I
>> propose Beam to use GCP Libraries BOM to set the Google-related library
>> versions, rather than the current way of making changes in each of ~30
>> Google libraries with individual PRs [2].
>> >>>>>>>> >>>
>> >>>>>>>> >>> After the proposal is implemented, Beam project upgrades the
>> BOM version to upgrade these Google-related libraries. This still needs to
>> ensure the libraries in GCP Library BOM are compatible with Beam's other
>> dependencies. (Linkage Checker will help with this job.) I believe
>> onboarding GCP Libraries BOM will solve lots of incompatibilities which we
>> have seen in gax, gRPC, google-cloud-core, and so on with minimal effort in
>> Beam's developers.
>> >>>>>>>> >>>
>> >>>>>>>> >>> Created an issue to track this: BEAM-9444 [3]. I appreciate
>> if you can share questions or feedback (thumbs-up / concerns).
>> >>>>>>>> >>>
>> >>>>>>>> >>> [1]:
>> https://github.com/GoogleCloudPlatform/cloud-opensource-java/wiki/The-Google-Cloud-Platform-Libraries-BOM
>> >>>>>>>> >>> [2]:
>> https://github.com/apache/beam/pulls?page=1&q=is%3Apr+author%3Asuztomo
>> >>>>>>>> >>> [3]: https://issues.apache.org/jira/browse/BEAM-9444
>> >>>>>>>> >>>
>> >>>>>>>> >>> --
>> >>>>>>>> >>> Regards,
>> >>>>>>>> >>> Tomo
>> >>>>
>> >>>>
>> >>>>
>> >>>> --
>> >>>> Regards,
>> >>>> Tomo
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> Regards,
>> >>> Tomo
>> >>
>> >>
>> >>
>> >> --
>> >> Regards,
>> >> Tomo
>>
>
Re: Proposal: Beam to use GCP Libraries BOM
Posted by Pablo Estrada <pa...@google.com>.
+1
On Fri, Mar 20, 2020 at 2:41 PM Ismaël Mejía <ie...@gmail.com> wrote:
> Well why we just don't merge it? I am unfamiliar with GCP deps to be
> confident to LGTM it. but given that 22 checks pass and given that
> Tomo addressed most comments and he has already a consistent track of
> good work on dependency improvements I think it is worth to merge it
> as it is. We still have some time to fix stuff if we find any
> regression. WDYT?
>
> On Thu, Mar 19, 2020 at 9:36 PM Luke Cwik <lc...@google.com> wrote:
> >
> > As much as I would like to spend time on these reviews, I believe I'll
> be delayed from reviewing them thoroughly till I finish other work that I'm
> targeting for the 2.21 release related to portability. It would be greatly
> appreciated if there are others that could review this in the meantime.
> >
> > On Thu, Mar 19, 2020 at 7:09 AM Tomo Suzuki <su...@google.com> wrote:
> >>
> >> PR is ready (22 successful check)
> >> https://github.com/apache/beam/pull/11156
> >> (Luke assigned himself as a reviewer)
> >>
> >> Regards,
> >> Tomo
> >>
> >> On Wed, Mar 11, 2020 at 3:50 PM Tomo Suzuki <su...@google.com> wrote:
> >>>
> >>> Thank you for favorable responses. I'll start implementation.
> >>>
> >>> On Thu, Mar 5, 2020 at 2:22 PM Tomo Suzuki <su...@google.com> wrote:
> >>>>
> >>>> > Do Spark or Flink have BOMs?
> >>>>
> >>>> Not that I know of. I couldn't find "bom" in their artifacts [1, 2].
> >>>>
> >>>> [1]: https://search.maven.org/search?q=g:org.apache.flink
> >>>> [2]: https://search.maven.org/search?q=g:org.apache.spark
> >>>>
> >>>>
> >>>> On Thu, Mar 5, 2020 at 1:46 PM Kenneth Knowles <ke...@apache.org>
> wrote:
> >>>>>
> >>>>> +1 and you have phrased the benefits and limitations well. We have
> plenty of not-Google-related dependencies that use Guava and protobuf (I
> know of Calcite, Cassandra, Kinesis, and Spark) so there's still work in
> managing deps, but the BOM should make it a lot easier to upgrade all these
> tightly coupled libraries that Google ships from their head.
> >>>>>
> >>>>> Do Spark or Flink have BOMs? I wonder if there's an opportunity to
> catch incompatible deps at a larger scale by comparing and merging a half
> dozen BOMs (although in the limit it approximately expands to one per
> runner and one per IO and projects mature and become independent)
> >>>>>
> >>>>> Kenn
> >>>>>
> >>>>> On Thu, Mar 5, 2020 at 10:05 AM Luke Cwik <lc...@google.com> wrote:
> >>>>>>
> >>>>>> How would the Apache Beam BOM and GCP BOM work together?
> >>>>>>
> >>>>>> On Thu, Mar 5, 2020 at 7:25 AM Filipe Regadas <
> filiperegadas@gmail.com> wrote:
> >>>>>>>
> >>>>>>> Big +1, this is a step in the right direction and checking with
> other Beam's direct and transitive deps is crucial since the referred bom
> only convers a small part of it. Apache Commons, Jackson, `com.google.{api,
> apis, cloud}`, slf4j comes to mind.
> >>>>>>>
> >>>>>>> Filipe Regadas
> >>>>>>>
> >>>>>>>
> >>>>>>> On Thu, Mar 5, 2020 at 3:33 AM Ismaël Mejía <ie...@gmail.com>
> wrote:
> >>>>>>>>
> >>>>>>>> +1 Sounds like a good improvement for users and maintainers !
> >>>>>>>>
> >>>>>>>> On Thu, Mar 5, 2020 at 6:59 AM Alex Van Boxel <al...@vanboxel.be>
> wrote:
> >>>>>>>> >
> >>>>>>>> > +1, I can remember the countless hours that we fought with
> Google dependencies.
> >>>>>>>> >
> >>>>>>>> > On Thu, Mar 5, 2020, 04:07 Chamikara Jayalath <
> chamikara@google.com> wrote:
> >>>>>>>> >>
> >>>>>>>> >> +1 for this.
> >>>>>>>> >>
> >>>>>>>> >> This will make life easy for many of our users and will help
> us keep GCP related dependencies compatible (which has not been easy in the
> past).
> >>>>>>>> >>
> >>>>>>>> >> On Wed, Mar 4, 2020 at 2:16 PM Tomo Suzuki <su...@google.com>
> wrote:
> >>>>>>>> >>>
> >>>>>>>> >>> Hi Beam developers,
> >>>>>>>> >>>
> >>>>>>>> >>> Shall we use GCP Libraries BOM [1] to specify the
> Google-related library versions in Beam?
> >>>>>>>> >>>
> >>>>>>>> >>> I've been working on Beam's dependency upgrades in the past
> few months. It's time to consider a long-term solution to keep the
> libraries up-to-date with small maintenance effort. To achieve that, I
> propose Beam to use GCP Libraries BOM to set the Google-related library
> versions, rather than the current way of making changes in each of ~30
> Google libraries with individual PRs [2].
> >>>>>>>> >>>
> >>>>>>>> >>> After the proposal is implemented, Beam project upgrades the
> BOM version to upgrade these Google-related libraries. This still needs to
> ensure the libraries in GCP Library BOM are compatible with Beam's other
> dependencies. (Linkage Checker will help with this job.) I believe
> onboarding GCP Libraries BOM will solve lots of incompatibilities which we
> have seen in gax, gRPC, google-cloud-core, and so on with minimal effort in
> Beam's developers.
> >>>>>>>> >>>
> >>>>>>>> >>> Created an issue to track this: BEAM-9444 [3]. I appreciate
> if you can share questions or feedback (thumbs-up / concerns).
> >>>>>>>> >>>
> >>>>>>>> >>> [1]:
> https://github.com/GoogleCloudPlatform/cloud-opensource-java/wiki/The-Google-Cloud-Platform-Libraries-BOM
> >>>>>>>> >>> [2]:
> https://github.com/apache/beam/pulls?page=1&q=is%3Apr+author%3Asuztomo
> >>>>>>>> >>> [3]: https://issues.apache.org/jira/browse/BEAM-9444
> >>>>>>>> >>>
> >>>>>>>> >>> --
> >>>>>>>> >>> Regards,
> >>>>>>>> >>> Tomo
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> Regards,
> >>>> Tomo
> >>>
> >>>
> >>>
> >>> --
> >>> Regards,
> >>> Tomo
> >>
> >>
> >>
> >> --
> >> Regards,
> >> Tomo
>
Re: Proposal: Beam to use GCP Libraries BOM
Posted by Ismaël Mejía <ie...@gmail.com>.
Well why we just don't merge it? I am unfamiliar with GCP deps to be
confident to LGTM it. but given that 22 checks pass and given that
Tomo addressed most comments and he has already a consistent track of
good work on dependency improvements I think it is worth to merge it
as it is. We still have some time to fix stuff if we find any
regression. WDYT?
On Thu, Mar 19, 2020 at 9:36 PM Luke Cwik <lc...@google.com> wrote:
>
> As much as I would like to spend time on these reviews, I believe I'll be delayed from reviewing them thoroughly till I finish other work that I'm targeting for the 2.21 release related to portability. It would be greatly appreciated if there are others that could review this in the meantime.
>
> On Thu, Mar 19, 2020 at 7:09 AM Tomo Suzuki <su...@google.com> wrote:
>>
>> PR is ready (22 successful check)
>> https://github.com/apache/beam/pull/11156
>> (Luke assigned himself as a reviewer)
>>
>> Regards,
>> Tomo
>>
>> On Wed, Mar 11, 2020 at 3:50 PM Tomo Suzuki <su...@google.com> wrote:
>>>
>>> Thank you for favorable responses. I'll start implementation.
>>>
>>> On Thu, Mar 5, 2020 at 2:22 PM Tomo Suzuki <su...@google.com> wrote:
>>>>
>>>> > Do Spark or Flink have BOMs?
>>>>
>>>> Not that I know of. I couldn't find "bom" in their artifacts [1, 2].
>>>>
>>>> [1]: https://search.maven.org/search?q=g:org.apache.flink
>>>> [2]: https://search.maven.org/search?q=g:org.apache.spark
>>>>
>>>>
>>>> On Thu, Mar 5, 2020 at 1:46 PM Kenneth Knowles <ke...@apache.org> wrote:
>>>>>
>>>>> +1 and you have phrased the benefits and limitations well. We have plenty of not-Google-related dependencies that use Guava and protobuf (I know of Calcite, Cassandra, Kinesis, and Spark) so there's still work in managing deps, but the BOM should make it a lot easier to upgrade all these tightly coupled libraries that Google ships from their head.
>>>>>
>>>>> Do Spark or Flink have BOMs? I wonder if there's an opportunity to catch incompatible deps at a larger scale by comparing and merging a half dozen BOMs (although in the limit it approximately expands to one per runner and one per IO and projects mature and become independent)
>>>>>
>>>>> Kenn
>>>>>
>>>>> On Thu, Mar 5, 2020 at 10:05 AM Luke Cwik <lc...@google.com> wrote:
>>>>>>
>>>>>> How would the Apache Beam BOM and GCP BOM work together?
>>>>>>
>>>>>> On Thu, Mar 5, 2020 at 7:25 AM Filipe Regadas <fi...@gmail.com> wrote:
>>>>>>>
>>>>>>> Big +1, this is a step in the right direction and checking with other Beam's direct and transitive deps is crucial since the referred bom only convers a small part of it. Apache Commons, Jackson, `com.google.{api, apis, cloud}`, slf4j comes to mind.
>>>>>>>
>>>>>>> Filipe Regadas
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Mar 5, 2020 at 3:33 AM Ismaël Mejía <ie...@gmail.com> wrote:
>>>>>>>>
>>>>>>>> +1 Sounds like a good improvement for users and maintainers !
>>>>>>>>
>>>>>>>> On Thu, Mar 5, 2020 at 6:59 AM Alex Van Boxel <al...@vanboxel.be> wrote:
>>>>>>>> >
>>>>>>>> > +1, I can remember the countless hours that we fought with Google dependencies.
>>>>>>>> >
>>>>>>>> > On Thu, Mar 5, 2020, 04:07 Chamikara Jayalath <ch...@google.com> wrote:
>>>>>>>> >>
>>>>>>>> >> +1 for this.
>>>>>>>> >>
>>>>>>>> >> This will make life easy for many of our users and will help us keep GCP related dependencies compatible (which has not been easy in the past).
>>>>>>>> >>
>>>>>>>> >> On Wed, Mar 4, 2020 at 2:16 PM Tomo Suzuki <su...@google.com> wrote:
>>>>>>>> >>>
>>>>>>>> >>> Hi Beam developers,
>>>>>>>> >>>
>>>>>>>> >>> Shall we use GCP Libraries BOM [1] to specify the Google-related library versions in Beam?
>>>>>>>> >>>
>>>>>>>> >>> I've been working on Beam's dependency upgrades in the past few months. It's time to consider a long-term solution to keep the libraries up-to-date with small maintenance effort. To achieve that, I propose Beam to use GCP Libraries BOM to set the Google-related library versions, rather than the current way of making changes in each of ~30 Google libraries with individual PRs [2].
>>>>>>>> >>>
>>>>>>>> >>> After the proposal is implemented, Beam project upgrades the BOM version to upgrade these Google-related libraries. This still needs to ensure the libraries in GCP Library BOM are compatible with Beam's other dependencies. (Linkage Checker will help with this job.) I believe onboarding GCP Libraries BOM will solve lots of incompatibilities which we have seen in gax, gRPC, google-cloud-core, and so on with minimal effort in Beam's developers.
>>>>>>>> >>>
>>>>>>>> >>> Created an issue to track this: BEAM-9444 [3]. I appreciate if you can share questions or feedback (thumbs-up / concerns).
>>>>>>>> >>>
>>>>>>>> >>> [1]: https://github.com/GoogleCloudPlatform/cloud-opensource-java/wiki/The-Google-Cloud-Platform-Libraries-BOM
>>>>>>>> >>> [2]: https://github.com/apache/beam/pulls?page=1&q=is%3Apr+author%3Asuztomo
>>>>>>>> >>> [3]: https://issues.apache.org/jira/browse/BEAM-9444
>>>>>>>> >>>
>>>>>>>> >>> --
>>>>>>>> >>> Regards,
>>>>>>>> >>> Tomo
>>>>
>>>>
>>>>
>>>> --
>>>> Regards,
>>>> Tomo
>>>
>>>
>>>
>>> --
>>> Regards,
>>> Tomo
>>
>>
>>
>> --
>> Regards,
>> Tomo
Re: Proposal: Beam to use GCP Libraries BOM
Posted by Luke Cwik <lc...@google.com>.
As much as I would like to spend time on these reviews, I believe I'll be
delayed from reviewing them thoroughly till I finish other work that I'm
targeting for the 2.21 release related to portability. It would be greatly
appreciated if there are others that could review this in the meantime.
On Thu, Mar 19, 2020 at 7:09 AM Tomo Suzuki <su...@google.com> wrote:
> PR is ready (22 successful check)
> https://github.com/apache/beam/pull/11156
> (Luke assigned himself as a reviewer)
>
> Regards,
> Tomo
>
> On Wed, Mar 11, 2020 at 3:50 PM Tomo Suzuki <su...@google.com> wrote:
>
>> Thank you for favorable responses. I'll start implementation.
>>
>> On Thu, Mar 5, 2020 at 2:22 PM Tomo Suzuki <su...@google.com> wrote:
>>
>>> > Do Spark or Flink have BOMs?
>>>
>>> Not that I know of. I couldn't find "bom" in their artifacts [1, 2].
>>>
>>> [1]: https://search.maven.org/search?q=g:org.apache.flink
>>> [2]: https://search.maven.org/search?q=g:org.apache.spark
>>>
>>>
>>> On Thu, Mar 5, 2020 at 1:46 PM Kenneth Knowles <ke...@apache.org> wrote:
>>>
>>>> +1 and you have phrased the benefits and limitations well. We have
>>>> plenty of not-Google-related dependencies that use Guava and protobuf (I
>>>> know of Calcite, Cassandra, Kinesis, and Spark) so there's still work in
>>>> managing deps, but the BOM should make it a lot easier to upgrade all these
>>>> tightly coupled libraries that Google ships from their head.
>>>>
>>>> Do Spark or Flink have BOMs? I wonder if there's an opportunity to
>>>> catch incompatible deps at a larger scale by comparing and merging a half
>>>> dozen BOMs (although in the limit it approximately expands to one per
>>>> runner and one per IO and projects mature and become independent)
>>>>
>>>> Kenn
>>>>
>>>> On Thu, Mar 5, 2020 at 10:05 AM Luke Cwik <lc...@google.com> wrote:
>>>>
>>>>> How would the Apache Beam BOM and GCP BOM work together?
>>>>>
>>>>> On Thu, Mar 5, 2020 at 7:25 AM Filipe Regadas <fi...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Big +1, this is a step in the right direction and checking with other
>>>>>> Beam's direct and transitive deps is crucial since the referred bom only
>>>>>> convers a small part of it. Apache Commons, Jackson, `com.google.{api,
>>>>>> apis, cloud}`, slf4j comes to mind.
>>>>>>
>>>>>> Filipe Regadas
>>>>>>
>>>>>>
>>>>>> On Thu, Mar 5, 2020 at 3:33 AM Ismaël Mejía <ie...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> +1 Sounds like a good improvement for users and maintainers !
>>>>>>>
>>>>>>> On Thu, Mar 5, 2020 at 6:59 AM Alex Van Boxel <al...@vanboxel.be>
>>>>>>> wrote:
>>>>>>> >
>>>>>>> > +1, I can remember the countless hours that we fought with Google
>>>>>>> dependencies.
>>>>>>> >
>>>>>>> > On Thu, Mar 5, 2020, 04:07 Chamikara Jayalath <
>>>>>>> chamikara@google.com> wrote:
>>>>>>> >>
>>>>>>> >> +1 for this.
>>>>>>> >>
>>>>>>> >> This will make life easy for many of our users and will help us
>>>>>>> keep GCP related dependencies compatible (which has not been easy in the
>>>>>>> past).
>>>>>>> >>
>>>>>>> >> On Wed, Mar 4, 2020 at 2:16 PM Tomo Suzuki <su...@google.com>
>>>>>>> wrote:
>>>>>>> >>>
>>>>>>> >>> Hi Beam developers,
>>>>>>> >>>
>>>>>>> >>> Shall we use GCP Libraries BOM [1] to specify the Google-related
>>>>>>> library versions in Beam?
>>>>>>> >>>
>>>>>>> >>> I've been working on Beam's dependency upgrades in the past few
>>>>>>> months. It's time to consider a long-term solution to keep the libraries
>>>>>>> up-to-date with small maintenance effort. To achieve that, I propose Beam
>>>>>>> to use GCP Libraries BOM to set the Google-related library versions, rather
>>>>>>> than the current way of making changes in each of ~30 Google libraries with
>>>>>>> individual PRs [2].
>>>>>>> >>>
>>>>>>> >>> After the proposal is implemented, Beam project upgrades the BOM
>>>>>>> version to upgrade these Google-related libraries. This still needs to
>>>>>>> ensure the libraries in GCP Library BOM are compatible with Beam's other
>>>>>>> dependencies. (Linkage Checker will help with this job.) I believe
>>>>>>> onboarding GCP Libraries BOM will solve lots of incompatibilities which we
>>>>>>> have seen in gax, gRPC, google-cloud-core, and so on with minimal effort in
>>>>>>> Beam's developers.
>>>>>>> >>>
>>>>>>> >>> Created an issue to track this: BEAM-9444 [3]. I appreciate if
>>>>>>> you can share questions or feedback (thumbs-up / concerns).
>>>>>>> >>>
>>>>>>> >>> [1]:
>>>>>>> https://github.com/GoogleCloudPlatform/cloud-opensource-java/wiki/The-Google-Cloud-Platform-Libraries-BOM
>>>>>>> >>> [2]:
>>>>>>> https://github.com/apache/beam/pulls?page=1&q=is%3Apr+author%3Asuztomo
>>>>>>> >>> [3]: https://issues.apache.org/jira/browse/BEAM-9444
>>>>>>> >>>
>>>>>>> >>> --
>>>>>>> >>> Regards,
>>>>>>> >>> Tomo
>>>>>>>
>>>>>>
>>>
>>> --
>>> Regards,
>>> Tomo
>>>
>>
>>
>> --
>> Regards,
>> Tomo
>>
>
>
> --
> Regards,
> Tomo
>
Re: Proposal: Beam to use GCP Libraries BOM
Posted by Tomo Suzuki <su...@google.com>.
PR is ready (22 successful check)
https://github.com/apache/beam/pull/11156
(Luke assigned himself as a reviewer)
Regards,
Tomo
On Wed, Mar 11, 2020 at 3:50 PM Tomo Suzuki <su...@google.com> wrote:
> Thank you for favorable responses. I'll start implementation.
>
> On Thu, Mar 5, 2020 at 2:22 PM Tomo Suzuki <su...@google.com> wrote:
>
>> > Do Spark or Flink have BOMs?
>>
>> Not that I know of. I couldn't find "bom" in their artifacts [1, 2].
>>
>> [1]: https://search.maven.org/search?q=g:org.apache.flink
>> [2]: https://search.maven.org/search?q=g:org.apache.spark
>>
>>
>> On Thu, Mar 5, 2020 at 1:46 PM Kenneth Knowles <ke...@apache.org> wrote:
>>
>>> +1 and you have phrased the benefits and limitations well. We have
>>> plenty of not-Google-related dependencies that use Guava and protobuf (I
>>> know of Calcite, Cassandra, Kinesis, and Spark) so there's still work in
>>> managing deps, but the BOM should make it a lot easier to upgrade all these
>>> tightly coupled libraries that Google ships from their head.
>>>
>>> Do Spark or Flink have BOMs? I wonder if there's an opportunity to catch
>>> incompatible deps at a larger scale by comparing and merging a half dozen
>>> BOMs (although in the limit it approximately expands to one per runner and
>>> one per IO and projects mature and become independent)
>>>
>>> Kenn
>>>
>>> On Thu, Mar 5, 2020 at 10:05 AM Luke Cwik <lc...@google.com> wrote:
>>>
>>>> How would the Apache Beam BOM and GCP BOM work together?
>>>>
>>>> On Thu, Mar 5, 2020 at 7:25 AM Filipe Regadas <fi...@gmail.com>
>>>> wrote:
>>>>
>>>>> Big +1, this is a step in the right direction and checking with other
>>>>> Beam's direct and transitive deps is crucial since the referred bom only
>>>>> convers a small part of it. Apache Commons, Jackson, `com.google.{api,
>>>>> apis, cloud}`, slf4j comes to mind.
>>>>>
>>>>> Filipe Regadas
>>>>>
>>>>>
>>>>> On Thu, Mar 5, 2020 at 3:33 AM Ismaël Mejía <ie...@gmail.com> wrote:
>>>>>
>>>>>> +1 Sounds like a good improvement for users and maintainers !
>>>>>>
>>>>>> On Thu, Mar 5, 2020 at 6:59 AM Alex Van Boxel <al...@vanboxel.be>
>>>>>> wrote:
>>>>>> >
>>>>>> > +1, I can remember the countless hours that we fought with Google
>>>>>> dependencies.
>>>>>> >
>>>>>> > On Thu, Mar 5, 2020, 04:07 Chamikara Jayalath <ch...@google.com>
>>>>>> wrote:
>>>>>> >>
>>>>>> >> +1 for this.
>>>>>> >>
>>>>>> >> This will make life easy for many of our users and will help us
>>>>>> keep GCP related dependencies compatible (which has not been easy in the
>>>>>> past).
>>>>>> >>
>>>>>> >> On Wed, Mar 4, 2020 at 2:16 PM Tomo Suzuki <su...@google.com>
>>>>>> wrote:
>>>>>> >>>
>>>>>> >>> Hi Beam developers,
>>>>>> >>>
>>>>>> >>> Shall we use GCP Libraries BOM [1] to specify the Google-related
>>>>>> library versions in Beam?
>>>>>> >>>
>>>>>> >>> I've been working on Beam's dependency upgrades in the past few
>>>>>> months. It's time to consider a long-term solution to keep the libraries
>>>>>> up-to-date with small maintenance effort. To achieve that, I propose Beam
>>>>>> to use GCP Libraries BOM to set the Google-related library versions, rather
>>>>>> than the current way of making changes in each of ~30 Google libraries with
>>>>>> individual PRs [2].
>>>>>> >>>
>>>>>> >>> After the proposal is implemented, Beam project upgrades the BOM
>>>>>> version to upgrade these Google-related libraries. This still needs to
>>>>>> ensure the libraries in GCP Library BOM are compatible with Beam's other
>>>>>> dependencies. (Linkage Checker will help with this job.) I believe
>>>>>> onboarding GCP Libraries BOM will solve lots of incompatibilities which we
>>>>>> have seen in gax, gRPC, google-cloud-core, and so on with minimal effort in
>>>>>> Beam's developers.
>>>>>> >>>
>>>>>> >>> Created an issue to track this: BEAM-9444 [3]. I appreciate if
>>>>>> you can share questions or feedback (thumbs-up / concerns).
>>>>>> >>>
>>>>>> >>> [1]:
>>>>>> https://github.com/GoogleCloudPlatform/cloud-opensource-java/wiki/The-Google-Cloud-Platform-Libraries-BOM
>>>>>> >>> [2]:
>>>>>> https://github.com/apache/beam/pulls?page=1&q=is%3Apr+author%3Asuztomo
>>>>>> >>> [3]: https://issues.apache.org/jira/browse/BEAM-9444
>>>>>> >>>
>>>>>> >>> --
>>>>>> >>> Regards,
>>>>>> >>> Tomo
>>>>>>
>>>>>
>>
>> --
>> Regards,
>> Tomo
>>
>
>
> --
> Regards,
> Tomo
>
--
Regards,
Tomo
Re: Proposal: Beam to use GCP Libraries BOM
Posted by Tomo Suzuki <su...@google.com>.
Thank you for favorable responses. I'll start implementation.
On Thu, Mar 5, 2020 at 2:22 PM Tomo Suzuki <su...@google.com> wrote:
> > Do Spark or Flink have BOMs?
>
> Not that I know of. I couldn't find "bom" in their artifacts [1, 2].
>
> [1]: https://search.maven.org/search?q=g:org.apache.flink
> [2]: https://search.maven.org/search?q=g:org.apache.spark
>
>
> On Thu, Mar 5, 2020 at 1:46 PM Kenneth Knowles <ke...@apache.org> wrote:
>
>> +1 and you have phrased the benefits and limitations well. We have plenty
>> of not-Google-related dependencies that use Guava and protobuf (I know of
>> Calcite, Cassandra, Kinesis, and Spark) so there's still work in managing
>> deps, but the BOM should make it a lot easier to upgrade all these tightly
>> coupled libraries that Google ships from their head.
>>
>> Do Spark or Flink have BOMs? I wonder if there's an opportunity to catch
>> incompatible deps at a larger scale by comparing and merging a half dozen
>> BOMs (although in the limit it approximately expands to one per runner and
>> one per IO and projects mature and become independent)
>>
>> Kenn
>>
>> On Thu, Mar 5, 2020 at 10:05 AM Luke Cwik <lc...@google.com> wrote:
>>
>>> How would the Apache Beam BOM and GCP BOM work together?
>>>
>>> On Thu, Mar 5, 2020 at 7:25 AM Filipe Regadas <fi...@gmail.com>
>>> wrote:
>>>
>>>> Big +1, this is a step in the right direction and checking with other
>>>> Beam's direct and transitive deps is crucial since the referred bom only
>>>> convers a small part of it. Apache Commons, Jackson, `com.google.{api,
>>>> apis, cloud}`, slf4j comes to mind.
>>>>
>>>> Filipe Regadas
>>>>
>>>>
>>>> On Thu, Mar 5, 2020 at 3:33 AM Ismaël Mejía <ie...@gmail.com> wrote:
>>>>
>>>>> +1 Sounds like a good improvement for users and maintainers !
>>>>>
>>>>> On Thu, Mar 5, 2020 at 6:59 AM Alex Van Boxel <al...@vanboxel.be>
>>>>> wrote:
>>>>> >
>>>>> > +1, I can remember the countless hours that we fought with Google
>>>>> dependencies.
>>>>> >
>>>>> > On Thu, Mar 5, 2020, 04:07 Chamikara Jayalath <ch...@google.com>
>>>>> wrote:
>>>>> >>
>>>>> >> +1 for this.
>>>>> >>
>>>>> >> This will make life easy for many of our users and will help us
>>>>> keep GCP related dependencies compatible (which has not been easy in the
>>>>> past).
>>>>> >>
>>>>> >> On Wed, Mar 4, 2020 at 2:16 PM Tomo Suzuki <su...@google.com>
>>>>> wrote:
>>>>> >>>
>>>>> >>> Hi Beam developers,
>>>>> >>>
>>>>> >>> Shall we use GCP Libraries BOM [1] to specify the Google-related
>>>>> library versions in Beam?
>>>>> >>>
>>>>> >>> I've been working on Beam's dependency upgrades in the past few
>>>>> months. It's time to consider a long-term solution to keep the libraries
>>>>> up-to-date with small maintenance effort. To achieve that, I propose Beam
>>>>> to use GCP Libraries BOM to set the Google-related library versions, rather
>>>>> than the current way of making changes in each of ~30 Google libraries with
>>>>> individual PRs [2].
>>>>> >>>
>>>>> >>> After the proposal is implemented, Beam project upgrades the BOM
>>>>> version to upgrade these Google-related libraries. This still needs to
>>>>> ensure the libraries in GCP Library BOM are compatible with Beam's other
>>>>> dependencies. (Linkage Checker will help with this job.) I believe
>>>>> onboarding GCP Libraries BOM will solve lots of incompatibilities which we
>>>>> have seen in gax, gRPC, google-cloud-core, and so on with minimal effort in
>>>>> Beam's developers.
>>>>> >>>
>>>>> >>> Created an issue to track this: BEAM-9444 [3]. I appreciate if you
>>>>> can share questions or feedback (thumbs-up / concerns).
>>>>> >>>
>>>>> >>> [1]:
>>>>> https://github.com/GoogleCloudPlatform/cloud-opensource-java/wiki/The-Google-Cloud-Platform-Libraries-BOM
>>>>> >>> [2]:
>>>>> https://github.com/apache/beam/pulls?page=1&q=is%3Apr+author%3Asuztomo
>>>>> >>> [3]: https://issues.apache.org/jira/browse/BEAM-9444
>>>>> >>>
>>>>> >>> --
>>>>> >>> Regards,
>>>>> >>> Tomo
>>>>>
>>>>
>
> --
> Regards,
> Tomo
>
--
Regards,
Tomo
Re: Proposal: Beam to use GCP Libraries BOM
Posted by Tomo Suzuki <su...@google.com>.
> Do Spark or Flink have BOMs?
Not that I know of. I couldn't find "bom" in their artifacts [1, 2].
[1]: https://search.maven.org/search?q=g:org.apache.flink
[2]: https://search.maven.org/search?q=g:org.apache.spark
On Thu, Mar 5, 2020 at 1:46 PM Kenneth Knowles <ke...@apache.org> wrote:
> +1 and you have phrased the benefits and limitations well. We have plenty
> of not-Google-related dependencies that use Guava and protobuf (I know of
> Calcite, Cassandra, Kinesis, and Spark) so there's still work in managing
> deps, but the BOM should make it a lot easier to upgrade all these tightly
> coupled libraries that Google ships from their head.
>
> Do Spark or Flink have BOMs? I wonder if there's an opportunity to catch
> incompatible deps at a larger scale by comparing and merging a half dozen
> BOMs (although in the limit it approximately expands to one per runner and
> one per IO and projects mature and become independent)
>
> Kenn
>
> On Thu, Mar 5, 2020 at 10:05 AM Luke Cwik <lc...@google.com> wrote:
>
>> How would the Apache Beam BOM and GCP BOM work together?
>>
>> On Thu, Mar 5, 2020 at 7:25 AM Filipe Regadas <fi...@gmail.com>
>> wrote:
>>
>>> Big +1, this is a step in the right direction and checking with other
>>> Beam's direct and transitive deps is crucial since the referred bom only
>>> convers a small part of it. Apache Commons, Jackson, `com.google.{api,
>>> apis, cloud}`, slf4j comes to mind.
>>>
>>> Filipe Regadas
>>>
>>>
>>> On Thu, Mar 5, 2020 at 3:33 AM Ismaël Mejía <ie...@gmail.com> wrote:
>>>
>>>> +1 Sounds like a good improvement for users and maintainers !
>>>>
>>>> On Thu, Mar 5, 2020 at 6:59 AM Alex Van Boxel <al...@vanboxel.be> wrote:
>>>> >
>>>> > +1, I can remember the countless hours that we fought with Google
>>>> dependencies.
>>>> >
>>>> > On Thu, Mar 5, 2020, 04:07 Chamikara Jayalath <ch...@google.com>
>>>> wrote:
>>>> >>
>>>> >> +1 for this.
>>>> >>
>>>> >> This will make life easy for many of our users and will help us keep
>>>> GCP related dependencies compatible (which has not been easy in the past).
>>>> >>
>>>> >> On Wed, Mar 4, 2020 at 2:16 PM Tomo Suzuki <su...@google.com>
>>>> wrote:
>>>> >>>
>>>> >>> Hi Beam developers,
>>>> >>>
>>>> >>> Shall we use GCP Libraries BOM [1] to specify the Google-related
>>>> library versions in Beam?
>>>> >>>
>>>> >>> I've been working on Beam's dependency upgrades in the past few
>>>> months. It's time to consider a long-term solution to keep the libraries
>>>> up-to-date with small maintenance effort. To achieve that, I propose Beam
>>>> to use GCP Libraries BOM to set the Google-related library versions, rather
>>>> than the current way of making changes in each of ~30 Google libraries with
>>>> individual PRs [2].
>>>> >>>
>>>> >>> After the proposal is implemented, Beam project upgrades the BOM
>>>> version to upgrade these Google-related libraries. This still needs to
>>>> ensure the libraries in GCP Library BOM are compatible with Beam's other
>>>> dependencies. (Linkage Checker will help with this job.) I believe
>>>> onboarding GCP Libraries BOM will solve lots of incompatibilities which we
>>>> have seen in gax, gRPC, google-cloud-core, and so on with minimal effort in
>>>> Beam's developers.
>>>> >>>
>>>> >>> Created an issue to track this: BEAM-9444 [3]. I appreciate if you
>>>> can share questions or feedback (thumbs-up / concerns).
>>>> >>>
>>>> >>> [1]:
>>>> https://github.com/GoogleCloudPlatform/cloud-opensource-java/wiki/The-Google-Cloud-Platform-Libraries-BOM
>>>> >>> [2]:
>>>> https://github.com/apache/beam/pulls?page=1&q=is%3Apr+author%3Asuztomo
>>>> >>> [3]: https://issues.apache.org/jira/browse/BEAM-9444
>>>> >>>
>>>> >>> --
>>>> >>> Regards,
>>>> >>> Tomo
>>>>
>>>
--
Regards,
Tomo
Re: Proposal: Beam to use GCP Libraries BOM
Posted by Kenneth Knowles <ke...@apache.org>.
+1 and you have phrased the benefits and limitations well. We have plenty
of not-Google-related dependencies that use Guava and protobuf (I know of
Calcite, Cassandra, Kinesis, and Spark) so there's still work in managing
deps, but the BOM should make it a lot easier to upgrade all these tightly
coupled libraries that Google ships from their head.
Do Spark or Flink have BOMs? I wonder if there's an opportunity to catch
incompatible deps at a larger scale by comparing and merging a half dozen
BOMs (although in the limit it approximately expands to one per runner and
one per IO and projects mature and become independent)
Kenn
On Thu, Mar 5, 2020 at 10:05 AM Luke Cwik <lc...@google.com> wrote:
> How would the Apache Beam BOM and GCP BOM work together?
>
> On Thu, Mar 5, 2020 at 7:25 AM Filipe Regadas <fi...@gmail.com>
> wrote:
>
>> Big +1, this is a step in the right direction and checking with other
>> Beam's direct and transitive deps is crucial since the referred bom only
>> convers a small part of it. Apache Commons, Jackson, `com.google.{api,
>> apis, cloud}`, slf4j comes to mind.
>>
>> Filipe Regadas
>>
>>
>> On Thu, Mar 5, 2020 at 3:33 AM Ismaël Mejía <ie...@gmail.com> wrote:
>>
>>> +1 Sounds like a good improvement for users and maintainers !
>>>
>>> On Thu, Mar 5, 2020 at 6:59 AM Alex Van Boxel <al...@vanboxel.be> wrote:
>>> >
>>> > +1, I can remember the countless hours that we fought with Google
>>> dependencies.
>>> >
>>> > On Thu, Mar 5, 2020, 04:07 Chamikara Jayalath <ch...@google.com>
>>> wrote:
>>> >>
>>> >> +1 for this.
>>> >>
>>> >> This will make life easy for many of our users and will help us keep
>>> GCP related dependencies compatible (which has not been easy in the past).
>>> >>
>>> >> On Wed, Mar 4, 2020 at 2:16 PM Tomo Suzuki <su...@google.com>
>>> wrote:
>>> >>>
>>> >>> Hi Beam developers,
>>> >>>
>>> >>> Shall we use GCP Libraries BOM [1] to specify the Google-related
>>> library versions in Beam?
>>> >>>
>>> >>> I've been working on Beam's dependency upgrades in the past few
>>> months. It's time to consider a long-term solution to keep the libraries
>>> up-to-date with small maintenance effort. To achieve that, I propose Beam
>>> to use GCP Libraries BOM to set the Google-related library versions, rather
>>> than the current way of making changes in each of ~30 Google libraries with
>>> individual PRs [2].
>>> >>>
>>> >>> After the proposal is implemented, Beam project upgrades the BOM
>>> version to upgrade these Google-related libraries. This still needs to
>>> ensure the libraries in GCP Library BOM are compatible with Beam's other
>>> dependencies. (Linkage Checker will help with this job.) I believe
>>> onboarding GCP Libraries BOM will solve lots of incompatibilities which we
>>> have seen in gax, gRPC, google-cloud-core, and so on with minimal effort in
>>> Beam's developers.
>>> >>>
>>> >>> Created an issue to track this: BEAM-9444 [3]. I appreciate if you
>>> can share questions or feedback (thumbs-up / concerns).
>>> >>>
>>> >>> [1]:
>>> https://github.com/GoogleCloudPlatform/cloud-opensource-java/wiki/The-Google-Cloud-Platform-Libraries-BOM
>>> >>> [2]:
>>> https://github.com/apache/beam/pulls?page=1&q=is%3Apr+author%3Asuztomo
>>> >>> [3]: https://issues.apache.org/jira/browse/BEAM-9444
>>> >>>
>>> >>> --
>>> >>> Regards,
>>> >>> Tomo
>>>
>>
Re: Proposal: Beam to use GCP Libraries BOM
Posted by Luke Cwik <lc...@google.com>.
How would the Apache Beam BOM and GCP BOM work together?
On Thu, Mar 5, 2020 at 7:25 AM Filipe Regadas <fi...@gmail.com>
wrote:
> Big +1, this is a step in the right direction and checking with other
> Beam's direct and transitive deps is crucial since the referred bom only
> convers a small part of it. Apache Commons, Jackson, `com.google.{api,
> apis, cloud}`, slf4j comes to mind.
>
> Filipe Regadas
>
>
> On Thu, Mar 5, 2020 at 3:33 AM Ismaël Mejía <ie...@gmail.com> wrote:
>
>> +1 Sounds like a good improvement for users and maintainers !
>>
>> On Thu, Mar 5, 2020 at 6:59 AM Alex Van Boxel <al...@vanboxel.be> wrote:
>> >
>> > +1, I can remember the countless hours that we fought with Google
>> dependencies.
>> >
>> > On Thu, Mar 5, 2020, 04:07 Chamikara Jayalath <ch...@google.com>
>> wrote:
>> >>
>> >> +1 for this.
>> >>
>> >> This will make life easy for many of our users and will help us keep
>> GCP related dependencies compatible (which has not been easy in the past).
>> >>
>> >> On Wed, Mar 4, 2020 at 2:16 PM Tomo Suzuki <su...@google.com> wrote:
>> >>>
>> >>> Hi Beam developers,
>> >>>
>> >>> Shall we use GCP Libraries BOM [1] to specify the Google-related
>> library versions in Beam?
>> >>>
>> >>> I've been working on Beam's dependency upgrades in the past few
>> months. It's time to consider a long-term solution to keep the libraries
>> up-to-date with small maintenance effort. To achieve that, I propose Beam
>> to use GCP Libraries BOM to set the Google-related library versions, rather
>> than the current way of making changes in each of ~30 Google libraries with
>> individual PRs [2].
>> >>>
>> >>> After the proposal is implemented, Beam project upgrades the BOM
>> version to upgrade these Google-related libraries. This still needs to
>> ensure the libraries in GCP Library BOM are compatible with Beam's other
>> dependencies. (Linkage Checker will help with this job.) I believe
>> onboarding GCP Libraries BOM will solve lots of incompatibilities which we
>> have seen in gax, gRPC, google-cloud-core, and so on with minimal effort in
>> Beam's developers.
>> >>>
>> >>> Created an issue to track this: BEAM-9444 [3]. I appreciate if you
>> can share questions or feedback (thumbs-up / concerns).
>> >>>
>> >>> [1]:
>> https://github.com/GoogleCloudPlatform/cloud-opensource-java/wiki/The-Google-Cloud-Platform-Libraries-BOM
>> >>> [2]:
>> https://github.com/apache/beam/pulls?page=1&q=is%3Apr+author%3Asuztomo
>> >>> [3]: https://issues.apache.org/jira/browse/BEAM-9444
>> >>>
>> >>> --
>> >>> Regards,
>> >>> Tomo
>>
>
Re: Proposal: Beam to use GCP Libraries BOM
Posted by Filipe Regadas <fi...@gmail.com>.
Big +1, this is a step in the right direction and checking with other
Beam's direct and transitive deps is crucial since the referred bom only
convers a small part of it. Apache Commons, Jackson, `com.google.{api,
apis, cloud}`, slf4j comes to mind.
Filipe Regadas
On Thu, Mar 5, 2020 at 3:33 AM Ismaël Mejía <ie...@gmail.com> wrote:
> +1 Sounds like a good improvement for users and maintainers !
>
> On Thu, Mar 5, 2020 at 6:59 AM Alex Van Boxel <al...@vanboxel.be> wrote:
> >
> > +1, I can remember the countless hours that we fought with Google
> dependencies.
> >
> > On Thu, Mar 5, 2020, 04:07 Chamikara Jayalath <ch...@google.com>
> wrote:
> >>
> >> +1 for this.
> >>
> >> This will make life easy for many of our users and will help us keep
> GCP related dependencies compatible (which has not been easy in the past).
> >>
> >> On Wed, Mar 4, 2020 at 2:16 PM Tomo Suzuki <su...@google.com> wrote:
> >>>
> >>> Hi Beam developers,
> >>>
> >>> Shall we use GCP Libraries BOM [1] to specify the Google-related
> library versions in Beam?
> >>>
> >>> I've been working on Beam's dependency upgrades in the past few
> months. It's time to consider a long-term solution to keep the libraries
> up-to-date with small maintenance effort. To achieve that, I propose Beam
> to use GCP Libraries BOM to set the Google-related library versions, rather
> than the current way of making changes in each of ~30 Google libraries with
> individual PRs [2].
> >>>
> >>> After the proposal is implemented, Beam project upgrades the BOM
> version to upgrade these Google-related libraries. This still needs to
> ensure the libraries in GCP Library BOM are compatible with Beam's other
> dependencies. (Linkage Checker will help with this job.) I believe
> onboarding GCP Libraries BOM will solve lots of incompatibilities which we
> have seen in gax, gRPC, google-cloud-core, and so on with minimal effort in
> Beam's developers.
> >>>
> >>> Created an issue to track this: BEAM-9444 [3]. I appreciate if you can
> share questions or feedback (thumbs-up / concerns).
> >>>
> >>> [1]:
> https://github.com/GoogleCloudPlatform/cloud-opensource-java/wiki/The-Google-Cloud-Platform-Libraries-BOM
> >>> [2]:
> https://github.com/apache/beam/pulls?page=1&q=is%3Apr+author%3Asuztomo
> >>> [3]: https://issues.apache.org/jira/browse/BEAM-9444
> >>>
> >>> --
> >>> Regards,
> >>> Tomo
>
Re: Proposal: Beam to use GCP Libraries BOM
Posted by Ismaël Mejía <ie...@gmail.com>.
+1 Sounds like a good improvement for users and maintainers !
On Thu, Mar 5, 2020 at 6:59 AM Alex Van Boxel <al...@vanboxel.be> wrote:
>
> +1, I can remember the countless hours that we fought with Google dependencies.
>
> On Thu, Mar 5, 2020, 04:07 Chamikara Jayalath <ch...@google.com> wrote:
>>
>> +1 for this.
>>
>> This will make life easy for many of our users and will help us keep GCP related dependencies compatible (which has not been easy in the past).
>>
>> On Wed, Mar 4, 2020 at 2:16 PM Tomo Suzuki <su...@google.com> wrote:
>>>
>>> Hi Beam developers,
>>>
>>> Shall we use GCP Libraries BOM [1] to specify the Google-related library versions in Beam?
>>>
>>> I've been working on Beam's dependency upgrades in the past few months. It's time to consider a long-term solution to keep the libraries up-to-date with small maintenance effort. To achieve that, I propose Beam to use GCP Libraries BOM to set the Google-related library versions, rather than the current way of making changes in each of ~30 Google libraries with individual PRs [2].
>>>
>>> After the proposal is implemented, Beam project upgrades the BOM version to upgrade these Google-related libraries. This still needs to ensure the libraries in GCP Library BOM are compatible with Beam's other dependencies. (Linkage Checker will help with this job.) I believe onboarding GCP Libraries BOM will solve lots of incompatibilities which we have seen in gax, gRPC, google-cloud-core, and so on with minimal effort in Beam's developers.
>>>
>>> Created an issue to track this: BEAM-9444 [3]. I appreciate if you can share questions or feedback (thumbs-up / concerns).
>>>
>>> [1]: https://github.com/GoogleCloudPlatform/cloud-opensource-java/wiki/The-Google-Cloud-Platform-Libraries-BOM
>>> [2]: https://github.com/apache/beam/pulls?page=1&q=is%3Apr+author%3Asuztomo
>>> [3]: https://issues.apache.org/jira/browse/BEAM-9444
>>>
>>> --
>>> Regards,
>>> Tomo
Re: Proposal: Beam to use GCP Libraries BOM
Posted by Alex Van Boxel <al...@vanboxel.be>.
+1, I can remember the countless hours that we fought with Google
dependencies.
On Thu, Mar 5, 2020, 04:07 Chamikara Jayalath <ch...@google.com> wrote:
> +1 for this.
>
> This will make life easy for many of our users and will help us keep GCP
> related dependencies compatible (which has not been easy in the past).
>
> On Wed, Mar 4, 2020 at 2:16 PM Tomo Suzuki <su...@google.com> wrote:
>
>> Hi Beam developers,
>>
>> Shall we use GCP Libraries BOM [1] to specify the Google-related library
>> versions in Beam?
>>
>> I've been working on Beam's dependency upgrades in the past few months.
>> It's time to consider a long-term solution to keep the libraries up-to-date
>> with small maintenance effort. To achieve that, I propose Beam to use GCP
>> Libraries BOM to set the Google-related library versions, rather than the
>> current way of making changes in each of ~30 Google libraries with
>> individual PRs [2].
>>
>> After the proposal is implemented, Beam project upgrades the BOM version
>> to upgrade these Google-related libraries. This still needs to ensure the
>> libraries in GCP Library BOM are compatible with Beam's other dependencies.
>> (Linkage Checker will help with this job.) I believe onboarding GCP
>> Libraries BOM will solve lots of incompatibilities which we have seen in
>> gax, gRPC, google-cloud-core, and so on with minimal effort in Beam's
>> developers.
>>
>> Created an issue to track this: BEAM-9444 [3]. I appreciate if you can
>> share questions or feedback (thumbs-up / concerns).
>>
>> [1]:
>> https://github.com/GoogleCloudPlatform/cloud-opensource-java/wiki/The-Google-Cloud-Platform-Libraries-BOM
>> [2]:
>> https://github.com/apache/beam/pulls?page=1&q=is%3Apr+author%3Asuztomo
>> [3]: https://issues.apache.org/jira/browse/BEAM-9444
>>
>> --
>> Regards,
>> Tomo
>>
>
Re: Proposal: Beam to use GCP Libraries BOM
Posted by Chamikara Jayalath <ch...@google.com>.
+1 for this.
This will make life easy for many of our users and will help us keep GCP
related dependencies compatible (which has not been easy in the past).
On Wed, Mar 4, 2020 at 2:16 PM Tomo Suzuki <su...@google.com> wrote:
> Hi Beam developers,
>
> Shall we use GCP Libraries BOM [1] to specify the Google-related library
> versions in Beam?
>
> I've been working on Beam's dependency upgrades in the past few months.
> It's time to consider a long-term solution to keep the libraries up-to-date
> with small maintenance effort. To achieve that, I propose Beam to use GCP
> Libraries BOM to set the Google-related library versions, rather than the
> current way of making changes in each of ~30 Google libraries with
> individual PRs [2].
>
> After the proposal is implemented, Beam project upgrades the BOM version
> to upgrade these Google-related libraries. This still needs to ensure the
> libraries in GCP Library BOM are compatible with Beam's other dependencies.
> (Linkage Checker will help with this job.) I believe onboarding GCP
> Libraries BOM will solve lots of incompatibilities which we have seen in
> gax, gRPC, google-cloud-core, and so on with minimal effort in Beam's
> developers.
>
> Created an issue to track this: BEAM-9444 [3]. I appreciate if you can
> share questions or feedback (thumbs-up / concerns).
>
> [1]:
> https://github.com/GoogleCloudPlatform/cloud-opensource-java/wiki/The-Google-Cloud-Platform-Libraries-BOM
> [2]:
> https://github.com/apache/beam/pulls?page=1&q=is%3Apr+author%3Asuztomo
> [3]: https://issues.apache.org/jira/browse/BEAM-9444
>
> --
> Regards,
> Tomo
>