You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Tomo Suzuki <su...@google.com> on 2019/11/11 18:14:29 UTC
Completeness of Beam Java Dependency Check Report
Hi Beam developers,
(I'm thinking to contribute to upgrades of Java dependencies of Beam; I
just read https://beam.apache.org/contribute/dependencies/)
As per the weekly report, Apache Beam Java SDK only has 8 outdated
dependencies based on the criteria. However, it seems many others are not
up-to-date. For example Guava 20.0 used in Beam
<https://github.com/apache/beam/blob/d692d2f/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L375>
is not the latest major release.
Why do some outdated dependencies not appear in this report?
Regards,
Tomo
On Mon, Nov 11, 2019 at 7:05 AM Apache Jenkins Server <
jenkins@builds.apache.org> wrote:
> High Priority Dependency Updates Of Beam Python SDK:
> *Dependency Name* *Current Version* *Latest Version* *Release Date Of the
> Current Used Version* *Release Date Of The Latest Release* *JIRA Issue*
> mock <https://pypi.org/project/mock> 2.0.0 3.0.5 2019-05-20 2019-05-20
> BEAM-7369 <https://issues.apache.org/jira/browse/BEAM-7369>
> oauth2client <https://pypi.org/project/oauth2client> 3.0.0 4.1.3
> 2018-12-10 2018-12-10 BEAM-6089
> <https://issues.apache.org/jira/browse/BEAM-6089>
> Sphinx <https://pypi.org/project/Sphinx> 1.8.5 2.2.1 2019-05-20 2019-10-28
> BEAM-7370 <https://issues.apache.org/jira/browse/BEAM-7370> High Priority
> Dependency Updates Of Beam Java SDK:
> *Dependency Name* *Current Version* *Latest Version* *Release Date Of the
> Current Used Version* *Release Date Of The Latest Release* *JIRA Issue*
> com.github.ben-manes.versions:com.github.ben-manes.versions.gradle.plugin
> <https://mvnrepository.com/artifact/com.github.ben-manes.versions/com.github.ben-manes.versions.gradle.plugin>
> 0.20.0 0.27.0 2019-02-11 2019-10-21 BEAM-6645
> <https://issues.apache.org/jira/browse/BEAM-6645>
> com.github.spotbugs:spotbugs
> <https://mvnrepository.com/artifact/com.github.spotbugs/spotbugs> 3.1.12
> 4.0.0-beta4 2019-03-01 2019-09-18 BEAM-7792
> <https://issues.apache.org/jira/browse/BEAM-7792>
> com.github.spotbugs:spotbugs-annotations
> <https://mvnrepository.com/artifact/com.github.spotbugs/spotbugs-annotations>
> 3.1.12 4.0.0-beta4 2019-03-01 2019-09-18 BEAM-6951
> <https://issues.apache.org/jira/browse/BEAM-6951>
> javax.servlet:javax.servlet-api
> <https://mvnrepository.com/artifact/javax.servlet/javax.servlet-api> 3.1.0
> 4.0.1 2013-04-25 2018-04-20 BEAM-5750
> <https://issues.apache.org/jira/browse/BEAM-5750>
> org.conscrypt:conscrypt-openjdk
> <https://mvnrepository.com/artifact/org.conscrypt/conscrypt-openjdk> 1.1.3
> 2.2.1 2018-06-04 2019-08-08 BEAM-5748
> <https://issues.apache.org/jira/browse/BEAM-5748>
> org.eclipse.jetty:jetty-server
> <https://mvnrepository.com/artifact/org.eclipse.jetty/jetty-server>
> 9.2.10.v20150310 10.0.0-alpha0 2015-03-10 2019-07-11 BEAM-5752
> <https://issues.apache.org/jira/browse/BEAM-5752>
> org.eclipse.jetty:jetty-servlet
> <https://mvnrepository.com/artifact/org.eclipse.jetty/jetty-servlet>
> 9.2.10.v20150310 10.0.0-alpha0 2015-03-10 2019-07-11 BEAM-5753
> <https://issues.apache.org/jira/browse/BEAM-5753>
> Gradle: <https://mvnrepository.com/artifact/Gradle/> 5.2.1 6.0 2019-08-19
> 2019-11-11 BEAM-8002 <https://issues.apache.org/jira/browse/BEAM-8002> A
> dependency update is high priority if it satisfies one of following
> criteria:
>
> - It has major versions update available, e.g.
> org.assertj:assertj-core 2.5.0 -> 3.10.0;
>
>
> - It is over 3 minor versions behind the latest version, e.g.
> org.tukaani:xz 1.5 -> 1.8;
>
>
> - The current version is behind the later version for over 180 days,
> e.g. com.google.auto.service:auto-service 2014-10-24 -> 2017-12-11.
>
> In Beam, we make a best-effort attempt at keeping all dependencies
> up-to-date. In the future, issues will be filed and tracked for these
> automatically, but in the meantime you can search for existing issues or
> open a new one. For more information: Beam Dependency Guide
> <https://beam.apache.org/contribute/dependencies/>
>
--
Regards,
Tomo
Re: Completeness of Beam Java Dependency Check Report
Posted by Yifan Zou <yi...@google.com>.
The dependency management tool is back. See the latest report
<https://builds.apache.org/job/beam_Dependency_Check/234/artifact/src/build/dependencyUpdates/beam-dependency-check-report.html>
.
On Tue, Nov 12, 2019 at 9:51 AM Yifan Zou <yi...@google.com> wrote:
> Thanks Tomo. I'll follow up in JIRA.
>
> On Tue, Nov 12, 2019 at 9:44 AM Tomo Suzuki <su...@google.com> wrote:
>
>> Yifan,
>> I created a ticket to track this finding:
>> https://issues.apache.org/jira/browse/BEAM-8621 .
>>
>>
>> On Mon, Nov 11, 2019 at 5:08 PM Tomo Suzuki <su...@google.com> wrote:
>>
>>> Kenn,
>>>
>>> Thank you for the analysis. Although Guava was randomly picked up, it's
>>> great learning for me to learn how you analyzed other modules using Guava.
>>>
>>> On Mon, Nov 11, 2019 at 4:29 PM Kenneth Knowles <ke...@apache.org> wrote:
>>>
>>>> BeamModulePlugin just contains lists of versions to ease coordination
>>>> across Beam modules, but mostly does not create dependencies. Most of
>>>> Beam's modules only depend on a few things there. For example Guava is not
>>>> a core dependency, but here is where it is actually depended upon:
>>>>
>>>> $ find . -name build.gradle | xargs grep library.java.guava
>>>> ./sdks/java/core/build.gradle: shadowTest library.java.guava_testlib
>>>> ./sdks/java/extensions/sql/jdbc/build.gradle: compile
>>>> library.java.guava
>>>> ./sdks/java/io/google-cloud-platform/build.gradle: compile
>>>> library.java.guava
>>>> ./sdks/java/io/kinesis/build.gradle: testCompile
>>>> library.java.guava_testlib
>>>>
>>>> These results appear to be misleading. Grepping for 'import
>>>> com.google.common', I see this as the actual state of things:
>>>>
>>>> - GCP connector does not appear to actually depend on Guava in compile
>>>> scope
>>>> - The Beam SQL JDBC driver does not appear to actually depend on Guava
>>>> in compile scope
>>>> - The Dataflow Java worker does depend on Guava at compile scope but
>>>> has incorrect dependencies (and it probably shouldn't)
>>>> - KinesisIO does depend on Guava at compile scope but has incorrect
>>>> dependencies (Kinesis libs have Guava on API surface so it is OK here, but
>>>> should be correctly declared)
>>>> - ZetaSQL translator does depend on Guava at compile scope but has
>>>> incorrect dependencies (ZetaSQL has it on API surface so it is OK here, but
>>>> should be correctly declared)
>>>>
>>>> We used to have an analysis that prevented this class of error.
>>>>
>>>> Once the errors are fixed, the guava_version is simply a version that
>>>> we have discovered that seems to work for both Kinesis and ZetaSQL,
>>>> libraries we do not control. Kinesis producer is built against 18.0.
>>>> Kinesis client against 26.0-jre. ZetaSQL against 26.0-android.
>>>>
>>>> (or maybe I messed up in my analysis)
>>>>
>>>> Kenn
>>>>
>>>> On Mon, Nov 11, 2019 at 12:07 PM Tomo Suzuki <su...@google.com>
>>>> wrote:
>>>>
>>>>>
>>>>> Chamikara and Yifan,
>>>>> Thank you for the responses! Looking forward to hearing the
>>>>> investigation result.
>>>>> In the meantime, I'll explore .test-infra/jenkins/dependency_check
>>>>> directory.
>>>>>
>>>>>
>>>
>>> --
>>> Regards,
>>> Tomo
>>>
>>
>>
>> --
>> Regards,
>> Tomo
>>
>
Re: Completeness of Beam Java Dependency Check Report
Posted by Yifan Zou <yi...@google.com>.
Thanks Tomo. I'll follow up in JIRA.
On Tue, Nov 12, 2019 at 9:44 AM Tomo Suzuki <su...@google.com> wrote:
> Yifan,
> I created a ticket to track this finding:
> https://issues.apache.org/jira/browse/BEAM-8621 .
>
>
> On Mon, Nov 11, 2019 at 5:08 PM Tomo Suzuki <su...@google.com> wrote:
>
>> Kenn,
>>
>> Thank you for the analysis. Although Guava was randomly picked up, it's
>> great learning for me to learn how you analyzed other modules using Guava.
>>
>> On Mon, Nov 11, 2019 at 4:29 PM Kenneth Knowles <ke...@apache.org> wrote:
>>
>>> BeamModulePlugin just contains lists of versions to ease coordination
>>> across Beam modules, but mostly does not create dependencies. Most of
>>> Beam's modules only depend on a few things there. For example Guava is not
>>> a core dependency, but here is where it is actually depended upon:
>>>
>>> $ find . -name build.gradle | xargs grep library.java.guava
>>> ./sdks/java/core/build.gradle: shadowTest library.java.guava_testlib
>>> ./sdks/java/extensions/sql/jdbc/build.gradle: compile library.java.guava
>>> ./sdks/java/io/google-cloud-platform/build.gradle: compile
>>> library.java.guava
>>> ./sdks/java/io/kinesis/build.gradle: testCompile
>>> library.java.guava_testlib
>>>
>>> These results appear to be misleading. Grepping for 'import
>>> com.google.common', I see this as the actual state of things:
>>>
>>> - GCP connector does not appear to actually depend on Guava in compile
>>> scope
>>> - The Beam SQL JDBC driver does not appear to actually depend on Guava
>>> in compile scope
>>> - The Dataflow Java worker does depend on Guava at compile scope but
>>> has incorrect dependencies (and it probably shouldn't)
>>> - KinesisIO does depend on Guava at compile scope but has incorrect
>>> dependencies (Kinesis libs have Guava on API surface so it is OK here, but
>>> should be correctly declared)
>>> - ZetaSQL translator does depend on Guava at compile scope but has
>>> incorrect dependencies (ZetaSQL has it on API surface so it is OK here, but
>>> should be correctly declared)
>>>
>>> We used to have an analysis that prevented this class of error.
>>>
>>> Once the errors are fixed, the guava_version is simply a version that we
>>> have discovered that seems to work for both Kinesis and ZetaSQL, libraries
>>> we do not control. Kinesis producer is built against 18.0. Kinesis client
>>> against 26.0-jre. ZetaSQL against 26.0-android.
>>>
>>> (or maybe I messed up in my analysis)
>>>
>>> Kenn
>>>
>>> On Mon, Nov 11, 2019 at 12:07 PM Tomo Suzuki <su...@google.com> wrote:
>>>
>>>>
>>>> Chamikara and Yifan,
>>>> Thank you for the responses! Looking forward to hearing the
>>>> investigation result.
>>>> In the meantime, I'll explore .test-infra/jenkins/dependency_check
>>>> directory.
>>>>
>>>>
>>
>> --
>> Regards,
>> Tomo
>>
>
>
> --
> Regards,
> Tomo
>
Re: Completeness of Beam Java Dependency Check Report
Posted by Tomo Suzuki <su...@google.com>.
Yifan,
I created a ticket to track this finding:
https://issues.apache.org/jira/browse/BEAM-8621 .
On Mon, Nov 11, 2019 at 5:08 PM Tomo Suzuki <su...@google.com> wrote:
> Kenn,
>
> Thank you for the analysis. Although Guava was randomly picked up, it's
> great learning for me to learn how you analyzed other modules using Guava.
>
> On Mon, Nov 11, 2019 at 4:29 PM Kenneth Knowles <ke...@apache.org> wrote:
>
>> BeamModulePlugin just contains lists of versions to ease coordination
>> across Beam modules, but mostly does not create dependencies. Most of
>> Beam's modules only depend on a few things there. For example Guava is not
>> a core dependency, but here is where it is actually depended upon:
>>
>> $ find . -name build.gradle | xargs grep library.java.guava
>> ./sdks/java/core/build.gradle: shadowTest library.java.guava_testlib
>> ./sdks/java/extensions/sql/jdbc/build.gradle: compile library.java.guava
>> ./sdks/java/io/google-cloud-platform/build.gradle: compile
>> library.java.guava
>> ./sdks/java/io/kinesis/build.gradle: testCompile
>> library.java.guava_testlib
>>
>> These results appear to be misleading. Grepping for 'import
>> com.google.common', I see this as the actual state of things:
>>
>> - GCP connector does not appear to actually depend on Guava in compile
>> scope
>> - The Beam SQL JDBC driver does not appear to actually depend on Guava
>> in compile scope
>> - The Dataflow Java worker does depend on Guava at compile scope but has
>> incorrect dependencies (and it probably shouldn't)
>> - KinesisIO does depend on Guava at compile scope but has incorrect
>> dependencies (Kinesis libs have Guava on API surface so it is OK here, but
>> should be correctly declared)
>> - ZetaSQL translator does depend on Guava at compile scope but has
>> incorrect dependencies (ZetaSQL has it on API surface so it is OK here, but
>> should be correctly declared)
>>
>> We used to have an analysis that prevented this class of error.
>>
>> Once the errors are fixed, the guava_version is simply a version that we
>> have discovered that seems to work for both Kinesis and ZetaSQL, libraries
>> we do not control. Kinesis producer is built against 18.0. Kinesis client
>> against 26.0-jre. ZetaSQL against 26.0-android.
>>
>> (or maybe I messed up in my analysis)
>>
>> Kenn
>>
>> On Mon, Nov 11, 2019 at 12:07 PM Tomo Suzuki <su...@google.com> wrote:
>>
>>>
>>> Chamikara and Yifan,
>>> Thank you for the responses! Looking forward to hearing the
>>> investigation result.
>>> In the meantime, I'll explore .test-infra/jenkins/dependency_check
>>> directory.
>>>
>>>
>
> --
> Regards,
> Tomo
>
--
Regards,
Tomo
Re: Completeness of Beam Java Dependency Check Report
Posted by Tomo Suzuki <su...@google.com>.
Kenn,
Thank you for the analysis. Although Guava was randomly picked up, it's
great learning for me to learn how you analyzed other modules using Guava.
On Mon, Nov 11, 2019 at 4:29 PM Kenneth Knowles <ke...@apache.org> wrote:
> BeamModulePlugin just contains lists of versions to ease coordination
> across Beam modules, but mostly does not create dependencies. Most of
> Beam's modules only depend on a few things there. For example Guava is not
> a core dependency, but here is where it is actually depended upon:
>
> $ find . -name build.gradle | xargs grep library.java.guava
> ./sdks/java/core/build.gradle: shadowTest library.java.guava_testlib
> ./sdks/java/extensions/sql/jdbc/build.gradle: compile library.java.guava
> ./sdks/java/io/google-cloud-platform/build.gradle: compile
> library.java.guava
> ./sdks/java/io/kinesis/build.gradle: testCompile
> library.java.guava_testlib
>
> These results appear to be misleading. Grepping for 'import
> com.google.common', I see this as the actual state of things:
>
> - GCP connector does not appear to actually depend on Guava in compile
> scope
> - The Beam SQL JDBC driver does not appear to actually depend on Guava in
> compile scope
> - The Dataflow Java worker does depend on Guava at compile scope but has
> incorrect dependencies (and it probably shouldn't)
> - KinesisIO does depend on Guava at compile scope but has incorrect
> dependencies (Kinesis libs have Guava on API surface so it is OK here, but
> should be correctly declared)
> - ZetaSQL translator does depend on Guava at compile scope but has
> incorrect dependencies (ZetaSQL has it on API surface so it is OK here, but
> should be correctly declared)
>
> We used to have an analysis that prevented this class of error.
>
> Once the errors are fixed, the guava_version is simply a version that we
> have discovered that seems to work for both Kinesis and ZetaSQL, libraries
> we do not control. Kinesis producer is built against 18.0. Kinesis client
> against 26.0-jre. ZetaSQL against 26.0-android.
>
> (or maybe I messed up in my analysis)
>
> Kenn
>
> On Mon, Nov 11, 2019 at 12:07 PM Tomo Suzuki <su...@google.com> wrote:
>
>>
>> Chamikara and Yifan,
>> Thank you for the responses! Looking forward to hearing the investigation
>> result.
>> In the meantime, I'll explore .test-infra/jenkins/dependency_check
>> directory.
>>
>>
--
Regards,
Tomo
Re: Completeness of Beam Java Dependency Check Report
Posted by Tomo Suzuki <su...@google.com>.
Hi Yifan,
I found resolutionStrategy is interfering gradle-versions-plugin (detail in
BEAM-8654). Would you check this PR?
https://github.com/apache/beam/pull/10127
On Thu, Nov 14, 2019 at 1:42 PM Kenneth Knowles <ke...@apache.org> wrote:
> On Thu, Nov 14, 2019 at 8:04 AM Alexey Romanenko <ar...@gmail.com>
> wrote:
>
>> Good example about Guava deps, let me go a bit deeper.
>>
>> $ find . -name build.gradle | xargs grep library.java.guava
>> ./sdks/java/core/build.gradle: shadowTest library.java.guava_testlib
>>
>> ./sdks/java/io/kinesis/build.gradle: testCompile
>> library.java.guava_testlib
>>
>>
>> Regarding using non-vendored Guava in KinesisIO (and "java/core" as
>> well), it’s all about *“library.java.guava_testlib” *and
>> *“com.google.common.testing.EqualsTester”* only in particular, which is
>> used for tests.
>> Do we need to vendor “*com.google.guava:guava-testlib*” for this in this
>> case?
>>
>
> I didn't worry about test scopes because they don't ship to users so much.
> It could be useful if there is a runner with a conflict and they want to
> run the integration tests.
>
>
>> - KinesisIO does depend on Guava at compile scope but has incorrect
>> dependencies (Kinesis libs have Guava on API surface so it is OK here, but
>> should be correctly declared)
>>
>>
>> Sorry, but I didn’t understand what do you mean by “*but should be
>> correctly declared*”.
>> Since Kinesis client libs have own Guava deps and we shade our own guava,
>> so it should be fine, no?
>>
>
> I mean that any module with an "import X" should have a dependency on X in
> its build.gradle. When you leave it off, the dep analysis (such as Maven's)
> calls it out as "used undeclared dependency".
>
> Kenn
>
>
>>
>> On 11 Nov 2019, at 22:29, Kenneth Knowles <ke...@apache.org> wrote:
>>
>> BeamModulePlugin just contains lists of versions to ease coordination
>> across Beam modules, but mostly does not create dependencies. Most of
>> Beam's modules only depend on a few things there. For example Guava is not
>> a core dependency, but here is where it is actually depended upon:
>>
>> $ find . -name build.gradle | xargs grep library.java.guava
>> ./sdks/java/core/build.gradle: shadowTest library.java.guava_testlib
>> ./sdks/java/extensions/sql/jdbc/build.gradle: compile library.java.guava
>> ./sdks/java/io/google-cloud-platform/build.gradle: compile
>> library.java.guava
>> ./sdks/java/io/kinesis/build.gradle: testCompile
>> library.java.guava_testlib
>>
>> These results appear to be misleading. Grepping for 'import
>> com.google.common', I see this as the actual state of things:
>>
>> - GCP connector does not appear to actually depend on Guava in compile
>> scope
>> - The Beam SQL JDBC driver does not appear to actually depend on Guava
>> in compile scope
>> - The Dataflow Java worker does depend on Guava at compile scope but has
>> incorrect dependencies (and it probably shouldn't)
>> - KinesisIO does depend on Guava at compile scope but has incorrect
>> dependencies (Kinesis libs have Guava on API surface so it is OK here, but
>> should be correctly declared)
>> - ZetaSQL translator does depend on Guava at compile scope but has
>> incorrect dependencies (ZetaSQL has it on API surface so it is OK here, but
>> should be correctly declared)
>>
>> We used to have an analysis that prevented this class of error.
>>
>> Once the errors are fixed, the guava_version is simply a version that we
>> have discovered that seems to work for both Kinesis and ZetaSQL, libraries
>> we do not control. Kinesis producer is built against 18.0. Kinesis client
>> against 26.0-jre. ZetaSQL against 26.0-android.
>>
>> (or maybe I messed up in my analysis)
>>
>> Kenn
>>
>> On Mon, Nov 11, 2019 at 12:07 PM Tomo Suzuki <su...@google.com> wrote:
>>
>>>
>>> Chamikara and Yifan,
>>> Thank you for the responses! Looking forward to hearing the
>>> investigation result.
>>> In the meantime, I'll explore .test-infra/jenkins/dependency_check
>>> directory.
>>>
>>>
>>
--
Regards,
Tomo
Re: Completeness of Beam Java Dependency Check Report
Posted by Kenneth Knowles <ke...@apache.org>.
On Thu, Nov 14, 2019 at 8:04 AM Alexey Romanenko <ar...@gmail.com>
wrote:
> Good example about Guava deps, let me go a bit deeper.
>
> $ find . -name build.gradle | xargs grep library.java.guava
> ./sdks/java/core/build.gradle: shadowTest library.java.guava_testlib
>
> ./sdks/java/io/kinesis/build.gradle: testCompile
> library.java.guava_testlib
>
>
> Regarding using non-vendored Guava in KinesisIO (and "java/core" as well),
> it’s all about *“library.java.guava_testlib” *and
> *“com.google.common.testing.EqualsTester”* only in particular, which is
> used for tests.
> Do we need to vendor “*com.google.guava:guava-testlib*” for this in this
> case?
>
I didn't worry about test scopes because they don't ship to users so much.
It could be useful if there is a runner with a conflict and they want to
run the integration tests.
> - KinesisIO does depend on Guava at compile scope but has incorrect
> dependencies (Kinesis libs have Guava on API surface so it is OK here, but
> should be correctly declared)
>
>
> Sorry, but I didn’t understand what do you mean by “*but should be
> correctly declared*”.
> Since Kinesis client libs have own Guava deps and we shade our own guava,
> so it should be fine, no?
>
I mean that any module with an "import X" should have a dependency on X in
its build.gradle. When you leave it off, the dep analysis (such as Maven's)
calls it out as "used undeclared dependency".
Kenn
>
> On 11 Nov 2019, at 22:29, Kenneth Knowles <ke...@apache.org> wrote:
>
> BeamModulePlugin just contains lists of versions to ease coordination
> across Beam modules, but mostly does not create dependencies. Most of
> Beam's modules only depend on a few things there. For example Guava is not
> a core dependency, but here is where it is actually depended upon:
>
> $ find . -name build.gradle | xargs grep library.java.guava
> ./sdks/java/core/build.gradle: shadowTest library.java.guava_testlib
> ./sdks/java/extensions/sql/jdbc/build.gradle: compile library.java.guava
> ./sdks/java/io/google-cloud-platform/build.gradle: compile
> library.java.guava
> ./sdks/java/io/kinesis/build.gradle: testCompile
> library.java.guava_testlib
>
> These results appear to be misleading. Grepping for 'import
> com.google.common', I see this as the actual state of things:
>
> - GCP connector does not appear to actually depend on Guava in compile
> scope
> - The Beam SQL JDBC driver does not appear to actually depend on Guava in
> compile scope
> - The Dataflow Java worker does depend on Guava at compile scope but has
> incorrect dependencies (and it probably shouldn't)
> - KinesisIO does depend on Guava at compile scope but has incorrect
> dependencies (Kinesis libs have Guava on API surface so it is OK here, but
> should be correctly declared)
> - ZetaSQL translator does depend on Guava at compile scope but has
> incorrect dependencies (ZetaSQL has it on API surface so it is OK here, but
> should be correctly declared)
>
> We used to have an analysis that prevented this class of error.
>
> Once the errors are fixed, the guava_version is simply a version that we
> have discovered that seems to work for both Kinesis and ZetaSQL, libraries
> we do not control. Kinesis producer is built against 18.0. Kinesis client
> against 26.0-jre. ZetaSQL against 26.0-android.
>
> (or maybe I messed up in my analysis)
>
> Kenn
>
> On Mon, Nov 11, 2019 at 12:07 PM Tomo Suzuki <su...@google.com> wrote:
>
>>
>> Chamikara and Yifan,
>> Thank you for the responses! Looking forward to hearing the investigation
>> result.
>> In the meantime, I'll explore .test-infra/jenkins/dependency_check
>> directory.
>>
>>
>
Re: Completeness of Beam Java Dependency Check Report
Posted by Alexey Romanenko <ar...@gmail.com>.
Good example about Guava deps, let me go a bit deeper.
> $ find . -name build.gradle | xargs grep library.java.guava
> ./sdks/java/core/build.gradle: shadowTest library.java.guava_testlib
> ./sdks/java/io/kinesis/build.gradle: testCompile library.java.guava_testlib
Regarding using non-vendored Guava in KinesisIO (and "java/core" as well), it’s all about “library.java.guava_testlib” and “com.google.common.testing.EqualsTester” only in particular, which is used for tests.
Do we need to vendor “com.google.guava:guava-testlib” for this in this case?
> - KinesisIO does depend on Guava at compile scope but has incorrect dependencies (Kinesis libs have Guava on API surface so it is OK here, but should be correctly declared)
Sorry, but I didn’t understand what do you mean by “but should be correctly declared”.
Since Kinesis client libs have own Guava deps and we shade our own guava, so it should be fine, no?
> On 11 Nov 2019, at 22:29, Kenneth Knowles <ke...@apache.org> wrote:
>
> BeamModulePlugin just contains lists of versions to ease coordination across Beam modules, but mostly does not create dependencies. Most of Beam's modules only depend on a few things there. For example Guava is not a core dependency, but here is where it is actually depended upon:
>
> $ find . -name build.gradle | xargs grep library.java.guava
> ./sdks/java/core/build.gradle: shadowTest library.java.guava_testlib
> ./sdks/java/extensions/sql/jdbc/build.gradle: compile library.java.guava
> ./sdks/java/io/google-cloud-platform/build.gradle: compile library.java.guava
> ./sdks/java/io/kinesis/build.gradle: testCompile library.java.guava_testlib
>
> These results appear to be misleading. Grepping for 'import com.google.common', I see this as the actual state of things:
>
> - GCP connector does not appear to actually depend on Guava in compile scope
> - The Beam SQL JDBC driver does not appear to actually depend on Guava in compile scope
> - The Dataflow Java worker does depend on Guava at compile scope but has incorrect dependencies (and it probably shouldn't)
> - KinesisIO does depend on Guava at compile scope but has incorrect dependencies (Kinesis libs have Guava on API surface so it is OK here, but should be correctly declared)
> - ZetaSQL translator does depend on Guava at compile scope but has incorrect dependencies (ZetaSQL has it on API surface so it is OK here, but should be correctly declared)
>
> We used to have an analysis that prevented this class of error.
>
> Once the errors are fixed, the guava_version is simply a version that we have discovered that seems to work for both Kinesis and ZetaSQL, libraries we do not control. Kinesis producer is built against 18.0. Kinesis client against 26.0-jre. ZetaSQL against 26.0-android.
>
> (or maybe I messed up in my analysis)
>
> Kenn
>
> On Mon, Nov 11, 2019 at 12:07 PM Tomo Suzuki <suztomo@google.com <ma...@google.com>> wrote:
>
> Chamikara and Yifan,
> Thank you for the responses! Looking forward to hearing the investigation result.
> In the meantime, I'll explore .test-infra/jenkins/dependency_check directory.
>
Re: Completeness of Beam Java Dependency Check Report
Posted by Kenneth Knowles <ke...@apache.org>.
BeamModulePlugin just contains lists of versions to ease coordination
across Beam modules, but mostly does not create dependencies. Most of
Beam's modules only depend on a few things there. For example Guava is not
a core dependency, but here is where it is actually depended upon:
$ find . -name build.gradle | xargs grep library.java.guava
./sdks/java/core/build.gradle: shadowTest library.java.guava_testlib
./sdks/java/extensions/sql/jdbc/build.gradle: compile library.java.guava
./sdks/java/io/google-cloud-platform/build.gradle: compile
library.java.guava
./sdks/java/io/kinesis/build.gradle: testCompile library.java.guava_testlib
These results appear to be misleading. Grepping for 'import
com.google.common', I see this as the actual state of things:
- GCP connector does not appear to actually depend on Guava in compile
scope
- The Beam SQL JDBC driver does not appear to actually depend on Guava in
compile scope
- The Dataflow Java worker does depend on Guava at compile scope but has
incorrect dependencies (and it probably shouldn't)
- KinesisIO does depend on Guava at compile scope but has incorrect
dependencies (Kinesis libs have Guava on API surface so it is OK here, but
should be correctly declared)
- ZetaSQL translator does depend on Guava at compile scope but has
incorrect dependencies (ZetaSQL has it on API surface so it is OK here, but
should be correctly declared)
We used to have an analysis that prevented this class of error.
Once the errors are fixed, the guava_version is simply a version that we
have discovered that seems to work for both Kinesis and ZetaSQL, libraries
we do not control. Kinesis producer is built against 18.0. Kinesis client
against 26.0-jre. ZetaSQL against 26.0-android.
(or maybe I messed up in my analysis)
Kenn
On Mon, Nov 11, 2019 at 12:07 PM Tomo Suzuki <su...@google.com> wrote:
>
> Chamikara and Yifan,
> Thank you for the responses! Looking forward to hearing the investigation
> result.
> In the meantime, I'll explore .test-infra/jenkins/dependency_check
> directory.
>
>
Re: Completeness of Beam Java Dependency Check Report
Posted by Tomo Suzuki <su...@google.com>.
Chamikara and Yifan,
Thank you for the responses! Looking forward to hearing the investigation
result.
In the meantime, I'll explore .test-infra/jenkins/dependency_check
directory.
Re: Completeness of Beam Java Dependency Check Report
Posted by Yifan Zou <yi...@google.com>.
Hi,
Thanks for taking care of Beam dependencies. The guava was tracked in
BEAM-5559 <https://issues.apache.org/jira/browse/BEAM-5559>. It was
filtered out by the tool because of the target version is x.y-jre.
On the other hand, I checked the logs of dependency job and found that the
high priority dependencies are way less than normal. I think the
gradle-versions-plugin sometimes couldn't get the versions which are
defined in BeamModulePlugin.groovy. Need more investigations on the
dependency management tool.
Yifan
On Mon, Nov 11, 2019 at 10:14 AM Tomo Suzuki <su...@google.com> wrote:
> Hi Beam developers,
> (I'm thinking to contribute to upgrades of Java dependencies of Beam; I
> just read https://beam.apache.org/contribute/dependencies/)
>
> As per the weekly report, Apache Beam Java SDK only has 8 outdated
> dependencies based on the criteria. However, it seems many others are not
> up-to-date. For example Guava 20.0 used in Beam
> <https://github.com/apache/beam/blob/d692d2f/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L375>
> is not the latest major release.
>
> Why do some outdated dependencies not appear in this report?
>
> Regards,
> Tomo
>
> On Mon, Nov 11, 2019 at 7:05 AM Apache Jenkins Server <
> jenkins@builds.apache.org> wrote:
>
>> High Priority Dependency Updates Of Beam Python SDK:
>> *Dependency Name* *Current Version* *Latest Version* *Release Date Of
>> the Current Used Version* *Release Date Of The Latest Release* *JIRA
>> Issue*
>> mock <https://pypi.org/project/mock> 2.0.0 3.0.5 2019-05-20 2019-05-20
>> BEAM-7369 <https://issues.apache.org/jira/browse/BEAM-7369>
>> oauth2client <https://pypi.org/project/oauth2client> 3.0.0 4.1.3
>> 2018-12-10 2018-12-10 BEAM-6089
>> <https://issues.apache.org/jira/browse/BEAM-6089>
>> Sphinx <https://pypi.org/project/Sphinx> 1.8.5 2.2.1 2019-05-20
>> 2019-10-28 BEAM-7370 <https://issues.apache.org/jira/browse/BEAM-7370> High
>> Priority Dependency Updates Of Beam Java SDK:
>> *Dependency Name* *Current Version* *Latest Version* *Release Date Of
>> the Current Used Version* *Release Date Of The Latest Release* *JIRA
>> Issue*
>> com.github.ben-manes.versions:com.github.ben-manes.versions.gradle.plugin
>> <https://mvnrepository.com/artifact/com.github.ben-manes.versions/com.github.ben-manes.versions.gradle.plugin>
>> 0.20.0 0.27.0 2019-02-11 2019-10-21 BEAM-6645
>> <https://issues.apache.org/jira/browse/BEAM-6645>
>> com.github.spotbugs:spotbugs
>> <https://mvnrepository.com/artifact/com.github.spotbugs/spotbugs> 3.1.12
>> 4.0.0-beta4 2019-03-01 2019-09-18 BEAM-7792
>> <https://issues.apache.org/jira/browse/BEAM-7792>
>> com.github.spotbugs:spotbugs-annotations
>> <https://mvnrepository.com/artifact/com.github.spotbugs/spotbugs-annotations>
>> 3.1.12 4.0.0-beta4 2019-03-01 2019-09-18 BEAM-6951
>> <https://issues.apache.org/jira/browse/BEAM-6951>
>> javax.servlet:javax.servlet-api
>> <https://mvnrepository.com/artifact/javax.servlet/javax.servlet-api>
>> 3.1.0 4.0.1 2013-04-25 2018-04-20 BEAM-5750
>> <https://issues.apache.org/jira/browse/BEAM-5750>
>> org.conscrypt:conscrypt-openjdk
>> <https://mvnrepository.com/artifact/org.conscrypt/conscrypt-openjdk>
>> 1.1.3 2.2.1 2018-06-04 2019-08-08 BEAM-5748
>> <https://issues.apache.org/jira/browse/BEAM-5748>
>> org.eclipse.jetty:jetty-server
>> <https://mvnrepository.com/artifact/org.eclipse.jetty/jetty-server>
>> 9.2.10.v20150310 10.0.0-alpha0 2015-03-10 2019-07-11 BEAM-5752
>> <https://issues.apache.org/jira/browse/BEAM-5752>
>> org.eclipse.jetty:jetty-servlet
>> <https://mvnrepository.com/artifact/org.eclipse.jetty/jetty-servlet>
>> 9.2.10.v20150310 10.0.0-alpha0 2015-03-10 2019-07-11 BEAM-5753
>> <https://issues.apache.org/jira/browse/BEAM-5753>
>> Gradle: <https://mvnrepository.com/artifact/Gradle/> 5.2.1 6.0 2019-08-19
>> 2019-11-11 BEAM-8002 <https://issues.apache.org/jira/browse/BEAM-8002> A
>> dependency update is high priority if it satisfies one of following
>> criteria:
>>
>> - It has major versions update available, e.g.
>> org.assertj:assertj-core 2.5.0 -> 3.10.0;
>>
>>
>> - It is over 3 minor versions behind the latest version, e.g.
>> org.tukaani:xz 1.5 -> 1.8;
>>
>>
>> - The current version is behind the later version for over 180 days,
>> e.g. com.google.auto.service:auto-service 2014-10-24 -> 2017-12-11.
>>
>> In Beam, we make a best-effort attempt at keeping all dependencies
>> up-to-date. In the future, issues will be filed and tracked for these
>> automatically, but in the meantime you can search for existing issues or
>> open a new one. For more information: Beam Dependency Guide
>> <https://beam.apache.org/contribute/dependencies/>
>>
>
>
> --
> Regards,
> Tomo
>
Re: Completeness of Beam Java Dependency Check Report
Posted by Chamikara Jayalath <ch...@google.com>.
On Mon, Nov 11, 2019 at 10:14 AM Tomo Suzuki <su...@google.com> wrote:
> Hi Beam developers,
> (I'm thinking to contribute to upgrades of Java dependencies of Beam; I
> just read https://beam.apache.org/contribute/dependencies/)
>
Thanks that will be great.
>
> As per the weekly report, Apache Beam Java SDK only has 8 outdated
> dependencies based on the criteria. However, it seems many others are not
> up-to-date. For example Guava 20.0 used in Beam
> <https://github.com/apache/beam/blob/d692d2f/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L375>
> is not the latest major release.
>
> Why do some outdated dependencies not appear in this report?
>
This is probably since artifact version changed from xy.z to xy.z-jre. +Yifan
Zou <yi...@google.com> any idea ?
>
> Regards,
> Tomo
>
> On Mon, Nov 11, 2019 at 7:05 AM Apache Jenkins Server <
> jenkins@builds.apache.org> wrote:
>
>> High Priority Dependency Updates Of Beam Python SDK:
>> *Dependency Name* *Current Version* *Latest Version* *Release Date Of
>> the Current Used Version* *Release Date Of The Latest Release* *JIRA
>> Issue*
>> mock <https://pypi.org/project/mock> 2.0.0 3.0.5 2019-05-20 2019-05-20
>> BEAM-7369 <https://issues.apache.org/jira/browse/BEAM-7369>
>> oauth2client <https://pypi.org/project/oauth2client> 3.0.0 4.1.3
>> 2018-12-10 2018-12-10 BEAM-6089
>> <https://issues.apache.org/jira/browse/BEAM-6089>
>> Sphinx <https://pypi.org/project/Sphinx> 1.8.5 2.2.1 2019-05-20
>> 2019-10-28 BEAM-7370 <https://issues.apache.org/jira/browse/BEAM-7370> High
>> Priority Dependency Updates Of Beam Java SDK:
>> *Dependency Name* *Current Version* *Latest Version* *Release Date Of
>> the Current Used Version* *Release Date Of The Latest Release* *JIRA
>> Issue*
>> com.github.ben-manes.versions:com.github.ben-manes.versions.gradle.plugin
>> <https://mvnrepository.com/artifact/com.github.ben-manes.versions/com.github.ben-manes.versions.gradle.plugin>
>> 0.20.0 0.27.0 2019-02-11 2019-10-21 BEAM-6645
>> <https://issues.apache.org/jira/browse/BEAM-6645>
>> com.github.spotbugs:spotbugs
>> <https://mvnrepository.com/artifact/com.github.spotbugs/spotbugs> 3.1.12
>> 4.0.0-beta4 2019-03-01 2019-09-18 BEAM-7792
>> <https://issues.apache.org/jira/browse/BEAM-7792>
>> com.github.spotbugs:spotbugs-annotations
>> <https://mvnrepository.com/artifact/com.github.spotbugs/spotbugs-annotations>
>> 3.1.12 4.0.0-beta4 2019-03-01 2019-09-18 BEAM-6951
>> <https://issues.apache.org/jira/browse/BEAM-6951>
>> javax.servlet:javax.servlet-api
>> <https://mvnrepository.com/artifact/javax.servlet/javax.servlet-api>
>> 3.1.0 4.0.1 2013-04-25 2018-04-20 BEAM-5750
>> <https://issues.apache.org/jira/browse/BEAM-5750>
>> org.conscrypt:conscrypt-openjdk
>> <https://mvnrepository.com/artifact/org.conscrypt/conscrypt-openjdk>
>> 1.1.3 2.2.1 2018-06-04 2019-08-08 BEAM-5748
>> <https://issues.apache.org/jira/browse/BEAM-5748>
>> org.eclipse.jetty:jetty-server
>> <https://mvnrepository.com/artifact/org.eclipse.jetty/jetty-server>
>> 9.2.10.v20150310 10.0.0-alpha0 2015-03-10 2019-07-11 BEAM-5752
>> <https://issues.apache.org/jira/browse/BEAM-5752>
>> org.eclipse.jetty:jetty-servlet
>> <https://mvnrepository.com/artifact/org.eclipse.jetty/jetty-servlet>
>> 9.2.10.v20150310 10.0.0-alpha0 2015-03-10 2019-07-11 BEAM-5753
>> <https://issues.apache.org/jira/browse/BEAM-5753>
>> Gradle: <https://mvnrepository.com/artifact/Gradle/> 5.2.1 6.0 2019-08-19
>> 2019-11-11 BEAM-8002 <https://issues.apache.org/jira/browse/BEAM-8002> A
>> dependency update is high priority if it satisfies one of following
>> criteria:
>>
>> - It has major versions update available, e.g.
>> org.assertj:assertj-core 2.5.0 -> 3.10.0;
>>
>>
>> - It is over 3 minor versions behind the latest version, e.g.
>> org.tukaani:xz 1.5 -> 1.8;
>>
>>
>> - The current version is behind the later version for over 180 days,
>> e.g. com.google.auto.service:auto-service 2014-10-24 -> 2017-12-11.
>>
>> In Beam, we make a best-effort attempt at keeping all dependencies
>> up-to-date. In the future, issues will be filed and tracked for these
>> automatically, but in the meantime you can search for existing issues or
>> open a new one. For more information: Beam Dependency Guide
>> <https://beam.apache.org/contribute/dependencies/>
>>
>
>
> --
> Regards,
> Tomo
>