You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Kenneth Knowles (Jira)" <ji...@apache.org> on 2019/11/02 20:35:00 UTC
[jira] [Commented] (BEAM-8497) Vendor or shade the dependencies of
beam-sdks-java-io-google-cloud-platform
[ https://issues.apache.org/jira/browse/BEAM-8497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16965494#comment-16965494 ]
Kenneth Knowles commented on BEAM-8497:
---------------------------------------
We are aware of the problem and generally agree with your suggestion. The problem is that upstream dependencies have Guava, etc, on their public API surface.
When storage system XYZ ships a client library with some dependency ABC, our default approach is assume users of storage system XYZ already have to deal with this leaked dependency ABC. In other words, we try not to make things worse, but we may not make things better. Often, the classes from the client library of XYZ are on the Beam connector's API surface. In this case, we cannot alter them, or you will get very confusing error messages "expected SomeXYZClass but found SomeXYZClass" because they have nonequal class signatures.
Would you be able to provide any more detail about which dependencies in particular? Beam's current approach is to detect problematic dependencies and attempt to create one global-to-Beam vendored version of the library. If we can find a way to vendor a dependency in a connector, we could do that.
> Vendor or shade the dependencies of beam-sdks-java-io-google-cloud-platform
> ---------------------------------------------------------------------------
>
> Key: BEAM-8497
> URL: https://issues.apache.org/jira/browse/BEAM-8497
> Project: Beam
> Issue Type: Improvement
> Components: io-java-gcp
> Reporter: Julien Tournay
> Priority: Major
>
> Here at Spotify, we regularly face conflicting dependencies issues, usually caused by guava, grpc-* and netty-*. Looking at this bug tracker, it seems that we are not alone and that conflicting versions are a recurring issue.
> In {{beam-sdks-java-core,}} {{beam-runners-direct-java}} and {{beam-runners-google-cloud-dataflow-java}} those dependencies are vendored to avoid such issue.
> I suggest to do the same for {{beam-sdks-java-io-google-cloud-platform}}.
>
> FIY, here's the list of all the dependencies pulled by {{beam-sdks-java-io-google-cloud-platform}}:
> {{com.fasterxml.jackson.core:jackson-annotations:2.9.10:default}}
> {{com.fasterxml.jackson.core:jackson-core:2.9.10:default}}
> {{com.fasterxml.jackson.core:jackson-databind:2.9.10:default}}
> {{com.google.api:api-common:1.7.0:default}}
> {{com.google.api:gax:1.38.0:default}}
> {{com.google.api:gax-grpc:1.38.0:default}}
> {{com.google.api:gax-httpjson:0.52.0:default}}
> {{com.google.api-client:google-api-client:1.27.0:default}}
> {{com.google.api-client:google-api-client-jackson2:1.27.0:default}}
> {{com.google.api-client:google-api-client-java6:1.27.0:default}}
> {{com.google.api.grpc:grpc-google-cloud-bigquerystorage-v1beta1:0.44.0:default}}
> {{com.google.api.grpc:grpc-google-cloud-bigtable-admin-v2:0.38.0:default}}
> {{com.google.api.grpc:grpc-google-cloud-bigtable-v2:0.38.0:default}}
> {{com.google.api.grpc:grpc-google-cloud-pubsub-v1:1.43.0:default}}
> {{com.google.api.grpc:grpc-google-cloud-spanner-admin-database-v1:1.6.0:default}}
> {{com.google.api.grpc:grpc-google-cloud-spanner-admin-instance-v1:1.6.0:default}}
> {{com.google.api.grpc:grpc-google-cloud-spanner-v1:1.6.0:default}}
> {{com.google.api.grpc:grpc-google-common-protos:1.12.0:default}}
> {{com.google.api.grpc:proto-google-cloud-bigquerystorage-v1beta1:0.44.0:default}}
> {{com.google.api.grpc:proto-google-cloud-bigtable-admin-v2:0.38.0:default}}
> {{com.google.api.grpc:proto-google-cloud-bigtable-v2:0.44.0:default}}
> {{com.google.api.grpc:proto-google-cloud-datastore-v1:0.44.0:default}}
> {{com.google.api.grpc:proto-google-cloud-pubsub-v1:1.43.0:default}}
> {{com.google.api.grpc:proto-google-cloud-spanner-admin-database-v1:1.6.0:default}}
> {{com.google.api.grpc:proto-google-cloud-spanner-admin-instance-v1:1.6.0:default}}
> {{com.google.api.grpc:proto-google-cloud-spanner-v1:1.6.0:default}}
> {{com.google.api.grpc:proto-google-common-protos:1.12.0:default}}
> {{com.google.api.grpc:proto-google-iam-v1:0.12.0:default}}
> {{com.google.apis:google-api-services-bigquery:v2-rev20181104-1.27.0:default}}
> {{com.google.apis:google-api-services-cloudresourcemanager:v1-rev20181015-1.27.0:default}}
> {{com.google.apis:google-api-services-pubsub:v1-rev20181105-1.27.0:default}}
> {{com.google.apis:google-api-services-storage:v1-rev20181109-1.27.0:default}}
> {{com.google.auth:google-auth-library-credentials:0.12.0:default}}
> {{com.google.auth:google-auth-library-oauth2-http:0.12.0:default}}
> {{com.google.auto.value:auto-value-annotations:1.6.3:default}}
> {{com.google.cloud:google-cloud-bigquerystorage:0.79.0-alpha:default}}
> {{com.google.cloud:google-cloud-bigtable:0.73.0-alpha:default}}
> {{com.google.cloud:google-cloud-bigtable-admin:0.73.0-alpha:default}}
> {{com.google.cloud:google-cloud-core:1.61.0:default}}
> {{com.google.cloud:google-cloud-core-grpc:1.61.0:default}}
> {{com.google.cloud:google-cloud-core-http:1.55.0:default}}
> {{com.google.cloud:google-cloud-spanner:1.6.0:default}}
> {{com.google.cloud.bigdataoss:gcsio:1.9.16:default}}
> {{com.google.cloud.bigdataoss:util:1.9.16:default}}
> {{com.google.cloud.bigtable:bigtable-client-core:1.8.0:default}}
> {{com.google.cloud.datastore:datastore-v1-proto-client:1.6.0:default}}
> {{com.google.code.findbugs:jsr305:3.0.2:default}}
> {{com.google.code.gson:gson:2.7:default}}
> {{com.google.errorprone:error_prone_annotations:2.3.2:default}}
> {{com.google.flogger:flogger:0.3.1:default}}
> {{com.google.flogger:flogger-system-backend:0.3.1:default}}
> {{com.google.flogger:google-extensions:0.3.1:default}}
> {{com.google.guava:failureaccess:1.0.1:default}}
> {{com.google.guava:guava:27.0.1-jre:default}}
> {{com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava:default}}
> {{com.google.http-client:google-http-client:1.27.0:default}}
> {{com.google.http-client:google-http-client-appengine:1.27.0:default}}
> {{com.google.http-client:google-http-client-jackson:1.20.0:default}}
> {{com.google.http-client:google-http-client-jackson2:1.27.0:default}}
> {{com.google.http-client:google-http-client-protobuf:1.20.0:default}}
> {{com.google.j2objc:j2objc-annotations:1.1:default}}
> {{com.google.oauth-client:google-oauth-client:1.27.0:default}}
> {{com.google.oauth-client:google-oauth-client-java6:1.27.0:default}}
> {{com.google.protobuf:protobuf-java:3.6.1:default}}
> {{com.google.protobuf:protobuf-java-util:3.6.1:default}}
> {{com.google.protobuf.nano:protobuf-javanano:3.0.0-alpha-5:default}}
> {{com.squareup.okhttp:okhttp:2.5.0:default}}
> {{com.squareup.okio:okio:1.13.0:default}}
> {{com.thoughtworks.paranamer:paranamer:2.7:default}}
> {{commons-codec:commons-codec:1.10:default}}
> {{commons-logging:commons-logging:1.2:default}}
> {{io.dropwizard.metrics:metrics-core:3.1.2:default}}
> {{io.grpc:grpc-all:1.17.1:default}}
> {{io.grpc:grpc-alts:1.17.1:default}}
> {{io.grpc:grpc-auth:1.17.1:default}}
> {{io.grpc:grpc-context:1.17.1:default}}
> {{io.grpc:grpc-core:1.17.1:default}}
> {{io.grpc:grpc-grpclb:1.17.1:default}}
> {{io.grpc:grpc-netty:1.17.1:default}}
> {{io.grpc:grpc-netty-shaded:1.17.1:default}}
> {{io.grpc:grpc-okhttp:1.17.1:default}}
> {{io.grpc:grpc-protobuf:1.17.1:default}}
> {{io.grpc:grpc-protobuf-lite:1.17.1:default}}
> {{io.grpc:grpc-protobuf-nano:1.17.1:default}}
> {{io.grpc:grpc-stub:1.17.1:default}}
> {{io.grpc:grpc-testing:1.17.1:default}}
> {{io.netty:netty-buffer:4.1.30.Final:default}}
> {{io.netty:netty-codec:4.1.30.Final:default}}
> {{io.netty:netty-codec-http:4.1.30.Final:default}}
> {{io.netty:netty-codec-http2:4.1.30.Final:default}}
> {{io.netty:netty-codec-socks:4.1.30.Final:default}}
> {{io.netty:netty-common:4.1.30.Final:default}}
> {{io.netty:netty-handler:4.1.30.Final:default}}
> {{io.netty:netty-handler-proxy:4.1.30.Final:default}}
> {{io.netty:netty-resolver:4.1.30.Final:default}}
> {{io.netty:netty-tcnative-boringssl-static:2.0.17.Final:default}}
> {{io.netty:netty-transport:4.1.30.Final:default}}
> {{io.opencensus:opencensus-api:0.17.0:default}}
> {{io.opencensus:opencensus-contrib-grpc-metrics:0.17.0:default}}
> {{io.opencensus:opencensus-contrib-grpc-util:0.17.0:default}}
> {{io.opencensus:opencensus-contrib-http-util:0.15.0:default}}
> {{javax.annotation:javax.annotation-api:1.2:default}}
> {{joda-time:joda-time:2.10.3:default}}
> {{junit:junit:4.12:default}}
> {{org.apache.avro:avro:1.8.2:default}}
> {{org.apache.beam:beam-model-job-management:2.16.0:default}}
> {{org.apache.beam:beam-model-pipeline:2.16.0:default}}
> {{org.apache.beam:beam-sdks-java-core:2.16.0:default}}
> {{org.apache.beam:beam-sdks-java-extensions-google-cloud-platform-core:2.16.0:default}}
> {{org.apache.beam:beam-sdks-java-extensions-protobuf:2.16.0:default}}
> {{org.apache.beam:beam-sdks-java-io-google-cloud-platform:2.16.0:default}}
> {{org.apache.beam:beam-vendor-bytebuddy-1_9_3:0.1:default}}
> {{org.apache.beam:beam-vendor-grpc-1_21_0:0.1:default}}
> {{org.apache.beam:beam-vendor-guava-26_0-jre:0.1:default}}
> {{org.apache.commons:commons-compress:1.8.1:default}}
> {{org.apache.commons:commons-lang3:3.5:default}}
> {{org.apache.httpcomponents:httpclient:4.5.5:default}}
> {{org.apache.httpcomponents:httpcore:4.4.9:default}}
> {{org.checkerframework:checker-qual:2.5.2:default}}
> {{org.codehaus.jackson:jackson-core-asl:1.9.13:default}}
> {{org.codehaus.jackson:jackson-mapper-asl:1.9.13:default}}
> {{org.codehaus.mojo:animal-sniffer-annotations:1.17:default}}
> {{org.hamcrest:hamcrest-core:1.3:default}}
> {{org.mockito:mockito-core:1.9.5:default}}
> {{org.objenesis:objenesis:1.0:default}}
> {{org.slf4j:slf4j-api:1.7.25:default}}
> {{org.threeten:threetenbp:1.3.3:default}}
> {{org.tukaani:xz:1.8:default}}
> {{org.xerial.snappy:snappy-java:1.1.4:default}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)