You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Tomo Suzuki (Jira)" <ji...@apache.org> on 2021/03/15 15:49:00 UTC
[jira] [Comment Edited] (BEAM-11968) Bump
com.google.cloud:libraries-bom to 19.0.0
[ https://issues.apache.org/jira/browse/BEAM-11968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17301718#comment-17301718 ]
Tomo Suzuki edited comment on BEAM-11968 at 3/15/21, 3:48 PM:
--------------------------------------------------------------
I agree that it's nice to upgrade the version of the Libraries BOM. Thank you for PR.
But I wonder why the NoSuchMethodError happened.
h1. Troubleshooting
The Libraries BOM version was 13.2.0 in 2.28.0 tree https://github.com/apache/beam/blob/release-2.28.0/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L535. That version of libraries BOM had google-http-java-client version 1.36.0.
{{com.google.api.services.storage.Storage$Builder}} is {{com.google.apis:google-api-services-storage}} artifact. At 2.28.0 release, it was "v1-rev20200611-1.30.10" (the Libraries BOM does not have the artifact in it). I don't see the {{Storage$Builder}} class has a reference to the missing method.
{noformat}
suztomo-macbookpro44% javap -private -verbose -cp google-api-services-storage-v1-rev20200611-1.30.10.jar 'com.google.api.services.storage.Storage$Builder' |grep isMtls
suztomo-macbookpro44% javap -private -verbose -cp google-api-services-storage-v1-rev20200611-1.30.10.jar 'com.google.api.services.storage.Storage$Builder' |grep chooseEndpoint
suztomo-macbookpro44%
{noformat}
[~fokko] From the observation above, I'm assuming that you specify the version of {{com.google.apis:google-api-services-storage}} by yourself or there are other dependencies that fetched the different version than "v1-rev20200611-1.30.10". *Would you provide the dependency graph* ({{mvn dependency:tree}} or {{./gradlew dependencies}}) of the problematic project? The tree might reveal the unexpected consequence in Beam's dependency management and I'd like to fix it if any.
was (Author: suztomo):
I agree that it's nice to upgrade the version of the Libraries BOM.
h1. Troubleshooting
The Libraries BOM version was 13.2.0 in 2.28.0 tree https://github.com/apache/beam/blob/release-2.28.0/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L535. That version of libraries BOM had google-http-java-client version 1.36.0.
{{com.google.api.services.storage.Storage$Builder}} is {{com.google.apis:google-api-services-storage}} artifact. At 2.28.0 release, it was "v1-rev20200611-1.30.10" (the Libraries BOM does not have the artifact in it). I don't see the {{Storage$Builder}} class has a reference to the missing method.
{noformat}
suztomo-macbookpro44% javap -private -verbose -cp google-api-services-storage-v1-rev20200611-1.30.10.jar 'com.google.api.services.storage.Storage$Builder' |grep isMtls
suztomo-macbookpro44% javap -private -verbose -cp google-api-services-storage-v1-rev20200611-1.30.10.jar 'com.google.api.services.storage.Storage$Builder' |grep chooseEndpoint
suztomo-macbookpro44%
{noformat}
[~fokko] From the observation above, I'm assuming that you specify the version of {{com.google.apis:google-api-services-storage}} by yourself or there are other dependencies that fetched the different version than "v1-rev20200611-1.30.10". *Would you provide the dependency graph* ({{mvn dependency:tree}} or {{./gradlew dependencies}}) of the problematic project? The tree might reveal the unexpected consequence in Beam's dependency management and I'd like to fix it if any.
> Bump com.google.cloud:libraries-bom to 19.0.0
> ---------------------------------------------
>
> Key: BEAM-11968
> URL: https://issues.apache.org/jira/browse/BEAM-11968
> Project: Beam
> Issue Type: Bug
> Components: extensions-java-gcp
> Affects Versions: 2.28.0
> Reporter: Fokko Driesprong
> Assignee: Fokko Driesprong
> Priority: P2
> Fix For: 2.29.0
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Missing the isMtls() in google-http-client. Looks like it has been added since 1.38.0:
> [https://github.com/googleapis/google-http-java-client/blob/master/google-http-client/src/main/java/com/google/api/client/http/HttpTransport.java#L136]
> Original stacktrace:
> java.lang.RuntimeException: Encountered checked exception when constructing an instance from factory method DataflowRunner#fromOptions(interface org.apache.beam.sdk.options.PipelineOptions)
> at org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod (InstanceBuilder.java:233)
> at org.apache.beam.sdk.util.InstanceBuilder.build (InstanceBuilder.java:158)
> at org.apache.beam.sdk.PipelineRunner.fromOptions (PipelineRunner.java:55)
> at org.apache.beam.sdk.Pipeline.create (Pipeline.java:154)
> ...
> at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
> at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:64)
> at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke (Method.java:564)
> at org.codehaus.mojo.exec.ExecJavaMojo$1.run (ExecJavaMojo.java:282)
> at java.lang.Thread.run (Thread.java:832)
> Caused by: java.lang.NoSuchMethodError: 'boolean com.google.api.client.http.HttpTransport.isMtls()'
> at com.google.api.services.storage.Storage$Builder.chooseEndpoint (Storage.java:11151)
> at com.google.api.services.storage.Storage$Builder.<init> (Storage.java:11184)
> at org.apache.beam.sdk.extensions.gcp.util.Transport.newStorageClient (Transport.java:97)
> at org.apache.beam.sdk.extensions.gcp.util.GcsUtil$GcsUtilFactory.create (GcsUtil.java:104)
> at org.apache.beam.sdk.extensions.gcp.util.GcsUtil$GcsUtilFactory.create (GcsUtil.java:93)
> at org.apache.beam.sdk.options.ProxyInvocationHandler.returnDefaultHelper (ProxyInvocationHandler.java:605)
> at org.apache.beam.sdk.options.ProxyInvocationHandler.getDefault (ProxyInvocationHandler.java:546)
> at org.apache.beam.sdk.options.ProxyInvocationHandler.invoke (ProxyInvocationHandler.java:171)
> at com.sun.proxy.$Proxy41.getGcsUtil (Unknown Source)
> at org.apache.beam.sdk.extensions.gcp.storage.GcsPathValidator.verifyPathIsAccessible (GcsPathValidator.java:83)
> at org.apache.beam.sdk.extensions.gcp.storage.GcsPathValidator.validateOutputFilePrefixSupported (GcsPathValidator.java:53)
> at org.apache.beam.runners.dataflow.DataflowRunner.fromOptions (DataflowRunner.java:291)
> at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
> at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:64)
> at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke (Method.java:564)
> at org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod (InstanceBuilder.java:217)
> at org.apache.beam.sdk.util.InstanceBuilder.build (InstanceBuilder.java:158)
> at org.apache.beam.sdk.PipelineRunner.fromOptions (PipelineRunner.java:55)
> at org.apache.beam.sdk.Pipeline.create (Pipeline.java:154)
> ...
> at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
> at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:64)
> at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke (Method.java:564)
> at org.codehaus.mojo.exec.ExecJavaMojo$1.run (ExecJavaMojo.java:282)
> at java.lang.Thread.run (Thread.java:832)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)