You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Zhiheng Huang <sy...@gmail.com> on 2020/03/12 04:37:09 UTC
BigQueryStorage API Dependency
Hi,
I am trying to use big query storage API on google dataflow, but I keep
getting:
java.lang.NoClassDefFoundError: Could not initialize class
com.google.cloud.bigquery.storage.v1beta1.Storage
at
com.google.cloud.bigquery.storage.v1beta1.Storage$CreateReadSessionRequest.internalGetFieldAccessorTable(Storage.java:5216)
at
com.google.protobuf.GeneratedMessageV3.getAllFieldsMutable(GeneratedMessageV3.java:135)
at
com.google.protobuf.GeneratedMessageV3.getAllFields(GeneratedMessageV3.java:211)
at com.google.protobuf.TextFormat$Printer.print(TextFormat.java:324)
at com.google.protobuf.TextFormat$Printer.access$000(TextFormat.java:307)
at com.google.protobuf.TextFormat.print(TextFormat.java:68)
at com.google.protobuf.TextFormat.printToString(TextFormat.java:148)
at com.google.protobuf.AbstractMessage.toString(AbstractMessage.java:117)
at
org.slf4j.helpers.MessageFormatter.safeObjectAppend(MessageFormatter.java:299)
at
org.slf4j.helpers.MessageFormatter.deeplyAppendParameter(MessageFormatter.java:271)
at org.slf4j.helpers.MessageFormatter.arrayFormat(MessageFormatter.java:233)
at org.slf4j.helpers.MessageFormatter.arrayFormat(MessageFormatter.java:173)
at org.slf4j.helpers.MessageFormatter.format(MessageFormatter.java:151)
at org.slf4j.impl.SimpleLogger.formatAndLog(SimpleLogger.java:354)
at org.slf4j.impl.SimpleLogger.info(SimpleLogger.java:496)
at
org.apache.beam.sdk.io.gcp.bigquery.BigQueryStorageSourceBase.split(BigQueryStorageSourceBase.java:140)
at
org.apache.beam.sdk.io.gcp.bigquery.BigQueryStorageTableSource.split(BigQueryStorageTableSource.java:42)
...
This seems like a library dependency problem. Anyone can point me to the
correct library dependency for using the API or know what might be wrong
here? I am using beam 2.19 and already included
"com.google.api.grpc:proto-google-cloud-bigquerystorage-v1beta1:0.83.0" and
"com.google.cloud:google-cloud-bigquerystorage:0.79.0-alpha" in my gradle
dependency.
Thanks!
Re: BigQueryStorage API Dependency
Posted by Tomo Suzuki <su...@google.com>.
Glad to hear it's resolved.
"./gradlew dependencies" tells why a particular version of artifact appears
in your class path.
On Thu, Mar 19, 2020 at 17:30 Zhiheng Huang <sy...@gmail.com> wrote:
> Hi,
>
> I spent more time on this and is able to figure out. There's indeed a
> static block in Storage.java class that tries to load the proto
> class com.google.api.ClientProto.
> I fixed it by forcing to use
> com.google.api.grpc:proto-google-common-protos:1.17.0 in build.gradle. For
> some reason(maybe transient dependency?), the project
> tries to pull 1.12.0 which doesn't have com.google.api.ClientProto
>
> Thanks for the help!
>
> On Mon, Mar 16, 2020 at 5:10 PM Tomo Suzuki <su...@google.com> wrote:
>
>> That TableReferenceProto class is in
>> com.google.api.grpc:proto-google-cloud-bigquerystorage-v1beta1:0.83.0 as
>> per the build.gradle. Doesn't your environment find that class?
>>
>> [image: tableproto.png]
>>
>> I believe the NoClassDefFoundError is caused by either
>> - your class path does not have the library, or
>> - the static initializer fails
>>
>>
>>
>> On Sun, Mar 15, 2020 at 12:30 AM Zhiheng Huang <sy...@gmail.com>
>> wrote:
>>
>>> Hi Tomo,
>>>
>>> I put a breakpoint and try tracing through it. I do not find any static
>>> initialization problems. com.google.cloud.bigquery.storage.v1beta1.Storage
>>> is an auto generated protobuf class.
>>> In fact, seems like when the code tries to access protobuf class in
>>> package "com.google.cloud.bigquery.storage.v1beta1" it will throw
>>> NoClassDefFound exception. For example, I
>>> also get NoClassDefFoundError for
>>> com.google.cloud.bigquery.storage.v1beta1.TableReferenceProto class.
>>>
>>> Thanks!
>>>
>>> On Thu, Mar 12, 2020 at 6:20 PM Tomo Suzuki <su...@google.com> wrote:
>>>
>>>> I tried looking at the dependency tree but couldn't identify any
>>>> discrepancy in versions. In general, NoClassDefFoundError occurs when a
>>>> class is missing as well as static initializers fails. From the error, I
>>>> feel that's the latter in this case.
>>>> Is it possible to set a breakpoint at
>>>> com.google.cloud.bigquery.storage.v1beta1.Storage line 5216 and see any
>>>> failure in static fields of the Storage class?
>>>>
>>>>
>>>>
>>>> On Thu, Mar 12, 2020 at 5:50 PM Zhiheng Huang <sy...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Tomo,
>>>>>
>>>>> Thanks for looking. Not the actual build.gradle, but this build.gradle
>>>>> <https://gist.github.com/sylvon/908bd08424cfbcc91f1819ab5325d6fa>
>>>>> shows the google/beam related dependencies I used.
>>>>>
>>>>> Thanks!
>>>>>
>>>>> On Thu, Mar 12, 2020 at 7:49 AM Tomo Suzuki <su...@google.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Zhiheng,
>>>>>> Would you share your pom.xml or build.gradle? I'd like to know the
>>>>>> dependencies of your project.
>>>>>>
>>>>>> On Thu, Mar 12, 2020 at 12:37 AM Zhiheng Huang <sy...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I am trying to use big query storage API on google dataflow, but I
>>>>>>> keep getting:
>>>>>>>
>>>>>>> java.lang.NoClassDefFoundError: Could not initialize class
>>>>>>> com.google.cloud.bigquery.storage.v1beta1.Storage
>>>>>>> at
>>>>>>> com.google.cloud.bigquery.storage.v1beta1.Storage$CreateReadSessionRequest.internalGetFieldAccessorTable(Storage.java:5216)
>>>>>>> at
>>>>>>> com.google.protobuf.GeneratedMessageV3.getAllFieldsMutable(GeneratedMessageV3.java:135)
>>>>>>> at
>>>>>>> com.google.protobuf.GeneratedMessageV3.getAllFields(GeneratedMessageV3.java:211)
>>>>>>> at com.google.protobuf.TextFormat$Printer.print(TextFormat.java:324)
>>>>>>> at
>>>>>>> com.google.protobuf.TextFormat$Printer.access$000(TextFormat.java:307)
>>>>>>> at com.google.protobuf.TextFormat.print(TextFormat.java:68)
>>>>>>> at com.google.protobuf.TextFormat.printToString(TextFormat.java:148)
>>>>>>> at
>>>>>>> com.google.protobuf.AbstractMessage.toString(AbstractMessage.java:117)
>>>>>>> at
>>>>>>> org.slf4j.helpers.MessageFormatter.safeObjectAppend(MessageFormatter.java:299)
>>>>>>> at
>>>>>>> org.slf4j.helpers.MessageFormatter.deeplyAppendParameter(MessageFormatter.java:271)
>>>>>>> at
>>>>>>> org.slf4j.helpers.MessageFormatter.arrayFormat(MessageFormatter.java:233)
>>>>>>> at
>>>>>>> org.slf4j.helpers.MessageFormatter.arrayFormat(MessageFormatter.java:173)
>>>>>>> at
>>>>>>> org.slf4j.helpers.MessageFormatter.format(MessageFormatter.java:151)
>>>>>>> at org.slf4j.impl.SimpleLogger.formatAndLog(SimpleLogger.java:354)
>>>>>>> at org.slf4j.impl.SimpleLogger.info(SimpleLogger.java:496)
>>>>>>> at
>>>>>>> org.apache.beam.sdk.io.gcp.bigquery.BigQueryStorageSourceBase.split(BigQueryStorageSourceBase.java:140)
>>>>>>> at
>>>>>>> org.apache.beam.sdk.io.gcp.bigquery.BigQueryStorageTableSource.split(BigQueryStorageTableSource.java:42)
>>>>>>> ...
>>>>>>>
>>>>>>>
>>>>>>> This seems like a library dependency problem. Anyone can point me to
>>>>>>> the correct library dependency for using the API or know what might be
>>>>>>> wrong here? I am using beam 2.19 and already included
>>>>>>> "com.google.api.grpc:proto-google-cloud-bigquerystorage-v1beta1:0.83.0" and
>>>>>>> "com.google.cloud:google-cloud-bigquerystorage:0.79.0-alpha" in my gradle
>>>>>>> dependency.
>>>>>>>
>>>>>>> Thanks!
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Regards,
>>>>>> Tomo
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Sylvon Huang
>>>>>
>>>>
>>>>
>>>> --
>>>> Regards,
>>>> Tomo
>>>>
>>>
>>>
>>> --
>>> Sylvon Huang
>>>
>>
>>
>> --
>> Regards,
>> Tomo
>>
> --
Regards,
Tomo
Re: BigQueryStorage API Dependency
Posted by Zhiheng Huang <sy...@gmail.com>.
Hi,
I spent more time on this and is able to figure out. There's indeed a
static block in Storage.java class that tries to load the proto
class com.google.api.ClientProto.
I fixed it by forcing to use
com.google.api.grpc:proto-google-common-protos:1.17.0 in build.gradle. For
some reason(maybe transient dependency?), the project
tries to pull 1.12.0 which doesn't have com.google.api.ClientProto
Thanks for the help!
On Mon, Mar 16, 2020 at 5:10 PM Tomo Suzuki <su...@google.com> wrote:
> That TableReferenceProto class is in
> com.google.api.grpc:proto-google-cloud-bigquerystorage-v1beta1:0.83.0 as
> per the build.gradle. Doesn't your environment find that class?
>
> [image: tableproto.png]
>
> I believe the NoClassDefFoundError is caused by either
> - your class path does not have the library, or
> - the static initializer fails
>
>
>
> On Sun, Mar 15, 2020 at 12:30 AM Zhiheng Huang <sy...@gmail.com>
> wrote:
>
>> Hi Tomo,
>>
>> I put a breakpoint and try tracing through it. I do not find any static
>> initialization problems. com.google.cloud.bigquery.storage.v1beta1.Storage
>> is an auto generated protobuf class.
>> In fact, seems like when the code tries to access protobuf class in
>> package "com.google.cloud.bigquery.storage.v1beta1" it will throw
>> NoClassDefFound exception. For example, I
>> also get NoClassDefFoundError for
>> com.google.cloud.bigquery.storage.v1beta1.TableReferenceProto class.
>>
>> Thanks!
>>
>> On Thu, Mar 12, 2020 at 6:20 PM Tomo Suzuki <su...@google.com> wrote:
>>
>>> I tried looking at the dependency tree but couldn't identify any
>>> discrepancy in versions. In general, NoClassDefFoundError occurs when a
>>> class is missing as well as static initializers fails. From the error, I
>>> feel that's the latter in this case.
>>> Is it possible to set a breakpoint at
>>> com.google.cloud.bigquery.storage.v1beta1.Storage line 5216 and see any
>>> failure in static fields of the Storage class?
>>>
>>>
>>>
>>> On Thu, Mar 12, 2020 at 5:50 PM Zhiheng Huang <sy...@gmail.com>
>>> wrote:
>>>
>>>> Hi Tomo,
>>>>
>>>> Thanks for looking. Not the actual build.gradle, but this build.gradle
>>>> <https://gist.github.com/sylvon/908bd08424cfbcc91f1819ab5325d6fa>
>>>> shows the google/beam related dependencies I used.
>>>>
>>>> Thanks!
>>>>
>>>> On Thu, Mar 12, 2020 at 7:49 AM Tomo Suzuki <su...@google.com> wrote:
>>>>
>>>>> Hi Zhiheng,
>>>>> Would you share your pom.xml or build.gradle? I'd like to know the
>>>>> dependencies of your project.
>>>>>
>>>>> On Thu, Mar 12, 2020 at 12:37 AM Zhiheng Huang <sy...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I am trying to use big query storage API on google dataflow, but I
>>>>>> keep getting:
>>>>>>
>>>>>> java.lang.NoClassDefFoundError: Could not initialize class
>>>>>> com.google.cloud.bigquery.storage.v1beta1.Storage
>>>>>> at
>>>>>> com.google.cloud.bigquery.storage.v1beta1.Storage$CreateReadSessionRequest.internalGetFieldAccessorTable(Storage.java:5216)
>>>>>> at
>>>>>> com.google.protobuf.GeneratedMessageV3.getAllFieldsMutable(GeneratedMessageV3.java:135)
>>>>>> at
>>>>>> com.google.protobuf.GeneratedMessageV3.getAllFields(GeneratedMessageV3.java:211)
>>>>>> at com.google.protobuf.TextFormat$Printer.print(TextFormat.java:324)
>>>>>> at
>>>>>> com.google.protobuf.TextFormat$Printer.access$000(TextFormat.java:307)
>>>>>> at com.google.protobuf.TextFormat.print(TextFormat.java:68)
>>>>>> at com.google.protobuf.TextFormat.printToString(TextFormat.java:148)
>>>>>> at
>>>>>> com.google.protobuf.AbstractMessage.toString(AbstractMessage.java:117)
>>>>>> at
>>>>>> org.slf4j.helpers.MessageFormatter.safeObjectAppend(MessageFormatter.java:299)
>>>>>> at
>>>>>> org.slf4j.helpers.MessageFormatter.deeplyAppendParameter(MessageFormatter.java:271)
>>>>>> at
>>>>>> org.slf4j.helpers.MessageFormatter.arrayFormat(MessageFormatter.java:233)
>>>>>> at
>>>>>> org.slf4j.helpers.MessageFormatter.arrayFormat(MessageFormatter.java:173)
>>>>>> at
>>>>>> org.slf4j.helpers.MessageFormatter.format(MessageFormatter.java:151)
>>>>>> at org.slf4j.impl.SimpleLogger.formatAndLog(SimpleLogger.java:354)
>>>>>> at org.slf4j.impl.SimpleLogger.info(SimpleLogger.java:496)
>>>>>> at
>>>>>> org.apache.beam.sdk.io.gcp.bigquery.BigQueryStorageSourceBase.split(BigQueryStorageSourceBase.java:140)
>>>>>> at
>>>>>> org.apache.beam.sdk.io.gcp.bigquery.BigQueryStorageTableSource.split(BigQueryStorageTableSource.java:42)
>>>>>> ...
>>>>>>
>>>>>>
>>>>>> This seems like a library dependency problem. Anyone can point me to
>>>>>> the correct library dependency for using the API or know what might be
>>>>>> wrong here? I am using beam 2.19 and already included
>>>>>> "com.google.api.grpc:proto-google-cloud-bigquerystorage-v1beta1:0.83.0" and
>>>>>> "com.google.cloud:google-cloud-bigquerystorage:0.79.0-alpha" in my gradle
>>>>>> dependency.
>>>>>>
>>>>>> Thanks!
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> Regards,
>>>>> Tomo
>>>>>
>>>>
>>>>
>>>> --
>>>> Sylvon Huang
>>>>
>>>
>>>
>>> --
>>> Regards,
>>> Tomo
>>>
>>
>>
>> --
>> Sylvon Huang
>>
>
>
> --
> Regards,
> Tomo
>
Re: BigQueryStorage API Dependency
Posted by Tomo Suzuki <su...@google.com>.
That TableReferenceProto class is in
com.google.api.grpc:proto-google-cloud-bigquerystorage-v1beta1:0.83.0 as
per the build.gradle. Doesn't your environment find that class?
[image: tableproto.png]
I believe the NoClassDefFoundError is caused by either
- your class path does not have the library, or
- the static initializer fails
On Sun, Mar 15, 2020 at 12:30 AM Zhiheng Huang <sy...@gmail.com>
wrote:
> Hi Tomo,
>
> I put a breakpoint and try tracing through it. I do not find any static
> initialization problems. com.google.cloud.bigquery.storage.v1beta1.Storage
> is an auto generated protobuf class.
> In fact, seems like when the code tries to access protobuf class in
> package "com.google.cloud.bigquery.storage.v1beta1" it will throw
> NoClassDefFound exception. For example, I
> also get NoClassDefFoundError for
> com.google.cloud.bigquery.storage.v1beta1.TableReferenceProto class.
>
> Thanks!
>
> On Thu, Mar 12, 2020 at 6:20 PM Tomo Suzuki <su...@google.com> wrote:
>
>> I tried looking at the dependency tree but couldn't identify any
>> discrepancy in versions. In general, NoClassDefFoundError occurs when a
>> class is missing as well as static initializers fails. From the error, I
>> feel that's the latter in this case.
>> Is it possible to set a breakpoint at
>> com.google.cloud.bigquery.storage.v1beta1.Storage line 5216 and see any
>> failure in static fields of the Storage class?
>>
>>
>>
>> On Thu, Mar 12, 2020 at 5:50 PM Zhiheng Huang <sy...@gmail.com>
>> wrote:
>>
>>> Hi Tomo,
>>>
>>> Thanks for looking. Not the actual build.gradle, but this build.gradle
>>> <https://gist.github.com/sylvon/908bd08424cfbcc91f1819ab5325d6fa> shows
>>> the google/beam related dependencies I used.
>>>
>>> Thanks!
>>>
>>> On Thu, Mar 12, 2020 at 7:49 AM Tomo Suzuki <su...@google.com> wrote:
>>>
>>>> Hi Zhiheng,
>>>> Would you share your pom.xml or build.gradle? I'd like to know the
>>>> dependencies of your project.
>>>>
>>>> On Thu, Mar 12, 2020 at 12:37 AM Zhiheng Huang <sy...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I am trying to use big query storage API on google dataflow, but I
>>>>> keep getting:
>>>>>
>>>>> java.lang.NoClassDefFoundError: Could not initialize class
>>>>> com.google.cloud.bigquery.storage.v1beta1.Storage
>>>>> at
>>>>> com.google.cloud.bigquery.storage.v1beta1.Storage$CreateReadSessionRequest.internalGetFieldAccessorTable(Storage.java:5216)
>>>>> at
>>>>> com.google.protobuf.GeneratedMessageV3.getAllFieldsMutable(GeneratedMessageV3.java:135)
>>>>> at
>>>>> com.google.protobuf.GeneratedMessageV3.getAllFields(GeneratedMessageV3.java:211)
>>>>> at com.google.protobuf.TextFormat$Printer.print(TextFormat.java:324)
>>>>> at
>>>>> com.google.protobuf.TextFormat$Printer.access$000(TextFormat.java:307)
>>>>> at com.google.protobuf.TextFormat.print(TextFormat.java:68)
>>>>> at com.google.protobuf.TextFormat.printToString(TextFormat.java:148)
>>>>> at
>>>>> com.google.protobuf.AbstractMessage.toString(AbstractMessage.java:117)
>>>>> at
>>>>> org.slf4j.helpers.MessageFormatter.safeObjectAppend(MessageFormatter.java:299)
>>>>> at
>>>>> org.slf4j.helpers.MessageFormatter.deeplyAppendParameter(MessageFormatter.java:271)
>>>>> at
>>>>> org.slf4j.helpers.MessageFormatter.arrayFormat(MessageFormatter.java:233)
>>>>> at
>>>>> org.slf4j.helpers.MessageFormatter.arrayFormat(MessageFormatter.java:173)
>>>>> at org.slf4j.helpers.MessageFormatter.format(MessageFormatter.java:151)
>>>>> at org.slf4j.impl.SimpleLogger.formatAndLog(SimpleLogger.java:354)
>>>>> at org.slf4j.impl.SimpleLogger.info(SimpleLogger.java:496)
>>>>> at
>>>>> org.apache.beam.sdk.io.gcp.bigquery.BigQueryStorageSourceBase.split(BigQueryStorageSourceBase.java:140)
>>>>> at
>>>>> org.apache.beam.sdk.io.gcp.bigquery.BigQueryStorageTableSource.split(BigQueryStorageTableSource.java:42)
>>>>> ...
>>>>>
>>>>>
>>>>> This seems like a library dependency problem. Anyone can point me to
>>>>> the correct library dependency for using the API or know what might be
>>>>> wrong here? I am using beam 2.19 and already included
>>>>> "com.google.api.grpc:proto-google-cloud-bigquerystorage-v1beta1:0.83.0" and
>>>>> "com.google.cloud:google-cloud-bigquerystorage:0.79.0-alpha" in my gradle
>>>>> dependency.
>>>>>
>>>>> Thanks!
>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>> Regards,
>>>> Tomo
>>>>
>>>
>>>
>>> --
>>> Sylvon Huang
>>>
>>
>>
>> --
>> Regards,
>> Tomo
>>
>
>
> --
> Sylvon Huang
>
--
Regards,
Tomo
Re: BigQueryStorage API Dependency
Posted by Zhiheng Huang <sy...@gmail.com>.
Hi Tomo,
I put a breakpoint and try tracing through it. I do not find any static
initialization problems. com.google.cloud.bigquery.storage.v1beta1.Storage
is an auto generated protobuf class.
In fact, seems like when the code tries to access protobuf class in package
"com.google.cloud.bigquery.storage.v1beta1" it will throw NoClassDefFound
exception. For example, I
also get NoClassDefFoundError for
com.google.cloud.bigquery.storage.v1beta1.TableReferenceProto class.
Thanks!
On Thu, Mar 12, 2020 at 6:20 PM Tomo Suzuki <su...@google.com> wrote:
> I tried looking at the dependency tree but couldn't identify any
> discrepancy in versions. In general, NoClassDefFoundError occurs when a
> class is missing as well as static initializers fails. From the error, I
> feel that's the latter in this case.
> Is it possible to set a breakpoint at
> com.google.cloud.bigquery.storage.v1beta1.Storage line 5216 and see any
> failure in static fields of the Storage class?
>
>
>
> On Thu, Mar 12, 2020 at 5:50 PM Zhiheng Huang <sy...@gmail.com>
> wrote:
>
>> Hi Tomo,
>>
>> Thanks for looking. Not the actual build.gradle, but this build.gradle
>> <https://gist.github.com/sylvon/908bd08424cfbcc91f1819ab5325d6fa> shows
>> the google/beam related dependencies I used.
>>
>> Thanks!
>>
>> On Thu, Mar 12, 2020 at 7:49 AM Tomo Suzuki <su...@google.com> wrote:
>>
>>> Hi Zhiheng,
>>> Would you share your pom.xml or build.gradle? I'd like to know the
>>> dependencies of your project.
>>>
>>> On Thu, Mar 12, 2020 at 12:37 AM Zhiheng Huang <sy...@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> I am trying to use big query storage API on google dataflow, but I keep
>>>> getting:
>>>>
>>>> java.lang.NoClassDefFoundError: Could not initialize class
>>>> com.google.cloud.bigquery.storage.v1beta1.Storage
>>>> at
>>>> com.google.cloud.bigquery.storage.v1beta1.Storage$CreateReadSessionRequest.internalGetFieldAccessorTable(Storage.java:5216)
>>>> at
>>>> com.google.protobuf.GeneratedMessageV3.getAllFieldsMutable(GeneratedMessageV3.java:135)
>>>> at
>>>> com.google.protobuf.GeneratedMessageV3.getAllFields(GeneratedMessageV3.java:211)
>>>> at com.google.protobuf.TextFormat$Printer.print(TextFormat.java:324)
>>>> at
>>>> com.google.protobuf.TextFormat$Printer.access$000(TextFormat.java:307)
>>>> at com.google.protobuf.TextFormat.print(TextFormat.java:68)
>>>> at com.google.protobuf.TextFormat.printToString(TextFormat.java:148)
>>>> at
>>>> com.google.protobuf.AbstractMessage.toString(AbstractMessage.java:117)
>>>> at
>>>> org.slf4j.helpers.MessageFormatter.safeObjectAppend(MessageFormatter.java:299)
>>>> at
>>>> org.slf4j.helpers.MessageFormatter.deeplyAppendParameter(MessageFormatter.java:271)
>>>> at
>>>> org.slf4j.helpers.MessageFormatter.arrayFormat(MessageFormatter.java:233)
>>>> at
>>>> org.slf4j.helpers.MessageFormatter.arrayFormat(MessageFormatter.java:173)
>>>> at org.slf4j.helpers.MessageFormatter.format(MessageFormatter.java:151)
>>>> at org.slf4j.impl.SimpleLogger.formatAndLog(SimpleLogger.java:354)
>>>> at org.slf4j.impl.SimpleLogger.info(SimpleLogger.java:496)
>>>> at
>>>> org.apache.beam.sdk.io.gcp.bigquery.BigQueryStorageSourceBase.split(BigQueryStorageSourceBase.java:140)
>>>> at
>>>> org.apache.beam.sdk.io.gcp.bigquery.BigQueryStorageTableSource.split(BigQueryStorageTableSource.java:42)
>>>> ...
>>>>
>>>>
>>>> This seems like a library dependency problem. Anyone can point me to
>>>> the correct library dependency for using the API or know what might be
>>>> wrong here? I am using beam 2.19 and already included
>>>> "com.google.api.grpc:proto-google-cloud-bigquerystorage-v1beta1:0.83.0" and
>>>> "com.google.cloud:google-cloud-bigquerystorage:0.79.0-alpha" in my gradle
>>>> dependency.
>>>>
>>>> Thanks!
>>>>
>>>>
>>>>
>>>
>>> --
>>> Regards,
>>> Tomo
>>>
>>
>>
>> --
>> Sylvon Huang
>>
>
>
> --
> Regards,
> Tomo
>
--
Sylvon Huang
Re: BigQueryStorage API Dependency
Posted by Tomo Suzuki <su...@google.com>.
I tried looking at the dependency tree but couldn't identify any
discrepancy in versions. In general, NoClassDefFoundError occurs when a
class is missing as well as static initializers fails. From the error, I
feel that's the latter in this case.
Is it possible to set a breakpoint at
com.google.cloud.bigquery.storage.v1beta1.Storage line 5216 and see any
failure in static fields of the Storage class?
On Thu, Mar 12, 2020 at 5:50 PM Zhiheng Huang <sy...@gmail.com> wrote:
> Hi Tomo,
>
> Thanks for looking. Not the actual build.gradle, but this build.gradle
> <https://gist.github.com/sylvon/908bd08424cfbcc91f1819ab5325d6fa> shows
> the google/beam related dependencies I used.
>
> Thanks!
>
> On Thu, Mar 12, 2020 at 7:49 AM Tomo Suzuki <su...@google.com> wrote:
>
>> Hi Zhiheng,
>> Would you share your pom.xml or build.gradle? I'd like to know the
>> dependencies of your project.
>>
>> On Thu, Mar 12, 2020 at 12:37 AM Zhiheng Huang <sy...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> I am trying to use big query storage API on google dataflow, but I keep
>>> getting:
>>>
>>> java.lang.NoClassDefFoundError: Could not initialize class
>>> com.google.cloud.bigquery.storage.v1beta1.Storage
>>> at
>>> com.google.cloud.bigquery.storage.v1beta1.Storage$CreateReadSessionRequest.internalGetFieldAccessorTable(Storage.java:5216)
>>> at
>>> com.google.protobuf.GeneratedMessageV3.getAllFieldsMutable(GeneratedMessageV3.java:135)
>>> at
>>> com.google.protobuf.GeneratedMessageV3.getAllFields(GeneratedMessageV3.java:211)
>>> at com.google.protobuf.TextFormat$Printer.print(TextFormat.java:324)
>>> at com.google.protobuf.TextFormat$Printer.access$000(TextFormat.java:307)
>>> at com.google.protobuf.TextFormat.print(TextFormat.java:68)
>>> at com.google.protobuf.TextFormat.printToString(TextFormat.java:148)
>>> at com.google.protobuf.AbstractMessage.toString(AbstractMessage.java:117)
>>> at
>>> org.slf4j.helpers.MessageFormatter.safeObjectAppend(MessageFormatter.java:299)
>>> at
>>> org.slf4j.helpers.MessageFormatter.deeplyAppendParameter(MessageFormatter.java:271)
>>> at
>>> org.slf4j.helpers.MessageFormatter.arrayFormat(MessageFormatter.java:233)
>>> at
>>> org.slf4j.helpers.MessageFormatter.arrayFormat(MessageFormatter.java:173)
>>> at org.slf4j.helpers.MessageFormatter.format(MessageFormatter.java:151)
>>> at org.slf4j.impl.SimpleLogger.formatAndLog(SimpleLogger.java:354)
>>> at org.slf4j.impl.SimpleLogger.info(SimpleLogger.java:496)
>>> at
>>> org.apache.beam.sdk.io.gcp.bigquery.BigQueryStorageSourceBase.split(BigQueryStorageSourceBase.java:140)
>>> at
>>> org.apache.beam.sdk.io.gcp.bigquery.BigQueryStorageTableSource.split(BigQueryStorageTableSource.java:42)
>>> ...
>>>
>>>
>>> This seems like a library dependency problem. Anyone can point me to the
>>> correct library dependency for using the API or know what might be wrong
>>> here? I am using beam 2.19 and already included
>>> "com.google.api.grpc:proto-google-cloud-bigquerystorage-v1beta1:0.83.0" and
>>> "com.google.cloud:google-cloud-bigquerystorage:0.79.0-alpha" in my gradle
>>> dependency.
>>>
>>> Thanks!
>>>
>>>
>>>
>>
>> --
>> Regards,
>> Tomo
>>
>
>
> --
> Sylvon Huang
>
--
Regards,
Tomo
Re: BigQueryStorage API Dependency
Posted by Zhiheng Huang <sy...@gmail.com>.
Hi Tomo,
Thanks for looking. Not the actual build.gradle, but this build.gradle
<https://gist.github.com/sylvon/908bd08424cfbcc91f1819ab5325d6fa> shows the
google/beam related dependencies I used.
Thanks!
On Thu, Mar 12, 2020 at 7:49 AM Tomo Suzuki <su...@google.com> wrote:
> Hi Zhiheng,
> Would you share your pom.xml or build.gradle? I'd like to know the
> dependencies of your project.
>
> On Thu, Mar 12, 2020 at 12:37 AM Zhiheng Huang <sy...@gmail.com>
> wrote:
>
>> Hi,
>>
>> I am trying to use big query storage API on google dataflow, but I keep
>> getting:
>>
>> java.lang.NoClassDefFoundError: Could not initialize class
>> com.google.cloud.bigquery.storage.v1beta1.Storage
>> at
>> com.google.cloud.bigquery.storage.v1beta1.Storage$CreateReadSessionRequest.internalGetFieldAccessorTable(Storage.java:5216)
>> at
>> com.google.protobuf.GeneratedMessageV3.getAllFieldsMutable(GeneratedMessageV3.java:135)
>> at
>> com.google.protobuf.GeneratedMessageV3.getAllFields(GeneratedMessageV3.java:211)
>> at com.google.protobuf.TextFormat$Printer.print(TextFormat.java:324)
>> at com.google.protobuf.TextFormat$Printer.access$000(TextFormat.java:307)
>> at com.google.protobuf.TextFormat.print(TextFormat.java:68)
>> at com.google.protobuf.TextFormat.printToString(TextFormat.java:148)
>> at com.google.protobuf.AbstractMessage.toString(AbstractMessage.java:117)
>> at
>> org.slf4j.helpers.MessageFormatter.safeObjectAppend(MessageFormatter.java:299)
>> at
>> org.slf4j.helpers.MessageFormatter.deeplyAppendParameter(MessageFormatter.java:271)
>> at
>> org.slf4j.helpers.MessageFormatter.arrayFormat(MessageFormatter.java:233)
>> at
>> org.slf4j.helpers.MessageFormatter.arrayFormat(MessageFormatter.java:173)
>> at org.slf4j.helpers.MessageFormatter.format(MessageFormatter.java:151)
>> at org.slf4j.impl.SimpleLogger.formatAndLog(SimpleLogger.java:354)
>> at org.slf4j.impl.SimpleLogger.info(SimpleLogger.java:496)
>> at
>> org.apache.beam.sdk.io.gcp.bigquery.BigQueryStorageSourceBase.split(BigQueryStorageSourceBase.java:140)
>> at
>> org.apache.beam.sdk.io.gcp.bigquery.BigQueryStorageTableSource.split(BigQueryStorageTableSource.java:42)
>> ...
>>
>>
>> This seems like a library dependency problem. Anyone can point me to the
>> correct library dependency for using the API or know what might be wrong
>> here? I am using beam 2.19 and already included
>> "com.google.api.grpc:proto-google-cloud-bigquerystorage-v1beta1:0.83.0" and
>> "com.google.cloud:google-cloud-bigquerystorage:0.79.0-alpha" in my gradle
>> dependency.
>>
>> Thanks!
>>
>>
>>
>
> --
> Regards,
> Tomo
>
--
Sylvon Huang
Re: BigQueryStorage API Dependency
Posted by Tomo Suzuki <su...@google.com>.
Hi Zhiheng,
Would you share your pom.xml or build.gradle? I'd like to know the
dependencies of your project.
On Thu, Mar 12, 2020 at 12:37 AM Zhiheng Huang <sy...@gmail.com>
wrote:
> Hi,
>
> I am trying to use big query storage API on google dataflow, but I keep
> getting:
>
> java.lang.NoClassDefFoundError: Could not initialize class
> com.google.cloud.bigquery.storage.v1beta1.Storage
> at
> com.google.cloud.bigquery.storage.v1beta1.Storage$CreateReadSessionRequest.internalGetFieldAccessorTable(Storage.java:5216)
> at
> com.google.protobuf.GeneratedMessageV3.getAllFieldsMutable(GeneratedMessageV3.java:135)
> at
> com.google.protobuf.GeneratedMessageV3.getAllFields(GeneratedMessageV3.java:211)
> at com.google.protobuf.TextFormat$Printer.print(TextFormat.java:324)
> at com.google.protobuf.TextFormat$Printer.access$000(TextFormat.java:307)
> at com.google.protobuf.TextFormat.print(TextFormat.java:68)
> at com.google.protobuf.TextFormat.printToString(TextFormat.java:148)
> at com.google.protobuf.AbstractMessage.toString(AbstractMessage.java:117)
> at
> org.slf4j.helpers.MessageFormatter.safeObjectAppend(MessageFormatter.java:299)
> at
> org.slf4j.helpers.MessageFormatter.deeplyAppendParameter(MessageFormatter.java:271)
> at
> org.slf4j.helpers.MessageFormatter.arrayFormat(MessageFormatter.java:233)
> at
> org.slf4j.helpers.MessageFormatter.arrayFormat(MessageFormatter.java:173)
> at org.slf4j.helpers.MessageFormatter.format(MessageFormatter.java:151)
> at org.slf4j.impl.SimpleLogger.formatAndLog(SimpleLogger.java:354)
> at org.slf4j.impl.SimpleLogger.info(SimpleLogger.java:496)
> at
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryStorageSourceBase.split(BigQueryStorageSourceBase.java:140)
> at
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryStorageTableSource.split(BigQueryStorageTableSource.java:42)
> ...
>
>
> This seems like a library dependency problem. Anyone can point me to the
> correct library dependency for using the API or know what might be wrong
> here? I am using beam 2.19 and already included
> "com.google.api.grpc:proto-google-cloud-bigquerystorage-v1beta1:0.83.0" and
> "com.google.cloud:google-cloud-bigquerystorage:0.79.0-alpha" in my gradle
> dependency.
>
> Thanks!
>
>
>
--
Regards,
Tomo