You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Jacob Marble <jm...@kochava.com> on 2018/01/31 00:55:54 UTC
Re: Dependencies and Datastore
Josh, what did you do to work around this?
This suddenly crept up on a production pipeline yesterday, without anything
changing on our side (we do rebuild at every run).
Jacob
On Fri, Dec 8, 2017 at 6:46 PM, Chamikara Jayalath <ch...@google.com>
wrote:
> Created https://issues.apache.org/jira/browse/BEAM-3321 to update
> the gax-grpc dependency of Beam.
>
> - Cham
>
> On Friday, December 8, 2017 at 6:12:20 PM UTC-8, Harsh Vardhan wrote:
>>
>> +chamikara@
>>
>>
>> On Friday, December 8, 2017 at 1:31:00 AM UTC-8, Joshua Fox wrote:
>>>
>>> I use Cloud Datastore API to check for Kinds in the Datastore, then use
>>> Dataflow -- now upgrading to Beam -- to copy one Datastore to another..
>>>
>>> After adding beam-sdks-java-io-google-cloud-platform to my pom, I start
>>> getting this when initializing the Cloud Datastore API
>>>
>>> Exception in thread "main" java.lang.NoClassDefFoundError:
>>> com/google/api/gax/retrying/ResultRetryAlgorithm
>>> at com.google.cloud.datastore.DatastoreOptions$DefaultDatastore
>>> Factory.create(DatastoreOptions.java:51)
>>> at com.google.cloud.datastore.DatastoreOptions$DefaultDatastore
>>> Factory.create(DatastoreOptions.java:45)
>>> at com.google.cloud.ServiceOptions.getService(ServiceOptions.java:426)
>>>
>>> It is caused by gax dependencies.
>>>
>>> Specifically, before I add beam-sdks-java-io-google-cloud-platform there
>>> is this gax dependency
>>>
>>> +- com.google.cloud:google-cloud-datastore:jar:1.12.0:compile
>>> | +- com.google.cloud:google-cloud-core:jar:1.12.0:compile
>>> | | +- com.google.api:gax:jar:1.15.0:compile
>>>
>>>
>>>
>>> and after I add it there is this
>>>
>>> +- org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:
>>> 2.2.0:compile
>>> | +- com.google.api:gax-grpc:jar:0.20.0:compile
>>> | | +- com.google.api:gax:jar:1.3.1:compile
>>>
>>>
>>> If I add gax 1.15.0 to my pom explicitly, I get
>>>
>>> Exception in thread "main" java.lang.NoClassDefFoundError:
>>> com/google/api/gax/retrying/ExceptionRetryAlgorithm
>>> at java.lang.ClassLoader.defineClass1(Native Method)
>>> ..
>>> at com.google.cloud.BaseService.<clinit>(BaseService.java:48)
>>> at com.google.cloud.datastore.DatastoreOptions$DefaultDatastore
>>> Factory.create(DatastoreOptions.java:51)
>>> at com.google.cloud.datastore.DatastoreOptions$DefaultDatastore
>>> Factory.create(DatastoreOptions.java:45)
>>> at com.google.cloud.ServiceOptions.getService(ServiceOptions.java:426)
>>>
>>> Clearly Datastore and Beam should work together. Yet there have been
>>> dependency problems between the two for a while. See this discussion
>>> <https://stackoverflow.com/questions/40830727/> from 1 year ago.
>>>
>>> How can I resolve this?
>>>
>>
Re: Dependencies and Datastore
Posted by Jacob Marble <jm...@kochava.com>.
Thanks for that, Joshua.
Late last night I moved my BigTable-related code (depending on
beam:beam-sdks-java-io-google-cloud-platform) into a separate pom/project
from GCS-related code (depending on google-cloud-storage).
pom A:
+- org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.2.0:compile
| +- com.google.api:gax-grpc:jar:0.20.0:compile
| | +- io.grpc:grpc-protobuf:jar:1.2.0:compile
| | +- com.google.api:api-common:jar:1.1.0:compile
| | +- com.google.api:gax:jar:1.3.1:compile
| | \- org.threeten:threetenbp:jar:1.3.3:compile
pom B:
| +- org.apache.beam:beam-sdks-java-core:jar:2.2.0:compile
| \-
org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.2.0:compile
| +- com.google.api:gax-grpc:jar:0.20.0:compile
| | +- io.grpc:grpc-protobuf:jar:1.2.0:compile
| | \- org.threeten:threetenbp:jar:1.3.3:compile
...
+- com.google.cloud:google-cloud-storage:jar:1.4.0:compile
| +- com.google.cloud:google-cloud-core:jar:1.4.0:compile
| | +- org.json:json:jar:20160810:compile
| | +- com.google.api:api-common:jar:1.1.0:compile
| | +- com.google.api:gax:jar:1.5.0:compile
Jacob
On Wed, Jan 31, 2018 at 3:24 AM, Joshua Fox <jo...@freightos.com> wrote:
> The problem does not occur for me now following a dependency update.
>
> Here are the parts of mvn dependency:tree that involve gax
>
> | +- org.apache.beam:beam-sdks-java-io-google-cloud-platform:
> jar:2.2.0:compile
> | | +- com.google.api:gax-grpc:jar:0.20.0:compile
>
> ...
>
> +- com.google.code.gson:gson:jar:2.8.2:compile
> \- com.google.api:gax:jar:1.15.0:compile
>
> Those that involve Beam
>
> +- org.apache.beam:beam-runners-google-cloud-dataflow-java:
> jar:2.2.0:compile
> | +- org.apache.beam:beam-sdks-java-core:jar:2.2.0:compile
> | +- org.apache.beam:beam-sdks-java-extensions-google-cloud-
> platform-core:jar:2.2.0:compile
> | +- org.apache.beam:beam-sdks-common-runner-api:jar:2.2.0:compile
> ...
>
> | +- org.apache.beam:beam-sdks-java-io-google-cloud-platform:
> jar:2.2.0:compile
> | | +- com.google.api:gax-grpc:jar:0.20.0:compile
>
>
> Here are the parts of my pom that involve gax and Beam
>
>
> <dependency>
> <groupId>org.apache.beam</groupId>
> <artifactId>beam-runners-google-cloud-dataflow-java</artifactId>
> <version>2.2.0</version>
> </dependency>
>
>
> <dependency>
> <groupId>com.google.api</groupId>
> <artifactId>gax</artifactId>
> <version>1.15.0</version>
> </dependency>
>
>
>
>
> On Wed, Jan 31, 2018 at 2:55 AM, Jacob Marble <jm...@kochava.com> wrote:
>
>> Josh, what did you do to work around this?
>>
>> This suddenly crept up on a production pipeline yesterday, without
>> anything changing on our side (we do rebuild at every run).
>>
>> Jacob
>>
>> On Fri, Dec 8, 2017 at 6:46 PM, Chamikara Jayalath <ch...@google.com>
>> wrote:
>>
>>> Created https://issues.apache.org/jira/browse/BEAM-3321 to update
>>> the gax-grpc dependency of Beam.
>>>
>>> - Cham
>>>
>>> On Friday, December 8, 2017 at 6:12:20 PM UTC-8, Harsh Vardhan wrote:
>>>>
>>>> +chamikara@
>>>>
>>>>
>>>> On Friday, December 8, 2017 at 1:31:00 AM UTC-8, Joshua Fox wrote:
>>>>>
>>>>> I use Cloud Datastore API to check for Kinds in the Datastore, then
>>>>> use Dataflow -- now upgrading to Beam -- to copy one Datastore to another..
>>>>>
>>>>> After adding beam-sdks-java-io-google-cloud-platform to my pom, I
>>>>> start getting this when initializing the Cloud Datastore API
>>>>>
>>>>> Exception in thread "main" java.lang.NoClassDefFoundError:
>>>>> com/google/api/gax/retrying/ResultRetryAlgorithm
>>>>> at com.google.cloud.datastore.DatastoreOptions$DefaultDatastore
>>>>> Factory.create(DatastoreOptions.java:51)
>>>>> at com.google.cloud.datastore.DatastoreOptions$DefaultDatastore
>>>>> Factory.create(DatastoreOptions.java:45)
>>>>> at com.google.cloud.ServiceOptions.getService(ServiceOptions.java:426)
>>>>>
>>>>> It is caused by gax dependencies.
>>>>>
>>>>> Specifically, before I add beam-sdks-java-io-google-cloud-platform there
>>>>> is this gax dependency
>>>>>
>>>>> +- com.google.cloud:google-cloud-datastore:jar:1.12.0:compile
>>>>> | +- com.google.cloud:google-cloud-core:jar:1.12.0:compile
>>>>> | | +- com.google.api:gax:jar:1.15.0:compile
>>>>>
>>>>>
>>>>>
>>>>> and after I add it there is this
>>>>>
>>>>> +- org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:
>>>>> 2.2.0:compile
>>>>> | +- com.google.api:gax-grpc:jar:0.20.0:compile
>>>>> | | +- com.google.api:gax:jar:1.3.1:compile
>>>>>
>>>>>
>>>>> If I add gax 1.15.0 to my pom explicitly, I get
>>>>>
>>>>> Exception in thread "main" java.lang.NoClassDefFoundError:
>>>>> com/google/api/gax/retrying/ExceptionRetryAlgorithm
>>>>> at java.lang.ClassLoader.defineClass1(Native Method)
>>>>> ..
>>>>> at com.google.cloud.BaseService.<clinit>(BaseService.java:48)
>>>>> at com.google.cloud.datastore.DatastoreOptions$DefaultDatastore
>>>>> Factory.create(DatastoreOptions.java:51)
>>>>> at com.google.cloud.datastore.DatastoreOptions$DefaultDatastore
>>>>> Factory.create(DatastoreOptions.java:45)
>>>>> at com.google.cloud.ServiceOptions.getService(ServiceOptions.java:426)
>>>>>
>>>>> Clearly Datastore and Beam should work together. Yet there have been
>>>>> dependency problems between the two for a while. See this discussion
>>>>> <https://stackoverflow.com/questions/40830727/> from 1 year ago.
>>>>>
>>>>> How can I resolve this?
>>>>>
>>>>
>>
>
>
> --
>
>
> *JOSHUA FOX*
> Principal Software Architect | Freightos
>
>
>
> *T (Israel): *+972-545691165 <+972%2054-569-1165> | *T (US)*: +
> 1-3123400953 <(312)%20340-0953>
> Smooth shipping.
>
>
>
>
Re: Dependencies and Datastore
Posted by Joshua Fox <jo...@freightos.com>.
The problem does not occur for me now following a dependency update.
Here are the parts of mvn dependency:tree that involve gax
| +-
org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.2.0:compile
| | +- com.google.api:gax-grpc:jar:0.20.0:compile
...
+- com.google.code.gson:gson:jar:2.8.2:compile
\- com.google.api:gax:jar:1.15.0:compile
Those that involve Beam
+-
org.apache.beam:beam-runners-google-cloud-dataflow-java:jar:2.2.0:compile
| +- org.apache.beam:beam-sdks-java-core:jar:2.2.0:compile
| +-
org.apache.beam:beam-sdks-java-extensions-google-cloud-platform-core:jar:2.2.0:compile
| +- org.apache.beam:beam-sdks-common-runner-api:jar:2.2.0:compile
...
| +-
org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.2.0:compile
| | +- com.google.api:gax-grpc:jar:0.20.0:compile
Here are the parts of my pom that involve gax and Beam
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-runners-google-cloud-dataflow-java</artifactId>
<version>2.2.0</version>
</dependency>
<dependency>
<groupId>com.google.api</groupId>
<artifactId>gax</artifactId>
<version>1.15.0</version>
</dependency>
On Wed, Jan 31, 2018 at 2:55 AM, Jacob Marble <jm...@kochava.com> wrote:
> Josh, what did you do to work around this?
>
> This suddenly crept up on a production pipeline yesterday, without
> anything changing on our side (we do rebuild at every run).
>
> Jacob
>
> On Fri, Dec 8, 2017 at 6:46 PM, Chamikara Jayalath <ch...@google.com>
> wrote:
>
>> Created https://issues.apache.org/jira/browse/BEAM-3321 to update
>> the gax-grpc dependency of Beam.
>>
>> - Cham
>>
>> On Friday, December 8, 2017 at 6:12:20 PM UTC-8, Harsh Vardhan wrote:
>>>
>>> +chamikara@
>>>
>>>
>>> On Friday, December 8, 2017 at 1:31:00 AM UTC-8, Joshua Fox wrote:
>>>>
>>>> I use Cloud Datastore API to check for Kinds in the Datastore, then
>>>> use Dataflow -- now upgrading to Beam -- to copy one Datastore to another..
>>>>
>>>> After adding beam-sdks-java-io-google-cloud-platform to my pom, I
>>>> start getting this when initializing the Cloud Datastore API
>>>>
>>>> Exception in thread "main" java.lang.NoClassDefFoundError:
>>>> com/google/api/gax/retrying/ResultRetryAlgorithm
>>>> at com.google.cloud.datastore.DatastoreOptions$DefaultDatastore
>>>> Factory.create(DatastoreOptions.java:51)
>>>> at com.google.cloud.datastore.DatastoreOptions$DefaultDatastore
>>>> Factory.create(DatastoreOptions.java:45)
>>>> at com.google.cloud.ServiceOptions.getService(ServiceOptions.java:426)
>>>>
>>>> It is caused by gax dependencies.
>>>>
>>>> Specifically, before I add beam-sdks-java-io-google-cloud-platform there
>>>> is this gax dependency
>>>>
>>>> +- com.google.cloud:google-cloud-datastore:jar:1.12.0:compile
>>>> | +- com.google.cloud:google-cloud-core:jar:1.12.0:compile
>>>> | | +- com.google.api:gax:jar:1.15.0:compile
>>>>
>>>>
>>>>
>>>> and after I add it there is this
>>>>
>>>> +- org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:
>>>> 2.2.0:compile
>>>> | +- com.google.api:gax-grpc:jar:0.20.0:compile
>>>> | | +- com.google.api:gax:jar:1.3.1:compile
>>>>
>>>>
>>>> If I add gax 1.15.0 to my pom explicitly, I get
>>>>
>>>> Exception in thread "main" java.lang.NoClassDefFoundError:
>>>> com/google/api/gax/retrying/ExceptionRetryAlgorithm
>>>> at java.lang.ClassLoader.defineClass1(Native Method)
>>>> ..
>>>> at com.google.cloud.BaseService.<clinit>(BaseService.java:48)
>>>> at com.google.cloud.datastore.DatastoreOptions$DefaultDatastore
>>>> Factory.create(DatastoreOptions.java:51)
>>>> at com.google.cloud.datastore.DatastoreOptions$DefaultDatastore
>>>> Factory.create(DatastoreOptions.java:45)
>>>> at com.google.cloud.ServiceOptions.getService(ServiceOptions.java:426)
>>>>
>>>> Clearly Datastore and Beam should work together. Yet there have been
>>>> dependency problems between the two for a while. See this discussion
>>>> <https://stackoverflow.com/questions/40830727/> from 1 year ago.
>>>>
>>>> How can I resolve this?
>>>>
>>>
>
--
*JOSHUA FOX*
Principal Software Architect | Freightos
*T (Israel): *+972-545691165 | *T (US)*: +1-3123400953
Smooth shipping.
Re: Dependencies and Datastore
Posted by Jacob Marble <jm...@kochava.com>.
Also weird, things work fine (no need to force gax to any version other
than 1.3.1) when I build and run from my workstation, but fail on our CI/CD
worker.
Jacob
On Tue, Jan 30, 2018 at 4:55 PM, Jacob Marble <jm...@kochava.com> wrote:
> Josh, what did you do to work around this?
>
> This suddenly crept up on a production pipeline yesterday, without
> anything changing on our side (we do rebuild at every run).
>
> Jacob
>
> On Fri, Dec 8, 2017 at 6:46 PM, Chamikara Jayalath <ch...@google.com>
> wrote:
>
>> Created https://issues.apache.org/jira/browse/BEAM-3321 to update
>> the gax-grpc dependency of Beam.
>>
>> - Cham
>>
>> On Friday, December 8, 2017 at 6:12:20 PM UTC-8, Harsh Vardhan wrote:
>>>
>>> +chamikara@
>>>
>>>
>>> On Friday, December 8, 2017 at 1:31:00 AM UTC-8, Joshua Fox wrote:
>>>>
>>>> I use Cloud Datastore API to check for Kinds in the Datastore, then
>>>> use Dataflow -- now upgrading to Beam -- to copy one Datastore to another..
>>>>
>>>> After adding beam-sdks-java-io-google-cloud-platform to my pom, I
>>>> start getting this when initializing the Cloud Datastore API
>>>>
>>>> Exception in thread "main" java.lang.NoClassDefFoundError:
>>>> com/google/api/gax/retrying/ResultRetryAlgorithm
>>>> at com.google.cloud.datastore.DatastoreOptions$DefaultDatastore
>>>> Factory.create(DatastoreOptions.java:51)
>>>> at com.google.cloud.datastore.DatastoreOptions$DefaultDatastore
>>>> Factory.create(DatastoreOptions.java:45)
>>>> at com.google.cloud.ServiceOptions.getService(ServiceOptions.java:426)
>>>>
>>>> It is caused by gax dependencies.
>>>>
>>>> Specifically, before I add beam-sdks-java-io-google-cloud-platform there
>>>> is this gax dependency
>>>>
>>>> +- com.google.cloud:google-cloud-datastore:jar:1.12.0:compile
>>>> | +- com.google.cloud:google-cloud-core:jar:1.12.0:compile
>>>> | | +- com.google.api:gax:jar:1.15.0:compile
>>>>
>>>>
>>>>
>>>> and after I add it there is this
>>>>
>>>> +- org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:
>>>> 2.2.0:compile
>>>> | +- com.google.api:gax-grpc:jar:0.20.0:compile
>>>> | | +- com.google.api:gax:jar:1.3.1:compile
>>>>
>>>>
>>>> If I add gax 1.15.0 to my pom explicitly, I get
>>>>
>>>> Exception in thread "main" java.lang.NoClassDefFoundError:
>>>> com/google/api/gax/retrying/ExceptionRetryAlgorithm
>>>> at java.lang.ClassLoader.defineClass1(Native Method)
>>>> ..
>>>> at com.google.cloud.BaseService.<clinit>(BaseService.java:48)
>>>> at com.google.cloud.datastore.DatastoreOptions$DefaultDatastore
>>>> Factory.create(DatastoreOptions.java:51)
>>>> at com.google.cloud.datastore.DatastoreOptions$DefaultDatastore
>>>> Factory.create(DatastoreOptions.java:45)
>>>> at com.google.cloud.ServiceOptions.getService(ServiceOptions.java:426)
>>>>
>>>> Clearly Datastore and Beam should work together. Yet there have been
>>>> dependency problems between the two for a while. See this discussion
>>>> <https://stackoverflow.com/questions/40830727/> from 1 year ago.
>>>>
>>>> How can I resolve this?
>>>>
>>>
>