Resolving Go SDK build/test failures when using gradle

[Robert Burke <ro...@frantil.com>]

If you haven't run into :beam-sdks-go:buildLinuxAmd64 or similar failing
with "undefined: passert.Sum" recently, stop reading now.

The root cause is that the gogradle plugin doesn't clean up the vendor
directories <https://golang.org/cmd/go/#hdr-Vendor_Directories> that it
sets up, in combination with trying to use gradle with Go, leads to gradle
vendoring the beam package in the sdks/go/test, sdks/go/examples,
sdks/go/container directories.
This vendoring is persistent on the local client, and isn't cleaned up by
the clean task.

The immediate fix is to navigate to each of the sdks/go, and
sdks/go/{test|examples|container} directories and delete the vendor and
.gogradle directories.

eg Run the following from your beam git root, if your'e using a *Unix or
similar:

rm -rf sdks/go/{vendor,.gogradle}
sdks/go/{test,examples,container}/{vendor,.gogradle}

Then try running your gradle command again.

Go gradle will create the same vendored directories, but will at least have
a more up to date version of the Go SDK.

The short term fix would be to fix the recursive clean tasks that affect
the go builds to also remove the vendor directories, if not targetting the
vendored copy of beam specifically.

This situation is awful and while the long term fix would probably be to
use Go Modules <https://github.com/golang/go/wiki/Modules> directly (see
BEAM-5379 <https://issues.apache.org/jira/browse/BEAM-5379>), and replace
gogradle plugin with targeted bash scripts, allowing the go tool to manage
dependencies and build artifacts directly.

If you're familiar with gradle and could provide guidance on customizing a
task, so it's properly invoked on the general clean task, it would be much
appreciated. BEAM-5465 <https://issues.apache.org/jira/browse/BEAM-5465> has
been filed to track that work.

Thank you for your patience,
Robert B

Re: Resolving Go SDK build/test failures when using gradle

[Lukasz Cwik <lc...@google.com>]

Are we setting all the intra-project dependencies[1] within each of our Go
based build.gradle files?

I ask because typically I would suspect that the build system would attempt
to get the dependency from its dependency management section if it wasn't
declared within.

1:
https://github.com/apache/beam/blob/a5bc2cbf07eb46d0af208190a2d828b96421fdab/sdks/go/test/build.gradle#L33

On Fri, Sep 21, 2018 at 4:24 PM Robert Burke <ro...@frantil.com> wrote:

> If you haven't run into :beam-sdks-go:buildLinuxAmd64 or similar failing
> with "undefined: passert.Sum" recently, stop reading now.
>
> The root cause is that the gogradle plugin doesn't clean up the vendor
> directories <https://golang.org/cmd/go/#hdr-Vendor_Directories> that it
> sets up, in combination with trying to use gradle with Go, leads to gradle
> vendoring the beam package in the sdks/go/test, sdks/go/examples,
> sdks/go/container directories.
> This vendoring is persistent on the local client, and isn't cleaned up by
> the clean task.
>
> The immediate fix is to navigate to each of the sdks/go, and
> sdks/go/{test|examples|container} directories and delete the vendor and
> .gogradle directories.
>
> eg Run the following from your beam git root, if your'e using a *Unix or
> similar:
>
> rm -rf sdks/go/{vendor,.gogradle}
> sdks/go/{test,examples,container}/{vendor,.gogradle}
>
> Then try running your gradle command again.
>
> Go gradle will create the same vendored directories, but will at least
> have a more up to date version of the Go SDK.
>
> The short term fix would be to fix the recursive clean tasks that affect
> the go builds to also remove the vendor directories, if not targetting the
> vendored copy of beam specifically.
>
> This situation is awful and while the long term fix would probably be to
> use Go Modules <https://github.com/golang/go/wiki/Modules> directly (see
> BEAM-5379 <https://issues.apache.org/jira/browse/BEAM-5379>), and replace
> gogradle plugin with targeted bash scripts, allowing the go tool to manage
> dependencies and build artifacts directly.
>
> If you're familiar with gradle and could provide guidance on customizing a
> task, so it's properly invoked on the general clean task, it would be much
> appreciated. BEAM-5465 <https://issues.apache.org/jira/browse/BEAM-5465> has
> been filed to track that work.
>
> Thank you for your patience,
> Robert B
>