Running SpannerWriteIT on dataflow

[Wout Scheepers <Wo...@vente-exclusive.com>]

Hey all,

I’m still running into a bug when streaming into spanner, which I describe in the comments of https://issues.apache.org/jira/browse/BEAM-4796.
I think the cause is a missing equals method on SpannerSchema, for which I get a warning in the worker logs when running on Dataflow.

To reproduce this, I would like to run the SpannerWriteIT integration test on dataflow. Could anyone point me into the right direction on how to do this?

Thanks in advance
- Wout

Re: Running SpannerWriteIT on dataflow

[Lukasz Cwik <lc...@google.com>]

You want to run this task[1] (either on your machine or by opening a Github
PR and using a trigger phrase). Tracing back from that task, you'll find
that the root ":javaPostCommit"[2]  task is responsible for running that
task and a bunch of others and that the Java SDK Post Commit Tests[3]
triggered via the phrase "Run Java PostCommit" is responsible for launching
the ":javaPostCommit" task.

You may want to update the include/exclude filter defined in the task to
reduce the number of ITs running and or modify the root build.gradle file
to only list the single task underneath the javaPostCommit. Note that
running the ITs via the trigger phrase without these modifications will run
a bunch of tests.

If executing on your machine, you'll want override the values set for the
GCP project and GCS location used during the tests by specifying the
command line flags -PdataflowProject=<myGCPproject> and
-PdataflowTempRoot=<gs://my/bucket>

1:
https://github.com/apache/beam/blob/71c6de321fb8648146f7bd877f074a0a7617c6e0/runners/google-cloud-dataflow-java/build.gradle#L235
2:
https://github.com/apache/beam/blob/3d8688bc540e6556e76232b2c7976916a432dbc9/build.gradle#L203
3:
https://github.com/apache/beam/blob/3d8688bc540e6556e76232b2c7976916a432dbc9/.test-infra/jenkins/job_PostCommit_Java_GradleBuild.groovy

On Wed, Nov 7, 2018 at 7:29 AM Wout Scheepers <
Wout.Scheepers@vente-exclusive.com> wrote:

> Hey all,
>
>
>
> I’m still running into a bug when streaming into spanner, which I describe
> in the comments of https://issues.apache.org/jira/browse/BEAM-4796.
>
> I think the cause is a missing equals method on SpannerSchema, for which I
> get a warning in the worker logs when running on Dataflow.
>
>
>
> To reproduce this, I would like to run the SpannerWriteIT integration test
> on dataflow. Could anyone point me into the right direction on how to do
> this?
>
>
>
> Thanks in advance
>
> - Wout
>
>
>
>
>