[jira] [Commented] (BEAM-6069) Bigquery Tornadoes example fails to run when we pass a custom temp location.

["Kenneth Knowles (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BEAM-6069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17547349#comment-17547349 ] 

Kenneth Knowles commented on BEAM-6069:
---------------------------------------

This issue has been migrated to https://github.com/apache/beam/issues/19116

> Bigquery Tornadoes example fails to run when we pass a custom temp location.
> ----------------------------------------------------------------------------
>
>                 Key: BEAM-6069
>                 URL: https://issues.apache.org/jira/browse/BEAM-6069
>             Project: Beam
>          Issue Type: Bug
>          Components: examples-java, io-java-gcp
>            Reporter: Valentyn Tymofieiev
>            Priority: P3
>
> Steps to reproduce:
> {noformat}
> PROJECT=$(gcloud config get-value project)
> BUCKET=${USER}_gcs_bucket
> BQ_DATASET=${USER}_bq_dataset
> TABLE_NAME=out
> bq mk --project=$PROJECT $BQ_DATASET
> gsutil mb gs://$BUCKET
> PATH_TO_REPO_CLONE=/path/to/beam
> mvn archetype:generate -DarchetypeGroupId=org.apache.beam -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples  -DarchetypeVersion=2.8.0  -DgroupId=org.example  -DartifactId=word-count-beam  -Dversion="0.1" -Dpackage=org.apache.beam.examples -DinteractiveMode=false
> cd word-count-beam/
> mkdir src/main/java/org/apache/beam/examples/cookbook
> cp $PATH_TO_REPO_CLONE/examples/java/src/main/java/org/apache/beam/examples/cookbook//BigQueryTornadoes.java ./src/main/java/org/apache/beam/examples/cookbook
> mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.cookbook.BigQueryTornadoes -Dexec.args="--runner=DataflowRunner --project=$PROJECT --input=clouddataflow-readonly:samples.weather_stations --gcpTempLocation=gs://$BUCKET/tmp --output=$BQ_DATASET.$TABLE_NAME " -Pdataflow-runner
> {noformat}
> This fails with:
> {noformat}
> java.lang.IllegalArgumentException: BigQueryIO.Read needs a GCS temp location to store temp files.
> at com.google.common.base.Preconditions.checkArgument(Preconditions.java:122)
> at org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO$TypedRead.validate(BigQueryIO.java:662)
> at org.apache.beam.sdk.Pipeline$ValidateVisitor.enterCompositeTransform(Pipeline.java:641)
> at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:645)
> at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:649)
> at org.apache.beam.sdk.runners.TransformHierarchy$Node.access$600(TransformHierarchy.java:311)
> at org.apache.beam.sdk.runners.TransformHierarchy.visit(TransformHierarchy.java:245)
> at org.apache.beam.sdk.Pipeline.traverseTopologically(Pipeline.java:458)
> at org.apache.beam.sdk.Pipeline.validate(Pipeline.java:577)
> at org.apache.beam.sdk.Pipeline.run(Pipeline.java:312)
> at org.apache.beam.sdk.Pipeline.run(Pipeline.java:299)
> at org.apache.beam.examples.cookbook.BigQueryTornadoes.runBigQueryTornadoes(BigQueryTornadoes.java:166)
> at org.apache.beam.examples.cookbook.BigQueryTornadoes.main(BigQueryTornadoes.java:172)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:282)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> Ironically, the example works if we remove --gcpTempLocation. From logs, we can see that in that case we use a bucket that looks like: gs://dataflow-staging-us-central1-927334603519.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)