You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Luke Cwik (Jira)" <ji...@apache.org> on 2020/03/17 22:56:00 UTC
[jira] [Commented] (BEAM-9390) [PostCommit_Java_PortabilityApi]
[BigQuery related ITs] UnsupportedOperationException: BigQuery source must
be split before being read
[ https://issues.apache.org/jira/browse/BEAM-9390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17061239#comment-17061239 ]
Luke Cwik commented on BEAM-9390:
---------------------------------
The issue is that the BigQuerySource only works with initial splitting since initial splitting kicks off the import job.
This can be fixed by:
1) Restore the JavaReadViaImpulse expansion for Dataflow runner only when using beam_fn_api
2) Update the JRH to support doing the expansion for SplittableDoFns itself
3) Migrate BigQuerySource to a SplittableDoFn
4) Implement "concat" source over the AvroIO files.
All but #3 is throw away work but #3 would make the source incompatible with non-portable runners. #1 is likely the easiest to do.
> [PostCommit_Java_PortabilityApi] [BigQuery related ITs] UnsupportedOperationException: BigQuery source must be split before being read
> --------------------------------------------------------------------------------------------------------------------------------------
>
> Key: BEAM-9390
> URL: https://issues.apache.org/jira/browse/BEAM-9390
> Project: Beam
> Issue Type: Bug
> Components: test-failures
> Reporter: Yueyang Qiu
> Assignee: Luke Cwik
> Priority: Major
> Labels: currently-failing
>
> Failed tests:
> org.apache.beam.examples.cookbook.BigQueryTornadoesIT.testE2EBigQueryTornadoesWithExport
> org.apache.beam.examples.cookbook.BigQueryTornadoesIT.testE2eBigQueryTornadoesWithStorageApi
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryTimePartitioningClusteringIT.testE2EBigQueryClusteringTableFunction
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryTimePartitioningClusteringIT.testE2EBigQueryClusteringDynamicDestinations
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryTimePartitioningClusteringIT.testE2EBigQueryTimePartitioning
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryTimePartitioningClusteringIT.testE2EBigQueryClustering
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryToTableIT.testNewTypesQueryWithReshuffle
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryToTableIT.testLegacyQueryWithoutReshuffle
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryToTableIT.testNewTypesQueryWithoutReshuffle
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryToTableIT.testStandardQueryWithoutCustom
> ([https://builds.apache.org/job/beam_PostCommit_Java_PortabilityApi/4226/#showFailuresLink)]
>
> Example failures:
> java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: Error received from SDK harness for instruction -596: java.lang.UnsupportedOperationException: BigQuery source must be split before being read at org.apache.beam.sdk.io.gcp.bigquery.BigQuerySourceBase.createReader(BigQuerySourceBase.java:173) at org.apache.beam.fn.harness.BoundedSourceRunner.runReadLoop(BoundedSourceRunner.java:159) at org.apache.beam.fn.harness.BoundedSourceRunner.start(BoundedSourceRunner.java:146)
> ...
--
This message was sent by Atlassian Jira
(v8.3.4#803005)