You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Luke Cwik (Jira)" <ji...@apache.org> on 2020/03/17 22:56:00 UTC

[jira] [Commented] (BEAM-9390) [PostCommit_Java_PortabilityApi] [BigQuery related ITs] UnsupportedOperationException: BigQuery source must be split before being read

    [ https://issues.apache.org/jira/browse/BEAM-9390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17061239#comment-17061239 ] 

Luke Cwik commented on BEAM-9390:
---------------------------------

The issue is that the BigQuerySource only works with initial splitting since initial splitting kicks off the import job.

 

This can be fixed by:

1) Restore the JavaReadViaImpulse expansion for Dataflow runner only when using beam_fn_api

2) Update the JRH to support doing the expansion for SplittableDoFns itself

3) Migrate BigQuerySource to a SplittableDoFn

4) Implement "concat" source over the AvroIO files.

 

All but #3 is throw away work but #3 would make the source incompatible with non-portable runners. #1 is likely the easiest to do.

> [PostCommit_Java_PortabilityApi] [BigQuery related ITs] UnsupportedOperationException: BigQuery source must be split before being read
> --------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: BEAM-9390
>                 URL: https://issues.apache.org/jira/browse/BEAM-9390
>             Project: Beam
>          Issue Type: Bug
>          Components: test-failures
>            Reporter: Yueyang Qiu
>            Assignee: Luke Cwik
>            Priority: Major
>              Labels: currently-failing
>
> Failed tests:
> org.apache.beam.examples.cookbook.BigQueryTornadoesIT.testE2EBigQueryTornadoesWithExport
>  org.apache.beam.examples.cookbook.BigQueryTornadoesIT.testE2eBigQueryTornadoesWithStorageApi
>  org.apache.beam.sdk.io.gcp.bigquery.BigQueryTimePartitioningClusteringIT.testE2EBigQueryClusteringTableFunction
>  org.apache.beam.sdk.io.gcp.bigquery.BigQueryTimePartitioningClusteringIT.testE2EBigQueryClusteringDynamicDestinations
>  org.apache.beam.sdk.io.gcp.bigquery.BigQueryTimePartitioningClusteringIT.testE2EBigQueryTimePartitioning
>  org.apache.beam.sdk.io.gcp.bigquery.BigQueryTimePartitioningClusteringIT.testE2EBigQueryClustering
>  org.apache.beam.sdk.io.gcp.bigquery.BigQueryToTableIT.testNewTypesQueryWithReshuffle
>  org.apache.beam.sdk.io.gcp.bigquery.BigQueryToTableIT.testLegacyQueryWithoutReshuffle
>  org.apache.beam.sdk.io.gcp.bigquery.BigQueryToTableIT.testNewTypesQueryWithoutReshuffle
>  org.apache.beam.sdk.io.gcp.bigquery.BigQueryToTableIT.testStandardQueryWithoutCustom
> ([https://builds.apache.org/job/beam_PostCommit_Java_PortabilityApi/4226/#showFailuresLink)]
>  
> Example failures:
> java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: Error received from SDK harness for instruction -596: java.lang.UnsupportedOperationException: BigQuery source must be split before being read at org.apache.beam.sdk.io.gcp.bigquery.BigQuerySourceBase.createReader(BigQuerySourceBase.java:173) at org.apache.beam.fn.harness.BoundedSourceRunner.runReadLoop(BoundedSourceRunner.java:159) at org.apache.beam.fn.harness.BoundedSourceRunner.start(BoundedSourceRunner.java:146)
> ...



--
This message was sent by Atlassian Jira
(v8.3.4#803005)