You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Filip Popić <fi...@gmail.com> on 2021/05/17 17:29:12 UTC

Beam 2.29.0 throwsing warning/error when reading by using BigQuery Storage Read API

Hi,

I am trying to read data from BigQuery table using BigQuery Storage/Direct
<https://cloud.google.com/bigquery/docs/reference/storage> Read API using
Beam 2.29.0, and providing simple negated boolean row restriction (`NOT
field_x`) but getting warning+errors:

```
*~*~*~ Channel ManagedChannelImpl{logId=1, target=
bigquerystorage.googleapis.com:443} was not shutdown properly!!! ~*~*~*
Make sure to call shutdown()/shutdownNow() and wait until
awaitTermination() returns true.
```

```
Error
2021-05-12 13:25:49.319 CESTError message from worker:
java.lang.NullPointerException
org.apache.beam.sdk.io.gcp.bigquery.BigQueryStorageSourceBase.split(BigQueryStorageSourceBase.java:105)
org.apache.beam.sdk.io.gcp.bigquery.BigQueryStorageTableSource.split(BigQueryStorageTableSource.java:40)
org.apache.beam.runners.dataflow.worker.WorkerCustomSources.splitAndValidate(WorkerCustomSources.java:294)
org.apache.beam.runners.dataflow.worker.WorkerCustomSources.performSplitTyped(WorkerCustomSources.java:216)
org.apache.beam.runners.dataflow.worker.WorkerCustomSources.performSplitWithApiLimit(WorkerCustomSources.java:200)
org.apache.beam.runners.dataflow.worker.WorkerCustomSources.performSplit(WorkerCustomSources.java:179)
org.apache.beam.runners.dataflow.worker.WorkerCustomSourceOperationExecutor.execute(WorkerCustomSourceOperationExecutor.java:82)
org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.executeWork(BatchDataflowWorker.java:420)
org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.doWork(BatchDataflowWorker.java:389)
org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.getAndPerformWork(BatchDataflowWorker.java:314)
org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.doWork(DataflowBatchWorkerHarness.java:140)
org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:120)
org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:107)
java.util.concurrent.FutureTask.run(FutureTask.java:266)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
```

I think it might be related to
https://github.com/apache/beam/pull/13896/files but not sure how to fix or
make a workaround?

Re: Re: Beam 2.29.0 throwsing warning/error when reading by using BigQuery Storage Read API

Posted by Filip Popi�� <fi...@gmail.com>.
Here it is https://issues.apache.org/jira/browse/BEAM-12359

On 2021/05/18 16:20:10 Chamikara Jayalath wrote:
> Filip, Could you create a Jira with steps to reproduce ?
> 
> +Kenneth Jung <km...@google.com>
> 
> Thanks,
> Cham
> 
> On Mon, May 17, 2021 at 10:29 AM Filip Popić <fi...@gmail.com> wrote:
> 
> > Hi,
> >
> > I am trying to read data from BigQuery table using BigQuery Storage/Direct
> > <https://cloud.google.com/bigquery/docs/reference/storage> Read API using
> > Beam 2.29.0, and providing simple negated boolean row restriction (`NOT
> > field_x`) but getting warning+errors:
> >
> > ```
> > *~*~*~ Channel ManagedChannelImpl{logId=1, target=
> > bigquerystorage.googleapis.com:443} was not shutdown properly!!! ~*~*~*
> > Make sure to call shutdown()/shutdownNow() and wait until
> > awaitTermination() returns true.
> > ```
> >
> > ```
> > Error
> > 2021-05-12 13:25:49.319 CESTError message from worker:
> > java.lang.NullPointerException
> > org.apache.beam.sdk.io.gcp.bigquery.BigQueryStorageSourceBase.split(BigQueryStorageSourceBase.java:105)
> > org.apache.beam.sdk.io.gcp.bigquery.BigQueryStorageTableSource.split(BigQueryStorageTableSource.java:40)
> > org.apache.beam.runners.dataflow.worker.WorkerCustomSources.splitAndValidate(WorkerCustomSources.java:294)
> > org.apache.beam.runners.dataflow.worker.WorkerCustomSources.performSplitTyped(WorkerCustomSources.java:216)
> > org.apache.beam.runners.dataflow.worker.WorkerCustomSources.performSplitWithApiLimit(WorkerCustomSources.java:200)
> > org.apache.beam.runners.dataflow.worker.WorkerCustomSources.performSplit(WorkerCustomSources.java:179)
> > org.apache.beam.runners.dataflow.worker.WorkerCustomSourceOperationExecutor.execute(WorkerCustomSourceOperationExecutor.java:82)
> > org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.executeWork(BatchDataflowWorker.java:420)
> > org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.doWork(BatchDataflowWorker.java:389)
> > org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.getAndPerformWork(BatchDataflowWorker.java:314)
> > org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.doWork(DataflowBatchWorkerHarness.java:140)
> > org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:120)
> > org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:107)
> > java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > java.lang.Thread.run(Thread.java:748)
> > ```
> >
> > I think it might be related to
> > https://github.com/apache/beam/pull/13896/files but not sure how to fix
> > or make a workaround?
> >
>

Re: Beam 2.29.0 throwsing warning/error when reading by using BigQuery Storage Read API

Posted by Chamikara Jayalath <ch...@google.com>.
Filip, Could you create a Jira with steps to reproduce ?

+Kenneth Jung <km...@google.com>

Thanks,
Cham

On Mon, May 17, 2021 at 10:29 AM Filip Popić <fi...@gmail.com> wrote:

> Hi,
>
> I am trying to read data from BigQuery table using BigQuery Storage/Direct
> <https://cloud.google.com/bigquery/docs/reference/storage> Read API using
> Beam 2.29.0, and providing simple negated boolean row restriction (`NOT
> field_x`) but getting warning+errors:
>
> ```
> *~*~*~ Channel ManagedChannelImpl{logId=1, target=
> bigquerystorage.googleapis.com:443} was not shutdown properly!!! ~*~*~*
> Make sure to call shutdown()/shutdownNow() and wait until
> awaitTermination() returns true.
> ```
>
> ```
> Error
> 2021-05-12 13:25:49.319 CESTError message from worker:
> java.lang.NullPointerException
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryStorageSourceBase.split(BigQueryStorageSourceBase.java:105)
> org.apache.beam.sdk.io.gcp.bigquery.BigQueryStorageTableSource.split(BigQueryStorageTableSource.java:40)
> org.apache.beam.runners.dataflow.worker.WorkerCustomSources.splitAndValidate(WorkerCustomSources.java:294)
> org.apache.beam.runners.dataflow.worker.WorkerCustomSources.performSplitTyped(WorkerCustomSources.java:216)
> org.apache.beam.runners.dataflow.worker.WorkerCustomSources.performSplitWithApiLimit(WorkerCustomSources.java:200)
> org.apache.beam.runners.dataflow.worker.WorkerCustomSources.performSplit(WorkerCustomSources.java:179)
> org.apache.beam.runners.dataflow.worker.WorkerCustomSourceOperationExecutor.execute(WorkerCustomSourceOperationExecutor.java:82)
> org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.executeWork(BatchDataflowWorker.java:420)
> org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.doWork(BatchDataflowWorker.java:389)
> org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.getAndPerformWork(BatchDataflowWorker.java:314)
> org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.doWork(DataflowBatchWorkerHarness.java:140)
> org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:120)
> org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:107)
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> java.lang.Thread.run(Thread.java:748)
> ```
>
> I think it might be related to
> https://github.com/apache/beam/pull/13896/files but not sure how to fix
> or make a workaround?
>