apache beam bigquery IO connector support for bigquery external tables

[Nirav Patel <ni...@gmail.com>]

Beam has bigquery IO support to be able to read and write to bq tables. I
am assuming it only supports bigquery internal tables however if anyone
knows if it supports writing to and reading from bq external tables
(parquet or Iceberg) ? I think reading should be possible but haven't tried
it myself.

Re: apache beam bigquery IO connector support for bigquery external tables

[Brian Hulette via user <us...@beam.apache.org>]

Hi Nirav,
BQ external tables are read-only, so you won't be able to write this way. I
also don't think reading a standard external table will work since the Read
API and tabledata.list are not supported for external tables [1].

BigLake tables [2] on the other hand, may "just work". I haven't
checked this though, and it would still be read-only.

Brian

[1] https://cloud.google.com/bigquery/docs/external-tables#limitations
[2] https://cloud.google.com/bigquery/docs/biglake-intro

On Tue, Apr 18, 2023 at 9:33 AM Nirav Patel <ni...@gmail.com> wrote:

> Beam has bigquery IO support to be able to read and write to bq tables. I
> am assuming it only supports bigquery internal tables however if anyone
> knows if it supports writing to and reading from bq external tables
> (parquet or Iceberg) ? I think reading should be possible but haven't tried
> it myself.
>