You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@beam.apache.org by Sayak Paul <sp...@gmail.com> on 2022/01/10 13:01:41 UTC

Using `external_table_definition` during `ReadFromBigQuery`

Hi folks,

BigQuery supports reading [1] from external sources such as BigTable [2]. I
was wondering if it's possible to use it to read from BigQuery in a Beam
pipeline. There's no explicit argument called `external_table_definition`
in `ReadFromBigQuery`.

*References*

[1] https://cloud.google.com/bigquery/external-table-definition
[2] https://cloud.google.com/bigquery/external-data-bigtable

Sayak Paul | sayak.dev

Re: Using `external_table_definition` during `ReadFromBigQuery`

Posted by Sayak Paul <sp...@gmail.com>.

Thanks. But it's still unclear how one could construct the query that reads
from external data sources supported by BigQuery.
Sayak Paul | sayak.dev



On Tue, Jan 11, 2022 at 7:00 AM Chamikara Jayalath <ch...@google.com>
wrote:

> For a general Python BQ example see here:
> https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/cookbook/bigquery_tornadoes.py
> (and specify the 'query' parameter of 'ReadFromBigQuery' transform instead
> of the 'table' parameter.
>
> I haven't tried with a BigTable external table as I mentioned.
>
> Thanks,
> Cham
>
> On Mon, Jan 10, 2022 at 5:13 PM Sayak Paul <sp...@gmail.com> wrote:
>
>> Thanks! Could you provide a reference?
>> Sayak Paul | sayak.dev
>>
>>
>>
>> On Tue, Jan 11, 2022 at 12:52 AM Chamikara Jayalath <ch...@google.com>
>> wrote:
>>
>>> I haven't tried it, but you can try including such a table definition in
>>> a BQ query. It should work if BQ supports exporting results of such a query
>>> to GCS using export jobs.
>>>
>>> Thanks,
>>> Cham
>>>
>>> On Mon, Jan 10, 2022 at 5:02 AM Sayak Paul <sp...@gmail.com>
>>> wrote:
>>>
>>>> Hi folks,
>>>>
>>>> BigQuery supports reading [1] from external sources such as BigTable
>>>> [2]. I was wondering if it's possible to use it to read from BigQuery in a
>>>> Beam pipeline. There's no explicit argument called
>>>> `external_table_definition` in `ReadFromBigQuery`.
>>>>
>>>> *References*
>>>>
>>>> [1] https://cloud.google.com/bigquery/external-table-definition
>>>> [2] https://cloud.google.com/bigquery/external-data-bigtable
>>>>
>>>> Sayak Paul | sayak.dev
>>>>
>>>>

Re: Using `external_table_definition` during `ReadFromBigQuery`

Posted by Chamikara Jayalath <ch...@google.com>.

For a general Python BQ example see here:
https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/cookbook/bigquery_tornadoes.py
(and specify the 'query' parameter of 'ReadFromBigQuery' transform instead
of the 'table' parameter.

I haven't tried with a BigTable external table as I mentioned.

Thanks,
Cham

On Mon, Jan 10, 2022 at 5:13 PM Sayak Paul <sp...@gmail.com> wrote:

> Thanks! Could you provide a reference?
> Sayak Paul | sayak.dev
>
>
>
> On Tue, Jan 11, 2022 at 12:52 AM Chamikara Jayalath <ch...@google.com>
> wrote:
>
>> I haven't tried it, but you can try including such a table definition in
>> a BQ query. It should work if BQ supports exporting results of such a query
>> to GCS using export jobs.
>>
>> Thanks,
>> Cham
>>
>> On Mon, Jan 10, 2022 at 5:02 AM Sayak Paul <sp...@gmail.com> wrote:
>>
>>> Hi folks,
>>>
>>> BigQuery supports reading [1] from external sources such as BigTable
>>> [2]. I was wondering if it's possible to use it to read from BigQuery in a
>>> Beam pipeline. There's no explicit argument called
>>> `external_table_definition` in `ReadFromBigQuery`.
>>>
>>> *References*
>>>
>>> [1] https://cloud.google.com/bigquery/external-table-definition
>>> [2] https://cloud.google.com/bigquery/external-data-bigtable
>>>
>>> Sayak Paul | sayak.dev
>>>
>>>

Re: Using `external_table_definition` during `ReadFromBigQuery`

Posted by Sayak Paul <sp...@gmail.com>.

Thanks! Could you provide a reference?
Sayak Paul | sayak.dev



On Tue, Jan 11, 2022 at 12:52 AM Chamikara Jayalath <ch...@google.com>
wrote:

> I haven't tried it, but you can try including such a table definition in a
> BQ query. It should work if BQ supports exporting results of such a query
> to GCS using export jobs.
>
> Thanks,
> Cham
>
> On Mon, Jan 10, 2022 at 5:02 AM Sayak Paul <sp...@gmail.com> wrote:
>
>> Hi folks,
>>
>> BigQuery supports reading [1] from external sources such as BigTable [2].
>> I was wondering if it's possible to use it to read from BigQuery in a Beam
>> pipeline. There's no explicit argument called `external_table_definition`
>> in `ReadFromBigQuery`.
>>
>> *References*
>>
>> [1] https://cloud.google.com/bigquery/external-table-definition
>> [2] https://cloud.google.com/bigquery/external-data-bigtable
>>
>> Sayak Paul | sayak.dev
>>
>>

Re: Using `external_table_definition` during `ReadFromBigQuery`

Posted by Chamikara Jayalath <ch...@google.com>.

I haven't tried it, but you can try including such a table definition in a
BQ query. It should work if BQ supports exporting results of such a query
to GCS using export jobs.

Thanks,
Cham

On Mon, Jan 10, 2022 at 5:02 AM Sayak Paul <sp...@gmail.com> wrote:

> Hi folks,
>
> BigQuery supports reading [1] from external sources such as BigTable [2].
> I was wondering if it's possible to use it to read from BigQuery in a Beam
> pipeline. There's no explicit argument called `external_table_definition`
> in `ReadFromBigQuery`.
>
> *References*
>
> [1] https://cloud.google.com/bigquery/external-table-definition
> [2] https://cloud.google.com/bigquery/external-data-bigtable
>
> Sayak Paul | sayak.dev
>
>