beam.io.BigQuerySource does not accept value providers

[Theodore Siu <th...@google.com>]

Hi,

We are attempting to build a Dataflow template in Beam Python and are
running into issues with using a value provider specifically
with beam.io.BigQuerySource which throws the following error.    ValueError:
Invalid DisplayDataItem. Value RuntimeValueProvider(option: input, type:
str, default_value: 'test') is of an unsupported type.

Tracing the code in Github it seems like the main culprits are the
following:

https://github.com/apache/beam/blob/d9add564c1c34065829f712074bdd3856b2b0982/sdks/python/apache_beam/io/gcp/bigquery.py#L470

https://github.com/apache/beam/blob/d9add564c1c34065829f712074bdd3856b2b0982/sdks/python/apache_beam/transforms/display.py#L244


Do we have any idea when a fix can be made?

-Theo

Re: beam.io.BigQuerySource does not accept value providers

[Pablo Estrada <pa...@google.com>]

Hi Theodore!
Display data is what's throwing the error, but the BigQuerySource does not
support value providers even despite that issue because it's a Dataflow
native source. Unfortunately, this is not currently possible.
Currently, you could do this executing a BQ export job (using a DoFn), and
use fileio to consume those newly exported files. We may prioritize
building a source for that, but it is not there ATM.
Best
-P.

On Fri, Oct 18, 2019 at 6:42 AM Theodore Siu <th...@google.com> wrote:

> Additionally, for reference
> https://stackoverflow.com/questions/46595149/dynamic-bigquery-query-in-dataflow-template
>
> On Fri, Oct 18, 2019 at 9:34 AM Theodore Siu <th...@google.com> wrote:
>
>> Hi,
>>
>> We are attempting to build a Dataflow template in Beam Python and are
>> running into issues with using a value provider specifically
>> with beam.io.BigQuerySource which throws the following error.    ValueError:
>> Invalid DisplayDataItem. Value RuntimeValueProvider(option: input, type:
>> str, default_value: 'test') is of an unsupported type.
>>
>> Tracing the code in Github it seems like the main culprits are the
>> following:
>>
>>
>> https://github.com/apache/beam/blob/d9add564c1c34065829f712074bdd3856b2b0982/sdks/python/apache_beam/io/gcp/bigquery.py#L470
>>
>>
>> https://github.com/apache/beam/blob/d9add564c1c34065829f712074bdd3856b2b0982/sdks/python/apache_beam/transforms/display.py#L244
>>
>>
>> Do we have any idea when a fix can be made?
>>
>> -Theo
>>
>>
>>
>>

Re: beam.io.BigQuerySource does not accept value providers

[Pablo Estrada <pa...@google.com>]

Hi Theodore!
Display data is what's throwing the error, but the BigQuerySource does not
support value providers even despite that issue because it's a Dataflow
native source. Unfortunately, this is not currently possible.
Currently, you could do this executing a BQ export job (using a DoFn), and
use fileio to consume those newly exported files. We may prioritize
building a source for that, but it is not there ATM.
Best
-P.

On Fri, Oct 18, 2019 at 6:42 AM Theodore Siu <th...@google.com> wrote:

> Additionally, for reference
> https://stackoverflow.com/questions/46595149/dynamic-bigquery-query-in-dataflow-template
>
> On Fri, Oct 18, 2019 at 9:34 AM Theodore Siu <th...@google.com> wrote:
>
>> Hi,
>>
>> We are attempting to build a Dataflow template in Beam Python and are
>> running into issues with using a value provider specifically
>> with beam.io.BigQuerySource which throws the following error.    ValueError:
>> Invalid DisplayDataItem. Value RuntimeValueProvider(option: input, type:
>> str, default_value: 'test') is of an unsupported type.
>>
>> Tracing the code in Github it seems like the main culprits are the
>> following:
>>
>>
>> https://github.com/apache/beam/blob/d9add564c1c34065829f712074bdd3856b2b0982/sdks/python/apache_beam/io/gcp/bigquery.py#L470
>>
>>
>> https://github.com/apache/beam/blob/d9add564c1c34065829f712074bdd3856b2b0982/sdks/python/apache_beam/transforms/display.py#L244
>>
>>
>> Do we have any idea when a fix can be made?
>>
>> -Theo
>>
>>
>>
>>

Re: beam.io.BigQuerySource does not accept value providers

[Theodore Siu <th...@google.com>]

Additionally, for reference
https://stackoverflow.com/questions/46595149/dynamic-bigquery-query-in-dataflow-template

On Fri, Oct 18, 2019 at 9:34 AM Theodore Siu <th...@google.com> wrote:

> Hi,
>
> We are attempting to build a Dataflow template in Beam Python and are
> running into issues with using a value provider specifically
> with beam.io.BigQuerySource which throws the following error.    ValueError:
> Invalid DisplayDataItem. Value RuntimeValueProvider(option: input, type:
> str, default_value: 'test') is of an unsupported type.
>
> Tracing the code in Github it seems like the main culprits are the
> following:
>
>
> https://github.com/apache/beam/blob/d9add564c1c34065829f712074bdd3856b2b0982/sdks/python/apache_beam/io/gcp/bigquery.py#L470
>
>
> https://github.com/apache/beam/blob/d9add564c1c34065829f712074bdd3856b2b0982/sdks/python/apache_beam/transforms/display.py#L244
>
>
> Do we have any idea when a fix can be made?
>
> -Theo
>
>
>
>

Re: beam.io.BigQuerySource does not accept value providers

[Theodore Siu <th...@google.com>]

Additionally, for reference
https://stackoverflow.com/questions/46595149/dynamic-bigquery-query-in-dataflow-template

On Fri, Oct 18, 2019 at 9:34 AM Theodore Siu <th...@google.com> wrote:

> Hi,
>
> We are attempting to build a Dataflow template in Beam Python and are
> running into issues with using a value provider specifically
> with beam.io.BigQuerySource which throws the following error.    ValueError:
> Invalid DisplayDataItem. Value RuntimeValueProvider(option: input, type:
> str, default_value: 'test') is of an unsupported type.
>
> Tracing the code in Github it seems like the main culprits are the
> following:
>
>
> https://github.com/apache/beam/blob/d9add564c1c34065829f712074bdd3856b2b0982/sdks/python/apache_beam/io/gcp/bigquery.py#L470
>
>
> https://github.com/apache/beam/blob/d9add564c1c34065829f712074bdd3856b2b0982/sdks/python/apache_beam/transforms/display.py#L244
>
>
> Do we have any idea when a fix can be made?
>
> -Theo
>
>
>
>