You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Asha Rostamianfar <ar...@google.com.INVALID> on 2017/08/23 21:51:33 UTC
Dynamically generate BigQuery schema based on input?
Hi,
I'm wondering whether it's possible to dynamically generate a BigQuery
schema based on input. For instance, the fields would be specified in one
or more input files that are read and processed as part of the pipeline.
Thanks,
Asha
Re: Dynamically generate BigQuery schema based on input?
Posted by Chamikara Jayalath <ch...@apache.org>.
Here are the JIRAs.
BigQuery custom sink: https://issues.apache.org/jira/browse/BEAM-2801
BigQuery custom source: https://issues.apache.org/jira/browse/BEAM-1440
Volunteers are welcome :)
Thanks,
Cham
On Wed, Aug 23, 2017 at 7:21 PM Chamikara Jayalath <ch...@apache.org>
wrote:
> I don't think we have JIRAs. I'll go ahead and create them.
>
> Thanks,
> Cham
>
>
> On Wed, Aug 23, 2017 at 6:33 PM Reuven Lax <re...@google.com.invalid>
> wrote:
>
>> Cham,
>>
>> Do we have JIRAs tracking these features?
>>
>> Reuven
>>
>> On Wed, Aug 23, 2017 at 6:26 PM, Asha Rostamianfar <
>> arostami@google.com.invalid> wrote:
>>
>> > It's for a batch job. I'll think about possible workarounds then
>> (perhaps I
>> > can have a prestage process that reads the necessary info from the files
>> > and passes as an argument). Thanks everyone!
>> >
>> > On Wed, Aug 23, 2017 at 9:17 PM, Chamikara Jayalath <
>> chamikara@apache.org>
>> > wrote:
>> >
>> > > Ah, actually for this feature I think we need a BigQuery custom sink.
>> > Which
>> > > also doesn't have an ETA currently.
>> > >
>> > > Thanks,
>> > > Cham
>> > >
>> > > On Wed, Aug 23, 2017 at 6:14 PM Chamikara Jayalath <
>> chamikara@apache.org
>> > >
>> > > wrote:
>> > >
>> > > > We have to develop a BigQuery BoundedSource for this first. So I
>> expect
>> > > > support for Python SDK to be at least few months away.
>> > > >
>> > > > Thanks,
>> > > > Cham
>> > > >
>> > > >
>> > > > On Wed, Aug 23, 2017 at 6:10 PM Asha Rostamianfar
>> > > > <ar...@google.com.invalid> wrote:
>> > > >
>> > > >> ah, ok..is there an existing JIRA entry with some approximate ETA?
>> > (i.e.
>> > > >> weeks, months, or quarters).
>> > > >>
>> > > >> Thanks again.
>> > > >>
>> > > >> On Wed, Aug 23, 2017 at 8:58 PM, Reuven Lax
>> <relax@google.com.invalid
>> > >
>> > > >> wrote:
>> > > >>
>> > > >> > Not yet, only in Java so far.
>> > > >> >
>> > > >> > On Wed, Aug 23, 2017 at 5:53 PM, Asha Rostamianfar <
>> > > >> > arostami@google.com.invalid> wrote:
>> > > >> >
>> > > >> > > Thanks, Eugene! Is this also supported in the python SDK?
>> > > >> > >
>> > > >> > > On Wed, Aug 23, 2017 at 8:39 PM, Eugene Kirpichov <
>> > > >> > > kirpichov@google.com.invalid> wrote:
>> > > >> > >
>> > > >> > > > Yes, this is possible using the BigQueryIO.write().to(
>> > > >> > > DynamicDestinations)
>> > > >> > > > API. It allows you to write different values to different
>> tables
>> > > >> with
>> > > >> > > > different schemas.
>> > > >> > > > On Wed, Aug 23, 2017, 2:51 PM Asha Rostamianfar
>> > > >> > > > <ar...@google.com.invalid>
>> > > >> > > > wrote:
>> > > >> > > >
>> > > >> > > > > Hi,
>> > > >> > > > >
>> > > >> > > > > I'm wondering whether it's possible to dynamically
>> generate a
>> > > >> > BigQuery
>> > > >> > > > > schema based on input. For instance, the fields would be
>> > > >> specified in
>> > > >> > > one
>> > > >> > > > > or more input files that are read and processed as part of
>> the
>> > > >> > > pipeline.
>> > > >> > > > >
>> > > >> > > > > Thanks,
>> > > >> > > > > Asha
>> > > >> > > > >
>> > > >> > > >
>> > > >> > >
>> > > >> >
>> > > >>
>> > > >
>> > >
>> >
>>
>
Re: Dynamically generate BigQuery schema based on input?
Posted by Chamikara Jayalath <ch...@apache.org>.
I don't think we have JIRAs. I'll go ahead and create them.
Thanks,
Cham
On Wed, Aug 23, 2017 at 6:33 PM Reuven Lax <re...@google.com.invalid> wrote:
> Cham,
>
> Do we have JIRAs tracking these features?
>
> Reuven
>
> On Wed, Aug 23, 2017 at 6:26 PM, Asha Rostamianfar <
> arostami@google.com.invalid> wrote:
>
> > It's for a batch job. I'll think about possible workarounds then
> (perhaps I
> > can have a prestage process that reads the necessary info from the files
> > and passes as an argument). Thanks everyone!
> >
> > On Wed, Aug 23, 2017 at 9:17 PM, Chamikara Jayalath <
> chamikara@apache.org>
> > wrote:
> >
> > > Ah, actually for this feature I think we need a BigQuery custom sink.
> > Which
> > > also doesn't have an ETA currently.
> > >
> > > Thanks,
> > > Cham
> > >
> > > On Wed, Aug 23, 2017 at 6:14 PM Chamikara Jayalath <
> chamikara@apache.org
> > >
> > > wrote:
> > >
> > > > We have to develop a BigQuery BoundedSource for this first. So I
> expect
> > > > support for Python SDK to be at least few months away.
> > > >
> > > > Thanks,
> > > > Cham
> > > >
> > > >
> > > > On Wed, Aug 23, 2017 at 6:10 PM Asha Rostamianfar
> > > > <ar...@google.com.invalid> wrote:
> > > >
> > > >> ah, ok..is there an existing JIRA entry with some approximate ETA?
> > (i.e.
> > > >> weeks, months, or quarters).
> > > >>
> > > >> Thanks again.
> > > >>
> > > >> On Wed, Aug 23, 2017 at 8:58 PM, Reuven Lax
> <relax@google.com.invalid
> > >
> > > >> wrote:
> > > >>
> > > >> > Not yet, only in Java so far.
> > > >> >
> > > >> > On Wed, Aug 23, 2017 at 5:53 PM, Asha Rostamianfar <
> > > >> > arostami@google.com.invalid> wrote:
> > > >> >
> > > >> > > Thanks, Eugene! Is this also supported in the python SDK?
> > > >> > >
> > > >> > > On Wed, Aug 23, 2017 at 8:39 PM, Eugene Kirpichov <
> > > >> > > kirpichov@google.com.invalid> wrote:
> > > >> > >
> > > >> > > > Yes, this is possible using the BigQueryIO.write().to(
> > > >> > > DynamicDestinations)
> > > >> > > > API. It allows you to write different values to different
> tables
> > > >> with
> > > >> > > > different schemas.
> > > >> > > > On Wed, Aug 23, 2017, 2:51 PM Asha Rostamianfar
> > > >> > > > <ar...@google.com.invalid>
> > > >> > > > wrote:
> > > >> > > >
> > > >> > > > > Hi,
> > > >> > > > >
> > > >> > > > > I'm wondering whether it's possible to dynamically generate
> a
> > > >> > BigQuery
> > > >> > > > > schema based on input. For instance, the fields would be
> > > >> specified in
> > > >> > > one
> > > >> > > > > or more input files that are read and processed as part of
> the
> > > >> > > pipeline.
> > > >> > > > >
> > > >> > > > > Thanks,
> > > >> > > > > Asha
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > > >
> > >
> >
>
Re: Dynamically generate BigQuery schema based on input?
Posted by Reuven Lax <re...@google.com.INVALID>.
Cham,
Do we have JIRAs tracking these features?
Reuven
On Wed, Aug 23, 2017 at 6:26 PM, Asha Rostamianfar <
arostami@google.com.invalid> wrote:
> It's for a batch job. I'll think about possible workarounds then (perhaps I
> can have a prestage process that reads the necessary info from the files
> and passes as an argument). Thanks everyone!
>
> On Wed, Aug 23, 2017 at 9:17 PM, Chamikara Jayalath <ch...@apache.org>
> wrote:
>
> > Ah, actually for this feature I think we need a BigQuery custom sink.
> Which
> > also doesn't have an ETA currently.
> >
> > Thanks,
> > Cham
> >
> > On Wed, Aug 23, 2017 at 6:14 PM Chamikara Jayalath <chamikara@apache.org
> >
> > wrote:
> >
> > > We have to develop a BigQuery BoundedSource for this first. So I expect
> > > support for Python SDK to be at least few months away.
> > >
> > > Thanks,
> > > Cham
> > >
> > >
> > > On Wed, Aug 23, 2017 at 6:10 PM Asha Rostamianfar
> > > <ar...@google.com.invalid> wrote:
> > >
> > >> ah, ok..is there an existing JIRA entry with some approximate ETA?
> (i.e.
> > >> weeks, months, or quarters).
> > >>
> > >> Thanks again.
> > >>
> > >> On Wed, Aug 23, 2017 at 8:58 PM, Reuven Lax <relax@google.com.invalid
> >
> > >> wrote:
> > >>
> > >> > Not yet, only in Java so far.
> > >> >
> > >> > On Wed, Aug 23, 2017 at 5:53 PM, Asha Rostamianfar <
> > >> > arostami@google.com.invalid> wrote:
> > >> >
> > >> > > Thanks, Eugene! Is this also supported in the python SDK?
> > >> > >
> > >> > > On Wed, Aug 23, 2017 at 8:39 PM, Eugene Kirpichov <
> > >> > > kirpichov@google.com.invalid> wrote:
> > >> > >
> > >> > > > Yes, this is possible using the BigQueryIO.write().to(
> > >> > > DynamicDestinations)
> > >> > > > API. It allows you to write different values to different tables
> > >> with
> > >> > > > different schemas.
> > >> > > > On Wed, Aug 23, 2017, 2:51 PM Asha Rostamianfar
> > >> > > > <ar...@google.com.invalid>
> > >> > > > wrote:
> > >> > > >
> > >> > > > > Hi,
> > >> > > > >
> > >> > > > > I'm wondering whether it's possible to dynamically generate a
> > >> > BigQuery
> > >> > > > > schema based on input. For instance, the fields would be
> > >> specified in
> > >> > > one
> > >> > > > > or more input files that are read and processed as part of the
> > >> > > pipeline.
> > >> > > > >
> > >> > > > > Thanks,
> > >> > > > > Asha
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> >
>
Re: Dynamically generate BigQuery schema based on input?
Posted by Asha Rostamianfar <ar...@google.com.INVALID>.
It's for a batch job. I'll think about possible workarounds then (perhaps I
can have a prestage process that reads the necessary info from the files
and passes as an argument). Thanks everyone!
On Wed, Aug 23, 2017 at 9:17 PM, Chamikara Jayalath <ch...@apache.org>
wrote:
> Ah, actually for this feature I think we need a BigQuery custom sink. Which
> also doesn't have an ETA currently.
>
> Thanks,
> Cham
>
> On Wed, Aug 23, 2017 at 6:14 PM Chamikara Jayalath <ch...@apache.org>
> wrote:
>
> > We have to develop a BigQuery BoundedSource for this first. So I expect
> > support for Python SDK to be at least few months away.
> >
> > Thanks,
> > Cham
> >
> >
> > On Wed, Aug 23, 2017 at 6:10 PM Asha Rostamianfar
> > <ar...@google.com.invalid> wrote:
> >
> >> ah, ok..is there an existing JIRA entry with some approximate ETA? (i.e.
> >> weeks, months, or quarters).
> >>
> >> Thanks again.
> >>
> >> On Wed, Aug 23, 2017 at 8:58 PM, Reuven Lax <re...@google.com.invalid>
> >> wrote:
> >>
> >> > Not yet, only in Java so far.
> >> >
> >> > On Wed, Aug 23, 2017 at 5:53 PM, Asha Rostamianfar <
> >> > arostami@google.com.invalid> wrote:
> >> >
> >> > > Thanks, Eugene! Is this also supported in the python SDK?
> >> > >
> >> > > On Wed, Aug 23, 2017 at 8:39 PM, Eugene Kirpichov <
> >> > > kirpichov@google.com.invalid> wrote:
> >> > >
> >> > > > Yes, this is possible using the BigQueryIO.write().to(
> >> > > DynamicDestinations)
> >> > > > API. It allows you to write different values to different tables
> >> with
> >> > > > different schemas.
> >> > > > On Wed, Aug 23, 2017, 2:51 PM Asha Rostamianfar
> >> > > > <ar...@google.com.invalid>
> >> > > > wrote:
> >> > > >
> >> > > > > Hi,
> >> > > > >
> >> > > > > I'm wondering whether it's possible to dynamically generate a
> >> > BigQuery
> >> > > > > schema based on input. For instance, the fields would be
> >> specified in
> >> > > one
> >> > > > > or more input files that are read and processed as part of the
> >> > > pipeline.
> >> > > > >
> >> > > > > Thanks,
> >> > > > > Asha
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
>
Re: Dynamically generate BigQuery schema based on input?
Posted by Chamikara Jayalath <ch...@apache.org>.
Ah, actually for this feature I think we need a BigQuery custom sink. Which
also doesn't have an ETA currently.
Thanks,
Cham
On Wed, Aug 23, 2017 at 6:14 PM Chamikara Jayalath <ch...@apache.org>
wrote:
> We have to develop a BigQuery BoundedSource for this first. So I expect
> support for Python SDK to be at least few months away.
>
> Thanks,
> Cham
>
>
> On Wed, Aug 23, 2017 at 6:10 PM Asha Rostamianfar
> <ar...@google.com.invalid> wrote:
>
>> ah, ok..is there an existing JIRA entry with some approximate ETA? (i.e.
>> weeks, months, or quarters).
>>
>> Thanks again.
>>
>> On Wed, Aug 23, 2017 at 8:58 PM, Reuven Lax <re...@google.com.invalid>
>> wrote:
>>
>> > Not yet, only in Java so far.
>> >
>> > On Wed, Aug 23, 2017 at 5:53 PM, Asha Rostamianfar <
>> > arostami@google.com.invalid> wrote:
>> >
>> > > Thanks, Eugene! Is this also supported in the python SDK?
>> > >
>> > > On Wed, Aug 23, 2017 at 8:39 PM, Eugene Kirpichov <
>> > > kirpichov@google.com.invalid> wrote:
>> > >
>> > > > Yes, this is possible using the BigQueryIO.write().to(
>> > > DynamicDestinations)
>> > > > API. It allows you to write different values to different tables
>> with
>> > > > different schemas.
>> > > > On Wed, Aug 23, 2017, 2:51 PM Asha Rostamianfar
>> > > > <ar...@google.com.invalid>
>> > > > wrote:
>> > > >
>> > > > > Hi,
>> > > > >
>> > > > > I'm wondering whether it's possible to dynamically generate a
>> > BigQuery
>> > > > > schema based on input. For instance, the fields would be
>> specified in
>> > > one
>> > > > > or more input files that are read and processed as part of the
>> > > pipeline.
>> > > > >
>> > > > > Thanks,
>> > > > > Asha
>> > > > >
>> > > >
>> > >
>> >
>>
>
Re: Dynamically generate BigQuery schema based on input?
Posted by Chamikara Jayalath <ch...@apache.org>.
We have to develop a BigQuery BoundedSource for this first. So I expect
support for Python SDK to be at least few months away.
Thanks,
Cham
On Wed, Aug 23, 2017 at 6:10 PM Asha Rostamianfar
<ar...@google.com.invalid> wrote:
> ah, ok..is there an existing JIRA entry with some approximate ETA? (i.e.
> weeks, months, or quarters).
>
> Thanks again.
>
> On Wed, Aug 23, 2017 at 8:58 PM, Reuven Lax <re...@google.com.invalid>
> wrote:
>
> > Not yet, only in Java so far.
> >
> > On Wed, Aug 23, 2017 at 5:53 PM, Asha Rostamianfar <
> > arostami@google.com.invalid> wrote:
> >
> > > Thanks, Eugene! Is this also supported in the python SDK?
> > >
> > > On Wed, Aug 23, 2017 at 8:39 PM, Eugene Kirpichov <
> > > kirpichov@google.com.invalid> wrote:
> > >
> > > > Yes, this is possible using the BigQueryIO.write().to(
> > > DynamicDestinations)
> > > > API. It allows you to write different values to different tables with
> > > > different schemas.
> > > > On Wed, Aug 23, 2017, 2:51 PM Asha Rostamianfar
> > > > <ar...@google.com.invalid>
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I'm wondering whether it's possible to dynamically generate a
> > BigQuery
> > > > > schema based on input. For instance, the fields would be specified
> in
> > > one
> > > > > or more input files that are read and processed as part of the
> > > pipeline.
> > > > >
> > > > > Thanks,
> > > > > Asha
> > > > >
> > > >
> > >
> >
>
Re: Dynamically generate BigQuery schema based on input?
Posted by Reuven Lax <re...@google.com.INVALID>.
Is this for a batch pipeline or a streaming pipeline? I don't believe that
Python yet has good support for BigQuery in batch jobs at all.
On Wed, Aug 23, 2017 at 6:10 PM, Asha Rostamianfar <
arostami@google.com.invalid> wrote:
> ah, ok..is there an existing JIRA entry with some approximate ETA? (i.e.
> weeks, months, or quarters).
>
> Thanks again.
>
> On Wed, Aug 23, 2017 at 8:58 PM, Reuven Lax <re...@google.com.invalid>
> wrote:
>
> > Not yet, only in Java so far.
> >
> > On Wed, Aug 23, 2017 at 5:53 PM, Asha Rostamianfar <
> > arostami@google.com.invalid> wrote:
> >
> > > Thanks, Eugene! Is this also supported in the python SDK?
> > >
> > > On Wed, Aug 23, 2017 at 8:39 PM, Eugene Kirpichov <
> > > kirpichov@google.com.invalid> wrote:
> > >
> > > > Yes, this is possible using the BigQueryIO.write().to(
> > > DynamicDestinations)
> > > > API. It allows you to write different values to different tables with
> > > > different schemas.
> > > > On Wed, Aug 23, 2017, 2:51 PM Asha Rostamianfar
> > > > <ar...@google.com.invalid>
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I'm wondering whether it's possible to dynamically generate a
> > BigQuery
> > > > > schema based on input. For instance, the fields would be specified
> in
> > > one
> > > > > or more input files that are read and processed as part of the
> > > pipeline.
> > > > >
> > > > > Thanks,
> > > > > Asha
> > > > >
> > > >
> > >
> >
>
Re: Dynamically generate BigQuery schema based on input?
Posted by Asha Rostamianfar <ar...@google.com.INVALID>.
ah, ok..is there an existing JIRA entry with some approximate ETA? (i.e.
weeks, months, or quarters).
Thanks again.
On Wed, Aug 23, 2017 at 8:58 PM, Reuven Lax <re...@google.com.invalid>
wrote:
> Not yet, only in Java so far.
>
> On Wed, Aug 23, 2017 at 5:53 PM, Asha Rostamianfar <
> arostami@google.com.invalid> wrote:
>
> > Thanks, Eugene! Is this also supported in the python SDK?
> >
> > On Wed, Aug 23, 2017 at 8:39 PM, Eugene Kirpichov <
> > kirpichov@google.com.invalid> wrote:
> >
> > > Yes, this is possible using the BigQueryIO.write().to(
> > DynamicDestinations)
> > > API. It allows you to write different values to different tables with
> > > different schemas.
> > > On Wed, Aug 23, 2017, 2:51 PM Asha Rostamianfar
> > > <ar...@google.com.invalid>
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > I'm wondering whether it's possible to dynamically generate a
> BigQuery
> > > > schema based on input. For instance, the fields would be specified in
> > one
> > > > or more input files that are read and processed as part of the
> > pipeline.
> > > >
> > > > Thanks,
> > > > Asha
> > > >
> > >
> >
>
Re: Dynamically generate BigQuery schema based on input?
Posted by Reuven Lax <re...@google.com.INVALID>.
Not yet, only in Java so far.
On Wed, Aug 23, 2017 at 5:53 PM, Asha Rostamianfar <
arostami@google.com.invalid> wrote:
> Thanks, Eugene! Is this also supported in the python SDK?
>
> On Wed, Aug 23, 2017 at 8:39 PM, Eugene Kirpichov <
> kirpichov@google.com.invalid> wrote:
>
> > Yes, this is possible using the BigQueryIO.write().to(
> DynamicDestinations)
> > API. It allows you to write different values to different tables with
> > different schemas.
> > On Wed, Aug 23, 2017, 2:51 PM Asha Rostamianfar
> > <ar...@google.com.invalid>
> > wrote:
> >
> > > Hi,
> > >
> > > I'm wondering whether it's possible to dynamically generate a BigQuery
> > > schema based on input. For instance, the fields would be specified in
> one
> > > or more input files that are read and processed as part of the
> pipeline.
> > >
> > > Thanks,
> > > Asha
> > >
> >
>
Re: Dynamically generate BigQuery schema based on input?
Posted by Asha Rostamianfar <ar...@google.com.INVALID>.
Thanks, Eugene! Is this also supported in the python SDK?
On Wed, Aug 23, 2017 at 8:39 PM, Eugene Kirpichov <
kirpichov@google.com.invalid> wrote:
> Yes, this is possible using the BigQueryIO.write().to(DynamicDestinations)
> API. It allows you to write different values to different tables with
> different schemas.
> On Wed, Aug 23, 2017, 2:51 PM Asha Rostamianfar
> <ar...@google.com.invalid>
> wrote:
>
> > Hi,
> >
> > I'm wondering whether it's possible to dynamically generate a BigQuery
> > schema based on input. For instance, the fields would be specified in one
> > or more input files that are read and processed as part of the pipeline.
> >
> > Thanks,
> > Asha
> >
>
Re: Dynamically generate BigQuery schema based on input?
Posted by Reuven Lax <re...@google.com.INVALID>.
Actually dynamic schemas can be supplied via the DynamicDestinations class.
withSchemaFromView is a convenience helper for the scenario where the
schema is a side input.
On Wed, Aug 23, 2017 at 11:54 PM, Chaim Turkel <ch...@behalf.com> wrote:
> it is not clear if you need dynamic destination or schema. For the
> destination the to will work, but for schema you need to use
> withSchemaFromView
>
> On Thu, Aug 24, 2017 at 3:39 AM, Eugene Kirpichov
> <ki...@google.com.invalid> wrote:
> > Yes, this is possible using the BigQueryIO.write().to(
> DynamicDestinations)
> > API. It allows you to write different values to different tables with
> > different schemas.
> > On Wed, Aug 23, 2017, 2:51 PM Asha Rostamianfar
> <ar...@google.com.invalid>
> > wrote:
> >
> >> Hi,
> >>
> >> I'm wondering whether it's possible to dynamically generate a BigQuery
> >> schema based on input. For instance, the fields would be specified in
> one
> >> or more input files that are read and processed as part of the pipeline.
> >>
> >> Thanks,
> >> Asha
> >>
>
Re: Dynamically generate BigQuery schema based on input?
Posted by Chaim Turkel <ch...@behalf.com>.
it is not clear if you need dynamic destination or schema. For the
destination the to will work, but for schema you need to use
withSchemaFromView
On Thu, Aug 24, 2017 at 3:39 AM, Eugene Kirpichov
<ki...@google.com.invalid> wrote:
> Yes, this is possible using the BigQueryIO.write().to(DynamicDestinations)
> API. It allows you to write different values to different tables with
> different schemas.
> On Wed, Aug 23, 2017, 2:51 PM Asha Rostamianfar <ar...@google.com.invalid>
> wrote:
>
>> Hi,
>>
>> I'm wondering whether it's possible to dynamically generate a BigQuery
>> schema based on input. For instance, the fields would be specified in one
>> or more input files that are read and processed as part of the pipeline.
>>
>> Thanks,
>> Asha
>>
Re: Dynamically generate BigQuery schema based on input?
Posted by Eugene Kirpichov <ki...@google.com.INVALID>.
Yes, this is possible using the BigQueryIO.write().to(DynamicDestinations)
API. It allows you to write different values to different tables with
different schemas.
On Wed, Aug 23, 2017, 2:51 PM Asha Rostamianfar <ar...@google.com.invalid>
wrote:
> Hi,
>
> I'm wondering whether it's possible to dynamically generate a BigQuery
> schema based on input. For instance, the fields would be specified in one
> or more input files that are read and processed as part of the pipeline.
>
> Thanks,
> Asha
>