You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@beam.apache.org by Chuck Yang <ch...@getcruise.com> on 2021/09/21 17:28:04 UTC

Change ReadFromBigQuery to submit queries with batch priority

Hi Beam devs,

I'd like to change the Python ReadFromBigQuery IO to submit queries
with batch priority by default to 1) enable queries to not count
toward the BQ concurrent rate limit and 2) allow queries to wait for
available idle resources. This was the default behavior when using the
now deprecated Dataflow native IO and AFAICT is also the default
behavior in the Java BigQueryIO.

I have a pull request to implement this change [1] and looking for
reviews/opinions. The PR adds a query_priority option to
ReadFromBigQuery, with a default value of BATCH.

Thanks!
Chuck

[1] https://github.com/apache/beam/pull/15536

-- 


*Confidentiality Note:* We care about protecting our proprietary 
information, confidential material, and trade secrets. This message may 
contain some or all of those things. Cruise will suffer material harm if 
anyone other than the intended recipient disseminates or takes any action 
based on this message. If you have received this message (including any 
attachments) in error, please delete it immediately and notify the sender 
promptly.