You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by gi...@apache.org on 2020/09/22 12:05:05 UTC
[beam] branch asf-site updated: Publishing website 2020/09/22
12:04:39 at commit dad8470
This is an automated email from the ASF dual-hosted git repository.
git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 3ff281d Publishing website 2020/09/22 12:04:39 at commit dad8470
3ff281d is described below
commit 3ff281d17cfbefdb5c30dcfba53f335a0bba9902
Author: jenkins <bu...@apache.org>
AuthorDate: Tue Sep 22 12:04:41 2020 +0000
Publishing website 2020/09/22 12:04:39 at commit dad8470
---
website/generated-content/documentation/index.xml | 38 ++++++++++------------
.../io/built-in/google-bigquery/index.html | 29 +++++++++--------
website/generated-content/sitemap.xml | 2 +-
3 files changed, 33 insertions(+), 36 deletions(-)
diff --git a/website/generated-content/documentation/index.xml b/website/generated-content/documentation/index.xml
index 0220a05..fa73c2c 100644
--- a/website/generated-content/documentation/index.xml
+++ b/website/generated-content/documentation/index.xml
@@ -10828,14 +10828,14 @@ BigQuery IO requires values of BYTES datatype to be encoded using base64
encoding when writing to BigQuery. When bytes are read from BigQuery they are
returned as base64-encoded bytes.</p>
<h2 id="reading-from-bigquery">Reading from BigQuery</h2>
-<p>BigQueryIO allows you to read from a BigQuery table, or read the results of an
-arbitrary SQL query string. By default, Beam invokes a <a href="https://cloud.google.com/bigquery/docs/exporting-data">BigQuery export
+<p>BigQueryIO allows you to read from a BigQuery table, or to execute a SQL query
+and read the results. By default, Beam invokes a <a href="https://cloud.google.com/bigquery/docs/exporting-data">BigQuery export
request</a> when you apply a
-BigQueryIO read transform. However, the Beam SDK for Java (version 2.11.0 and
-later) adds support for the beta release of the <a href="https://cloud.google.com/bigquery/docs/reference/storage/">BigQuery Storage API</a>
-as an <a href="https://beam.apache.org/releases/javadoc/current/index.html?org/apache/beam/sdk/annotations/Experimental.html">experimental feature</a>.
-See <a href="#storage-api">Using the BigQuery Storage API</a> for more information and a
-list of limitations.</p>
+BigQueryIO read transform. However, the Beam SDK for Java also supports using
+the <a href="https://cloud.google.com/bigquery/docs/reference/storage">BigQuery Storage
+API</a> to read directly
+from BigQuery storage. See <a href="#storage-api">Using the BigQuery Storage API</a> for
+more information.</p>
<blockquote>
<p>Beam’s use of BigQuery APIs is subject to BigQuery&rsquo;s
<a href="https://cloud.google.com/bigquery/quota-policy">Quota</a>
@@ -10984,20 +10984,16 @@ in the following example:</p>
</div>
<h3 id="storage-api">Using the BigQuery Storage API</h3>
<p>The <a href="https://cloud.google.com/bigquery/docs/reference/storage/">BigQuery Storage API</a>
-allows you to directly access tables in BigQuery storage. As a result, your
-pipeline can read from BigQuery storage faster than previously possible.</p>
-<p>The Beam SDK for Java (version 2.11.0 and later) adds support for the beta
-release of the BigQuery Storage API as an <a href="https://beam.apache.org/releases/javadoc/current/index.html?org/apache/beam/sdk/annotations/Experimental.html">experimental feature</a>.
-Beam&rsquo;s support for the BigQuery Storage API has the following limitations:</p>
-<ul>
-<li>The SDK for Python does not support the BigQuery Storage API.</li>
-<li>SDK versions 2.11.0 and 2.12.0 do not support reading with a query string; you
-can only read from a table.</li>
-<li>SDK versions before 2.15.0 do not support dynamic work rebalancing. As a
-result, reads might be less efficient in the presence of stragglers.</li>
-</ul>
-<p>Because this is currently a Beam experimental feature, export based reads are
-recommended for production jobs.</p>
+allows you to directly access tables in BigQuery storage, and supports features
+such as column selection and predicate filter push-down which can allow more
+efficient pipeline execution.</p>
+<p>The Beam SDK for Java supports using the BigQuery Storage API when reading from
+BigQuery. SDK versions before 2.24.0 support the BigQuery Storage API as an
+<a href="https://beam.apache.org/releases/javadoc/current/index.html?org/apache/beam/sdk/annotations/Experimental.html">experimental feature</a>
+and use the pre-GA BigQuery Storage API surface. Callers should migrate
+pipelines which use the BigQuery Storage API to use SDK version 2.24.0 or later.</p>
+<p>The Beam SDK for Python does not support the BigQuery Storage API. See
+<a href="https://issues.apache.org/jira/browse/BEAM-10917">BEAM-10917</a>).</p>
<h4 id="updating-your-code">Updating your code</h4>
<p>Use the following methods when you read from a table:</p>
<ul>
diff --git a/website/generated-content/documentation/io/built-in/google-bigquery/index.html b/website/generated-content/documentation/io/built-in/google-bigquery/index.html
index 0758714..672b70b 100644
--- a/website/generated-content/documentation/io/built-in/google-bigquery/index.html
+++ b/website/generated-content/documentation/io/built-in/google-bigquery/index.html
@@ -128,14 +128,14 @@ The GEOGRAPHY data type works with Well-Known Text (See <a href=https://en.wikip
format for reading and writing to BigQuery.
BigQuery IO requires values of BYTES datatype to be encoded using base64
encoding when writing to BigQuery. When bytes are read from BigQuery they are
-returned as base64-encoded bytes.</p><h2 id=reading-from-bigquery>Reading from BigQuery</h2><p>BigQueryIO allows you to read from a BigQuery table, or read the results of an
-arbitrary SQL query string. By default, Beam invokes a <a href=https://cloud.google.com/bigquery/docs/exporting-data>BigQuery export
+returned as base64-encoded bytes.</p><h2 id=reading-from-bigquery>Reading from BigQuery</h2><p>BigQueryIO allows you to read from a BigQuery table, or to execute a SQL query
+and read the results. By default, Beam invokes a <a href=https://cloud.google.com/bigquery/docs/exporting-data>BigQuery export
request</a> when you apply a
-BigQueryIO read transform. However, the Beam SDK for Java (version 2.11.0 and
-later) adds support for the beta release of the <a href=https://cloud.google.com/bigquery/docs/reference/storage/>BigQuery Storage API</a>
-as an <a href=https://beam.apache.org/releases/javadoc/current/index.html?org/apache/beam/sdk/annotations/Experimental.html>experimental feature</a>.
-See <a href=#storage-api>Using the BigQuery Storage API</a> for more information and a
-list of limitations.</p><blockquote><p>Beam’s use of BigQuery APIs is subject to BigQuery’s
+BigQueryIO read transform. However, the Beam SDK for Java also supports using
+the <a href=https://cloud.google.com/bigquery/docs/reference/storage>BigQuery Storage
+API</a> to read directly
+from BigQuery storage. See <a href=#storage-api>Using the BigQuery Storage API</a> for
+more information.</p><blockquote><p>Beam’s use of BigQuery APIs is subject to BigQuery’s
<a href=https://cloud.google.com/bigquery/quota-policy>Quota</a>
and <a href=https://cloud.google.com/bigquery/pricing>Pricing</a> policies.</p></blockquote><p class=language-java>The Beam SDK for Java has two BigQueryIO read methods. Both of these methods
allow you to read from a table, or read fields using a query string.</p><span class=language-java><ol><li><p><code>read(SerializableFunction)</code> reads Avro-formatted records and uses a
@@ -244,13 +244,14 @@ in the following example:</p><div class=language-java><div class=highlight><pre
<span class=n>use_standard_sql</span><span class=o>=</span><span class=bp>True</span><span class=p>))</span>
<span class=c1># Each row is a dictionary where the keys are the BigQuery columns</span>
<span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>lambda</span> <span class=n>elem</span><span class=p>:</span> <span class=n>elem</span><span class=p>[</span><span class=s1>'max_temperature'</span><span class=p>]))</span></code></pre></div></div><h3 id=storage-api>Using the BigQuery Storage API</h3><p>The <a href=https://cloud.google.com/bigquery/docs/reference/storage/>BigQuery Storage API</a>
-allows you to directly access tables in BigQuery storage. As a result, your
-pipeline can read from BigQuery storage faster than previously possible.</p><p>The Beam SDK for Java (version 2.11.0 and later) adds support for the beta
-release of the BigQuery Storage API as an <a href=https://beam.apache.org/releases/javadoc/current/index.html?org/apache/beam/sdk/annotations/Experimental.html>experimental feature</a>.
-Beam’s support for the BigQuery Storage API has the following limitations:</p><ul><li>The SDK for Python does not support the BigQuery Storage API.</li><li>SDK versions 2.11.0 and 2.12.0 do not support reading with a query string; you
-can only read from a table.</li><li>SDK versions before 2.15.0 do not support dynamic work rebalancing. As a
-result, reads might be less efficient in the presence of stragglers.</li></ul><p>Because this is currently a Beam experimental feature, export based reads are
-recommended for production jobs.</p><h4 id=updating-your-code>Updating your code</h4><p>Use the following methods when you read from a table:</p><ul><li>Required: Specify <a href=https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.TypedRead.html#withMethod-org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.TypedRead.Method->withMethod(Method.DIRECT_READ)</a> to use the BigQuery Storage API for the read operation.</li><li>Optional: To use features s [...]
+allows you to directly access tables in BigQuery storage, and supports features
+such as column selection and predicate filter push-down which can allow more
+efficient pipeline execution.</p><p>The Beam SDK for Java supports using the BigQuery Storage API when reading from
+BigQuery. SDK versions before 2.24.0 support the BigQuery Storage API as an
+<a href=https://beam.apache.org/releases/javadoc/current/index.html?org/apache/beam/sdk/annotations/Experimental.html>experimental feature</a>
+and use the pre-GA BigQuery Storage API surface. Callers should migrate
+pipelines which use the BigQuery Storage API to use SDK version 2.24.0 or later.</p><p>The Beam SDK for Python does not support the BigQuery Storage API. See
+<a href=https://issues.apache.org/jira/browse/BEAM-10917>BEAM-10917</a>).</p><h4 id=updating-your-code>Updating your code</h4><p>Use the following methods when you read from a table:</p><ul><li>Required: Specify <a href=https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.TypedRead.html#withMethod-org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.TypedRead.Method->withMethod(Method.DIRECT_READ)</a> to use the BigQuery Storage API for the read opera [...]
example</a>.
When the example’s read method option is set to <code>DIRECT_READ</code>, the pipeline uses
the BigQuery Storage API and column projection to read public samples of weather
diff --git a/website/generated-content/sitemap.xml b/website/generated-content/sitemap.xml
index 7d844fd..a2bde42 100644
--- a/website/generated-content/sitemap.xml
+++ b/website/generated-content/sitemap.xml
@@ -1 +1 @@
-<?xml version="1.0" encoding="utf-8" standalone="yes"?><urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:xhtml="http://www.w3.org/1999/xhtml"><url><loc>/blog/beam-2.24.0/</loc><lastmod>2020-09-18T12:38:38-07:00</lastmod></url><url><loc>/categories/blog/</loc><lastmod>2020-09-18T12:38:38-07:00</lastmod></url><url><loc>/blog/</loc><lastmod>2020-09-18T12:38:38-07:00</lastmod></url><url><loc>/categories/</loc><lastmod>2020-09-18T12:38:38-07:00</lastmod></url><url><loc>/blog/p [...]
\ No newline at end of file
+<?xml version="1.0" encoding="utf-8" standalone="yes"?><urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:xhtml="http://www.w3.org/1999/xhtml"><url><loc>/blog/beam-2.24.0/</loc><lastmod>2020-09-18T12:38:38-07:00</lastmod></url><url><loc>/categories/blog/</loc><lastmod>2020-09-18T12:38:38-07:00</lastmod></url><url><loc>/blog/</loc><lastmod>2020-09-18T12:38:38-07:00</lastmod></url><url><loc>/categories/</loc><lastmod>2020-09-18T12:38:38-07:00</lastmod></url><url><loc>/blog/p [...]
\ No newline at end of file