You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by gi...@apache.org on 2021/10/02 06:03:17 UTC
[beam] branch asf-site updated: Publishing website 2021/10/02
06:02:45 at commit b9457b7
This is an automated email from the ASF dual-hosted git repository.
git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 39fc301 Publishing website 2021/10/02 06:02:45 at commit b9457b7
39fc301 is described below
commit 39fc301df36a239e676d921c95f0b09321b0775e
Author: jenkins <bu...@apache.org>
AuthorDate: Sat Oct 2 06:02:45 2021 +0000
Publishing website 2021/10/02 06:02:45 at commit b9457b7
---
website/generated-content/documentation/index.xml | 259 +++++++++++++++++++--
.../documentation/programming-guide/index.html | 211 ++++++++++++++---
website/generated-content/sitemap.xml | 2 +-
3 files changed, 425 insertions(+), 47 deletions(-)
diff --git a/website/generated-content/documentation/index.xml b/website/generated-content/documentation/index.xml
index 9c698e5..56112c0 100644
--- a/website/generated-content/documentation/index.xml
+++ b/website/generated-content/documentation/index.xml
@@ -5898,7 +5898,7 @@ determined by the input data, or depend on a different branch of your pipeline.&
<span class="kd">var</span> <span class="nx">cutOff</span> <span class="kt">float64</span>
<span class="nx">ok</span> <span class="o">:=</span> <span class="nf">lengthCutOffIter</span><span class="p">(</span><span class="o">&amp;</span><span class="nx">cutOff</span><span class="p">)</span>
<span class="k">if</span> <span class="p">!</span><span class="nx">ok</span> <span class="p">{</span>
-<span class="k">return</span> <span class="nx">fmt</span><span class="p">.</span><span class="nf">Errorf</span><span class="p">(</span><span class="s">&#34;No length cutoff provided.&#34;</span><span class="p">)</span>
+<span class="k">return</span> <span class="nx">fmt</span><span class="p">.</span><span class="nf">Errorf</span><span class="p">(</span><span class="s">&#34;no length cutoff provided&#34;</span><span class="p">)</span>
<span class="p">}</span>
<span class="k">if</span> <span class="nb">float64</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="nx">word</span><span class="p">))</span> <span class="p">&gt;</span> <span class="nx">cutOff</span> <span class="p">{</span>
<span class="nf">emitAboveCutoff</span><span class="p">(</span><span class="nx">word</span><span class="p">)</span>
@@ -5948,7 +5948,7 @@ a different view of the side input each time.</p>
latest trigger firing. This is particularly useful if you use a side input with
a single global window and specify a trigger.</p>
<h3 id="additional-outputs">4.5. Additional outputs</h3>
-<p class="language-java language-python">While <code>ParDo</code> always produces a main output <code>PCollection</code> (as the return value
+<p class="language-java language-py">While <code>ParDo</code> always produces a main output <code>PCollection</code> (as the return value
from <code>apply</code>), you can also have your <code>ParDo</code> produce any number of additional
output <code>PCollection</code>s. If you choose to have multiple outputs, your <code>ParDo</code>
returns all of the output <code>PCollection</code>s (including the main output) bundled
@@ -6875,6 +6875,8 @@ having Beam understand their element schemas.</p>
<p class="language-java">In Java you could use the following set of classes to represent the purchase schema. Beam will automatically
infer the correct schema based on the members of the class.</p>
<p class="language-py">In Python you can use the following set of classes to represent the purchase schema. Beam will automatically infer the correct schema based on the members of the class.</p>
+<p class="language-go">In Go, schema encoding is used by default for struct types, with Exported fields becoming part of the schema.
+Beam will automatically infer the schema based on the fields and field tags of the struct, and their order.</p>
<div class='language-java snippet'>
<div class="notebook-skip code-snippet">
<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard">
@@ -6941,6 +6943,38 @@ infer the correct schema based on the members of the class.</p>
<span class="n">purchase_amount</span><span class="p">:</span> <span class="nb">float</span></code></pre></div>
</div>
</div>
+<div class='language-go snippet'>
+<div class="notebook-skip code-snippet">
+<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<div class="highlight"><pre class="chroma"><code class="language-go" data-lang="go"><span class="kd">type</span> <span class="nx">Purchase</span> <span class="kd">struct</span> <span class="p">{</span>
+<span class="c1">// ID of the user who made the purchase.
+</span><span class="c1"></span> <span class="nx">UserID</span> <span class="kt">string</span> <span class="s">`beam:&#34;userId&#34;`</span>
+<span class="c1">// Identifier of the item that was purchased.
+</span><span class="c1"></span> <span class="nx">ItemID</span> <span class="kt">int64</span> <span class="s">`beam:&#34;itemId&#34;`</span>
+<span class="c1">// The shipping address, a nested type.
+</span><span class="c1"></span> <span class="nx">ShippingAddress</span> <span class="nx">ShippingAddress</span> <span class="s">`beam:&#34;shippingAddress&#34;`</span>
+<span class="c1">// The cost of the item in cents.
+</span><span class="c1"></span> <span class="nx">Cost</span> <span class="kt">int64</span> <span class="s">`beam:&#34;cost&#34;`</span>
+<span class="c1">// The transactions that paid for this purchase.
+</span><span class="c1"></span> <span class="c1">// A slice since the purchase might be spread out over multiple
+</span><span class="c1"></span> <span class="c1">// credit cards.
+</span><span class="c1"></span> <span class="nx">Transactions</span> <span class="p">[]</span><span class="nx">Transaction</span> <span class="s">`beam:&#34;transactions&#34;`</span>
+<span class="p">}</span>
+<span class="kd">type</span> <span class="nx">ShippingAddress</span> <span class="kd">struct</span> <span class="p">{</span>
+<span class="nx">StreetAddress</span> <span class="kt">string</span> <span class="s">`beam:&#34;streetAddress&#34;`</span>
+<span class="nx">City</span> <span class="kt">string</span> <span class="s">`beam:&#34;city&#34;`</span>
+<span class="nx">State</span> <span class="o">*</span><span class="kt">string</span> <span class="s">`beam:&#34;state&#34;`</span>
+<span class="nx">Country</span> <span class="kt">string</span> <span class="s">`beam:&#34;country&#34;`</span>
+<span class="nx">PostCode</span> <span class="kt">string</span> <span class="s">`beam:&#34;postCode&#34;`</span>
+<span class="p">}</span>
+<span class="kd">type</span> <span class="nx">Transaction</span> <span class="kd">struct</span> <span class="p">{</span>
+<span class="nx">Bank</span> <span class="kt">string</span> <span class="s">`beam:&#34;bank&#34;`</span>
+<span class="nx">PurchaseAmount</span> <span class="kt">float64</span> <span class="s">`beam:&#34;purchaseAmount&#34;`</span>
+<span class="p">}</span></code></pre></div>
+</div>
+</div>
<p class="language-java">Using JavaBean classes as above is one way to map a schema to Java classes. However multiple Java classes might have
the same schema, in which case the different Java types can often be used interchangeably. Beam will add implicit
conversions between types that have matching schemas. For example, the above
@@ -7087,6 +7121,9 @@ a nanosecond timestamp, with the INT64 field representing seconds and the INT32
limited-precision decimal type would have an integer argument indicating how many digits of precision are represented.
The argument is represented by a schema type, so can itself be a complex type.</p>
<p class="language-java">In Java, a logical type is specified as a subclass of the <code>LogicalType</code> class. A custom Java class can be specified to represent the logical type and conversion functions must be supplied to convert back and forth between this Java class and the underlying Schema type representation. For example, the logical type representing nanosecond timestamp might be implemented as follows</p>
+<p class="language-go">In Go, a logical type is specified with a custom implementation of the <code>beam.SchemaProvider</code> interface.
+For example, the logical type provider representing nanosecond timestamps
+might be implemented as follows</p>
<div class='language-java snippet'>
<div class="notebook-skip code-snippet">
<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard">
@@ -7110,9 +7147,91 @@ The argument is represented by a schema type, so can itself be a complex type.&l
<span class="o">}</span></code></pre></div>
</div>
</div>
+<div class='language-go snippet'>
+<div class="notebook-skip code-snippet">
+<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<div class="highlight"><pre class="chroma"><code class="language-go" data-lang="go"><span class="c1">// Define a logical provider like so:
+</span><span class="c1"></span>
+<span class="c1">// TimestampNanos is a logical type using time.Time, but
+</span><span class="c1">// encodes as a schema type.
+</span><span class="c1"></span><span class="kd">type</span> <span class="nx">TimestampNanos</span> <span class="nx">time</span><span class="p">.</span><span class="nx">Time</span>
+<span class="kd">func</span> <span class="p">(</span><span class="nx">tn</span> <span class="nx">TimestampNanos</span><span class="p">)</span> <span class="nf">Seconds</span><span class="p">()</span> <span class="kt">int64</span> <span class="p">{</span>
+<span class="k">return</span> <span class="nx">time</span><span class="p">.</span><span class="nf">Time</span><span class="p">(</span><span class="nx">tn</span><span class="p">).</span><span class="nf">Unix</span><span class="p">()</span>
+<span class="p">}</span>
+<span class="kd">func</span> <span class="p">(</span><span class="nx">tn</span> <span class="nx">TimestampNanos</span><span class="p">)</span> <span class="nf">Nanos</span><span class="p">()</span> <span class="kt">int32</span> <span class="p">{</span>
+<span class="k">return</span> <span class="nb">int32</span><span class="p">(</span><span class="nx">time</span><span class="p">.</span><span class="nf">Time</span><span class="p">(</span><span class="nx">tn</span><span class="p">).</span><span class="nf">UnixNano</span><span class="p">()</span> <span class="o">%</span> <span class="mi">1000000000</span><span class="p">)</span>
+<span class="p">}</span>
+<span class="c1">// tnStorage is the storage schema for TimestampNanos.
+</span><span class="c1"></span><span class="kd">type</span> <span class="nx">tnStorage</span> <span class="kd">struct</span> <span class="p">{</span>
+<span class="nx">Seconds</span> <span class="kt">int64</span> <span class="s">`beam:&#34;seconds&#34;`</span>
+<span class="nx">Nanos</span> <span class="kt">int32</span> <span class="s">`beam:&#34;nanos&#34;`</span>
+<span class="p">}</span>
+<span class="kd">var</span> <span class="p">(</span>
+<span class="c1">// reflect.Type of the Value type of TimestampNanos
+</span><span class="c1"></span> <span class="nx">tnType</span> <span class="p">=</span> <span class="nx">reflect</span><span class="p">.</span><span class="nf">TypeOf</span><span class="p">((</span><span class="o">*</span><span class="nx">TimestampNanos</span><span class="p">)(</span><span class="kc">nil</span><span class="p">)).</span><span class="nf">Elem</span><span class="p">()</span>
+<span class="nx">tnStorageType</span> <span class="p">=</span> <span class="nx">reflect</span><span class="p">.</span><span class="nf">TypeOf</span><span class="p">((</span><span class="o">*</span><span class="nx">tnStorage</span><span class="p">)(</span><span class="kc">nil</span><span class="p">)).</span><span class="nf">Elem</span><span class="p">()</span>
+<span class="p">)</span>
+<span class="c1">// TimestampNanosProvider implements the beam.SchemaProvider interface.
+</span><span class="c1"></span><span class="kd">type</span> <span class="nx">TimestampNanosProvider</span> <span class="kd">struct</span><span class="p">{}</span>
+<span class="c1">// FromLogicalType converts checks if the given type is TimestampNanos, and if so
+</span><span class="c1">// returns the storage type.
+</span><span class="c1"></span><span class="kd">func</span> <span class="p">(</span><span class="nx">p</span> <span class="o">*</span><span class="nx">TimestampNanosProvider</span><span class="p">)</span> <span class="nf">FromLogicalType</span><span class="p">(</span><span class="nx">rt</span> <span class="nx">reflect</span><span class="p">.</span><span class="nx">Type</span><span class="p">)</span> &l [...]
+<span class="k">if</span> <span class="nx">rt</span> <span class="o">!=</span> <span class="nx">tnType</span> <span class="p">{</span>
+<span class="k">return</span> <span class="kc">nil</span><span class="p">,</span> <span class="nx">fmt</span><span class="p">.</span><span class="nf">Errorf</span><span class="p">(</span><span class="s">&#34;unable to provide schema.LogicalType for type %v, want %v&#34;</span><span class="p">,</span> <span class="nx">rt</span><span class="p">,</span> <span class="nx">tnType</span><span class="p">)</span>
+<span class="p">}</span>
+<span class="k">return</span> <span class="nx">tnStorageType</span><span class="p">,</span> <span class="kc">nil</span>
+<span class="p">}</span>
+<span class="c1">// BuildEncoder builds a Beam schema encoder for the TimestampNanos type.
+</span><span class="c1"></span><span class="kd">func</span> <span class="p">(</span><span class="nx">p</span> <span class="o">*</span><span class="nx">TimestampNanosProvider</span><span class="p">)</span> <span class="nf">BuildEncoder</span><span class="p">(</span><span class="nx">rt</span> <span class="nx">reflect</span><span class="p">.</span><span class="nx">Type</span><span class="p">)</span> <s [...]
+<span class="k">if</span> <span class="nx">_</span><span class="p">,</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">p</span><span class="p">.</span><span class="nf">FromLogicalType</span><span class="p">(</span><span class="nx">rt</span><span class="p">);</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
+<span class="k">return</span> <span class="kc">nil</span><span class="p">,</span> <span class="nx">err</span>
+<span class="p">}</span>
+<span class="nx">enc</span><span class="p">,</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">coder</span><span class="p">.</span><span class="nf">RowEncoderForStruct</span><span class="p">(</span><span class="nx">tnStorageType</span><span class="p">)</span>
+<span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
+<span class="k">return</span> <span class="kc">nil</span><span class="p">,</span> <span class="nx">err</span>
+<span class="p">}</span>
+<span class="k">return</span> <span class="kd">func</span><span class="p">(</span><span class="nx">iface</span> <span class="kd">interface</span><span class="p">{},</span> <span class="nx">w</span> <span class="nx">io</span><span class="p">.</span><span class="nx">Writer</span><span class="p">)</span> <span class="kt">error</span> <span class="p">{</span>
+<span class="nx">v</span> <span class="o">:=</span> <span class="nx">iface</span><span class="p">.(</span><span class="nx">TimestampNanos</span><span class="p">)</span>
+<span class="k">return</span> <span class="nf">enc</span><span class="p">(</span><span class="nx">tnStorage</span><span class="p">{</span>
+<span class="nx">Seconds</span><span class="p">:</span> <span class="nx">v</span><span class="p">.</span><span class="nf">Seconds</span><span class="p">(),</span>
+<span class="nx">Nanos</span><span class="p">:</span> <span class="nx">v</span><span class="p">.</span><span class="nf">Nanos</span><span class="p">(),</span>
+<span class="p">},</span> <span class="nx">w</span><span class="p">)</span>
+<span class="p">},</span> <span class="kc">nil</span>
+<span class="p">}</span>
+<span class="c1">// BuildDecoder builds a Beam schema decoder for the TimestampNanos type.
+</span><span class="c1"></span><span class="kd">func</span> <span class="p">(</span><span class="nx">p</span> <span class="o">*</span><span class="nx">TimestampNanosProvider</span><span class="p">)</span> <span class="nf">BuildDecoder</span><span class="p">(</span><span class="nx">rt</span> <span class="nx">reflect</span><span class="p">.</span><span class="nx">Type</span><span class="p">)</span> <s [...]
+<span class="k">if</span> <span class="nx">_</span><span class="p">,</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">p</span><span class="p">.</span><span class="nf">FromLogicalType</span><span class="p">(</span><span class="nx">rt</span><span class="p">);</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
+<span class="k">return</span> <span class="kc">nil</span><span class="p">,</span> <span class="nx">err</span>
+<span class="p">}</span>
+<span class="nx">dec</span><span class="p">,</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nx">coder</span><span class="p">.</span><span class="nf">RowDecoderForStruct</span><span class="p">(</span><span class="nx">tnStorageType</span><span class="p">)</span>
+<span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
+<span class="k">return</span> <span class="kc">nil</span><span class="p">,</span> <span class="nx">err</span>
+<span class="p">}</span>
+<span class="k">return</span> <span class="kd">func</span><span class="p">(</span><span class="nx">r</span> <span class="nx">io</span><span class="p">.</span><span class="nx">Reader</span><span class="p">)</span> <span class="p">(</span><span class="kd">interface</span><span class="p">{},</span> <span class="kt">error</span><span class="p">)</span> <span class="p">{</span>
+<span class="nx">s</span><span class="p">,</span> <span class="nx">err</span> <span class="o">:=</span> <span class="nf">dec</span><span class="p">(</span><span class="nx">r</span><span class="p">)</span>
+<span class="k">if</span> <span class="nx">err</span> <span class="o">!=</span> <span class="kc">nil</span> <span class="p">{</span>
+<span class="k">return</span> <span class="kc">nil</span><span class="p">,</span> <span class="nx">err</span>
+<span class="p">}</span>
+<span class="nx">tn</span> <span class="o">:=</span> <span class="nx">s</span><span class="p">.(</span><span class="nx">tnStorage</span><span class="p">)</span>
+<span class="k">return</span> <span class="nf">TimestampNanos</span><span class="p">(</span><span class="nx">time</span><span class="p">.</span><span class="nf">Unix</span><span class="p">(</span><span class="nx">tn</span><span class="p">.</span><span class="nx">Seconds</span><span class="p">,</span> <span class="nb">int64</span><span class="p">(</span><span class="nx">tn</span><span class="p">.</sp [...]
+<span class="p">},</span> <span class="kc">nil</span>
+<span class="p">}</span>
+<span class="c1">// Register it like so:
+</span><span class="c1"></span>
+<span class="nx">beam</span><span class="p">.</span><span class="nf">RegisterSchemaProvider</span><span class="p">(</span><span class="nx">tnType</span><span class="p">,</span> <span class="o">&amp;</span><span class="nx">TimestampNanosProvider</span><span class="p">{})</span></code></pre></div>
+</div>
+</div>
<h4 id="built-in-logical-types">6.4.2. Useful logical types</h4>
+<p class="language-py">Currently the Python SDK provides minimal convenience logical types,
+other than to handle <code>MicrosInstant</code>.</p>
+<p class="language-go">Currently the Go SDK provides minimal convenience logical types,
+other than to handle additional integer primitives, and <code>time.Time</code>.</p>
<h5 id="enumerationtype"><strong>EnumerationType</strong></h5>
-<p>This logical type allows creating an enumeration type consisting of a set of named constants.</p>
+<p class="language-py">This convenience builder doesn&rsquo;t yet exist for the Python SDK.</p>
+<p class="language-go">This convenience builder doesn&rsquo;t yet exist for the Go SDK.</p>
+<p class="language-java">This logical type allows creating an enumeration type consisting of a set of named constants.</p>
<div class='language-java snippet'>
<div class="notebook-skip code-snippet">
<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard">
@@ -7124,7 +7243,7 @@ The argument is represented by a schema type, so can itself be a complex type.&l
<span class="o">.</span><span class="na">build</span><span class="o">();</span></code></pre></div>
</div>
</div>
-<p>The value of this field is stored in the row as an INT32 type, however the logical type defines a value type that lets
+<p class="language-java">The value of this field is stored in the row as an INT32 type, however the logical type defines a value type that lets
you access the enumeration either as a string or a value. For example:</p>
<div class='language-java snippet'>
<div class="notebook-skip code-snippet">
@@ -7136,7 +7255,7 @@ you access the enumeration either as a string or a value. For example:</p>
</span><span class="c1"></span><span class="n">enumValue</span><span class="o">.</span><span class="na">toString</span><span class="o">();</span> <span class="o">//</span> <span class="n">Returns</span> <span class="s">&#34;RED&#34;</span><span class="o">,</span> <span class="n">the</span> <span class="n">string</span> <span class="n">value</span> <span class="n">of</span> <span class="n">the</ [...]
</div>
</div>
-<p>Given a row object with an enumeration field, you can also extract the field as the enumeration value.</p>
+<p class="language-java">Given a row object with an enumeration field, you can also extract the field as the enumeration value.</p>
<div class='language-java snippet'>
<div class="notebook-skip code-snippet">
<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard">
@@ -7145,10 +7264,12 @@ you access the enumeration either as a string or a value. For example:</p>
<div class="highlight"><pre class="chroma"><code class="language-java" data-lang="java"><span class="n">EnumerationType</span><span class="o">.</span><span class="na">Value</span> <span class="n">enumValue</span> <span class="o">=</span> <span class="n">row</span><span class="o">.</span><span class="na">getLogicalTypeValue</span><span class="o">(</span><span class="s">&#34;color&#34;</span><span class="o">,&l [...]
</div>
</div>
-<p>Automatic schema inference from Java POJOs and JavaBeans automatically converts Java enums to EnumerationType logical
+<p class="language-java">Automatic schema inference from Java POJOs and JavaBeans automatically converts Java enums to EnumerationType logical
types.</p>
<h5 id="oneoftype"><strong>OneOfType</strong></h5>
-<p>OneOfType allows creating a disjoint union type over a set of schema fields. For example:</p>
+<p class="language-py">This convenience builder doesn&rsquo;t yet exist for the Python SDK.</p>
+<p class="language-go">This convenience builder doesn&rsquo;t yet exist for the Go SDK.</p>
+<p class="language-java">OneOfType allows creating a disjoint union type over a set of schema fields. For example:</p>
<div class='language-java snippet'>
<div class="notebook-skip code-snippet">
<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard">
@@ -7163,7 +7284,7 @@ types.</p>
<span class="o">.</span><span class="na">build</span><span class="o">();</span></code></pre></div>
</div>
</div>
-<p>The value of this field is stored in the row as another Row type, where all the fields are marked as nullable. The
+<p class="language-java">The value of this field is stored in the row as another Row type, where all the fields are marked as nullable. The
logical type however defines a Value object that contains an enumeration value indicating which field was set and allows
getting just that field:</p>
<div class='language-java snippet'>
@@ -7188,12 +7309,27 @@ getting just that field:</p>
<span class="o">}</span></code></pre></div>
</div>
</div>
-<p>In the above example we used the field names in the switch statement for clarity, however the enum integer values could
+<p class="language-java">In the above example we used the field names in the switch statement for clarity, however the enum integer values could
also be used.</p>
<h3 id="creating-schemas">6.5. Creating Schemas</h3>
-<p>In order to take advantage of schemas, your <code>PCollection</code>s must have a schema attached to it. Often, the source itself will attach a schema to the PCollection. For example, when using <code>AvroIO</code> to read Avro files, the source can automatically infer a Beam schema from the Avro schema and attach that to the Beam <code>PCollection</code>. However not all sources produce schemas. In addition, often Beam pipelines have intermediate stages and types [...]
+<p>In order to take advantage of schemas, your <code>PCollection</code>s must have a schema attached to it.
+Often, the source itself will attach a schema to the PCollection.
+For example, when using <code>AvroIO</code> to read Avro files, the source can automatically infer a Beam schema from the Avro schema and attach that to the Beam <code>PCollection</code>.
+However not all sources produce schemas.
+In addition, often Beam pipelines have intermediate stages and types, and those also can benefit from the expressiveness of schemas.</p>
<h4 id="inferring-schemas">6.5.1. Inferring schemas</h4>
-<p class="language-java">Beam is able to infer schemas from a variety of common Java types. The <code>@DefaultSchema</code> annotation can be used to tell Beam to infer schemas from a specific type. The annotation takes a <code>SchemaProvider</code> as an argument, and <code>SchemaProvider</code> classes are already built in for common Java types. The <code>SchemaRegistry</code> can also be invoked programmatically for cases where it is not practical to annotat [...]
+<nav class="language-switcher">
+<strong>Adapt for:</strong>
+<ul>
+<li data-type="language-java" class="active">Java SDK</li>
+<li data-type="language-py">Python SDK</li>
+<li data-type="language-go">Go SDK</li>
+</ul>
+</nav>
+<p class="language-java">Beam is able to infer schemas from a variety of common Java types.
+The <code>@DefaultSchema</code> annotation can be used to tell Beam to infer schemas from a specific type.
+The annotation takes a <code>SchemaProvider</code> as an argument, and <code>SchemaProvider</code> classes are already built in for common Java types.
+The <code>SchemaRegistry</code> can also be invoked programmatically for cases where it is not practical to annotate the Java type itself.</p>
<p class="language-java"><strong>Java POJOs</strong></p>
<p class="language-java">A POJO (Plain Old Java Object) is a Java object that is not bound by any restriction other than the Java Language
Specification. A POJO can contain member variables that are primitives, that are other POJOs, or are collections maps or
@@ -7359,10 +7495,48 @@ correctly infers types with <code>beam.Row</code> or with <code>Select&
<span class="n">purchase_amount</span><span class="o">=</span><span class="nb">float</span><span class="p">(</span><span class="n">item</span><span class="p">[</span><span class="s2">&#34;purchase_amount&#34;</span><span class="p">])))</span></code></pre></div>
</div>
</div>
+<p class="language-go">Beam currently only infers schemas for exported fields in Go structs.</p>
+<p class="language-go"><strong>Structs</strong></p>
+<p class="language-go">Beam will automatically infer schemas for all Go structs used
+as PCollection elements, and default to encoding them using
+schema encoding.</p>
+<div class='language-go snippet'>
+<div class="notebook-skip code-snippet">
+<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<div class="highlight"><pre class="chroma"><code class="language-go" data-lang="go"><span class="kd">type</span> <span class="nx">Transaction</span> <span class="kd">struct</span><span class="p">{</span>
+<span class="nx">Bank</span> <span class="kt">string</span>
+<span class="nx">PurchaseAmount</span> <span class="kt">float64</span>
+<span class="nx">checksum</span> <span class="p">[]</span><span class="kt">byte</span> <span class="c1">// ignored
+</span><span class="c1"></span><span class="p">}</span></code></pre></div>
+</div>
+</div>
+<p class="language-go">Unexported fields are ignored, and cannot be automatically infered as part of the schema.
+Fields of type func, channel, unsafe.Pointer, or uintptr will be ignored by inference.
+Fields of interface types are ignored, unless a schema provider
+is registered for them.</p>
+<p class="language-go">By default, schema field names will match the exported struct field names.
+In the above example, &ldquo;Bank&rdquo; and &ldquo;PurchaseAmount&rdquo; are the schema field names.
+A schema field name can be overridden with a struct tag for the field.</p>
+<div class='language-go snippet'>
+<div class="notebook-skip code-snippet">
+<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<div class="highlight"><pre class="chroma"><code class="language-go" data-lang="go"><span class="kd">type</span> <span class="nx">Transaction</span> <span class="kd">struct</span><span class="p">{</span>
+<span class="nx">Bank</span> <span class="kt">string</span> <span class="s">`beam:&#34;bank&#34;`</span>
+<span class="nx">PurchaseAmount</span> <span class="kt">float64</span> <span class="s">`beam:&#34;purchase_amount&#34;`</span>
+<span class="p">}</span></code></pre></div>
+</div>
+</div>
+<p class="language-go">Overriding schema field names is useful for compatibility cross language transforms,
+as schema fields may have different requirements or restrictions from Go exported fields.</p>
<h3 id="using-schemas">6.6. Using Schema Transforms</h3>
<p>A schema on a <code>PCollection</code> enables a rich variety of relational transforms. The fact that each record is composed of
named fields allows for simple and readable aggregations that reference fields by name, similar to the aggregations in
a SQL expression.</p>
+<p class="language-go">Beam does not yet support Schema transforms natively in Go. However, it will be implemented with the following behavior.</p>
<h4 id="661-field-selection-syntax">6.6.1. Field selection syntax</h4>
<p>The advantage of schemas is that they allow referencing of element fields by name. Beam provides a selection syntax for
referencing fields, including nested and repeated fields. This syntax is used by all of the schema transforms when
@@ -7892,6 +8066,7 @@ pipeline will fail to launch.</p>
<p>A <code>PCollection</code> with a schema can apply a <code>ParDo</code>, just like any other <code>PCollection</code>. However the Beam runner is aware
of schemas when applying a <code>ParDo</code>, which enables additional functionality.</p>
<h5 id="input-conversion"><strong>Input conversion</strong></h5>
+<p class="language-go">Beam does not yet support input conversion in Go.</p>
<p>Since Beam knows the schema of the source <code>PCollection</code>, it can automatically convert the elements to any Java type for
which a matching schema is known. For example, using the above-mentioned Transaction schema, say we have the following
<code>PCollection</code>:</p>
@@ -7967,6 +8142,14 @@ using the above-described selection expressions, as follows:</p>
<p>For more information, see the section on field-selection expressions. When selecting subschemas, Beam will
automatically convert to any matching schema type, just like when reading the entire row.</p>
<h2 id="data-encoding-and-type-safety">7. Data encoding and type safety</h2>
+<nav class="language-switcher">
+<strong>Adapt for:</strong>
+<ul>
+<li data-type="language-java" class="active">Java SDK</li>
+<li data-type="language-py">Python SDK</li>
+<li data-type="language-go">Go SDK</li>
+</ul>
+</nav>
<p>When Beam runners execute your pipeline, they often need to materialize the
intermediate data in your <code>PCollection</code>s, which requires converting elements to
and from byte strings. The Beam SDKs use objects called <code>Coder</code>s to describe how
@@ -7989,6 +8172,12 @@ types, Tuple, Iterable, StringUtf8 and more. You can find all of the available
Coder subclasses in the
<a href="https://github.com/apache/beam/tree/master/sdks/python/apache_beam/coders">apache_beam.coders</a>
package.</p>
+<p class="language-go">Standard Go types like <code>int</code>, <code>int64</code> <code>float64</code>, <code>[]byte</code>, and <code>string</code> and more are coded using builtin coders.
+Structs and pointers to structs default using Beam Schema Row encoding.
+However, users can build and register custom coders with <code>beam.RegisterCoder</code>.
+You can find available Coder functions in the
+<a href="https://pkg.go.dev/github.com/apache/beam/sdks/v2/go/pkg/beam/core/graph/coders">coder</a>
+package.</p>
<blockquote>
<p>Note that coders do not necessarily have a 1:1 relationship with types. For
example, the Integer type can have multiple valid coders, and input and output
@@ -8016,6 +8205,8 @@ mapping of Java types to the default coders that the pipeline should use for
<p class="language-py">The Beam SDK for Python has a <code>CoderRegistry</code> that represents a mapping of
Python types to the default coder that should be used for <code>PCollection</code>s of each
type.</p>
+<p class="language-go">The Beam SDK for Go allows users to register default coder
+implementations with <code>beam.RegisterCoder</code>.</p>
<p class="language-java">By default, the Beam SDK for Java automatically infers the <code>Coder</code> for the
elements of a <code>PCollection</code> produced by a <code>PTransform</code> using the type parameter
from the transform&rsquo;s function object, such as <code>DoFn</code>. In the case of <code>ParDo</code>,
@@ -8032,13 +8223,20 @@ with the typehints <code>@beam.typehints.with_input_types(int)</code> and
and produces an output element of type str. In such a case, the Beam SDK for
Python will automatically infer the default <code>Coder</code> for the output <code>PCollection</code>
(in the default pipeline <code>CoderRegistry</code>, this is <code>BytesCoder</code>).</p>
+<p class="language-go">By default, the Beam SDK for Go automatically infers the <code>Coder</code> for the elements of an output <code>PCollection</code> by the output of the transform&rsquo;s function object, such as a <code>DoFn</code>.
+In the case of <code>ParDo</code>, for example a <code>DoFn</code>
+with the parameters of <code>v int, emit func(string)</code> accepts an input element of type <code>int</code>
+and produces an output element of type <code>string</code>.
+In such a case, the Beam SDK for Go will automatically infer the default <code>Coder</code> for the output <code>PCollection</code> to be the <code>string_utf8</code> coder.</p>
+<span class="language-java">
<blockquote>
-<p>NOTE: If you create your <code>PCollection</code> from in-memory data by using the
+<p><strong>Note:</strong> If you create your <code>PCollection</code> from in-memory data by using the
<code>Create</code> transform, you cannot rely on coder inference and default coders.
<code>Create</code> does not have access to any typing information for its arguments, and
may not be able to infer a coder if the argument list contains a value whose
exact run-time class doesn&rsquo;t have a default coder registered.</p>
</blockquote>
+</span>
<p class="language-java">When using <code>Create</code>, the simplest way to ensure that you have the correct coder
is by invoking <code>withCoder</code> when you apply the <code>Create</code> transform.</p>
<h3 id="default-coders-and-the-coderregistry">7.2. Default coders and the CoderRegistry</h3>
@@ -8153,8 +8351,9 @@ this pipeline, verify that Integer values are encoded using
<p class="language-py">You can use the method <code>CoderRegistry.get_coder</code> to determine the default Coder
for a Python type. You can use <code>coders.registry</code> to access the <code>CoderRegistry</code>.
This allows you to determine (or set) the default Coder for a Python type.</p>
+<p class="language-go">You can use the <code>beam.NewCoder</code> function to determine the default Coder for a Go type.</p>
<h4 id="setting-default-coder">7.2.2. Setting the default coder for a type</h4>
-<p>To set the default Coder for a
+<p class="language-java language-py">To set the default Coder for a
<span class="language-java">Java</span><span class="language-py">Python</span>
type for a particular pipeline, you obtain and modify the pipeline&rsquo;s
<code>CoderRegistry</code>. You use the method
@@ -8164,10 +8363,13 @@ to get the <code>CoderRegistry</code> object, and then use the method
<span class="language-java"><code>CoderRegistry.registerCoder</code></span>
<span class="language-py"><code>CoderRegistry.register_coder</code></span>
to register a new <code>Coder</code> for the target type.</p>
-<p>The following example code demonstrates how to set a default Coder, in this case
+<p class="language-go">To set the default Coder for a Go type you use the function <code>beam.RegisterCoder</code> to register a encoder and decoder functions for the target type.
+However, built in types like <code>int</code>, <code>string</code>, <code>float64</code>, etc cannot have their coders overridde.</p>
+<p class="language-java language-py">The following example code demonstrates how to set a default Coder, in this case
<code>BigEndianIntegerCoder</code>, for
<span class="language-java">Integer</span><span class="language-py">int</span>
values for a pipeline.</p>
+<p class="language-go">The following example code demonstrates how to set a custom Coder for <code>MyCustomType</code> elements.</p>
<div class='language-java snippet'>
<div class="notebook-skip code-snippet">
<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard">
@@ -8187,8 +8389,26 @@ values for a pipeline.</p>
<div class="highlight"><pre class="chroma"><code class="language-py" data-lang="py"><span class="n">apache_beam</span><span class="o">.</span><span class="n">coders</span><span class="o">.</span><span class="n">registry</span><span class="o">.</span><span class="n">register_coder</span><span class="p">(</span><span class="nb">int</span><span class="p">,</span> <span class="n">BigEndianIntegerCoder</span><span c [...]
</div>
</div>
+<div class='language-go snippet'>
+<div class="notebook-skip code-snippet">
+<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard">
+<img src="/images/copy-icon.svg"/>
+</a>
+<div class="highlight"><pre class="chroma"><code class="language-go" data-lang="go"><span class="kd">type</span> <span class="nx">MyCustomType</span> <span class="kd">struct</span><span class="p">{</span>
+<span class="o">...</span>
+<span class="p">}</span>
+<span class="c1">// See documentation on beam.RegisterCoder for other supported coder forms.
+</span><span class="c1"></span>
+<span class="kd">func</span> <span class="nf">encode</span><span class="p">(</span><span class="nx">MyCustomType</span><span class="p">)</span> <span class="p">[]</span><span class="kt">byte</span> <span class="p">{</span> <span class="o">...</span> <span class="p">}</span>
+<span class="kd">func</span> <span class="nf">decode</span><span class="p">(</span><span class="nx">b</span> <span class="p">[]</span><span class="kt">byte</span><span class="p">)</span> <span class="nx">MyCustomType</span> <span class="p">{</span> <span class="o">...</span> <span class="p">}</span>
+<span class="kd">func</span> <span class="nf">init</span><span class="p">()</span> <span class="p">{</span>
+<span class="nx">beam</span><span class="p">.</span><span class="nf">RegisterCoder</span><span class="p">(</span><span class="nx">reflect</span><span class="p">.</span><span class="nf">TypeOf</span><span class="p">((</span><span class="o">*</span><span class="nx">MyCustomType</span><span class="p">)(</span><span class="kc">nil</span><span class="p">)).</span><span class="nf">Elem</span><span class="p"> [...]
+<span class="p">}</span></code></pre></div>
+</div>
+</div>
<h4 id="annotating-custom-type-default-coder">7.2.3. Annotating a custom data type with a default coder</h4>
-<p class="language-java"><p>If your pipeline program defines a custom data type, you can use the
+<span class="language-java">
+<p>If your pipeline program defines a custom data type, you can use the
<code>@DefaultCoder</code> annotation to specify the coder to use with that type.
By default, Beam will use <code>SerializableCoder</code> which uses Java serialization,
but it has drawbacks:</p>
@@ -8202,11 +8422,11 @@ See this <a href="https://blog.softwaremill.com/the-best-serialization-strate
equivalent objects.</p>
<p>For key/value pairs, the correctness of key-based operations
(GroupByKey, Combine) and per-key State depends on having a deterministic
-coder for the key.</p>
+coder for the key</p>
</li>
</ol>
<p>You can use the <code>@DefaultCoder</code> annotation to set a new default as follows:</p>
-</p>
+</span>
<div class='language-java snippet'>
<div class="notebook-skip code-snippet">
<a class="copy" type="button" data-bs-toggle="tooltip" data-bs-placement="bottom" title="Copy to clipboard">
@@ -8236,8 +8456,9 @@ the <code>@DefaultCoder</code> annotation, your coder class must implement
<span class="o">}</span></code></pre></div>
</div>
</div>
-<p class="language-py">The Beam SDK for Python does not support annotating data types with a default
-coder. If you would like to set a default coder, use the method described in the
+<p class="language-py language-go">The Beam SDK for <span class="language-py">Python</span><span class="language-go">Go</span>
+does not support annotating data types with a default coder.
+If you would like to set a default coder, use the method described in the
previous section, <em>Setting the default coder for a type</em>.</p>
<h2 id="windowing">8. Windowing</h2>
<p>Windowing subdivides a <code>PCollection</code> according to the timestamps of its
diff --git a/website/generated-content/documentation/programming-guide/index.html b/website/generated-content/documentation/programming-guide/index.html
index b31a31e..9cc9792 100644
--- a/website/generated-content/documentation/programming-guide/index.html
+++ b/website/generated-content/documentation/programming-guide/index.html
@@ -1367,7 +1367,7 @@ determined by the input data, or depend on a different branch of your pipeline.<
<span class=kd>var</span> <span class=nx>cutOff</span> <span class=kt>float64</span>
<span class=nx>ok</span> <span class=o>:=</span> <span class=nf>lengthCutOffIter</span><span class=p>(</span><span class=o>&</span><span class=nx>cutOff</span><span class=p>)</span>
<span class=k>if</span> <span class=p>!</span><span class=nx>ok</span> <span class=p>{</span>
- <span class=k>return</span> <span class=nx>fmt</span><span class=p>.</span><span class=nf>Errorf</span><span class=p>(</span><span class=s>"No length cutoff provided."</span><span class=p>)</span>
+ <span class=k>return</span> <span class=nx>fmt</span><span class=p>.</span><span class=nf>Errorf</span><span class=p>(</span><span class=s>"no length cutoff provided"</span><span class=p>)</span>
<span class=p>}</span>
<span class=k>if</span> <span class=nb>float64</span><span class=p>(</span><span class=nb>len</span><span class=p>(</span><span class=nx>word</span><span class=p>))</span> <span class=p>></span> <span class=nx>cutOff</span> <span class=p>{</span>
<span class=nf>emitAboveCutoff</span><span class=p>(</span><span class=nx>word</span><span class=p>)</span>
@@ -1412,7 +1412,7 @@ gets called multiple times, once for each window. Each call to <code>processElem
projects the “current” window for the main input element, and thus might provide
a different view of the side input each time.</p><p>If the side input has multiple trigger firings, Beam uses the value from the
latest trigger firing. This is particularly useful if you use a side input with
-a single global window and specify a trigger.</p><h3 id=additional-outputs>4.5. Additional outputs</h3><p class="language-java language-python">While <code>ParDo</code> always produces a main output <code>PCollection</code> (as the return value
+a single global window and specify a trigger.</p><h3 id=additional-outputs>4.5. Additional outputs</h3><p class="language-java language-py">While <code>ParDo</code> always produces a main output <code>PCollection</code> (as the return value
from <code>apply</code>), you can also have your <code>ParDo</code> produce any number of additional
output <code>PCollection</code>s. If you choose to have multiple outputs, your <code>ParDo</code>
returns all of the output <code>PCollection</code>s (including the main output) bundled
@@ -1889,7 +1889,8 @@ types across different programming-language APIs.</p><p>A <code>PCollection</cod
Schema rows; Beam uses a special coder to encode schema types.</p><h3 id=schemas-for-pl-types>6.2. Schemas for programming language types</h3><p>While schemas themselves are language independent, they are designed to embed naturally into the programming languages
of the Beam SDK being used. This allows Beam users to continue using native types while reaping the advantage of
having Beam understand their element schemas.</p><p class=language-java>In Java you could use the following set of classes to represent the purchase schema. Beam will automatically
-infer the correct schema based on the members of the class.</p><p class=language-py>In Python you can use the following set of classes to represent the purchase schema. Beam will automatically infer the correct schema based on the members of the class.</p><div class="language-java snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre [...]
+infer the correct schema based on the members of the class.</p><p class=language-py>In Python you can use the following set of classes to represent the purchase schema. Beam will automatically infer the correct schema based on the members of the class.</p><p class=language-go>In Go, schema encoding is used by default for struct types, with Exported fields becoming part of the schema.
+Beam will automatically infer the schema based on the fields and field tags of the struct, and their order.</p><div class="language-java snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-java data-lang=java><span class=nd>@DefaultSchema</span><span class=o>(</span><span class=n>JavaBeanSchema</spa [...]
<span class=kd>public</span> <span class=kd>class</span> <span class=nc>Purchase</span> <span class=o>{</span>
<span class=kd>public</span> <span class=n>String</span> <span class=nf>getUserId</span><span class=o>();</span> <span class=c1>// Returns the id of the user who made the purchase.
</span><span class=c1></span> <span class=kd>public</span> <span class=kt>long</span> <span class=nf>getItemId</span><span class=o>();</span> <span class=c1>// Returns the identifier of the item that was purchased.
@@ -1946,7 +1947,33 @@ infer the correct schema based on the members of the class.</p><p class=language
<span class=k>class</span> <span class=nc>Transaction</span><span class=p>(</span><span class=n>typing</span><span class=o>.</span><span class=n>NamedTuple</span><span class=p>):</span>
<span class=n>bank</span><span class=p>:</span> <span class=nb>str</span>
- <span class=n>purchase_amount</span><span class=p>:</span> <span class=nb>float</span></code></pre></div></div></div><p class=language-java>Using JavaBean classes as above is one way to map a schema to Java classes. However multiple Java classes might have
+ <span class=n>purchase_amount</span><span class=p>:</span> <span class=nb>float</span></code></pre></div></div></div><div class="language-go snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-go data-lang=go><span class=kd>type</span> <span class=nx>Purchase</span> <span class=kd>struct</span> <s [...]
+ <span class=c1>// ID of the user who made the purchase.
+</span><span class=c1></span> <span class=nx>UserID</span> <span class=kt>string</span> <span class=s>`beam:"userId"`</span>
+ <span class=c1>// Identifier of the item that was purchased.
+</span><span class=c1></span> <span class=nx>ItemID</span> <span class=kt>int64</span> <span class=s>`beam:"itemId"`</span>
+ <span class=c1>// The shipping address, a nested type.
+</span><span class=c1></span> <span class=nx>ShippingAddress</span> <span class=nx>ShippingAddress</span> <span class=s>`beam:"shippingAddress"`</span>
+ <span class=c1>// The cost of the item in cents.
+</span><span class=c1></span> <span class=nx>Cost</span> <span class=kt>int64</span> <span class=s>`beam:"cost"`</span>
+ <span class=c1>// The transactions that paid for this purchase.
+</span><span class=c1></span> <span class=c1>// A slice since the purchase might be spread out over multiple
+</span><span class=c1></span> <span class=c1>// credit cards.
+</span><span class=c1></span> <span class=nx>Transactions</span> <span class=p>[]</span><span class=nx>Transaction</span> <span class=s>`beam:"transactions"`</span>
+<span class=p>}</span>
+
+<span class=kd>type</span> <span class=nx>ShippingAddress</span> <span class=kd>struct</span> <span class=p>{</span>
+ <span class=nx>StreetAddress</span> <span class=kt>string</span> <span class=s>`beam:"streetAddress"`</span>
+ <span class=nx>City</span> <span class=kt>string</span> <span class=s>`beam:"city"`</span>
+ <span class=nx>State</span> <span class=o>*</span><span class=kt>string</span> <span class=s>`beam:"state"`</span>
+ <span class=nx>Country</span> <span class=kt>string</span> <span class=s>`beam:"country"`</span>
+ <span class=nx>PostCode</span> <span class=kt>string</span> <span class=s>`beam:"postCode"`</span>
+<span class=p>}</span>
+
+<span class=kd>type</span> <span class=nx>Transaction</span> <span class=kd>struct</span> <span class=p>{</span>
+ <span class=nx>Bank</span> <span class=kt>string</span> <span class=s>`beam:"bank"`</span>
+ <span class=nx>PurchaseAmount</span> <span class=kt>float64</span> <span class=s>`beam:"purchaseAmount"`</span>
+<span class=p>}</span></code></pre></div></div></div><p class=language-java>Using JavaBean classes as above is one way to map a schema to Java classes. However multiple Java classes might have
the same schema, in which case the different Java types can often be used interchangeably. Beam will add implicit
conversions between types that have matching schemas. For example, the above
<code>Transaction</code> class has the same schema as the following class:</p><div class="language-java snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-java data-lang=java><span class=nd>@DefaultSchema</span><span class=o>(</span><span class=n>JavaFieldSchema</span><span class=o>.</span><span cl [...]
@@ -1985,7 +2012,9 @@ type to represent nanosecond timestamps is represented as a schema containing an
alone does not say anything about how to interpret this type, however the logical type tells you that this represents
a nanosecond timestamp, with the INT64 field representing seconds and the INT32 field representing nanoseconds.</p><p>Logical types are also specified by an argument, which allows creating a class of related types. For example, a
limited-precision decimal type would have an integer argument indicating how many digits of precision are represented.
-The argument is represented by a schema type, so can itself be a complex type.</p><p class=language-java>In Java, a logical type is specified as a subclass of the <code>LogicalType</code> class. A custom Java class can be specified to represent the logical type and conversion functions must be supplied to convert back and forth between this Java class and the underlying Schema type representation. For example, the logical type representing nanosecond timestamp might be implemented as fol [...]
+The argument is represented by a schema type, so can itself be a complex type.</p><p class=language-java>In Java, a logical type is specified as a subclass of the <code>LogicalType</code> class. A custom Java class can be specified to represent the logical type and conversion functions must be supplied to convert back and forth between this Java class and the underlying Schema type representation. For example, the logical type representing nanosecond timestamp might be implemented as fol [...]
+For example, the logical type provider representing nanosecond timestamps
+might be implemented as follows</p><div class="language-java snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-java data-lang=java><span class=c1>// A Logical type using java.time.Instant to represent the logical type.
</span><span class=c1></span><span class=kd>public</span> <span class=kd>class</span> <span class=nc>TimestampNanos</span> <span class=kd>implements</span> <span class=n>LogicalType</span><span class=o><</span><span class=n>Instant</span><span class=o>,</span> <span class=n>Row</span><span class=o>></span> <span class=o>{</span>
<span class=c1>// The underlying schema used to represent rows.
</span><span class=c1></span> <span class=kd>private</span> <span class=kd>final</span> <span class=n>Schema</span> <span class=n>SCHEMA</span> <span class=o>=</span> <span class=n>Schema</span><span class=o>.</span><span class=na>builder</span><span class=o>().</span><span class=na>addInt64Field</span><span class=o>(</span><span class=s>"seconds"</span><span class=o>).</span><span class=na>addInt32Field</span><span class=o>(</span><span class=s>"nanos"</span><span class [...]
@@ -2003,20 +2032,101 @@ The argument is represented by a schema type, so can itself be a complex type.</
<span class=o>}</span>
<span class=o>...</span>
-<span class=o>}</span></code></pre></div></div></div><h4 id=built-in-logical-types>6.4.2. Useful logical types</h4><h5 id=enumerationtype><strong>EnumerationType</strong></h5><p>This logical type allows creating an enumeration type consisting of a set of named constants.</p><div class="language-java snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div cl [...]
+<span class=o>}</span></code></pre></div></div></div><div class="language-go snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-go data-lang=go><span class=c1>// Define a logical provider like so:
+</span><span class=c1></span>
+
+<span class=c1>// TimestampNanos is a logical type using time.Time, but
+</span><span class=c1>// encodes as a schema type.
+</span><span class=c1></span><span class=kd>type</span> <span class=nx>TimestampNanos</span> <span class=nx>time</span><span class=p>.</span><span class=nx>Time</span>
+
+<span class=kd>func</span> <span class=p>(</span><span class=nx>tn</span> <span class=nx>TimestampNanos</span><span class=p>)</span> <span class=nf>Seconds</span><span class=p>()</span> <span class=kt>int64</span> <span class=p>{</span>
+ <span class=k>return</span> <span class=nx>time</span><span class=p>.</span><span class=nf>Time</span><span class=p>(</span><span class=nx>tn</span><span class=p>).</span><span class=nf>Unix</span><span class=p>()</span>
+<span class=p>}</span>
+<span class=kd>func</span> <span class=p>(</span><span class=nx>tn</span> <span class=nx>TimestampNanos</span><span class=p>)</span> <span class=nf>Nanos</span><span class=p>()</span> <span class=kt>int32</span> <span class=p>{</span>
+ <span class=k>return</span> <span class=nb>int32</span><span class=p>(</span><span class=nx>time</span><span class=p>.</span><span class=nf>Time</span><span class=p>(</span><span class=nx>tn</span><span class=p>).</span><span class=nf>UnixNano</span><span class=p>()</span> <span class=o>%</span> <span class=mi>1000000000</span><span class=p>)</span>
+<span class=p>}</span>
+
+<span class=c1>// tnStorage is the storage schema for TimestampNanos.
+</span><span class=c1></span><span class=kd>type</span> <span class=nx>tnStorage</span> <span class=kd>struct</span> <span class=p>{</span>
+ <span class=nx>Seconds</span> <span class=kt>int64</span> <span class=s>`beam:"seconds"`</span>
+ <span class=nx>Nanos</span> <span class=kt>int32</span> <span class=s>`beam:"nanos"`</span>
+<span class=p>}</span>
+
+<span class=kd>var</span> <span class=p>(</span>
+ <span class=c1>// reflect.Type of the Value type of TimestampNanos
+</span><span class=c1></span> <span class=nx>tnType</span> <span class=p>=</span> <span class=nx>reflect</span><span class=p>.</span><span class=nf>TypeOf</span><span class=p>((</span><span class=o>*</span><span class=nx>TimestampNanos</span><span class=p>)(</span><span class=kc>nil</span><span class=p>)).</span><span class=nf>Elem</span><span class=p>()</span>
+ <span class=nx>tnStorageType</span> <span class=p>=</span> <span class=nx>reflect</span><span class=p>.</span><span class=nf>TypeOf</span><span class=p>((</span><span class=o>*</span><span class=nx>tnStorage</span><span class=p>)(</span><span class=kc>nil</span><span class=p>)).</span><span class=nf>Elem</span><span class=p>()</span>
+<span class=p>)</span>
+
+<span class=c1>// TimestampNanosProvider implements the beam.SchemaProvider interface.
+</span><span class=c1></span><span class=kd>type</span> <span class=nx>TimestampNanosProvider</span> <span class=kd>struct</span><span class=p>{}</span>
+
+<span class=c1>// FromLogicalType converts checks if the given type is TimestampNanos, and if so
+</span><span class=c1>// returns the storage type.
+</span><span class=c1></span><span class=kd>func</span> <span class=p>(</span><span class=nx>p</span> <span class=o>*</span><span class=nx>TimestampNanosProvider</span><span class=p>)</span> <span class=nf>FromLogicalType</span><span class=p>(</span><span class=nx>rt</span> <span class=nx>reflect</span><span class=p>.</span><span class=nx>Type</span><span class=p>)</span> <span class=p>(</span><span class=nx>reflect</span><span class=p>.</span><span class=nx>Type</span><span class=p>,</s [...]
+ <span class=k>if</span> <span class=nx>rt</span> <span class=o>!=</span> <span class=nx>tnType</span> <span class=p>{</span>
+ <span class=k>return</span> <span class=kc>nil</span><span class=p>,</span> <span class=nx>fmt</span><span class=p>.</span><span class=nf>Errorf</span><span class=p>(</span><span class=s>"unable to provide schema.LogicalType for type %v, want %v"</span><span class=p>,</span> <span class=nx>rt</span><span class=p>,</span> <span class=nx>tnType</span><span class=p>)</span>
+ <span class=p>}</span>
+ <span class=k>return</span> <span class=nx>tnStorageType</span><span class=p>,</span> <span class=kc>nil</span>
+<span class=p>}</span>
+
+<span class=c1>// BuildEncoder builds a Beam schema encoder for the TimestampNanos type.
+</span><span class=c1></span><span class=kd>func</span> <span class=p>(</span><span class=nx>p</span> <span class=o>*</span><span class=nx>TimestampNanosProvider</span><span class=p>)</span> <span class=nf>BuildEncoder</span><span class=p>(</span><span class=nx>rt</span> <span class=nx>reflect</span><span class=p>.</span><span class=nx>Type</span><span class=p>)</span> <span class=p>(</span><span class=kd>func</span><span class=p>(</span><span class=kd>interface</span><span class=p>{},</ [...]
+ <span class=k>if</span> <span class=nx>_</span><span class=p>,</span> <span class=nx>err</span> <span class=o>:=</span> <span class=nx>p</span><span class=p>.</span><span class=nf>FromLogicalType</span><span class=p>(</span><span class=nx>rt</span><span class=p>);</span> <span class=nx>err</span> <span class=o>!=</span> <span class=kc>nil</span> <span class=p>{</span>
+ <span class=k>return</span> <span class=kc>nil</span><span class=p>,</span> <span class=nx>err</span>
+ <span class=p>}</span>
+ <span class=nx>enc</span><span class=p>,</span> <span class=nx>err</span> <span class=o>:=</span> <span class=nx>coder</span><span class=p>.</span><span class=nf>RowEncoderForStruct</span><span class=p>(</span><span class=nx>tnStorageType</span><span class=p>)</span>
+ <span class=k>if</span> <span class=nx>err</span> <span class=o>!=</span> <span class=kc>nil</span> <span class=p>{</span>
+ <span class=k>return</span> <span class=kc>nil</span><span class=p>,</span> <span class=nx>err</span>
+ <span class=p>}</span>
+ <span class=k>return</span> <span class=kd>func</span><span class=p>(</span><span class=nx>iface</span> <span class=kd>interface</span><span class=p>{},</span> <span class=nx>w</span> <span class=nx>io</span><span class=p>.</span><span class=nx>Writer</span><span class=p>)</span> <span class=kt>error</span> <span class=p>{</span>
+ <span class=nx>v</span> <span class=o>:=</span> <span class=nx>iface</span><span class=p>.(</span><span class=nx>TimestampNanos</span><span class=p>)</span>
+ <span class=k>return</span> <span class=nf>enc</span><span class=p>(</span><span class=nx>tnStorage</span><span class=p>{</span>
+ <span class=nx>Seconds</span><span class=p>:</span> <span class=nx>v</span><span class=p>.</span><span class=nf>Seconds</span><span class=p>(),</span>
+ <span class=nx>Nanos</span><span class=p>:</span> <span class=nx>v</span><span class=p>.</span><span class=nf>Nanos</span><span class=p>(),</span>
+ <span class=p>},</span> <span class=nx>w</span><span class=p>)</span>
+ <span class=p>},</span> <span class=kc>nil</span>
+<span class=p>}</span>
+
+<span class=c1>// BuildDecoder builds a Beam schema decoder for the TimestampNanos type.
+</span><span class=c1></span><span class=kd>func</span> <span class=p>(</span><span class=nx>p</span> <span class=o>*</span><span class=nx>TimestampNanosProvider</span><span class=p>)</span> <span class=nf>BuildDecoder</span><span class=p>(</span><span class=nx>rt</span> <span class=nx>reflect</span><span class=p>.</span><span class=nx>Type</span><span class=p>)</span> <span class=p>(</span><span class=kd>func</span><span class=p>(</span><span class=nx>io</span><span class=p>.</span><spa [...]
+ <span class=k>if</span> <span class=nx>_</span><span class=p>,</span> <span class=nx>err</span> <span class=o>:=</span> <span class=nx>p</span><span class=p>.</span><span class=nf>FromLogicalType</span><span class=p>(</span><span class=nx>rt</span><span class=p>);</span> <span class=nx>err</span> <span class=o>!=</span> <span class=kc>nil</span> <span class=p>{</span>
+ <span class=k>return</span> <span class=kc>nil</span><span class=p>,</span> <span class=nx>err</span>
+ <span class=p>}</span>
+ <span class=nx>dec</span><span class=p>,</span> <span class=nx>err</span> <span class=o>:=</span> <span class=nx>coder</span><span class=p>.</span><span class=nf>RowDecoderForStruct</span><span class=p>(</span><span class=nx>tnStorageType</span><span class=p>)</span>
+ <span class=k>if</span> <span class=nx>err</span> <span class=o>!=</span> <span class=kc>nil</span> <span class=p>{</span>
+ <span class=k>return</span> <span class=kc>nil</span><span class=p>,</span> <span class=nx>err</span>
+ <span class=p>}</span>
+ <span class=k>return</span> <span class=kd>func</span><span class=p>(</span><span class=nx>r</span> <span class=nx>io</span><span class=p>.</span><span class=nx>Reader</span><span class=p>)</span> <span class=p>(</span><span class=kd>interface</span><span class=p>{},</span> <span class=kt>error</span><span class=p>)</span> <span class=p>{</span>
+ <span class=nx>s</span><span class=p>,</span> <span class=nx>err</span> <span class=o>:=</span> <span class=nf>dec</span><span class=p>(</span><span class=nx>r</span><span class=p>)</span>
+ <span class=k>if</span> <span class=nx>err</span> <span class=o>!=</span> <span class=kc>nil</span> <span class=p>{</span>
+ <span class=k>return</span> <span class=kc>nil</span><span class=p>,</span> <span class=nx>err</span>
+ <span class=p>}</span>
+ <span class=nx>tn</span> <span class=o>:=</span> <span class=nx>s</span><span class=p>.(</span><span class=nx>tnStorage</span><span class=p>)</span>
+ <span class=k>return</span> <span class=nf>TimestampNanos</span><span class=p>(</span><span class=nx>time</span><span class=p>.</span><span class=nf>Unix</span><span class=p>(</span><span class=nx>tn</span><span class=p>.</span><span class=nx>Seconds</span><span class=p>,</span> <span class=nb>int64</span><span class=p>(</span><span class=nx>tn</span><span class=p>.</span><span class=nx>Nanos</span><span class=p>))),</span> <span class=kc>nil</span>
+ <span class=p>},</span> <span class=kc>nil</span>
+<span class=p>}</span>
+
+
+
+<span class=c1>// Register it like so:
+</span><span class=c1></span>
+<span class=nx>beam</span><span class=p>.</span><span class=nf>RegisterSchemaProvider</span><span class=p>(</span><span class=nx>tnType</span><span class=p>,</span> <span class=o>&</span><span class=nx>TimestampNanosProvider</span><span class=p>{})</span></code></pre></div></div></div><h4 id=built-in-logical-types>6.4.2. Useful logical types</h4><p class=language-py>Currently the Python SDK provides minimal convenience logical types,
+other than to handle <code>MicrosInstant</code>.</p><p class=language-go>Currently the Go SDK provides minimal convenience logical types,
+other than to handle additional integer primitives, and <code>time.Time</code>.</p><h5 id=enumerationtype><strong>EnumerationType</strong></h5><p class=language-py>This convenience builder doesn’t yet exist for the Python SDK.</p><p class=language-go>This convenience builder doesn’t yet exist for the Go SDK.</p><p class=language-java>This logical type allows creating an enumeration type consisting of a set of named constants.</p><div class="language-java snippet"><div class=" [...]
<span class=err>…</span>
<span class=o>.</span><span class=na>addLogicalTypeField</span><span class=o>(</span><span class=s>"color"</span><span class=o>,</span> <span class=n>EnumerationType</span><span class=o>.</span><span class=na>create</span><span class=o>(</span><span class=s>"RED"</span><span class=o>,</span> <span class=s>"GREEN"</span><span class=o>,</span> <span class=s>"BLUE"</span><span class=o>))</span>
- <span class=o>.</span><span class=na>build</span><span class=o>();</span></code></pre></div></div></div><p>The value of this field is stored in the row as an INT32 type, however the logical type defines a value type that lets
+ <span class=o>.</span><span class=na>build</span><span class=o>();</span></code></pre></div></div></div><p class=language-java>The value of this field is stored in the row as an INT32 type, however the logical type defines a value type that lets
you access the enumeration either as a string or a value. For example:</p><div class="language-java snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-java data-lang=java><span class=n>EnumerationType</span><span class=o>.</span><span class=na>Value</span> <span class=n>enumValue</span> <span class [...]
<span class=n>enumValue</span><span class=o>.</span><span class=na>getValue</span><span class=o>();</span> <span class=c1>// Returns 0, the integer value of the constant.
-</span><span class=c1></span><span class=n>enumValue</span><span class=o>.</span><span class=na>toString</span><span class=o>();</span> <span class=o>//</span> <span class=n>Returns</span> <span class=s>"RED"</span><span class=o>,</span> <span class=n>the</span> <span class=n>string</span> <span class=n>value</span> <span class=n>of</span> <span class=n>the</span> <span class=n>constant</span></code></pre></div></div></div><p>Given a row object with an enumeration field, you can [...]
-types.</p><h5 id=oneoftype><strong>OneOfType</strong></h5><p>OneOfType allows creating a disjoint union type over a set of schema fields. For example:</p><div class="language-java snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-java data-lang=java><span class=n>Schema</span> <span class=n>schema [...]
+</span><span class=c1></span><span class=n>enumValue</span><span class=o>.</span><span class=na>toString</span><span class=o>();</span> <span class=o>//</span> <span class=n>Returns</span> <span class=s>"RED"</span><span class=o>,</span> <span class=n>the</span> <span class=n>string</span> <span class=n>value</span> <span class=n>of</span> <span class=n>the</span> <span class=n>constant</span></code></pre></div></div></div><p class=language-java>Given a row object with an enumer [...]
+types.</p><h5 id=oneoftype><strong>OneOfType</strong></h5><p class=language-py>This convenience builder doesn’t yet exist for the Python SDK.</p><p class=language-go>This convenience builder doesn’t yet exist for the Go SDK.</p><p class=language-java>OneOfType allows creating a disjoint union type over a set of schema fields. For example:</p><div class="language-java snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-plac [...]
<span class=err>…</span>
<span class=o>.</span><span class=na>addLogicalTypeField</span><span class=o>(</span><span class=s>"oneOfField"</span><span class=o>,</span>
<span class=n>OneOfType</span><span class=o>.</span><span class=na>create</span><span class=o>(</span><span class=n>Field</span><span class=o>.</span><span class=na>of</span><span class=o>(</span><span class=s>"intField"</span><span class=o>,</span> <span class=n>FieldType</span><span class=o>.</span><span class=na>INT32</span><span class=o>),</span>
<span class=n>Field</span><span class=o>.</span><span class=na>of</span><span class=o>(</span><span class=s>"stringField"</span><span class=o>,</span> <span class=n>FieldType</span><span class=o>.</span><span class=na>STRING</span><span class=o>),</span>
<span class=n>Field</span><span class=o>.</span><span class=na>of</span><span class=o>(</span><span class=s>"bytesField"</span><span class=o>,</span> <span class=n>FieldType</span><span class=o>.</span><span class=na>BYTES</span><span class=o>)))</span>
- <span class=o>.</span><span class=na>build</span><span class=o>();</span></code></pre></div></div></div><p>The value of this field is stored in the row as another Row type, where all the fields are marked as nullable. The
+ <span class=o>.</span><span class=na>build</span><span class=o>();</span></code></pre></div></div></div><p class=language-java>The value of this field is stored in the row as another Row type, where all the fields are marked as nullable. The
logical type however defines a Value object that contains an enumeration value indicating which field was set and allows
getting just that field:</p><div class="language-java snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-java data-lang=java><span class=c1>// Returns an enumeration indicating all possible case values for the enum.
</span><span class=c1>// For the above example, this will be
@@ -2034,8 +2144,15 @@ getting just that field:</p><div class="language-java snippet"><div class="noteb
<span class=k>return</span> <span class=n>processString</span><span class=o>(</span><span class=n>oneOfValue</span><span class=o>.</span><span class=na>getValue</span><span class=o>(</span><span class=n>String</span><span class=o>.</span><span class=na>class</span><span class=o>));</span>
<span class=k>case</span> <span class=s>"bytesField"</span><span class=o>:</span>
<span class=k>return</span> <span class=n>processBytes</span><span class=o>(</span><span class=n>oneOfValue</span><span class=o>.</span><span class=na>getValue</span><span class=o>(</span><span class=n>bytes</span><span class=o>[].</span><span class=na>class</span><span class=o>));</span>
-<span class=o>}</span></code></pre></div></div></div><p>In the above example we used the field names in the switch statement for clarity, however the enum integer values could
-also be used.</p><h3 id=creating-schemas>6.5. Creating Schemas</h3><p>In order to take advantage of schemas, your <code>PCollection</code>s must have a schema attached to it. Often, the source itself will attach a schema to the PCollection. For example, when using <code>AvroIO</code> to read Avro files, the source can automatically infer a Beam schema from the Avro schema and attach that to the Beam <code>PCollection</code>. However not all sources produce schemas. In addition, often Bea [...]
+<span class=o>}</span></code></pre></div></div></div><p class=language-java>In the above example we used the field names in the switch statement for clarity, however the enum integer values could
+also be used.</p><h3 id=creating-schemas>6.5. Creating Schemas</h3><p>In order to take advantage of schemas, your <code>PCollection</code>s must have a schema attached to it.
+Often, the source itself will attach a schema to the PCollection.
+For example, when using <code>AvroIO</code> to read Avro files, the source can automatically infer a Beam schema from the Avro schema and attach that to the Beam <code>PCollection</code>.
+However not all sources produce schemas.
+In addition, often Beam pipelines have intermediate stages and types, and those also can benefit from the expressiveness of schemas.</p><h4 id=inferring-schemas>6.5.1. Inferring schemas</h4><nav class=language-switcher><strong>Adapt for:</strong><ul><li data-type=language-java class=active>Java SDK</li><li data-type=language-py>Python SDK</li><li data-type=language-go>Go SDK</li></ul></nav><p class=language-java>Beam is able to infer schemas from a variety of common Java types.
+The <code>@DefaultSchema</code> annotation can be used to tell Beam to infer schemas from a specific type.
+The annotation takes a <code>SchemaProvider</code> as an argument, and <code>SchemaProvider</code> classes are already built in for common Java types.
+The <code>SchemaRegistry</code> can also be invoked programmatically for cases where it is not practical to annotate the Java type itself.</p><p class=language-java><strong>Java POJOs</strong></p><p class=language-java>A POJO (Plain Old Java Object) is a Java object that is not bound by any restriction other than the Java Language
Specification. A POJO can contain member variables that are primitives, that are other POJOs, or are collections maps or
arrays thereof. POJOs do not have to extend prespecified classes or extend any specific interfaces.</p><p class=language-java>If a POJO class is annotated with <code>@DefaultSchema(JavaFieldSchema.class)</code>, Beam will automatically infer a schema for
this class. Nested classes are supported as are classes with <code>List</code>, array, and <code>Map</code> fields.</p><p class=language-java>For example, annotating the following class tells Beam to infer a schema from this POJO class and apply it to any
@@ -2106,9 +2223,25 @@ type information. If it’s unable to it will fall back to the generic type
<code>Any</code>. Sometimes this is not ideal, you can use casts to make sure Beam
correctly infers types with <code>beam.Row</code> or with <code>Select</code>:</p><div class="language-py snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-py data-lang=py><span class=n>input_pc</span> <span class=o>=</span> <span class=o>...</span> <span class=c1># {"bank": ..., "purc [...]
<span class=n>output_pc</span> <span class=o>=</span> <span class=n>input_pc</span> <span class=o>|</span> <span class=n>beam</span><span class=o>.</span><span class=n>Map</span><span class=p>(</span><span class=k>lambda</span> <span class=n>item</span><span class=p>:</span> <span class=n>beam</span><span class=o>.</span><span class=n>Row</span><span class=p>(</span><span class=n>bank</span><span class=o>=</span><span class=nb>str</span><span class=p>(</span><span class=n>item</span><spa [...]
- <span class=n>purchase_amount</span><span class=o>=</span><span class=nb>float</span><span class=p>(</span><span class=n>item</span><span class=p>[</span><span class=s2>"purchase_amount"</span><span class=p>])))</span></code></pre></div></div></div><h3 id=using-schemas>6.6. Using Schema Transforms</h3><p>A schema on a <code>PCollection</code> enables a rich variety of relational transforms. The fact that each record is composed of
+ <span class=n>purchase_amount</span><span class=o>=</span><span class=nb>float</span><span class=p>(</span><span class=n>item</span><span class=p>[</span><span class=s2>"purchase_amount"</span><span class=p>])))</span></code></pre></div></div></div><p class=language-go>Beam currently only infers schemas for exported fields in Go structs.</p><p class=language-go><strong>Structs</strong></p><p class=language-go>Beam will automat [...]
+as PCollection elements, and default to encoding them using
+schema encoding.</p><div class="language-go snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-go data-lang=go><span class=kd>type</span> <span class=nx>Transaction</span> <span class=kd>struct</span><span class=p>{</span>
+ <span class=nx>Bank</span> <span class=kt>string</span>
+ <span class=nx>PurchaseAmount</span> <span class=kt>float64</span>
+
+ <span class=nx>checksum</span> <span class=p>[]</span><span class=kt>byte</span> <span class=c1>// ignored
+</span><span class=c1></span><span class=p>}</span></code></pre></div></div></div><p class=language-go>Unexported fields are ignored, and cannot be automatically infered as part of the schema.
+Fields of type func, channel, unsafe.Pointer, or uintptr will be ignored by inference.
+Fields of interface types are ignored, unless a schema provider
+is registered for them.</p><p class=language-go>By default, schema field names will match the exported struct field names.
+In the above example, “Bank” and “PurchaseAmount” are the schema field names.
+A schema field name can be overridden with a struct tag for the field.</p><div class="language-go snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-go data-lang=go><span class=kd>type</span> <span class=nx>Transaction</span> <span class=kd>struct</span><span class=p>{</span>
+ <span class=nx>Bank</span> <span class=kt>string</span> <span class=s>`beam:"bank"`</span>
+ <span class=nx>PurchaseAmount</span> <span class=kt>float64</span> <span class=s>`beam:"purchase_amount"`</span>
+<span class=p>}</span></code></pre></div></div></div><p class=language-go>Overriding schema field names is useful for compatibility cross language transforms,
+as schema fields may have different requirements or restrictions from Go exported fields.</p><h3 id=using-schemas>6.6. Using Schema Transforms</h3><p>A schema on a <code>PCollection</code> enables a rich variety of relational transforms. The fact that each record is composed of
named fields allows for simple and readable aggregations that reference fields by name, similar to the aggregations in
-a SQL expression.</p><h4 id=661-field-selection-syntax>6.6.1. Field selection syntax</h4><p>The advantage of schemas is that they allow referencing of element fields by name. Beam provides a selection syntax for
+a SQL expression.</p><p class=language-go>Beam does not yet support Schema transforms natively in Go. However, it will be implemented with the following behavior.</p><h4 id=661-field-selection-syntax>6.6.1. Field selection syntax</h4><p>The advantage of schemas is that they allow referencing of element fields by name. Beam provides a selection syntax for
referencing fields, including nested and repeated fields. This syntax is used by all of the schema transforms when
referencing the fields they operate on. The syntax can also be used inside of a DoFn to specify which schema fields to
process.</p><p>Addressing fields by name still retains type safety as Beam will check that schemas match at the time the pipeline graph
@@ -2198,7 +2331,7 @@ follows.</p><div class="language-java snippet"><div class="notebook-skip code-sn
unboxing the row. For example, give a schema with a single INT64 field, the following will convert it to a
<code>PCollection<Long></code></p><div class="language-java snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-java data-lang=java><span class=n>PCollection</span><span class=o><</span><span class=n>Long</span><span class=o>></span> <span class=n>longs</span> <span class=o>=</span> <span c [...]
pipeline will fail to launch.</p><h4 id=663-schemas-in-pardo>6.6.3. Schemas in ParDo</h4><p>A <code>PCollection</code> with a schema can apply a <code>ParDo</code>, just like any other <code>PCollection</code>. However the Beam runner is aware
-of schemas when applying a <code>ParDo</code>, which enables additional functionality.</p><h5 id=input-conversion><strong>Input conversion</strong></h5><p>Since Beam knows the schema of the source <code>PCollection</code>, it can automatically convert the elements to any Java type for
+of schemas when applying a <code>ParDo</code>, which enables additional functionality.</p><h5 id=input-conversion><strong>Input conversion</strong></h5><p class=language-go>Beam does not yet support input conversion in Go.</p><p>Since Beam knows the schema of the source <code>PCollection</code>, it can automatically convert the elements to any Java type for
which a matching schema is known. For example, using the above-mentioned Transaction schema, say we have the following
<code>PCollection</code>:</p><div class="language-java snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-java data-lang=java><span class=n>PCollection</span><span class=o><</span><span class=n>PurchasePojo</span><span class=o>></span> <span class=n>purchases</span> <span class=o>=</span> <sp [...]
since there is a schema, you could apply the following DoFn:</p><div class="language-java snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-java data-lang=java><span class=n>purchases</span><span class=o>.</span><span class=na>appy</span><span class=o>(</span><span class=n>ParDo</span><span class= [...]
@@ -2223,7 +2356,7 @@ using the above-described selection expressions, as follows:</p><div class="lang
<span class=o>...</span>
<span class=o>}</span>
<span class=o>}));</span></code></pre></div></div></div><p>For more information, see the section on field-selection expressions. When selecting subschemas, Beam will
-automatically convert to any matching schema type, just like when reading the entire row.</p><h2 id=data-encoding-and-type-safety>7. Data encoding and type safety</h2><p>When Beam runners execute your pipeline, they often need to materialize the
+automatically convert to any matching schema type, just like when reading the entire row.</p><h2 id=data-encoding-and-type-safety>7. Data encoding and type safety</h2><nav class=language-switcher><strong>Adapt for:</strong><ul><li data-type=language-java class=active>Java SDK</li><li data-type=language-py>Python SDK</li><li data-type=language-go>Go SDK</li></ul></nav><p>When Beam runners execute your pipeline, they often need to materialize the
intermediate data in your <code>PCollection</code>s, which requires converting elements to
and from byte strings. The Beam SDKs use objects called <code>Coder</code>s to describe how
the elements of a given <code>PCollection</code> may be encoded and decoded.</p><blockquote><p>Note that coders are unrelated to parsing or formatting data when interacting
@@ -2239,6 +2372,11 @@ subclasses that work with a variety of standard Python types, such as primitive
types, Tuple, Iterable, StringUtf8 and more. You can find all of the available
Coder subclasses in the
<a href=https://github.com/apache/beam/tree/master/sdks/python/apache_beam/coders>apache_beam.coders</a>
+package.</p><p class=language-go>Standard Go types like <code>int</code>, <code>int64</code> <code>float64</code>, <code>[]byte</code>, and <code>string</code> and more are coded using builtin coders.
+Structs and pointers to structs default using Beam Schema Row encoding.
+However, users can build and register custom coders with <code>beam.RegisterCoder</code>.
+You can find available Coder functions in the
+<a href=https://pkg.go.dev/github.com/apache/beam/sdks/v2/go/pkg/beam/core/graph/coders>coder</a>
package.</p><blockquote><p>Note that coders do not necessarily have a 1:1 relationship with types. For
example, the Integer type can have multiple valid coders, and input and output
data can use different Integer coders. A transform might have Integer-typed
@@ -2256,7 +2394,8 @@ not been set and cannot be inferred for the given <code>PCollection</code>.</p><
mapping of Java types to the default coders that the pipeline should use for
<code>PCollection</code>s of each type.</p><p class=language-py>The Beam SDK for Python has a <code>CoderRegistry</code> that represents a mapping of
Python types to the default coder that should be used for <code>PCollection</code>s of each
-type.</p><p class=language-java>By default, the Beam SDK for Java automatically infers the <code>Coder</code> for the
+type.</p><p class=language-go>The Beam SDK for Go allows users to register default coder
+implementations with <code>beam.RegisterCoder</code>.</p><p class=language-java>By default, the Beam SDK for Java automatically infers the <code>Coder</code> for the
elements of a <code>PCollection</code> produced by a <code>PTransform</code> using the type parameter
from the transform’s function object, such as <code>DoFn</code>. In the case of <code>ParDo</code>,
for example, a <code>DoFn<Integer, String></code> function object accepts an input element
@@ -2270,11 +2409,15 @@ with the typehints <code>@beam.typehints.with_input_types(int)</code> and
<code>@beam.typehints.with_output_types(str)</code> accepts an input element of type int
and produces an output element of type str. In such a case, the Beam SDK for
Python will automatically infer the default <code>Coder</code> for the output <code>PCollection</code>
-(in the default pipeline <code>CoderRegistry</code>, this is <code>BytesCoder</code>).</p><blockquote><p>NOTE: If you create your <code>PCollection</code> from in-memory data by using the
+(in the default pipeline <code>CoderRegistry</code>, this is <code>BytesCoder</code>).</p><p class=language-go>By default, the Beam SDK for Go automatically infers the <code>Coder</code> for the elements of an output <code>PCollection</code> by the output of the transform’s function object, such as a <code>DoFn</code>.
+In the case of <code>ParDo</code>, for example a <code>DoFn</code>
+with the parameters of <code>v int, emit func(string)</code> accepts an input element of type <code>int</code>
+and produces an output element of type <code>string</code>.
+In such a case, the Beam SDK for Go will automatically infer the default <code>Coder</code> for the output <code>PCollection</code> to be the <code>string_utf8</code> coder.</p><span class=language-java><blockquote><p><strong>Note:</strong> If you create your <code>PCollection</code> from in-memory data by using the
<code>Create</code> transform, you cannot rely on coder inference and default coders.
<code>Create</code> does not have access to any typing information for its arguments, and
may not be able to infer a coder if the argument list contains a value whose
-exact run-time class doesn’t have a default coder registered.</p></blockquote><p class=language-java>When using <code>Create</code>, the simplest way to ensure that you have the correct coder
+exact run-time class doesn’t have a default coder registered.</p></blockquote></span><p class=language-java>When using <code>Create</code>, the simplest way to ensure that you have the correct coder
is by invoking <code>withCoder</code> when you apply the <code>Create</code> transform.</p><h3 id=default-coders-and-the-coderregistry>7.2. Default coders and the CoderRegistry</h3><p>Each Pipeline object has a <code>CoderRegistry</code> object, which maps language types to
the default coder the pipeline should use for those types. You can use the
<code>CoderRegistry</code> yourself to look up the default coder for a given type, or to
@@ -2289,7 +2432,7 @@ by using the method <code>Pipeline.getCoderRegistry</code>. This allows you to d
this pipeline, verify that Integer values are encoded using
<code>BigEndianIntegerCoder</code>.”</p><p class=language-py>You can use the method <code>CoderRegistry.get_coder</code> to determine the default Coder
for a Python type. You can use <code>coders.registry</code> to access the <code>CoderRegistry</code>.
-This allows you to determine (or set) the default Coder for a Python type.</p><h4 id=setting-default-coder>7.2.2. Setting the default coder for a type</h4><p>To set the default Coder for a
+This allows you to determine (or set) the default Coder for a Python type.</p><p class=language-go>You can use the <code>beam.NewCoder</code> function to determine the default Coder for a Go type.</p><h4 id=setting-default-coder>7.2.2. Setting the default coder for a type</h4><p class="language-java language-py">To set the default Coder for a
<span class=language-java>Java</span><span class=language-py>Python</span>
type for a particular pipeline, you obtain and modify the pipeline’s
<code>CoderRegistry</code>. You use the method
@@ -2298,21 +2441,34 @@ type for a particular pipeline, you obtain and modify the pipeline’s
to get the <code>CoderRegistry</code> object, and then use the method
<span class=language-java><code>CoderRegistry.registerCoder</code></span>
<span class=language-py><code>CoderRegistry.register_coder</code></span>
-to register a new <code>Coder</code> for the target type.</p><p>The following example code demonstrates how to set a default Coder, in this case
+to register a new <code>Coder</code> for the target type.</p><p class=language-go>To set the default Coder for a Go type you use the function <code>beam.RegisterCoder</code> to register a encoder and decoder functions for the target type.
+However, built in types like <code>int</code>, <code>string</code>, <code>float64</code>, etc cannot have their coders overridde.</p><p class="language-java language-py">The following example code demonstrates how to set a default Coder, in this case
<code>BigEndianIntegerCoder</code>, for
<span class=language-java>Integer</span><span class=language-py>int</span>
-values for a pipeline.</p><div class="language-java snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-java data-lang=java><span class=n>PipelineOptions</span> <span class=n>options</span> <span class=o>=</span> <span class=n>PipelineOptionsFactory</span><span class=o>.</span><span class=na>create< [...]
+values for a pipeline.</p><p class=language-go>The following example code demonstrates how to set a custom Coder for <code>MyCustomType</code> elements.</p><div class="language-java snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-java data-lang=java><span class=n>PipelineOptions</span> <span cla [...]
<span class=n>Pipeline</span> <span class=n>p</span> <span class=o>=</span> <span class=n>Pipeline</span><span class=o>.</span><span class=na>create</span><span class=o>(</span><span class=n>options</span><span class=o>);</span>
<span class=n>CoderRegistry</span> <span class=n>cr</span> <span class=o>=</span> <span class=n>p</span><span class=o>.</span><span class=na>getCoderRegistry</span><span class=o>();</span>
-<span class=n>cr</span><span class=o>.</span><span class=na>registerCoder</span><span class=o>(</span><span class=n>Integer</span><span class=o>.</span><span class=na>class</span><span class=o>,</span> <span class=n>BigEndianIntegerCoder</span><span class=o>.</span><span class=na>class</span><span class=o>);</span></code></pre></div></div></div><div class="language-py snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=botto [...]
+<span class=n>cr</span><span class=o>.</span><span class=na>registerCoder</span><span class=o>(</span><span class=n>Integer</span><span class=o>.</span><span class=na>class</span><span class=o>,</span> <span class=n>BigEndianIntegerCoder</span><span class=o>.</span><span class=na>class</span><span class=o>);</span></code></pre></div></div></div><div class="language-py snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=botto [...]
+ <span class=o>...</span>
+<span class=p>}</span>
+
+<span class=c1>// See documentation on beam.RegisterCoder for other supported coder forms.
+</span><span class=c1></span>
+<span class=kd>func</span> <span class=nf>encode</span><span class=p>(</span><span class=nx>MyCustomType</span><span class=p>)</span> <span class=p>[]</span><span class=kt>byte</span> <span class=p>{</span> <span class=o>...</span> <span class=p>}</span>
+
+<span class=kd>func</span> <span class=nf>decode</span><span class=p>(</span><span class=nx>b</span> <span class=p>[]</span><span class=kt>byte</span><span class=p>)</span> <span class=nx>MyCustomType</span> <span class=p>{</span> <span class=o>...</span> <span class=p>}</span>
+
+<span class=kd>func</span> <span class=nf>init</span><span class=p>()</span> <span class=p>{</span>
+ <span class=nx>beam</span><span class=p>.</span><span class=nf>RegisterCoder</span><span class=p>(</span><span class=nx>reflect</span><span class=p>.</span><span class=nf>TypeOf</span><span class=p>((</span><span class=o>*</span><span class=nx>MyCustomType</span><span class=p>)(</span><span class=kc>nil</span><span class=p>)).</span><span class=nf>Elem</span><span class=p>(),</span> <span class=nx>encode</span><span class=p>,</span> <span class=nx>decode</span><span class=p>)</span>
+<span class=p>}</span></code></pre></div></div></div><h4 id=annotating-custom-type-default-coder>7.2.3. Annotating a custom data type with a default coder</h4><span class=language-java><p>If your pipeline program defines a custom data type, you can use the
<code>@DefaultCoder</code> annotation to specify the coder to use with that type.
By default, Beam will use <code>SerializableCoder</code> which uses Java serialization,
but it has drawbacks:</p><ol><li><p>It is inefficient in encoding size and speed.
See this <a href=https://blog.softwaremill.com/the-best-serialization-strategy-for-event-sourcing-9321c299632b>comparison of Java serialization methods.</a></p></li><li><p>It is non-deterministic: it may produce different binary encodings for two
equivalent objects.</p><p>For key/value pairs, the correctness of key-based operations
(GroupByKey, Combine) and per-key State depends on having a deterministic
-coder for the key.</p></li></ol><p>You can use the <code>@DefaultCoder</code> annotation to set a new default as follows:</p></p><div class="language-java snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-java data-lang=java><span class=nd>@DefaultCoder</span><span class=o>(</span><span class=n>Av [...]
+coder for the key</p></li></ol><p>You can use the <code>@DefaultCoder</code> annotation to set a new default as follows:</p></span><div class="language-java snippet"><div class="notebook-skip code-snippet"><a class=copy type=button data-bs-toggle=tooltip data-bs-placement=bottom title="Copy to clipboard"><img src=/images/copy-icon.svg></a><div class=highlight><pre class=chroma><code class=language-java data-lang=java><span class=nd>@DefaultCoder</span><span class=o>(</span><span class=n> [...]
<span class=kd>public</span> <span class=kd>class</span> <span class=nc>MyCustomDataType</span> <span class=o>{</span>
<span class=o>...</span>
<span class=o>}</span></code></pre></div></div></div><p class=language-java>If you’ve created a custom coder to match your data type, and you want to use
@@ -2325,8 +2481,9 @@ the <code>@DefaultCoder</code> annotation, your coder class must implement a sta
<span class=nd>@DefaultCoder</span><span class=o>(</span><span class=n>MyCustomCoder</span><span class=o>.</span><span class=na>class</span><span class=o>)</span>
<span class=kd>public</span> <span class=kd>class</span> <span class=nc>MyCustomDataType</span> <span class=o>{</span>
<span class=o>...</span>
-<span class=o>}</span></code></pre></div></div></div><p class=language-py>The Beam SDK for Python does not support annotating data types with a default
-coder. If you would like to set a default coder, use the method described in the
+<span class=o>}</span></code></pre></div></div></div><p class="language-py language-go">The Beam SDK for <span class=language-py>Python</span><span class=language-go>Go</span>
+does not support annotating data types with a default coder.
+If you would like to set a default coder, use the method described in the
previous section, <em>Setting the default coder for a type</em>.</p><h2 id=windowing>8. Windowing</h2><p>Windowing subdivides a <code>PCollection</code> according to the timestamps of its
individual elements. Transforms that aggregate multiple elements, such as
<code>GroupByKey</code> and <code>Combine</code>, work implicitly on a per-window basis — they process
@@ -3974,7 +4131,7 @@ expansionAddr := "localhost:8097"
outT := beam.UnnamedOutput(typex.New(reflectx.String))
res := beam.CrossLanguage(s, urn, payload, expansionAddr, beam.UnnamedInput(inputPCol), outT)
</code></pre></div></div></li><li><p>After the job has been submitted to the Beam runner, shutdown the expansion service by
-terminating the expansion service process.</p></li></ol><h3 id=x-lang-transform-runner-support>13.3. Runner Support</h3><p>Currently, portable runners such as Flink, Spark, and the Direct runner can be used with multi-language pipelines.</p><p>Google Cloud Dataflow supports multi-language pipelines through the Dataflow Runner v2 backend architecture.</p><div class=feedback><p class=update>Last updated on 2021/09/24</p><h3>Have you found everything you were looking for?</h3><p class=descr [...]
+terminating the expansion service process.</p></li></ol><h3 id=x-lang-transform-runner-support>13.3. Runner Support</h3><p>Currently, portable runners such as Flink, Spark, and the Direct runner can be used with multi-language pipelines.</p><p>Google Cloud Dataflow supports multi-language pipelines through the Dataflow Runner v2 backend architecture.</p><div class=feedback><p class=update>Last updated on 2021/10/01</p><h3>Have you found everything you were looking for?</h3><p class=descr [...]
<a href=http://www.apache.org>The Apache Software Foundation</a>
| <a href=/privacy_policy>Privacy Policy</a>
| <a href=/feed.xml>RSS Feed</a><br><br>Apache Beam, Apache, Beam, the Beam logo, and the Apache feather logo are either registered trademarks or trademarks of The Apache Software Foundation. All other products or name brands are trademarks of their respective holders, including The Apache Software Foundation.</div></div></div></div></footer></body></html>
\ No newline at end of file
diff --git a/website/generated-content/sitemap.xml b/website/generated-content/sitemap.xml
index 265a3d4..980082e 100644
--- a/website/generated-content/sitemap.xml
+++ b/website/generated-content/sitemap.xml
@@ -1 +1 @@
-<?xml version="1.0" encoding="utf-8" standalone="yes"?><urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:xhtml="http://www.w3.org/1999/xhtml"><url><loc>/blog/beam-2.32.0/</loc><lastmod>2021-09-16T12:21:14-07:00</lastmod></url><url><loc>/categories/blog/</loc><lastmod>2021-09-16T12:21:14-07:00</lastmod></url><url><loc>/blog/</loc><lastmod>2021-09-16T12:21:14-07:00</lastmod></url><url><loc>/categories/</loc><lastmod>2021-09-17T14:05:48-07:00</lastmod></url><url><loc>/blog/b [...]
\ No newline at end of file
+<?xml version="1.0" encoding="utf-8" standalone="yes"?><urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:xhtml="http://www.w3.org/1999/xhtml"><url><loc>/blog/beam-2.32.0/</loc><lastmod>2021-09-16T12:21:14-07:00</lastmod></url><url><loc>/categories/blog/</loc><lastmod>2021-09-16T12:21:14-07:00</lastmod></url><url><loc>/blog/</loc><lastmod>2021-09-16T12:21:14-07:00</lastmod></url><url><loc>/categories/</loc><lastmod>2021-09-17T14:05:48-07:00</lastmod></url><url><loc>/blog/b [...]
\ No newline at end of file